Repository: louischatriot/nedb Branch: master Commit: 35be491ca67d Files: 55 Total size: 1.0 MB Directory structure: gitextract_fo51vwzk/ ├── .gitignore ├── LICENSE ├── README.md ├── benchmarks/ │ ├── commonUtilities.js │ ├── ensureIndex.js │ ├── find.js │ ├── findOne.js │ ├── findWithIn.js │ ├── insert.js │ ├── loadDatabase.js │ ├── remove.js │ └── update.js ├── bower.json ├── browser-version/ │ ├── browser-specific/ │ │ └── lib/ │ │ ├── customUtils.js │ │ └── storage.js │ ├── build.js │ ├── out/ │ │ └── nedb.js │ ├── package.json │ └── test/ │ ├── async.js │ ├── chai.js │ ├── index.html │ ├── localforage.js │ ├── mocha.css │ ├── mocha.js │ ├── nedb-browser.js │ ├── playground.html │ ├── testLoad.html │ ├── testLoad.js │ ├── testPersistence.html │ ├── testPersistence.js │ ├── testPersistence2.html │ └── testPersistence2.js ├── index.js ├── lib/ │ ├── cursor.js │ ├── customUtils.js │ ├── datastore.js │ ├── executor.js │ ├── indexes.js │ ├── model.js │ ├── persistence.js │ └── storage.js ├── package.json ├── test/ │ ├── cursor.test.js │ ├── customUtil.test.js │ ├── db.test.js │ ├── executor.test.js │ ├── indexes.test.js │ ├── mocha.opts │ ├── model.test.js │ └── persistence.test.js └── test_lac/ ├── loadAndCrash.test.js ├── openFds.test.js ├── openFdsLaunch.sh ├── openFdsTestFile └── openFdsTestFile2 ================================================ FILE CONTENTS ================================================ ================================================ FILE: .gitignore ================================================ lib-cov *.seed *.log *.csv *.dat *.out *.pid *.gz .idea pids logs results npm-debug.log workspace node_modules browser-version/src browser-version/node_modules *.swp *~ *.swo ================================================ FILE: LICENSE ================================================ (The MIT License) Copyright (c) 2013 Louis Chatriot <louis.chatriot@gmail.com> Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the 'Software'), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ================================================ FILE: README.md ================================================ ## The JavaScript Database > :warning: :warning: :warning: **WARNING:** this library is no longer maintained, and may have bugs and security issues. Feel free to fork but no pull request or security alert will be answered. **Embedded persistent or in memory database for Node.js, nw.js, Electron and browsers, 100% JavaScript, no binary dependency**. API is a subset of MongoDB's and it's plenty fast. **IMPORTANT NOTE**: Please don't submit issues for questions regarding your code. Only actual bugs or feature requests will be answered, all others will be closed without comment. Also, please follow the bug reporting guidelines and check the change log before submitting an already fixed bug :) ## Installation, tests Module name on npm and bower is `nedb`. ``` npm install nedb --save # Put latest version in your package.json npm test # You'll need the dev dependencies to launch tests bower install nedb # For the browser versions, which will be in browser-version/out ``` ## API It is a subset of MongoDB's API (the most used operations). * Creating/loading a database * Persistence * Inserting documents * Finding documents * Basic Querying * Operators ($lt, $lte, $gt, $gte, $in, $nin, $ne, $exists, $regex) * Array fields * Logical operators $or, $and, $not, $where * Sorting and paginating * Projections * Counting documents * Updating documents * Removing documents * Indexing * Browser version ### Creating/loading a database You can use NeDB as an in-memory only datastore or as a persistent datastore. One datastore is the equivalent of a MongoDB collection. The constructor is used as follows `new Datastore(options)` where `options` is an object with the following fields: * `filename` (optional): path to the file where the data is persisted. If left blank, the datastore is automatically considered in-memory only. It cannot end with a `~` which is used in the temporary files NeDB uses to perform crash-safe writes. * `inMemoryOnly` (optional, defaults to `false`): as the name implies. * `timestampData` (optional, defaults to `false`): timestamp the insertion and last update of all documents, with the fields `createdAt` and `updatedAt`. User-specified values override automatic generation, usually useful for testing. * `autoload` (optional, defaults to `false`): if used, the database will automatically be loaded from the datafile upon creation (you don't need to call `loadDatabase`). Any command issued before load is finished is buffered and will be executed when load is done. * `onload` (optional): if you use autoloading, this is the handler called after the `loadDatabase`. It takes one `error` argument. If you use autoloading without specifying this handler, and an error happens during load, an error will be thrown. * `afterSerialization` (optional): hook you can use to transform data after it was serialized and before it is written to disk. Can be used for example to encrypt data before writing database to disk. This function takes a string as parameter (one line of an NeDB data file) and outputs the transformed string, **which must absolutely not contain a `\n` character** (or data will be lost). * `beforeDeserialization` (optional): inverse of `afterSerialization`. Make sure to include both and not just one or you risk data loss. For the same reason, make sure both functions are inverses of one another. Some failsafe mechanisms are in place to prevent data loss if you misuse the serialization hooks: NeDB checks that never one is declared without the other, and checks that they are reverse of one another by testing on random strings of various lengths. In addition, if too much data is detected as corrupt, NeDB will refuse to start as it could mean you're not using the deserialization hook corresponding to the serialization hook used before (see below). * `corruptAlertThreshold` (optional): between 0 and 1, defaults to 10%. NeDB will refuse to start if more than this percentage of the datafile is corrupt. 0 means you don't tolerate any corruption, 1 means you don't care. * `compareStrings` (optional): function compareStrings(a, b) compares strings a and b and return -1, 0 or 1. If specified, it overrides default string comparison which is not well adapted to non-US characters in particular accented letters. Native `localCompare` will most of the time be the right choice * `nodeWebkitAppName` (optional, **DEPRECATED**): if you are using NeDB from whithin a Node Webkit app, specify its name (the same one you use in the `package.json`) in this field and the `filename` will be relative to the directory Node Webkit uses to store the rest of the application's data (local storage etc.). It works on Linux, OS X and Windows. Now that you can use `require('nw.gui').App.dataPath` in Node Webkit to get the path to the data directory for your application, you should not use this option anymore and it will be removed. If you use a persistent datastore without the `autoload` option, you need to call `loadDatabase` manually. This function fetches the data from datafile and prepares the database. **Don't forget it!** If you use a persistent datastore, no command (insert, find, update, remove) will be executed before `loadDatabase` is called, so make sure to call it yourself or use the `autoload` option. Also, if `loadDatabase` fails, all commands registered to the executor afterwards will not be executed. They will be registered and executed, in sequence, only after a successful `loadDatabase`. ```javascript // Type 1: In-memory only datastore (no need to load the database) var Datastore = require('nedb') , db = new Datastore(); // Type 2: Persistent datastore with manual loading var Datastore = require('nedb') , db = new Datastore({ filename: 'path/to/datafile' }); db.loadDatabase(function (err) { // Callback is optional // Now commands will be executed }); // Type 3: Persistent datastore with automatic loading var Datastore = require('nedb') , db = new Datastore({ filename: 'path/to/datafile', autoload: true }); // You can issue commands right away // Type 4: Persistent datastore for a Node Webkit app called 'nwtest' // For example on Linux, the datafile will be ~/.config/nwtest/nedb-data/something.db var Datastore = require('nedb') , path = require('path') , db = new Datastore({ filename: path.join(require('nw.gui').App.dataPath, 'something.db') }); // Of course you can create multiple datastores if you need several // collections. In this case it's usually a good idea to use autoload for all collections. db = {}; db.users = new Datastore('path/to/users.db'); db.robots = new Datastore('path/to/robots.db'); // You need to load each database (here we do it asynchronously) db.users.loadDatabase(); db.robots.loadDatabase(); ``` ### Persistence Under the hood, NeDB's persistence uses an append-only format, meaning that all updates and deletes actually result in lines added at the end of the datafile, for performance reasons. The database is automatically compacted (i.e. put back in the one-line-per-document format) every time you load each database within your application. You can manually call the compaction function with `yourDatabase.persistence.compactDatafile` which takes no argument. It queues a compaction of the datafile in the executor, to be executed sequentially after all pending operations. The datastore will fire a `compaction.done` event once compaction is finished. You can also set automatic compaction at regular intervals with `yourDatabase.persistence.setAutocompactionInterval(interval)`, `interval` in milliseconds (a minimum of 5s is enforced), and stop automatic compaction with `yourDatabase.persistence.stopAutocompaction()`. Keep in mind that compaction takes a bit of time (not too much: 130ms for 50k records on a typical development machine) and no other operation can happen when it does, so most projects actually don't need to use it. Compaction will also immediately remove any documents whose data line has become corrupted, assuming that the total percentage of all corrupted documents in that database still falls below the specified `corruptAlertThreshold` option's value. Durability works similarly to major databases: compaction forces the OS to physically flush data to disk, while appends to the data file do not (the OS is responsible for flushing the data). That guarantees that a server crash can never cause complete data loss, while preserving performance. The worst that can happen is a crash between two syncs, causing a loss of all data between the two syncs. Usually syncs are 30 seconds appart so that's at most 30 seconds of data. This post by Antirez on Redis persistence explains this in more details, NeDB being very close to Redis AOF persistence with `appendfsync` option set to `no`. ### Inserting documents The native types are `String`, `Number`, `Boolean`, `Date` and `null`. You can also use arrays and subdocuments (objects). If a field is `undefined`, it will not be saved (this is different from MongoDB which transforms `undefined` in `null`, something I find counter-intuitive). If the document does not contain an `_id` field, NeDB will automatically generated one for you (a 16-characters alphanumerical string). The `_id` of a document, once set, cannot be modified. Field names cannot begin by '$' or contain a '.'. ```javascript var doc = { hello: 'world' , n: 5 , today: new Date() , nedbIsAwesome: true , notthere: null , notToBeSaved: undefined // Will not be saved , fruits: [ 'apple', 'orange', 'pear' ] , infos: { name: 'nedb' } }; db.insert(doc, function (err, newDoc) { // Callback is optional // newDoc is the newly inserted document, including its _id // newDoc has no key called notToBeSaved since its value was undefined }); ``` You can also bulk-insert an array of documents. This operation is atomic, meaning that if one insert fails due to a unique constraint being violated, all changes are rolled back. ```javascript db.insert([{ a: 5 }, { a: 42 }], function (err, newDocs) { // Two documents were inserted in the database // newDocs is an array with these documents, augmented with their _id }); // If there is a unique constraint on field 'a', this will fail db.insert([{ a: 5 }, { a: 42 }, { a: 5 }], function (err) { // err is a 'uniqueViolated' error // The database was not modified }); ``` ### Finding documents Use `find` to look for multiple documents matching you query, or `findOne` to look for one specific document. You can select documents based on field equality or use comparison operators (`$lt`, `$lte`, `$gt`, `$gte`, `$in`, `$nin`, `$ne`). You can also use logical operators `$or`, `$and`, `$not` and `$where`. See below for the syntax. You can use regular expressions in two ways: in basic querying in place of a string, or with the `$regex` operator. You can sort and paginate results using the cursor API (see below). You can use standard projections to restrict the fields to appear in the results (see below). #### Basic querying Basic querying means are looking for documents whose fields match the ones you specify. You can use regular expression to match strings. You can use the dot notation to navigate inside nested documents, arrays, arrays of subdocuments and to match a specific element of an array. ```javascript // Let's say our datastore contains the following collection // { _id: 'id1', planet: 'Mars', system: 'solar', inhabited: false, satellites: ['Phobos', 'Deimos'] } // { _id: 'id2', planet: 'Earth', system: 'solar', inhabited: true, humans: { genders: 2, eyes: true } } // { _id: 'id3', planet: 'Jupiter', system: 'solar', inhabited: false } // { _id: 'id4', planet: 'Omicron Persei 8', system: 'futurama', inhabited: true, humans: { genders: 7 } } // { _id: 'id5', completeData: { planets: [ { name: 'Earth', number: 3 }, { name: 'Mars', number: 2 }, { name: 'Pluton', number: 9 } ] } } // Finding all planets in the solar system db.find({ system: 'solar' }, function (err, docs) { // docs is an array containing documents Mars, Earth, Jupiter // If no document is found, docs is equal to [] }); // Finding all planets whose name contain the substring 'ar' using a regular expression db.find({ planet: /ar/ }, function (err, docs) { // docs contains Mars and Earth }); // Finding all inhabited planets in the solar system db.find({ system: 'solar', inhabited: true }, function (err, docs) { // docs is an array containing document Earth only }); // Use the dot-notation to match fields in subdocuments db.find({ "humans.genders": 2 }, function (err, docs) { // docs contains Earth }); // Use the dot-notation to navigate arrays of subdocuments db.find({ "completeData.planets.name": "Mars" }, function (err, docs) { // docs contains document 5 }); db.find({ "completeData.planets.name": "Jupiter" }, function (err, docs) { // docs is empty }); db.find({ "completeData.planets.0.name": "Earth" }, function (err, docs) { // docs contains document 5 // If we had tested against "Mars" docs would be empty because we are matching against a specific array element }); // You can also deep-compare objects. Don't confuse this with dot-notation! db.find({ humans: { genders: 2 } }, function (err, docs) { // docs is empty, because { genders: 2 } is not equal to { genders: 2, eyes: true } }); // Find all documents in the collection db.find({}, function (err, docs) { }); // The same rules apply when you want to only find one document db.findOne({ _id: 'id1' }, function (err, doc) { // doc is the document Mars // If no document is found, doc is null }); ``` #### Operators ($lt, $lte, $gt, $gte, $in, $nin, $ne, $exists, $regex) The syntax is `{ field: { $op: value } }` where `$op` is any comparison operator: * `$lt`, `$lte`: less than, less than or equal * `$gt`, `$gte`: greater than, greater than or equal * `$in`: member of. `value` must be an array of values * `$ne`, `$nin`: not equal, not a member of * `$exists`: checks whether the document posses the property `field`. `value` should be true or false * `$regex`: checks whether a string is matched by the regular expression. Contrary to MongoDB, the use of `$options` with `$regex` is not supported, because it doesn't give you more power than regex flags. Basic queries are more readable so only use the `$regex` operator when you need to use another operator with it (see example below) ```javascript // $lt, $lte, $gt and $gte work on numbers and strings db.find({ "humans.genders": { $gt: 5 } }, function (err, docs) { // docs contains Omicron Persei 8, whose humans have more than 5 genders (7). }); // When used with strings, lexicographical order is used db.find({ planet: { $gt: 'Mercury' }}, function (err, docs) { // docs contains Omicron Persei 8 }) // Using $in. $nin is used in the same way db.find({ planet: { $in: ['Earth', 'Jupiter'] }}, function (err, docs) { // docs contains Earth and Jupiter }); // Using $exists db.find({ satellites: { $exists: true } }, function (err, docs) { // docs contains only Mars }); // Using $regex with another operator db.find({ planet: { $regex: /ar/, $nin: ['Jupiter', 'Earth'] } }, function (err, docs) { // docs only contains Mars because Earth was excluded from the match by $nin }); ``` #### Array fields When a field in a document is an array, NeDB first tries to see if the query value is an array to perform an exact match, then whether there is an array-specific comparison function (for now there is only `$size` and `$elemMatch`) being used. If not, the query is treated as a query on every element and there is a match if at least one element matches. * `$size`: match on the size of the array * `$elemMatch`: matches if at least one array element matches the query entirely ```javascript // Exact match db.find({ satellites: ['Phobos', 'Deimos'] }, function (err, docs) { // docs contains Mars }) db.find({ satellites: ['Deimos', 'Phobos'] }, function (err, docs) { // docs is empty }) // Using an array-specific comparison function // $elemMatch operator will provide match for a document, if an element from the array field satisfies all the conditions specified with the `$elemMatch` operator db.find({ completeData: { planets: { $elemMatch: { name: 'Earth', number: 3 } } } }, function (err, docs) { // docs contains documents with id 5 (completeData) }); db.find({ completeData: { planets: { $elemMatch: { name: 'Earth', number: 5 } } } }, function (err, docs) { // docs is empty }); // You can use inside #elemMatch query any known document query operator db.find({ completeData: { planets: { $elemMatch: { name: 'Earth', number: { $gt: 2 } } } } }, function (err, docs) { // docs contains documents with id 5 (completeData) }); // Note: you can't use nested comparison functions, e.g. { $size: { $lt: 5 } } will throw an error db.find({ satellites: { $size: 2 } }, function (err, docs) { // docs contains Mars }); db.find({ satellites: { $size: 1 } }, function (err, docs) { // docs is empty }); // If a document's field is an array, matching it means matching any element of the array db.find({ satellites: 'Phobos' }, function (err, docs) { // docs contains Mars. Result would have been the same if query had been { satellites: 'Deimos' } }); // This also works for queries that use comparison operators db.find({ satellites: { $lt: 'Amos' } }, function (err, docs) { // docs is empty since Phobos and Deimos are after Amos in lexicographical order }); // This also works with the $in and $nin operator db.find({ satellites: { $in: ['Moon', 'Deimos'] } }, function (err, docs) { // docs contains Mars (the Earth document is not complete!) }); ``` #### Logical operators $or, $and, $not, $where You can combine queries using logical operators: * For `$or` and `$and`, the syntax is `{ $op: [query1, query2, ...] }`. * For `$not`, the syntax is `{ $not: query }` * For `$where`, the syntax is `{ $where: function () { /* object is "this", return a boolean */ } }` ```javascript db.find({ $or: [{ planet: 'Earth' }, { planet: 'Mars' }] }, function (err, docs) { // docs contains Earth and Mars }); db.find({ $not: { planet: 'Earth' } }, function (err, docs) { // docs contains Mars, Jupiter, Omicron Persei 8 }); db.find({ $where: function () { return Object.keys(this) > 6; } }, function (err, docs) { // docs with more than 6 properties }); // You can mix normal queries, comparison queries and logical operators db.find({ $or: [{ planet: 'Earth' }, { planet: 'Mars' }], inhabited: true }, function (err, docs) { // docs contains Earth }); ``` #### Sorting and paginating If you don't specify a callback to `find`, `findOne` or `count`, a `Cursor` object is returned. You can modify the cursor with `sort`, `skip` and `limit` and then execute it with `exec(callback)`. ```javascript // Let's say the database contains these 4 documents // doc1 = { _id: 'id1', planet: 'Mars', system: 'solar', inhabited: false, satellites: ['Phobos', 'Deimos'] } // doc2 = { _id: 'id2', planet: 'Earth', system: 'solar', inhabited: true, humans: { genders: 2, eyes: true } } // doc3 = { _id: 'id3', planet: 'Jupiter', system: 'solar', inhabited: false } // doc4 = { _id: 'id4', planet: 'Omicron Persei 8', system: 'futurama', inhabited: true, humans: { genders: 7 } } // No query used means all results are returned (before the Cursor modifiers) db.find({}).sort({ planet: 1 }).skip(1).limit(2).exec(function (err, docs) { // docs is [doc3, doc1] }); // You can sort in reverse order like this db.find({ system: 'solar' }).sort({ planet: -1 }).exec(function (err, docs) { // docs is [doc1, doc3, doc2] }); // You can sort on one field, then another, and so on like this: db.find({}).sort({ firstField: 1, secondField: -1 }) ... // You understand how this works! ``` #### Projections You can give `find` and `findOne` an optional second argument, `projections`. The syntax is the same as MongoDB: `{ a: 1, b: 1 }` to return only the `a` and `b` fields, `{ a: 0, b: 0 }` to omit these two fields. You cannot use both modes at the time, except for `_id` which is by default always returned and which you can choose to omit. You can project on nested documents. ```javascript // Same database as above // Keeping only the given fields db.find({ planet: 'Mars' }, { planet: 1, system: 1 }, function (err, docs) { // docs is [{ planet: 'Mars', system: 'solar', _id: 'id1' }] }); // Keeping only the given fields but removing _id db.find({ planet: 'Mars' }, { planet: 1, system: 1, _id: 0 }, function (err, docs) { // docs is [{ planet: 'Mars', system: 'solar' }] }); // Omitting only the given fields and removing _id db.find({ planet: 'Mars' }, { planet: 0, system: 0, _id: 0 }, function (err, docs) { // docs is [{ inhabited: false, satellites: ['Phobos', 'Deimos'] }] }); // Failure: using both modes at the same time db.find({ planet: 'Mars' }, { planet: 0, system: 1 }, function (err, docs) { // err is the error message, docs is undefined }); // You can also use it in a Cursor way but this syntax is not compatible with MongoDB db.find({ planet: 'Mars' }).projection({ planet: 1, system: 1 }).exec(function (err, docs) { // docs is [{ planet: 'Mars', system: 'solar', _id: 'id1' }] }); // Project on a nested document db.findOne({ planet: 'Earth' }).projection({ planet: 1, 'humans.genders': 1 }).exec(function (err, doc) { // doc is { planet: 'Earth', _id: 'id2', humans: { genders: 2 } } }); ``` ### Counting documents You can use `count` to count documents. It has the same syntax as `find`. For example: ```javascript // Count all planets in the solar system db.count({ system: 'solar' }, function (err, count) { // count equals to 3 }); // Count all documents in the datastore db.count({}, function (err, count) { // count equals to 4 }); ``` ### Updating documents `db.update(query, update, options, callback)` will update all documents matching `query` according to the `update` rules: * `query` is the same kind of finding query you use with `find` and `findOne` * `update` specifies how the documents should be modified. It is either a new document or a set of modifiers (you cannot use both together, it doesn't make sense!) * A new document will replace the matched docs * The modifiers create the fields they need to modify if they don't exist, and you can apply them to subdocs. Available field modifiers are `$set` to change a field's value, `$unset` to delete a field, `$inc` to increment a field's value and `$min`/`$max` to change field's value, only if provided value is less/greater than current value. To work on arrays, you have `$push`, `$pop`, `$addToSet`, `$pull`, and the special `$each` and `$slice`. See examples below for the syntax. * `options` is an object with two possible parameters * `multi` (defaults to `false`) which allows the modification of several documents if set to true * `upsert` (defaults to `false`) if you want to insert a new document corresponding to the `update` rules if your `query` doesn't match anything. If your `update` is a simple object with no modifiers, it is the inserted document. In the other case, the `query` is stripped from all operator recursively, and the `update` is applied to it. * `returnUpdatedDocs` (defaults to `false`, not MongoDB-compatible) if set to true and update is not an upsert, will return the array of documents matched by the find query and updated. Updated documents will be returned even if the update did not actually modify them. * `callback` (optional) signature: `(err, numAffected, affectedDocuments, upsert)`. **Warning**: the API was changed between v1.7.4 and v1.8. Please refer to the change log to see the change. * For an upsert, `affectedDocuments` contains the inserted document and the `upsert` flag is set to `true`. * For a standard update with `returnUpdatedDocs` flag set to `false`, `affectedDocuments` is not set. * For a standard update with `returnUpdatedDocs` flag set to `true` and `multi` to `false`, `affectedDocuments` is the updated document. * For a standard update with `returnUpdatedDocs` flag set to `true` and `multi` to `true`, `affectedDocuments` is the array of updated documents. **Note**: you can't change a document's _id. ```javascript // Let's use the same example collection as in the "finding document" part // { _id: 'id1', planet: 'Mars', system: 'solar', inhabited: false } // { _id: 'id2', planet: 'Earth', system: 'solar', inhabited: true } // { _id: 'id3', planet: 'Jupiter', system: 'solar', inhabited: false } // { _id: 'id4', planet: 'Omicron Persia 8', system: 'futurama', inhabited: true } // Replace a document by another db.update({ planet: 'Jupiter' }, { planet: 'Pluton'}, {}, function (err, numReplaced) { // numReplaced = 1 // The doc #3 has been replaced by { _id: 'id3', planet: 'Pluton' } // Note that the _id is kept unchanged, and the document has been replaced // (the 'system' and inhabited fields are not here anymore) }); // Set an existing field's value db.update({ system: 'solar' }, { $set: { system: 'solar system' } }, { multi: true }, function (err, numReplaced) { // numReplaced = 3 // Field 'system' on Mars, Earth, Jupiter now has value 'solar system' }); // Setting the value of a non-existing field in a subdocument by using the dot-notation db.update({ planet: 'Mars' }, { $set: { "data.satellites": 2, "data.red": true } }, {}, function () { // Mars document now is { _id: 'id1', system: 'solar', inhabited: false // , data: { satellites: 2, red: true } // } // Not that to set fields in subdocuments, you HAVE to use dot-notation // Using object-notation will just replace the top-level field db.update({ planet: 'Mars' }, { $set: { data: { satellites: 3 } } }, {}, function () { // Mars document now is { _id: 'id1', system: 'solar', inhabited: false // , data: { satellites: 3 } // } // You lost the "data.red" field which is probably not the intended behavior }); }); // Deleting a field db.update({ planet: 'Mars' }, { $unset: { planet: true } }, {}, function () { // Now the document for Mars doesn't contain the planet field // You can unset nested fields with the dot notation of course }); // Upserting a document db.update({ planet: 'Pluton' }, { planet: 'Pluton', inhabited: false }, { upsert: true }, function (err, numReplaced, upsert) { // numReplaced = 1, upsert = { _id: 'id5', planet: 'Pluton', inhabited: false } // A new document { _id: 'id5', planet: 'Pluton', inhabited: false } has been added to the collection }); // If you upsert with a modifier, the upserted doc is the query modified by the modifier // This is simpler than it sounds :) db.update({ planet: 'Pluton' }, { $inc: { distance: 38 } }, { upsert: true }, function () { // A new document { _id: 'id5', planet: 'Pluton', distance: 38 } has been added to the collection }); // If we insert a new document { _id: 'id6', fruits: ['apple', 'orange', 'pear'] } in the collection, // let's see how we can modify the array field atomically // $push inserts new elements at the end of the array db.update({ _id: 'id6' }, { $push: { fruits: 'banana' } }, {}, function () { // Now the fruits array is ['apple', 'orange', 'pear', 'banana'] }); // $pop removes an element from the end (if used with 1) or the front (if used with -1) of the array db.update({ _id: 'id6' }, { $pop: { fruits: 1 } }, {}, function () { // Now the fruits array is ['apple', 'orange'] // With { $pop: { fruits: -1 } }, it would have been ['orange', 'pear'] }); // $addToSet adds an element to an array only if it isn't already in it // Equality is deep-checked (i.e. $addToSet will not insert an object in an array already containing the same object) // Note that it doesn't check whether the array contained duplicates before or not db.update({ _id: 'id6' }, { $addToSet: { fruits: 'apple' } }, {}, function () { // The fruits array didn't change // If we had used a fruit not in the array, e.g. 'banana', it would have been added to the array }); // $pull removes all values matching a value or even any NeDB query from the array db.update({ _id: 'id6' }, { $pull: { fruits: 'apple' } }, {}, function () { // Now the fruits array is ['orange', 'pear'] }); db.update({ _id: 'id6' }, { $pull: { fruits: $in: ['apple', 'pear'] } }, {}, function () { // Now the fruits array is ['orange'] }); // $each can be used to $push or $addToSet multiple values at once // This example works the same way with $addToSet db.update({ _id: 'id6' }, { $push: { fruits: { $each: ['banana', 'orange'] } } }, {}, function () { // Now the fruits array is ['apple', 'orange', 'pear', 'banana', 'orange'] }); // $slice can be used in cunjunction with $push and $each to limit the size of the resulting array. // A value of 0 will update the array to an empty array. A positive value n will keep only the n first elements // A negative value -n will keep only the last n elements. // If $slice is specified but not $each, $each is set to [] db.update({ _id: 'id6' }, { $push: { fruits: { $each: ['banana'], $slice: 2 } } }, {}, function () { // Now the fruits array is ['apple', 'orange'] }); // $min/$max to update only if provided value is less/greater than current value // Let's say the database contains this document // doc = { _id: 'id', name: 'Name', value: 5 } db.update({ _id: 'id1' }, { $min: { value: 2 } }, {}, function () { // The document will be updated to { _id: 'id', name: 'Name', value: 2 } }); db.update({ _id: 'id1' }, { $min: { value: 8 } }, {}, function () { // The document will not be modified }); ``` ### Removing documents `db.remove(query, options, callback)` will remove all documents matching `query` according to `options` * `query` is the same as the ones used for finding and updating * `options` only one option for now: `multi` which allows the removal of multiple documents if set to true. Default is false * `callback` is optional, signature: err, numRemoved ```javascript // Let's use the same example collection as in the "finding document" part // { _id: 'id1', planet: 'Mars', system: 'solar', inhabited: false } // { _id: 'id2', planet: 'Earth', system: 'solar', inhabited: true } // { _id: 'id3', planet: 'Jupiter', system: 'solar', inhabited: false } // { _id: 'id4', planet: 'Omicron Persia 8', system: 'futurama', inhabited: true } // Remove one document from the collection // options set to {} since the default for multi is false db.remove({ _id: 'id2' }, {}, function (err, numRemoved) { // numRemoved = 1 }); // Remove multiple documents db.remove({ system: 'solar' }, { multi: true }, function (err, numRemoved) { // numRemoved = 3 // All planets from the solar system were removed }); // Removing all documents with the 'match-all' query db.remove({}, { multi: true }, function (err, numRemoved) { }); ``` ### Indexing NeDB supports indexing. It gives a very nice speed boost and can be used to enforce a unique constraint on a field. You can index any field, including fields in nested documents using the dot notation. For now, indexes are only used to speed up basic queries and queries using `$in`, `$lt`, `$lte`, `$gt` and `$gte`. The indexed values cannot be of type array of object. To create an index, use `datastore.ensureIndex(options, cb)`, where callback is optional and get passed an error if any (usually a unique constraint that was violated). `ensureIndex` can be called when you want, even after some data was inserted, though it's best to call it at application startup. The options are: * **fieldName** (required): name of the field to index. Use the dot notation to index a field in a nested document. * **unique** (optional, defaults to `false`): enforce field uniqueness. Note that a unique index will raise an error if you try to index two documents for which the field is not defined. * **sparse** (optional, defaults to `false`): don't index documents for which the field is not defined. Use this option along with "unique" if you want to accept multiple documents for which it is not defined. * **expireAfterSeconds** (number of seconds, optional): if set, the created index is a TTL (time to live) index, that will automatically remove documents when the system date becomes larger than the date on the indexed field plus `expireAfterSeconds`. Documents where the indexed field is not specified or not a `Date` object are ignored Note: the `_id` is automatically indexed with a unique constraint, no need to call `ensureIndex` on it. You can remove a previously created index with `datastore.removeIndex(fieldName, cb)`. If your datastore is persistent, the indexes you created are persisted in the datafile, when you load the database a second time they are automatically created for you. No need to remove any `ensureIndex` though, if it is called on a database that already has the index, nothing happens. ```javascript db.ensureIndex({ fieldName: 'somefield' }, function (err) { // If there was an error, err is not null }); // Using a unique constraint with the index db.ensureIndex({ fieldName: 'somefield', unique: true }, function (err) { }); // Using a sparse unique index db.ensureIndex({ fieldName: 'somefield', unique: true, sparse: true }, function (err) { }); // Format of the error message when the unique constraint is not met db.insert({ somefield: 'nedb' }, function (err) { // err is null db.insert({ somefield: 'nedb' }, function (err) { // err is { errorType: 'uniqueViolated' // , key: 'name' // , message: 'Unique constraint violated for key name' } }); }); // Remove index on field somefield db.removeIndex('somefield', function (err) { }); // Example of using expireAfterSeconds to remove documents 1 hour // after their creation (db's timestampData option is true here) db.ensureIndex({ fieldName: 'createdAt', expireAfterSeconds: 3600 }, function (err) { }); // You can also use the option to set an expiration date like so db.ensureIndex({ fieldName: 'expirationDate', expireAfterSeconds: 0 }, function (err) { // Now all documents will expire when system time reaches the date in their // expirationDate field }); ``` **Note:** the `ensureIndex` function creates the index synchronously, so it's best to use it at application startup. It's quite fast so it doesn't increase startup time much (35 ms for a collection containing 10,000 documents). ## Browser version The browser version and its minified counterpart are in the `browser-version/out` directory. You only need to require `nedb.js` or `nedb.min.js` in your HTML file and the global object `Nedb` can be used right away, with the same API as the server version: ``` ``` If you specify a `filename`, the database will be persistent, and automatically select the best storage method available (IndexedDB, WebSQL or localStorage) depending on the browser. In most cases that means a lot of data can be stored, typically in hundreds of MB. **WARNING**: the storage system changed between v1.3 and v1.4 and is NOT back-compatible! Your application needs to resync client-side when you upgrade NeDB. NeDB is compatible with all major browsers: Chrome, Safari, Firefox, IE9+. Tests are in the `browser-version/test` directory (files `index.html` and `testPersistence.html`). If you fork and modify nedb, you can build the browser version from the sources, the build script is `browser-version/build.js`. ## Performance ### Speed NeDB is not intended to be a replacement of large-scale databases such as MongoDB, and as such was not designed for speed. That said, it is still pretty fast on the expected datasets, especially if you use indexing. On a typical, not-so-fast dev machine, for a collection containing 10,000 documents, with indexing: * Insert: **10,680 ops/s** * Find: **43,290 ops/s** * Update: **8,000 ops/s** * Remove: **11,750 ops/s** You can run these simple benchmarks by executing the scripts in the `benchmarks` folder. Run them with the `--help` flag to see how they work. ### Memory footprint A copy of the whole database is kept in memory. This is not much on the expected kind of datasets (20MB for 10,000 2KB documents). ## Use in other services * connect-nedb-session is a session store for Connect and Express, backed by nedb * If you mostly use NeDB for logging purposes and don't want the memory footprint of your application to grow too large, you can use NeDB Logger to insert documents in a NeDB-readable database * If you've outgrown NeDB, switching to MongoDB won't be too hard as it is the same API. Use this utility to transfer the data from a NeDB database to a MongoDB collection * An ODM for NeDB: Camo ## Pull requests **Important: I consider NeDB to be feature-complete, i.e. it does everything I think it should and nothing more. As a general rule I will not accept pull requests anymore, except for bugfixes (of course) or if I get convinced I overlook a strong usecase. Please make sure to open an issue before spending time on any PR.** If you submit a pull request, thanks! There are a couple rules to follow though to make it manageable: * The pull request should be atomic, i.e. contain only one feature. If it contains more, please submit multiple pull requests. Reviewing massive, 1000 loc+ pull requests is extremely hard. * Likewise, if for one unique feature the pull request grows too large (more than 200 loc tests not included), please get in touch first. * Please stick to the current coding style. It's important that the code uses a coherent style for readability. * Do not include sylistic improvements ("housekeeping"). If you think one part deserves lots of housekeeping, use a separate pull request so as not to pollute the code. * Don't forget tests for your new feature. Also don't forget to run the whole test suite before submitting to make sure you didn't introduce regressions. * Do not build the browser version in your branch, I'll take care of it once the code is merged. * Update the readme accordingly. * Last but not least: keep in mind what NeDB's mindset is! The goal is not to be a replacement for MongoDB, but to have a pure JS database, easy to use, cross platform, fast and expressive enough for the target projects (small and self contained apps on server/desktop/browser/mobile). Sometimes it's better to shoot for simplicity than for API completeness with regards to MongoDB. ## Bug reporting guidelines If you report a bug, thank you! That said for the process to be manageable please strictly adhere to the following guidelines. I'll not be able to handle bug reports that don't: * Your bug report should be a self-containing gist complete with a package.json for any dependencies you need. I need to run through a simple `git clone gist; npm install; node bugreport.js`, nothing more. * It should use assertions to showcase the expected vs actual behavior and be hysteresis-proof. It's quite simple in fact, see this example: https://gist.github.com/louischatriot/220cf6bd29c7de06a486 * Simplify as much as you can. Strip all your application-specific code. Most of the time you will see that there is no bug but an error in your code :) * 50 lines max. If you need more, read the above point and rework your bug report. If you're **really** convinced you need more, please explain precisely in the issue. * The code should be Javascript, not Coffeescript. ### Bitcoins You don't have time? You can support NeDB by sending bitcoins to this address: 1dDZLnWpBbodPiN8sizzYrgaz5iahFyb1 ## License See [License](LICENSE) ================================================ FILE: benchmarks/commonUtilities.js ================================================ /** * Functions that are used in several benchmark tests */ var customUtils = require('../lib/customUtils') , fs = require('fs') , path = require('path') , Datastore = require('../lib/datastore') , Persistence = require('../lib/persistence') , executeAsap // process.nextTick or setImmediate depending on your Node version ; try { executeAsap = setImmediate; } catch (e) { executeAsap = process.nextTick; } /** * Configure the benchmark */ module.exports.getConfiguration = function (benchDb) { var d, n , program = require('commander') ; program .option('-n --number [number]', 'Size of the collection to test on', parseInt) .option('-i --with-index', 'Use an index') .option('-m --in-memory', 'Test with an in-memory only store') .parse(process.argv); n = program.number || 10000; console.log("----------------------------"); console.log("Test with " + n + " documents"); console.log(program.withIndex ? "Use an index" : "Don't use an index"); console.log(program.inMemory ? "Use an in-memory datastore" : "Use a persistent datastore"); console.log("----------------------------"); d = new Datastore({ filename: benchDb , inMemoryOnly: program.inMemory }); return { n: n, d: d, program: program }; }; /** * Ensure the workspace exists and the db datafile is empty */ module.exports.prepareDb = function (filename, cb) { Persistence.ensureDirectoryExists(path.dirname(filename), function () { fs.exists(filename, function (exists) { if (exists) { fs.unlink(filename, cb); } else { return cb(); } }); }); }; /** * Return an array with the numbers from 0 to n-1, in a random order * Uses Fisher Yates algorithm * Useful to get fair tests */ function getRandomArray (n) { var res = [] , i, j, temp ; for (i = 0; i < n; i += 1) { res[i] = i; } for (i = n - 1; i >= 1; i -= 1) { j = Math.floor((i + 1) * Math.random()); temp = res[i]; res[i] = res[j]; res[j] = temp; } return res; }; module.exports.getRandomArray = getRandomArray; /** * Insert a certain number of documents for testing */ module.exports.insertDocs = function (d, n, profiler, cb) { var beg = new Date() , order = getRandomArray(n) ; profiler.step('Begin inserting ' + n + ' docs'); function runFrom(i) { if (i === n) { // Finished var opsPerSecond = Math.floor(1000* n / profiler.elapsedSinceLastStep()); console.log("===== RESULT (insert) ===== " + opsPerSecond + " ops/s"); profiler.step('Finished inserting ' + n + ' docs'); profiler.insertOpsPerSecond = opsPerSecond; return cb(); } d.insert({ docNumber: order[i] }, function (err) { executeAsap(function () { runFrom(i + 1); }); }); } runFrom(0); }; /** * Find documents with find */ module.exports.findDocs = function (d, n, profiler, cb) { var beg = new Date() , order = getRandomArray(n) ; profiler.step("Finding " + n + " documents"); function runFrom(i) { if (i === n) { // Finished console.log("===== RESULT (find) ===== " + Math.floor(1000* n / profiler.elapsedSinceLastStep()) + " ops/s"); profiler.step('Finished finding ' + n + ' docs'); return cb(); } d.find({ docNumber: order[i] }, function (err, docs) { if (docs.length !== 1 || docs[0].docNumber !== order[i]) { return cb('One find didnt work'); } executeAsap(function () { runFrom(i + 1); }); }); } runFrom(0); }; /** * Find documents with find and the $in operator */ module.exports.findDocsWithIn = function (d, n, profiler, cb) { var beg = new Date() , order = getRandomArray(n) , ins = [], i, j , arraySize = Math.min(10, n) // The array for $in needs to be smaller than n (inclusive) ; // Preparing all the $in arrays, will take some time for (i = 0; i < n; i += 1) { ins[i] = []; for (j = 0; j < arraySize; j += 1) { ins[i].push((i + j) % n); } } profiler.step("Finding " + n + " documents WITH $IN OPERATOR"); function runFrom(i) { if (i === n) { // Finished console.log("===== RESULT (find with in selector) ===== " + Math.floor(1000* n / profiler.elapsedSinceLastStep()) + " ops/s"); profiler.step('Finished finding ' + n + ' docs'); return cb(); } d.find({ docNumber: { $in: ins[i] } }, function (err, docs) { if (docs.length !== arraySize) { return cb('One find didnt work'); } executeAsap(function () { runFrom(i + 1); }); }); } runFrom(0); }; /** * Find documents with findOne */ module.exports.findOneDocs = function (d, n, profiler, cb) { var beg = new Date() , order = getRandomArray(n) ; profiler.step("FindingOne " + n + " documents"); function runFrom(i) { if (i === n) { // Finished console.log("===== RESULT (findOne) ===== " + Math.floor(1000* n / profiler.elapsedSinceLastStep()) + " ops/s"); profiler.step('Finished finding ' + n + ' docs'); return cb(); } d.findOne({ docNumber: order[i] }, function (err, doc) { if (!doc || doc.docNumber !== order[i]) { return cb('One find didnt work'); } executeAsap(function () { runFrom(i + 1); }); }); } runFrom(0); }; /** * Update documents * options is the same as the options object for update */ module.exports.updateDocs = function (options, d, n, profiler, cb) { var beg = new Date() , order = getRandomArray(n) ; profiler.step("Updating " + n + " documents"); function runFrom(i) { if (i === n) { // Finished console.log("===== RESULT (update) ===== " + Math.floor(1000* n / profiler.elapsedSinceLastStep()) + " ops/s"); profiler.step('Finished updating ' + n + ' docs'); return cb(); } // Will not actually modify the document but will take the same time d.update({ docNumber: order[i] }, { docNumber: order[i] }, options, function (err, nr) { if (err) { return cb(err); } if (nr !== 1) { return cb('One update didnt work'); } executeAsap(function () { runFrom(i + 1); }); }); } runFrom(0); }; /** * Remove documents * options is the same as the options object for update */ module.exports.removeDocs = function (options, d, n, profiler, cb) { var beg = new Date() , order = getRandomArray(n) ; profiler.step("Removing " + n + " documents"); function runFrom(i) { if (i === n) { // Finished // opsPerSecond corresponds to 1 insert + 1 remove, needed to keep collection size at 10,000 // We need to subtract the time taken by one insert to get the time actually taken by one remove var opsPerSecond = Math.floor(1000 * n / profiler.elapsedSinceLastStep()); var removeOpsPerSecond = Math.floor(1 / ((1 / opsPerSecond) - (1 / profiler.insertOpsPerSecond))) console.log("===== RESULT (remove) ===== " + removeOpsPerSecond + " ops/s"); profiler.step('Finished removing ' + n + ' docs'); return cb(); } d.remove({ docNumber: order[i] }, options, function (err, nr) { if (err) { return cb(err); } if (nr !== 1) { return cb('One remove didnt work'); } d.insert({ docNumber: order[i] }, function (err) { // We need to reinsert the doc so that we keep the collection's size at n // So actually we're calculating the average time taken by one insert + one remove executeAsap(function () { runFrom(i + 1); }); }); }); } runFrom(0); }; /** * Load database */ module.exports.loadDatabase = function (d, n, profiler, cb) { var beg = new Date() , order = getRandomArray(n) ; profiler.step("Loading the database " + n + " times"); function runFrom(i) { if (i === n) { // Finished console.log("===== RESULT ===== " + Math.floor(1000* n / profiler.elapsedSinceLastStep()) + " ops/s"); profiler.step('Finished loading a database' + n + ' times'); return cb(); } d.loadDatabase(function (err) { executeAsap(function () { runFrom(i + 1); }); }); } runFrom(0); }; ================================================ FILE: benchmarks/ensureIndex.js ================================================ var Datastore = require('../lib/datastore') , benchDb = 'workspace/insert.bench.db' , async = require('async') , commonUtilities = require('./commonUtilities') , execTime = require('exec-time') , profiler = new execTime('INSERT BENCH') , d = new Datastore(benchDb) , program = require('commander') , n ; program .option('-n --number [number]', 'Size of the collection to test on', parseInt) .option('-i --with-index', 'Test with an index') .parse(process.argv); n = program.number || 10000; console.log("----------------------------"); console.log("Test with " + n + " documents"); console.log("----------------------------"); async.waterfall([ async.apply(commonUtilities.prepareDb, benchDb) , function (cb) { d.loadDatabase(function (err) { if (err) { return cb(err); } cb(); }); } , function (cb) { profiler.beginProfiling(); return cb(); } , async.apply(commonUtilities.insertDocs, d, n, profiler) , function (cb) { var i; profiler.step('Begin calling ensureIndex ' + n + ' times'); for (i = 0; i < n; i += 1) { d.ensureIndex({ fieldName: 'docNumber' }); delete d.indexes.docNumber; } console.log("Average time for one ensureIndex: " + (profiler.elapsedSinceLastStep() / n) + "ms"); profiler.step('Finished calling ensureIndex ' + n + ' times'); } ], function (err) { profiler.step("Benchmark finished"); if (err) { return console.log("An error was encountered: ", err); } }); ================================================ FILE: benchmarks/find.js ================================================ var Datastore = require('../lib/datastore') , benchDb = 'workspace/find.bench.db' , fs = require('fs') , path = require('path') , async = require('async') , execTime = require('exec-time') , profiler = new execTime('FIND BENCH') , commonUtilities = require('./commonUtilities') , config = commonUtilities.getConfiguration(benchDb) , d = config.d , n = config.n ; async.waterfall([ async.apply(commonUtilities.prepareDb, benchDb) , function (cb) { d.loadDatabase(function (err) { if (err) { return cb(err); } if (config.program.withIndex) { d.ensureIndex({ fieldName: 'docNumber' }); } cb(); }); } , function (cb) { profiler.beginProfiling(); return cb(); } , async.apply(commonUtilities.insertDocs, d, n, profiler) , async.apply(commonUtilities.findDocs, d, n, profiler) ], function (err) { profiler.step("Benchmark finished"); if (err) { return console.log("An error was encountered: ", err); } }); ================================================ FILE: benchmarks/findOne.js ================================================ var Datastore = require('../lib/datastore') , benchDb = 'workspace/findOne.bench.db' , fs = require('fs') , path = require('path') , async = require('async') , execTime = require('exec-time') , profiler = new execTime('FINDONE BENCH') , commonUtilities = require('./commonUtilities') , config = commonUtilities.getConfiguration(benchDb) , d = config.d , n = config.n ; async.waterfall([ async.apply(commonUtilities.prepareDb, benchDb) , function (cb) { d.loadDatabase(function (err) { if (err) { return cb(err); } if (config.program.withIndex) { d.ensureIndex({ fieldName: 'docNumber' }); } cb(); }); } , function (cb) { profiler.beginProfiling(); return cb(); } , async.apply(commonUtilities.insertDocs, d, n, profiler) , function (cb) { setTimeout(function () {cb();}, 500); } , async.apply(commonUtilities.findOneDocs, d, n, profiler) ], function (err) { profiler.step("Benchmark finished"); if (err) { return console.log("An error was encountered: ", err); } }); ================================================ FILE: benchmarks/findWithIn.js ================================================ var Datastore = require('../lib/datastore') , benchDb = 'workspace/find.bench.db' , fs = require('fs') , path = require('path') , async = require('async') , execTime = require('exec-time') , profiler = new execTime('FIND BENCH') , commonUtilities = require('./commonUtilities') , config = commonUtilities.getConfiguration(benchDb) , d = config.d , n = config.n ; async.waterfall([ async.apply(commonUtilities.prepareDb, benchDb) , function (cb) { d.loadDatabase(function (err) { if (err) { return cb(err); } if (config.program.withIndex) { d.ensureIndex({ fieldName: 'docNumber' }); } cb(); }); } , function (cb) { profiler.beginProfiling(); return cb(); } , async.apply(commonUtilities.insertDocs, d, n, profiler) , async.apply(commonUtilities.findDocsWithIn, d, n, profiler) ], function (err) { profiler.step("Benchmark finished"); if (err) { return console.log("An error was encountered: ", err); } }); ================================================ FILE: benchmarks/insert.js ================================================ var Datastore = require('../lib/datastore') , benchDb = 'workspace/insert.bench.db' , async = require('async') , execTime = require('exec-time') , profiler = new execTime('INSERT BENCH') , commonUtilities = require('./commonUtilities') , config = commonUtilities.getConfiguration(benchDb) , d = config.d , n = config.n ; async.waterfall([ async.apply(commonUtilities.prepareDb, benchDb) , function (cb) { d.loadDatabase(function (err) { if (err) { return cb(err); } if (config.program.withIndex) { d.ensureIndex({ fieldName: 'docNumber' }); n = 2 * n; // We will actually insert twice as many documents // because the index is slower when the collection is already // big. So the result given by the algorithm will be a bit worse than // actual performance } cb(); }); } , function (cb) { profiler.beginProfiling(); return cb(); } , async.apply(commonUtilities.insertDocs, d, n, profiler) ], function (err) { profiler.step("Benchmark finished"); if (err) { return console.log("An error was encountered: ", err); } }); ================================================ FILE: benchmarks/loadDatabase.js ================================================ var Datastore = require('../lib/datastore') , benchDb = 'workspace/loaddb.bench.db' , fs = require('fs') , path = require('path') , async = require('async') , commonUtilities = require('./commonUtilities') , execTime = require('exec-time') , profiler = new execTime('LOADDB BENCH') , d = new Datastore(benchDb) , program = require('commander') , n ; program .option('-n --number [number]', 'Size of the collection to test on', parseInt) .option('-i --with-index', 'Test with an index') .parse(process.argv); n = program.number || 10000; console.log("----------------------------"); console.log("Test with " + n + " documents"); console.log(program.withIndex ? "Use an index" : "Don't use an index"); console.log("----------------------------"); async.waterfall([ async.apply(commonUtilities.prepareDb, benchDb) , function (cb) { d.loadDatabase(cb); } , function (cb) { profiler.beginProfiling(); return cb(); } , async.apply(commonUtilities.insertDocs, d, n, profiler) , async.apply(commonUtilities.loadDatabase, d, n, profiler) ], function (err) { profiler.step("Benchmark finished"); if (err) { return console.log("An error was encountered: ", err); } }); ================================================ FILE: benchmarks/remove.js ================================================ var Datastore = require('../lib/datastore') , benchDb = 'workspace/remove.bench.db' , fs = require('fs') , path = require('path') , async = require('async') , execTime = require('exec-time') , profiler = new execTime('REMOVE BENCH') , commonUtilities = require('./commonUtilities') , config = commonUtilities.getConfiguration(benchDb) , d = config.d , n = config.n ; async.waterfall([ async.apply(commonUtilities.prepareDb, benchDb) , function (cb) { d.loadDatabase(function (err) { if (err) { return cb(err); } if (config.program.withIndex) { d.ensureIndex({ fieldName: 'docNumber' }); } cb(); }); } , function (cb) { profiler.beginProfiling(); return cb(); } , async.apply(commonUtilities.insertDocs, d, n, profiler) // Test with remove only one document , function (cb) { profiler.step('MULTI: FALSE'); return cb(); } , async.apply(commonUtilities.removeDocs, { multi: false }, d, n, profiler) // Test with multiple documents , function (cb) { d.remove({}, { multi: true }, function () { return cb(); }); } , async.apply(commonUtilities.insertDocs, d, n, profiler) , function (cb) { profiler.step('MULTI: TRUE'); return cb(); } , async.apply(commonUtilities.removeDocs, { multi: true }, d, n, profiler) ], function (err) { profiler.step("Benchmark finished"); if (err) { return console.log("An error was encountered: ", err); } }); ================================================ FILE: benchmarks/update.js ================================================ var Datastore = require('../lib/datastore') , benchDb = 'workspace/update.bench.db' , fs = require('fs') , path = require('path') , async = require('async') , execTime = require('exec-time') , profiler = new execTime('UPDATE BENCH') , commonUtilities = require('./commonUtilities') , config = commonUtilities.getConfiguration(benchDb) , d = config.d , n = config.n ; async.waterfall([ async.apply(commonUtilities.prepareDb, benchDb) , function (cb) { d.loadDatabase(function (err) { if (err) { return cb(err); } if (config.program.withIndex) { d.ensureIndex({ fieldName: 'docNumber' }); } cb(); }); } , function (cb) { profiler.beginProfiling(); return cb(); } , async.apply(commonUtilities.insertDocs, d, n, profiler) // Test with update only one document , function (cb) { profiler.step('MULTI: FALSE'); return cb(); } , async.apply(commonUtilities.updateDocs, { multi: false }, d, n, profiler) // Test with multiple documents , function (cb) { d.remove({}, { multi: true }, function (err) { return cb(); }); } , async.apply(commonUtilities.insertDocs, d, n, profiler) , function (cb) { profiler.step('MULTI: TRUE'); return cb(); } , async.apply(commonUtilities.updateDocs, { multi: true }, d, n, profiler) ], function (err) { profiler.step("Benchmark finished"); if (err) { return console.log("An error was encountered: ", err); } }); ================================================ FILE: bower.json ================================================ { "name": "nedb", "description": "The Javascript Database for Node, nwjs, Electron and the browser", "ignore": ["benchmarks", "lib", "test", "test_lac"], "main": ["browser-version/nedb.js", "browser-version/nedb.min.js"] } ================================================ FILE: browser-version/browser-specific/lib/customUtils.js ================================================ /** * Specific customUtils for the browser, where we don't have access to the Crypto and Buffer modules */ /** * Taken from the crypto-browserify module * https://github.com/dominictarr/crypto-browserify * NOTE: Math.random() does not guarantee "cryptographic quality" but we actually don't need it */ function randomBytes (size) { var bytes = new Array(size); var r; for (var i = 0, r; i < size; i++) { if ((i & 0x03) == 0) r = Math.random() * 0x100000000; bytes[i] = r >>> ((i & 0x03) << 3) & 0xff; } return bytes; } /** * Taken from the base64-js module * https://github.com/beatgammit/base64-js/ */ function byteArrayToBase64 (uint8) { var lookup = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/' , extraBytes = uint8.length % 3 // if we have 1 byte left, pad 2 bytes , output = "" , temp, length, i; function tripletToBase64 (num) { return lookup[num >> 18 & 0x3F] + lookup[num >> 12 & 0x3F] + lookup[num >> 6 & 0x3F] + lookup[num & 0x3F]; }; // go through the array every three bytes, we'll deal with trailing stuff later for (i = 0, length = uint8.length - extraBytes; i < length; i += 3) { temp = (uint8[i] << 16) + (uint8[i + 1] << 8) + (uint8[i + 2]); output += tripletToBase64(temp); } // pad the end with zeros, but make sure to not forget the extra bytes switch (extraBytes) { case 1: temp = uint8[uint8.length - 1]; output += lookup[temp >> 2]; output += lookup[(temp << 4) & 0x3F]; output += '=='; break; case 2: temp = (uint8[uint8.length - 2] << 8) + (uint8[uint8.length - 1]); output += lookup[temp >> 10]; output += lookup[(temp >> 4) & 0x3F]; output += lookup[(temp << 2) & 0x3F]; output += '='; break; } return output; } /** * Return a random alphanumerical string of length len * There is a very small probability (less than 1/1,000,000) for the length to be less than len * (il the base64 conversion yields too many pluses and slashes) but * that's not an issue here * The probability of a collision is extremely small (need 3*10^12 documents to have one chance in a million of a collision) * See http://en.wikipedia.org/wiki/Birthday_problem */ function uid (len) { return byteArrayToBase64(randomBytes(Math.ceil(Math.max(8, len * 2)))).replace(/[+\/]/g, '').slice(0, len); } module.exports.uid = uid; ================================================ FILE: browser-version/browser-specific/lib/storage.js ================================================ /** * Way data is stored for this database * For a Node.js/Node Webkit database it's the file system * For a browser-side database it's localforage, which uses the best backend available (IndexedDB then WebSQL then localStorage) * * This version is the browser version */ var localforage = require('localforage') // Configure localforage to display NeDB name for now. Would be a good idea to let user use his own app name localforage.config({ name: 'NeDB' , storeName: 'nedbdata' }); function exists (filename, callback) { localforage.getItem(filename, function (err, value) { if (value !== null) { // Even if value is undefined, localforage returns null return callback(true); } else { return callback(false); } }); } function rename (filename, newFilename, callback) { localforage.getItem(filename, function (err, value) { if (value === null) { localforage.removeItem(newFilename, function () { return callback(); }); } else { localforage.setItem(newFilename, value, function () { localforage.removeItem(filename, function () { return callback(); }); }); } }); } function writeFile (filename, contents, options, callback) { // Options do not matter in browser setup if (typeof options === 'function') { callback = options; } localforage.setItem(filename, contents, function () { return callback(); }); } function appendFile (filename, toAppend, options, callback) { // Options do not matter in browser setup if (typeof options === 'function') { callback = options; } localforage.getItem(filename, function (err, contents) { contents = contents || ''; contents += toAppend; localforage.setItem(filename, contents, function () { return callback(); }); }); } function readFile (filename, options, callback) { // Options do not matter in browser setup if (typeof options === 'function') { callback = options; } localforage.getItem(filename, function (err, contents) { return callback(null, contents || ''); }); } function unlink (filename, callback) { localforage.removeItem(filename, function () { return callback(); }); } // Nothing to do, no directories will be used on the browser function mkdirp (dir, callback) { return callback(); } // Nothing to do, no data corruption possible in the brower function ensureDatafileIntegrity (filename, callback) { return callback(null); } // Interface module.exports.exists = exists; module.exports.rename = rename; module.exports.writeFile = writeFile; module.exports.crashSafeWriteFile = writeFile; // No need for a crash safe function in the browser module.exports.appendFile = appendFile; module.exports.readFile = readFile; module.exports.unlink = unlink; module.exports.mkdirp = mkdirp; module.exports.ensureDatafileIntegrity = ensureDatafileIntegrity; ================================================ FILE: browser-version/build.js ================================================ /** * Build the browser version of nedb */ var fs = require('fs') , path = require('path') , child_process = require('child_process') , toCopy = ['lib', 'node_modules'] , async, browserify, uglify ; // Ensuring both node_modules (the source one and build one), src and out directories exist function ensureDirExists (name) { try { fs.mkdirSync(path.join(__dirname, name)); } catch (e) { if (e.code !== 'EEXIST') { console.log("Error ensuring that node_modules exists"); process.exit(1); } } } ensureDirExists('../node_modules'); ensureDirExists('node_modules'); ensureDirExists('out'); ensureDirExists('src'); // Installing build dependencies and require them console.log("Installing build dependencies"); child_process.exec('npm install', { cwd: __dirname }, function (err, stdout, stderr) { if (err) { console.log("Error reinstalling dependencies"); process.exit(1); } fs = require('fs-extra'); async = require('async'); browserify = require('browserify'); uglify = require('uglify-js'); async.waterfall([ function (cb) { console.log("Installing source dependencies if needed"); child_process.exec('npm install', { cwd: path.join(__dirname, '..') }, function (err) { return cb(err); }); } , function (cb) { console.log("Removing contents of the src directory"); async.eachSeries(fs.readdirSync(path.join(__dirname, 'src')), function (item, _cb) { fs.remove(path.join(__dirname, 'src', item), _cb); }, cb); } , function (cb) { console.log("Copying source files"); async.eachSeries(toCopy, function (item, _cb) { fs.copy(path.join(__dirname, '..', item), path.join(__dirname, 'src', item), _cb); }, cb); } , function (cb) { console.log("Copying browser specific files to replace their server-specific counterparts"); async.eachSeries(fs.readdirSync(path.join(__dirname, 'browser-specific')), function (item, _cb) { fs.copy(path.join(__dirname, 'browser-specific', item), path.join(__dirname, 'src', item), _cb); }, cb); } , function (cb) { console.log("Browserifying the code"); var b = browserify() , srcPath = path.join(__dirname, 'src/lib/datastore.js'); b.add(srcPath); b.bundle({ standalone: 'Nedb' }, function (err, out) { if (err) { return cb(err); } fs.writeFile(path.join(__dirname, 'out/nedb.js'), out, 'utf8', function (err) { if (err) { return cb(err); } else { return cb(null, out); } }); }); } , function (out, cb) { console.log("Creating the minified version"); var compressedCode = uglify.minify(out, { fromString: true }); fs.writeFile(path.join(__dirname, 'out/nedb.min.js'), compressedCode.code, 'utf8', cb); } ], function (err) { if (err) { console.log("Error during build"); console.log(err); } else { console.log("Build finished with success"); } }); }); ================================================ FILE: browser-version/out/nedb.js ================================================ (function(e){if("function"==typeof bootstrap)bootstrap("nedb",e);else if("object"==typeof exports)module.exports=e();else if("function"==typeof define&&define.amd)define(e);else if("undefined"!=typeof ses){if(!ses.ok())return;ses.makeNedb=e}else"undefined"!=typeof window?window.Nedb=e():global.Nedb=e()})(function(){var define,ses,bootstrap,module,exports; return (function(e,t,n){function i(n,s){if(!t[n]){if(!e[n]){var o=typeof require=="function"&&require;if(!s&&o)return o(n,!0);if(r)return r(n,!0);throw new Error("Cannot find module '"+n+"'")}var u=t[n]={exports:{}};e[n][0].call(u.exports,function(t){var r=e[n][1][t];return i(r?r:t)},u,u.exports)}return t[n].exports}var r=typeof require=="function"&&require;for(var s=0;s 0 && this._events[type].length > m) { this._events[type].warned = true; console.error('(node) warning: possible EventEmitter memory ' + 'leak detected. %d listeners added. ' + 'Use emitter.setMaxListeners() to increase limit.', this._events[type].length); console.trace(); } } // If we've already got an array, just append. this._events[type].push(listener); } else { // Adding the second element, need to change to array. this._events[type] = [this._events[type], listener]; } return this; }; EventEmitter.prototype.on = EventEmitter.prototype.addListener; EventEmitter.prototype.once = function(type, listener) { var self = this; self.on(type, function g() { self.removeListener(type, g); listener.apply(this, arguments); }); return this; }; EventEmitter.prototype.removeListener = function(type, listener) { if ('function' !== typeof listener) { throw new Error('removeListener only takes instances of Function'); } // does not use listeners(), so no side effect of creating _events[type] if (!this._events || !this._events[type]) return this; var list = this._events[type]; if (isArray(list)) { var i = indexOf(list, listener); if (i < 0) return this; list.splice(i, 1); if (list.length == 0) delete this._events[type]; } else if (this._events[type] === listener) { delete this._events[type]; } return this; }; EventEmitter.prototype.removeAllListeners = function(type) { if (arguments.length === 0) { this._events = {}; return this; } // does not use listeners(), so no side effect of creating _events[type] if (type && this._events && this._events[type]) this._events[type] = null; return this; }; EventEmitter.prototype.listeners = function(type) { if (!this._events) this._events = {}; if (!this._events[type]) this._events[type] = []; if (!isArray(this._events[type])) { this._events[type] = [this._events[type]]; } return this._events[type]; }; EventEmitter.listenerCount = function(emitter, type) { var ret; if (!emitter._events || !emitter._events[type]) ret = 0; else if (typeof emitter._events[type] === 'function') ret = 1; else ret = emitter._events[type].length; return ret; }; },{"__browserify_process":4}],2:[function(require,module,exports){ var process=require("__browserify_process");function filter (xs, fn) { var res = []; for (var i = 0; i < xs.length; i++) { if (fn(xs[i], i, xs)) res.push(xs[i]); } return res; } // resolves . and .. elements in a path array with directory names there // must be no slashes, empty elements, or device names (c:\) in the array // (so also no leading and trailing slashes - it does not distinguish // relative and absolute paths) function normalizeArray(parts, allowAboveRoot) { // if the path tries to go above the root, `up` ends up > 0 var up = 0; for (var i = parts.length; i >= 0; i--) { var last = parts[i]; if (last == '.') { parts.splice(i, 1); } else if (last === '..') { parts.splice(i, 1); up++; } else if (up) { parts.splice(i, 1); up--; } } // if the path is allowed to go above the root, restore leading ..s if (allowAboveRoot) { for (; up--; up) { parts.unshift('..'); } } return parts; } // Regex to split a filename into [*, dir, basename, ext] // posix version var splitPathRe = /^(.+\/(?!$)|\/)?((?:.+?)?(\.[^.]*)?)$/; // path.resolve([from ...], to) // posix version exports.resolve = function() { var resolvedPath = '', resolvedAbsolute = false; for (var i = arguments.length; i >= -1 && !resolvedAbsolute; i--) { var path = (i >= 0) ? arguments[i] : process.cwd(); // Skip empty and invalid entries if (typeof path !== 'string' || !path) { continue; } resolvedPath = path + '/' + resolvedPath; resolvedAbsolute = path.charAt(0) === '/'; } // At this point the path should be resolved to a full absolute path, but // handle relative paths to be safe (might happen when process.cwd() fails) // Normalize the path resolvedPath = normalizeArray(filter(resolvedPath.split('/'), function(p) { return !!p; }), !resolvedAbsolute).join('/'); return ((resolvedAbsolute ? '/' : '') + resolvedPath) || '.'; }; // path.normalize(path) // posix version exports.normalize = function(path) { var isAbsolute = path.charAt(0) === '/', trailingSlash = path.slice(-1) === '/'; // Normalize the path path = normalizeArray(filter(path.split('/'), function(p) { return !!p; }), !isAbsolute).join('/'); if (!path && !isAbsolute) { path = '.'; } if (path && trailingSlash) { path += '/'; } return (isAbsolute ? '/' : '') + path; }; // posix version exports.join = function() { var paths = Array.prototype.slice.call(arguments, 0); return exports.normalize(filter(paths, function(p, index) { return p && typeof p === 'string'; }).join('/')); }; exports.dirname = function(path) { var dir = splitPathRe.exec(path)[1] || ''; var isWindows = false; if (!dir) { // No dirname return '.'; } else if (dir.length === 1 || (isWindows && dir.length <= 3 && dir.charAt(1) === ':')) { // It is just a slash or a drive letter with a slash return dir; } else { // It is a full dirname, strip trailing slash return dir.substring(0, dir.length - 1); } }; exports.basename = function(path, ext) { var f = splitPathRe.exec(path)[2] || ''; // TODO: make this comparison case-insensitive on windows? if (ext && f.substr(-1 * ext.length) === ext) { f = f.substr(0, f.length - ext.length); } return f; }; exports.extname = function(path) { return splitPathRe.exec(path)[3] || ''; }; exports.relative = function(from, to) { from = exports.resolve(from).substr(1); to = exports.resolve(to).substr(1); function trim(arr) { var start = 0; for (; start < arr.length; start++) { if (arr[start] !== '') break; } var end = arr.length - 1; for (; end >= 0; end--) { if (arr[end] !== '') break; } if (start > end) return []; return arr.slice(start, end - start + 1); } var fromParts = trim(from.split('/')); var toParts = trim(to.split('/')); var length = Math.min(fromParts.length, toParts.length); var samePartsLength = length; for (var i = 0; i < length; i++) { if (fromParts[i] !== toParts[i]) { samePartsLength = i; break; } } var outputParts = []; for (var i = samePartsLength; i < fromParts.length; i++) { outputParts.push('..'); } outputParts = outputParts.concat(toParts.slice(samePartsLength)); return outputParts.join('/'); }; exports.sep = '/'; },{"__browserify_process":4}],3:[function(require,module,exports){ var events = require('events'); exports.isArray = isArray; exports.isDate = function(obj){return Object.prototype.toString.call(obj) === '[object Date]'}; exports.isRegExp = function(obj){return Object.prototype.toString.call(obj) === '[object RegExp]'}; exports.print = function () {}; exports.puts = function () {}; exports.debug = function() {}; exports.inspect = function(obj, showHidden, depth, colors) { var seen = []; var stylize = function(str, styleType) { // http://en.wikipedia.org/wiki/ANSI_escape_code#graphics var styles = { 'bold' : [1, 22], 'italic' : [3, 23], 'underline' : [4, 24], 'inverse' : [7, 27], 'white' : [37, 39], 'grey' : [90, 39], 'black' : [30, 39], 'blue' : [34, 39], 'cyan' : [36, 39], 'green' : [32, 39], 'magenta' : [35, 39], 'red' : [31, 39], 'yellow' : [33, 39] }; var style = { 'special': 'cyan', 'number': 'blue', 'boolean': 'yellow', 'undefined': 'grey', 'null': 'bold', 'string': 'green', 'date': 'magenta', // "name": intentionally not styling 'regexp': 'red' }[styleType]; if (style) { return '\u001b[' + styles[style][0] + 'm' + str + '\u001b[' + styles[style][1] + 'm'; } else { return str; } }; if (! colors) { stylize = function(str, styleType) { return str; }; } function format(value, recurseTimes) { // Provide a hook for user-specified inspect functions. // Check that value is an object with an inspect function on it if (value && typeof value.inspect === 'function' && // Filter out the util module, it's inspect function is special value !== exports && // Also filter out any prototype objects using the circular check. !(value.constructor && value.constructor.prototype === value)) { return value.inspect(recurseTimes); } // Primitive types cannot have properties switch (typeof value) { case 'undefined': return stylize('undefined', 'undefined'); case 'string': var simple = '\'' + JSON.stringify(value).replace(/^"|"$/g, '') .replace(/'/g, "\\'") .replace(/\\"/g, '"') + '\''; return stylize(simple, 'string'); case 'number': return stylize('' + value, 'number'); case 'boolean': return stylize('' + value, 'boolean'); } // For some reason typeof null is "object", so special case here. if (value === null) { return stylize('null', 'null'); } // Look up the keys of the object. var visible_keys = Object_keys(value); var keys = showHidden ? Object_getOwnPropertyNames(value) : visible_keys; // Functions without properties can be shortcutted. if (typeof value === 'function' && keys.length === 0) { if (isRegExp(value)) { return stylize('' + value, 'regexp'); } else { var name = value.name ? ': ' + value.name : ''; return stylize('[Function' + name + ']', 'special'); } } // Dates without properties can be shortcutted if (isDate(value) && keys.length === 0) { return stylize(value.toUTCString(), 'date'); } var base, type, braces; // Determine the object type if (isArray(value)) { type = 'Array'; braces = ['[', ']']; } else { type = 'Object'; braces = ['{', '}']; } // Make functions say that they are functions if (typeof value === 'function') { var n = value.name ? ': ' + value.name : ''; base = (isRegExp(value)) ? ' ' + value : ' [Function' + n + ']'; } else { base = ''; } // Make dates with properties first say the date if (isDate(value)) { base = ' ' + value.toUTCString(); } if (keys.length === 0) { return braces[0] + base + braces[1]; } if (recurseTimes < 0) { if (isRegExp(value)) { return stylize('' + value, 'regexp'); } else { return stylize('[Object]', 'special'); } } seen.push(value); var output = keys.map(function(key) { var name, str; if (value.__lookupGetter__) { if (value.__lookupGetter__(key)) { if (value.__lookupSetter__(key)) { str = stylize('[Getter/Setter]', 'special'); } else { str = stylize('[Getter]', 'special'); } } else { if (value.__lookupSetter__(key)) { str = stylize('[Setter]', 'special'); } } } if (visible_keys.indexOf(key) < 0) { name = '[' + key + ']'; } if (!str) { if (seen.indexOf(value[key]) < 0) { if (recurseTimes === null) { str = format(value[key]); } else { str = format(value[key], recurseTimes - 1); } if (str.indexOf('\n') > -1) { if (isArray(value)) { str = str.split('\n').map(function(line) { return ' ' + line; }).join('\n').substr(2); } else { str = '\n' + str.split('\n').map(function(line) { return ' ' + line; }).join('\n'); } } } else { str = stylize('[Circular]', 'special'); } } if (typeof name === 'undefined') { if (type === 'Array' && key.match(/^\d+$/)) { return str; } name = JSON.stringify('' + key); if (name.match(/^"([a-zA-Z_][a-zA-Z_0-9]*)"$/)) { name = name.substr(1, name.length - 2); name = stylize(name, 'name'); } else { name = name.replace(/'/g, "\\'") .replace(/\\"/g, '"') .replace(/(^"|"$)/g, "'"); name = stylize(name, 'string'); } } return name + ': ' + str; }); seen.pop(); var numLinesEst = 0; var length = output.reduce(function(prev, cur) { numLinesEst++; if (cur.indexOf('\n') >= 0) numLinesEst++; return prev + cur.length + 1; }, 0); if (length > 50) { output = braces[0] + (base === '' ? '' : base + '\n ') + ' ' + output.join(',\n ') + ' ' + braces[1]; } else { output = braces[0] + base + ' ' + output.join(', ') + ' ' + braces[1]; } return output; } return format(obj, (typeof depth === 'undefined' ? 2 : depth)); }; function isArray(ar) { return Array.isArray(ar) || (typeof ar === 'object' && Object.prototype.toString.call(ar) === '[object Array]'); } function isRegExp(re) { typeof re === 'object' && Object.prototype.toString.call(re) === '[object RegExp]'; } function isDate(d) { return typeof d === 'object' && Object.prototype.toString.call(d) === '[object Date]'; } function pad(n) { return n < 10 ? '0' + n.toString(10) : n.toString(10); } var months = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']; // 26 Feb 16:19:34 function timestamp() { var d = new Date(); var time = [pad(d.getHours()), pad(d.getMinutes()), pad(d.getSeconds())].join(':'); return [d.getDate(), months[d.getMonth()], time].join(' '); } exports.log = function (msg) {}; exports.pump = null; var Object_keys = Object.keys || function (obj) { var res = []; for (var key in obj) res.push(key); return res; }; var Object_getOwnPropertyNames = Object.getOwnPropertyNames || function (obj) { var res = []; for (var key in obj) { if (Object.hasOwnProperty.call(obj, key)) res.push(key); } return res; }; var Object_create = Object.create || function (prototype, properties) { // from es5-shim var object; if (prototype === null) { object = { '__proto__' : null }; } else { if (typeof prototype !== 'object') { throw new TypeError( 'typeof prototype[' + (typeof prototype) + '] != \'object\'' ); } var Type = function () {}; Type.prototype = prototype; object = new Type(); object.__proto__ = prototype; } if (typeof properties !== 'undefined' && Object.defineProperties) { Object.defineProperties(object, properties); } return object; }; exports.inherits = function(ctor, superCtor) { ctor.super_ = superCtor; ctor.prototype = Object_create(superCtor.prototype, { constructor: { value: ctor, enumerable: false, writable: true, configurable: true } }); }; var formatRegExp = /%[sdj%]/g; exports.format = function(f) { if (typeof f !== 'string') { var objects = []; for (var i = 0; i < arguments.length; i++) { objects.push(exports.inspect(arguments[i])); } return objects.join(' '); } var i = 1; var args = arguments; var len = args.length; var str = String(f).replace(formatRegExp, function(x) { if (x === '%%') return '%'; if (i >= len) return x; switch (x) { case '%s': return String(args[i++]); case '%d': return Number(args[i++]); case '%j': return JSON.stringify(args[i++]); default: return x; } }); for(var x = args[i]; i < len; x = args[++i]){ if (x === null || typeof x !== 'object') { str += ' ' + x; } else { str += ' ' + exports.inspect(x); } } return str; }; },{"events":1}],4:[function(require,module,exports){ // shim for using process in browser var process = module.exports = {}; process.nextTick = (function () { var canSetImmediate = typeof window !== 'undefined' && window.setImmediate; var canPost = typeof window !== 'undefined' && window.postMessage && window.addEventListener ; if (canSetImmediate) { return function (f) { return window.setImmediate(f) }; } if (canPost) { var queue = []; window.addEventListener('message', function (ev) { var source = ev.source; if ((source === window || source === null) && ev.data === 'process-tick') { ev.stopPropagation(); if (queue.length > 0) { var fn = queue.shift(); fn(); } } }, true); return function nextTick(fn) { queue.push(fn); window.postMessage('process-tick', '*'); }; } return function nextTick(fn) { setTimeout(fn, 0); }; })(); process.title = 'browser'; process.browser = true; process.env = {}; process.argv = []; process.binding = function (name) { throw new Error('process.binding is not supported'); } // TODO(shtylman) process.cwd = function () { return '/' }; process.chdir = function (dir) { throw new Error('process.chdir is not supported'); }; },{}],5:[function(require,module,exports){ /** * Manage access to data, be it to find, update or remove it */ var model = require('./model') , _ = require('underscore') ; /** * Create a new cursor for this collection * @param {Datastore} db - The datastore this cursor is bound to * @param {Query} query - The query this cursor will operate on * @param {Function} execFn - Handler to be executed after cursor has found the results and before the callback passed to find/findOne/update/remove */ function Cursor (db, query, execFn) { this.db = db; this.query = query || {}; if (execFn) { this.execFn = execFn; } } /** * Set a limit to the number of results */ Cursor.prototype.limit = function(limit) { this._limit = limit; return this; }; /** * Skip a the number of results */ Cursor.prototype.skip = function(skip) { this._skip = skip; return this; }; /** * Sort results of the query * @param {SortQuery} sortQuery - SortQuery is { field: order }, field can use the dot-notation, order is 1 for ascending and -1 for descending */ Cursor.prototype.sort = function(sortQuery) { this._sort = sortQuery; return this; }; /** * Add the use of a projection * @param {Object} projection - MongoDB-style projection. {} means take all fields. Then it's { key1: 1, key2: 1 } to take only key1 and key2 * { key1: 0, key2: 0 } to omit only key1 and key2. Except _id, you can't mix takes and omits */ Cursor.prototype.projection = function(projection) { this._projection = projection; return this; }; /** * Apply the projection */ Cursor.prototype.project = function (candidates) { var res = [], self = this , keepId, action, keys ; if (this._projection === undefined || Object.keys(this._projection).length === 0) { return candidates; } keepId = this._projection._id === 0 ? false : true; this._projection = _.omit(this._projection, '_id'); // Check for consistency keys = Object.keys(this._projection); keys.forEach(function (k) { if (action !== undefined && self._projection[k] !== action) { throw new Error("Can't both keep and omit fields except for _id"); } action = self._projection[k]; }); // Do the actual projection candidates.forEach(function (candidate) { var toPush; if (action === 1) { // pick-type projection toPush = { $set: {} }; keys.forEach(function (k) { toPush.$set[k] = model.getDotValue(candidate, k); if (toPush.$set[k] === undefined) { delete toPush.$set[k]; } }); toPush = model.modify({}, toPush); } else { // omit-type projection toPush = { $unset: {} }; keys.forEach(function (k) { toPush.$unset[k] = true }); toPush = model.modify(candidate, toPush); } if (keepId) { toPush._id = candidate._id; } else { delete toPush._id; } res.push(toPush); }); return res; }; /** * Get all matching elements * Will return pointers to matched elements (shallow copies), returning full copies is the role of find or findOne * This is an internal function, use exec which uses the executor * * @param {Function} callback - Signature: err, results */ Cursor.prototype._exec = function(_callback) { var res = [], added = 0, skipped = 0, self = this , error = null , i, keys, key ; function callback (error, res) { if (self.execFn) { return self.execFn(error, res, _callback); } else { return _callback(error, res); } } this.db.getCandidates(this.query, function (err, candidates) { if (err) { return callback(err); } try { for (i = 0; i < candidates.length; i += 1) { if (model.match(candidates[i], self.query)) { // If a sort is defined, wait for the results to be sorted before applying limit and skip if (!self._sort) { if (self._skip && self._skip > skipped) { skipped += 1; } else { res.push(candidates[i]); added += 1; if (self._limit && self._limit <= added) { break; } } } else { res.push(candidates[i]); } } } } catch (err) { return callback(err); } // Apply all sorts if (self._sort) { keys = Object.keys(self._sort); // Sorting var criteria = []; for (i = 0; i < keys.length; i++) { key = keys[i]; criteria.push({ key: key, direction: self._sort[key] }); } res.sort(function(a, b) { var criterion, compare, i; for (i = 0; i < criteria.length; i++) { criterion = criteria[i]; compare = criterion.direction * model.compareThings(model.getDotValue(a, criterion.key), model.getDotValue(b, criterion.key), self.db.compareStrings); if (compare !== 0) { return compare; } } return 0; }); // Applying limit and skip var limit = self._limit || res.length , skip = self._skip || 0; res = res.slice(skip, skip + limit); } // Apply projection try { res = self.project(res); } catch (e) { error = e; res = undefined; } return callback(error, res); }); }; Cursor.prototype.exec = function () { this.db.executor.push({ this: this, fn: this._exec, arguments: arguments }); }; // Interface module.exports = Cursor; },{"./model":10,"underscore":19}],6:[function(require,module,exports){ /** * Specific customUtils for the browser, where we don't have access to the Crypto and Buffer modules */ /** * Taken from the crypto-browserify module * https://github.com/dominictarr/crypto-browserify * NOTE: Math.random() does not guarantee "cryptographic quality" but we actually don't need it */ function randomBytes (size) { var bytes = new Array(size); var r; for (var i = 0, r; i < size; i++) { if ((i & 0x03) == 0) r = Math.random() * 0x100000000; bytes[i] = r >>> ((i & 0x03) << 3) & 0xff; } return bytes; } /** * Taken from the base64-js module * https://github.com/beatgammit/base64-js/ */ function byteArrayToBase64 (uint8) { var lookup = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/' , extraBytes = uint8.length % 3 // if we have 1 byte left, pad 2 bytes , output = "" , temp, length, i; function tripletToBase64 (num) { return lookup[num >> 18 & 0x3F] + lookup[num >> 12 & 0x3F] + lookup[num >> 6 & 0x3F] + lookup[num & 0x3F]; }; // go through the array every three bytes, we'll deal with trailing stuff later for (i = 0, length = uint8.length - extraBytes; i < length; i += 3) { temp = (uint8[i] << 16) + (uint8[i + 1] << 8) + (uint8[i + 2]); output += tripletToBase64(temp); } // pad the end with zeros, but make sure to not forget the extra bytes switch (extraBytes) { case 1: temp = uint8[uint8.length - 1]; output += lookup[temp >> 2]; output += lookup[(temp << 4) & 0x3F]; output += '=='; break; case 2: temp = (uint8[uint8.length - 2] << 8) + (uint8[uint8.length - 1]); output += lookup[temp >> 10]; output += lookup[(temp >> 4) & 0x3F]; output += lookup[(temp << 2) & 0x3F]; output += '='; break; } return output; } /** * Return a random alphanumerical string of length len * There is a very small probability (less than 1/1,000,000) for the length to be less than len * (il the base64 conversion yields too many pluses and slashes) but * that's not an issue here * The probability of a collision is extremely small (need 3*10^12 documents to have one chance in a million of a collision) * See http://en.wikipedia.org/wiki/Birthday_problem */ function uid (len) { return byteArrayToBase64(randomBytes(Math.ceil(Math.max(8, len * 2)))).replace(/[+\/]/g, '').slice(0, len); } module.exports.uid = uid; },{}],7:[function(require,module,exports){ var customUtils = require('./customUtils') , model = require('./model') , async = require('async') , Executor = require('./executor') , Index = require('./indexes') , util = require('util') , _ = require('underscore') , Persistence = require('./persistence') , Cursor = require('./cursor') ; /** * Create a new collection * @param {String} options.filename Optional, datastore will be in-memory only if not provided * @param {Boolean} options.timestampData Optional, defaults to false. If set to true, createdAt and updatedAt will be created and populated automatically (if not specified by user) * @param {Boolean} options.inMemoryOnly Optional, defaults to false * @param {String} options.nodeWebkitAppName Optional, specify the name of your NW app if you want options.filename to be relative to the directory where * Node Webkit stores application data such as cookies and local storage (the best place to store data in my opinion) * @param {Boolean} options.autoload Optional, defaults to false * @param {Function} options.onload Optional, if autoload is used this will be called after the load database with the error object as parameter. If you don't pass it the error will be thrown * @param {Function} options.afterSerialization/options.beforeDeserialization Optional, serialization hooks * @param {Number} options.corruptAlertThreshold Optional, threshold after which an alert is thrown if too much data is corrupt * @param {Function} options.compareStrings Optional, string comparison function that overrides default for sorting * * Event Emitter - Events * * compaction.done - Fired whenever a compaction operation was finished */ function Datastore (options) { var filename; // Retrocompatibility with v0.6 and before if (typeof options === 'string') { filename = options; this.inMemoryOnly = false; // Default } else { options = options || {}; filename = options.filename; this.inMemoryOnly = options.inMemoryOnly || false; this.autoload = options.autoload || false; this.timestampData = options.timestampData || false; } // Determine whether in memory or persistent if (!filename || typeof filename !== 'string' || filename.length === 0) { this.filename = null; this.inMemoryOnly = true; } else { this.filename = filename; } // String comparison function this.compareStrings = options.compareStrings; // Persistence handling this.persistence = new Persistence({ db: this, nodeWebkitAppName: options.nodeWebkitAppName , afterSerialization: options.afterSerialization , beforeDeserialization: options.beforeDeserialization , corruptAlertThreshold: options.corruptAlertThreshold }); // This new executor is ready if we don't use persistence // If we do, it will only be ready once loadDatabase is called this.executor = new Executor(); if (this.inMemoryOnly) { this.executor.ready = true; } // Indexed by field name, dot notation can be used // _id is always indexed and since _ids are generated randomly the underlying // binary is always well-balanced this.indexes = {}; this.indexes._id = new Index({ fieldName: '_id', unique: true }); this.ttlIndexes = {}; // Queue a load of the database right away and call the onload handler // By default (no onload handler), if there is an error there, no operation will be possible so warn the user by throwing an exception if (this.autoload) { this.loadDatabase(options.onload || function (err) { if (err) { throw err; } }); } } util.inherits(Datastore, require('events').EventEmitter); /** * Load the database from the datafile, and trigger the execution of buffered commands if any */ Datastore.prototype.loadDatabase = function () { this.executor.push({ this: this.persistence, fn: this.persistence.loadDatabase, arguments: arguments }, true); }; /** * Get an array of all the data in the database */ Datastore.prototype.getAllData = function () { return this.indexes._id.getAll(); }; /** * Reset all currently defined indexes */ Datastore.prototype.resetIndexes = function (newData) { var self = this; Object.keys(this.indexes).forEach(function (i) { self.indexes[i].reset(newData); }); }; /** * Ensure an index is kept for this field. Same parameters as lib/indexes * For now this function is synchronous, we need to test how much time it takes * We use an async API for consistency with the rest of the code * @param {String} options.fieldName * @param {Boolean} options.unique * @param {Boolean} options.sparse * @param {Number} options.expireAfterSeconds - Optional, if set this index becomes a TTL index (only works on Date fields, not arrays of Date) * @param {Function} cb Optional callback, signature: err */ Datastore.prototype.ensureIndex = function (options, cb) { var err , callback = cb || function () {}; options = options || {}; if (!options.fieldName) { err = new Error("Cannot create an index without a fieldName"); err.missingFieldName = true; return callback(err); } if (this.indexes[options.fieldName]) { return callback(null); } this.indexes[options.fieldName] = new Index(options); if (options.expireAfterSeconds !== undefined) { this.ttlIndexes[options.fieldName] = options.expireAfterSeconds; } // With this implementation index creation is not necessary to ensure TTL but we stick with MongoDB's API here try { this.indexes[options.fieldName].insert(this.getAllData()); } catch (e) { delete this.indexes[options.fieldName]; return callback(e); } // We may want to force all options to be persisted including defaults, not just the ones passed the index creation function this.persistence.persistNewState([{ $$indexCreated: options }], function (err) { if (err) { return callback(err); } return callback(null); }); }; /** * Remove an index * @param {String} fieldName * @param {Function} cb Optional callback, signature: err */ Datastore.prototype.removeIndex = function (fieldName, cb) { var callback = cb || function () {}; delete this.indexes[fieldName]; this.persistence.persistNewState([{ $$indexRemoved: fieldName }], function (err) { if (err) { return callback(err); } return callback(null); }); }; /** * Add one or several document(s) to all indexes */ Datastore.prototype.addToIndexes = function (doc) { var i, failingIndex, error , keys = Object.keys(this.indexes) ; for (i = 0; i < keys.length; i += 1) { try { this.indexes[keys[i]].insert(doc); } catch (e) { failingIndex = i; error = e; break; } } // If an error happened, we need to rollback the insert on all other indexes if (error) { for (i = 0; i < failingIndex; i += 1) { this.indexes[keys[i]].remove(doc); } throw error; } }; /** * Remove one or several document(s) from all indexes */ Datastore.prototype.removeFromIndexes = function (doc) { var self = this; Object.keys(this.indexes).forEach(function (i) { self.indexes[i].remove(doc); }); }; /** * Update one or several documents in all indexes * To update multiple documents, oldDoc must be an array of { oldDoc, newDoc } pairs * If one update violates a constraint, all changes are rolled back */ Datastore.prototype.updateIndexes = function (oldDoc, newDoc) { var i, failingIndex, error , keys = Object.keys(this.indexes) ; for (i = 0; i < keys.length; i += 1) { try { this.indexes[keys[i]].update(oldDoc, newDoc); } catch (e) { failingIndex = i; error = e; break; } } // If an error happened, we need to rollback the update on all other indexes if (error) { for (i = 0; i < failingIndex; i += 1) { this.indexes[keys[i]].revertUpdate(oldDoc, newDoc); } throw error; } }; /** * Return the list of candidates for a given query * Crude implementation for now, we return the candidates given by the first usable index if any * We try the following query types, in this order: basic match, $in match, comparison match * One way to make it better would be to enable the use of multiple indexes if the first usable index * returns too much data. I may do it in the future. * * Returned candidates will be scanned to find and remove all expired documents * * @param {Query} query * @param {Boolean} dontExpireStaleDocs Optional, defaults to false, if true don't remove stale docs. Useful for the remove function which shouldn't be impacted by expirations * @param {Function} callback Signature err, candidates */ Datastore.prototype.getCandidates = function (query, dontExpireStaleDocs, callback) { var indexNames = Object.keys(this.indexes) , self = this , usableQueryKeys; if (typeof dontExpireStaleDocs === 'function') { callback = dontExpireStaleDocs; dontExpireStaleDocs = false; } async.waterfall([ // STEP 1: get candidates list by checking indexes from most to least frequent usecase function (cb) { // For a basic match usableQueryKeys = []; Object.keys(query).forEach(function (k) { if (typeof query[k] === 'string' || typeof query[k] === 'number' || typeof query[k] === 'boolean' || util.isDate(query[k]) || query[k] === null) { usableQueryKeys.push(k); } }); usableQueryKeys = _.intersection(usableQueryKeys, indexNames); if (usableQueryKeys.length > 0) { return cb(null, self.indexes[usableQueryKeys[0]].getMatching(query[usableQueryKeys[0]])); } // For a $in match usableQueryKeys = []; Object.keys(query).forEach(function (k) { if (query[k] && query[k].hasOwnProperty('$in')) { usableQueryKeys.push(k); } }); usableQueryKeys = _.intersection(usableQueryKeys, indexNames); if (usableQueryKeys.length > 0) { return cb(null, self.indexes[usableQueryKeys[0]].getMatching(query[usableQueryKeys[0]].$in)); } // For a comparison match usableQueryKeys = []; Object.keys(query).forEach(function (k) { if (query[k] && (query[k].hasOwnProperty('$lt') || query[k].hasOwnProperty('$lte') || query[k].hasOwnProperty('$gt') || query[k].hasOwnProperty('$gte'))) { usableQueryKeys.push(k); } }); usableQueryKeys = _.intersection(usableQueryKeys, indexNames); if (usableQueryKeys.length > 0) { return cb(null, self.indexes[usableQueryKeys[0]].getBetweenBounds(query[usableQueryKeys[0]])); } // By default, return all the DB data return cb(null, self.getAllData()); } // STEP 2: remove all expired documents , function (docs) { if (dontExpireStaleDocs) { return callback(null, docs); } var expiredDocsIds = [], validDocs = [], ttlIndexesFieldNames = Object.keys(self.ttlIndexes); docs.forEach(function (doc) { var valid = true; ttlIndexesFieldNames.forEach(function (i) { if (doc[i] !== undefined && util.isDate(doc[i]) && Date.now() > doc[i].getTime() + self.ttlIndexes[i] * 1000) { valid = false; } }); if (valid) { validDocs.push(doc); } else { expiredDocsIds.push(doc._id); } }); async.eachSeries(expiredDocsIds, function (_id, cb) { self._remove({ _id: _id }, {}, function (err) { if (err) { return callback(err); } return cb(); }); }, function (err) { return callback(null, validDocs); }); }]); }; /** * Insert a new document * @param {Function} cb Optional callback, signature: err, insertedDoc * * @api private Use Datastore.insert which has the same signature */ Datastore.prototype._insert = function (newDoc, cb) { var callback = cb || function () {} , preparedDoc ; try { preparedDoc = this.prepareDocumentForInsertion(newDoc) this._insertInCache(preparedDoc); } catch (e) { return callback(e); } this.persistence.persistNewState(util.isArray(preparedDoc) ? preparedDoc : [preparedDoc], function (err) { if (err) { return callback(err); } return callback(null, model.deepCopy(preparedDoc)); }); }; /** * Create a new _id that's not already in use */ Datastore.prototype.createNewId = function () { var tentativeId = customUtils.uid(16); // Try as many times as needed to get an unused _id. As explained in customUtils, the probability of this ever happening is extremely small, so this is O(1) if (this.indexes._id.getMatching(tentativeId).length > 0) { tentativeId = this.createNewId(); } return tentativeId; }; /** * Prepare a document (or array of documents) to be inserted in a database * Meaning adds _id and timestamps if necessary on a copy of newDoc to avoid any side effect on user input * @api private */ Datastore.prototype.prepareDocumentForInsertion = function (newDoc) { var preparedDoc, self = this; if (util.isArray(newDoc)) { preparedDoc = []; newDoc.forEach(function (doc) { preparedDoc.push(self.prepareDocumentForInsertion(doc)); }); } else { preparedDoc = model.deepCopy(newDoc); if (preparedDoc._id === undefined) { preparedDoc._id = this.createNewId(); } var now = new Date(); if (this.timestampData && preparedDoc.createdAt === undefined) { preparedDoc.createdAt = now; } if (this.timestampData && preparedDoc.updatedAt === undefined) { preparedDoc.updatedAt = now; } model.checkObject(preparedDoc); } return preparedDoc; }; /** * If newDoc is an array of documents, this will insert all documents in the cache * @api private */ Datastore.prototype._insertInCache = function (preparedDoc) { if (util.isArray(preparedDoc)) { this._insertMultipleDocsInCache(preparedDoc); } else { this.addToIndexes(preparedDoc); } }; /** * If one insertion fails (e.g. because of a unique constraint), roll back all previous * inserts and throws the error * @api private */ Datastore.prototype._insertMultipleDocsInCache = function (preparedDocs) { var i, failingI, error; for (i = 0; i < preparedDocs.length; i += 1) { try { this.addToIndexes(preparedDocs[i]); } catch (e) { error = e; failingI = i; break; } } if (error) { for (i = 0; i < failingI; i += 1) { this.removeFromIndexes(preparedDocs[i]); } throw error; } }; Datastore.prototype.insert = function () { this.executor.push({ this: this, fn: this._insert, arguments: arguments }); }; /** * Count all documents matching the query * @param {Object} query MongoDB-style query */ Datastore.prototype.count = function(query, callback) { var cursor = new Cursor(this, query, function(err, docs, callback) { if (err) { return callback(err); } return callback(null, docs.length); }); if (typeof callback === 'function') { cursor.exec(callback); } else { return cursor; } }; /** * Find all documents matching the query * If no callback is passed, we return the cursor so that user can limit, skip and finally exec * @param {Object} query MongoDB-style query * @param {Object} projection MongoDB-style projection */ Datastore.prototype.find = function (query, projection, callback) { switch (arguments.length) { case 1: projection = {}; // callback is undefined, will return a cursor break; case 2: if (typeof projection === 'function') { callback = projection; projection = {}; } // If not assume projection is an object and callback undefined break; } var cursor = new Cursor(this, query, function(err, docs, callback) { var res = [], i; if (err) { return callback(err); } for (i = 0; i < docs.length; i += 1) { res.push(model.deepCopy(docs[i])); } return callback(null, res); }); cursor.projection(projection); if (typeof callback === 'function') { cursor.exec(callback); } else { return cursor; } }; /** * Find one document matching the query * @param {Object} query MongoDB-style query * @param {Object} projection MongoDB-style projection */ Datastore.prototype.findOne = function (query, projection, callback) { switch (arguments.length) { case 1: projection = {}; // callback is undefined, will return a cursor break; case 2: if (typeof projection === 'function') { callback = projection; projection = {}; } // If not assume projection is an object and callback undefined break; } var cursor = new Cursor(this, query, function(err, docs, callback) { if (err) { return callback(err); } if (docs.length === 1) { return callback(null, model.deepCopy(docs[0])); } else { return callback(null, null); } }); cursor.projection(projection).limit(1); if (typeof callback === 'function') { cursor.exec(callback); } else { return cursor; } }; /** * Update all docs matching query * @param {Object} query * @param {Object} updateQuery * @param {Object} options Optional options * options.multi If true, can update multiple documents (defaults to false) * options.upsert If true, document is inserted if the query doesn't match anything * options.returnUpdatedDocs Defaults to false, if true return as third argument the array of updated matched documents (even if no change actually took place) * @param {Function} cb Optional callback, signature: (err, numAffected, affectedDocuments, upsert) * If update was an upsert, upsert flag is set to true * affectedDocuments can be one of the following: * * For an upsert, the upserted document * * For an update with returnUpdatedDocs option false, null * * For an update with returnUpdatedDocs true and multi false, the updated document * * For an update with returnUpdatedDocs true and multi true, the array of updated documents * * WARNING: The API was changed between v1.7.4 and v1.8, for consistency and readability reasons. Prior and including to v1.7.4, * the callback signature was (err, numAffected, updated) where updated was the updated document in case of an upsert * or the array of updated documents for an update if the returnUpdatedDocs option was true. That meant that the type of * affectedDocuments in a non multi update depended on whether there was an upsert or not, leaving only two ways for the * user to check whether an upsert had occured: checking the type of affectedDocuments or running another find query on * the whole dataset to check its size. Both options being ugly, the breaking change was necessary. * * @api private Use Datastore.update which has the same signature */ Datastore.prototype._update = function (query, updateQuery, options, cb) { var callback , self = this , numReplaced = 0 , multi, upsert , i ; if (typeof options === 'function') { cb = options; options = {}; } callback = cb || function () {}; multi = options.multi !== undefined ? options.multi : false; upsert = options.upsert !== undefined ? options.upsert : false; async.waterfall([ function (cb) { // If upsert option is set, check whether we need to insert the doc if (!upsert) { return cb(); } // Need to use an internal function not tied to the executor to avoid deadlock var cursor = new Cursor(self, query); cursor.limit(1)._exec(function (err, docs) { if (err) { return callback(err); } if (docs.length === 1) { return cb(); } else { var toBeInserted; try { model.checkObject(updateQuery); // updateQuery is a simple object with no modifier, use it as the document to insert toBeInserted = updateQuery; } catch (e) { // updateQuery contains modifiers, use the find query as the base, // strip it from all operators and update it according to updateQuery try { toBeInserted = model.modify(model.deepCopy(query, true), updateQuery); } catch (err) { return callback(err); } } return self._insert(toBeInserted, function (err, newDoc) { if (err) { return callback(err); } return callback(null, 1, newDoc, true); }); } }); } , function () { // Perform the update var modifiedDoc , modifications = [], createdAt; self.getCandidates(query, function (err, candidates) { if (err) { return callback(err); } // Preparing update (if an error is thrown here neither the datafile nor // the in-memory indexes are affected) try { for (i = 0; i < candidates.length; i += 1) { if (model.match(candidates[i], query) && (multi || numReplaced === 0)) { numReplaced += 1; if (self.timestampData) { createdAt = candidates[i].createdAt; } modifiedDoc = model.modify(candidates[i], updateQuery); if (self.timestampData) { modifiedDoc.createdAt = createdAt; modifiedDoc.updatedAt = new Date(); } modifications.push({ oldDoc: candidates[i], newDoc: modifiedDoc }); } } } catch (err) { return callback(err); } // Change the docs in memory try { self.updateIndexes(modifications); } catch (err) { return callback(err); } // Update the datafile var updatedDocs = _.pluck(modifications, 'newDoc'); self.persistence.persistNewState(updatedDocs, function (err) { if (err) { return callback(err); } if (!options.returnUpdatedDocs) { return callback(null, numReplaced); } else { var updatedDocsDC = []; updatedDocs.forEach(function (doc) { updatedDocsDC.push(model.deepCopy(doc)); }); if (! multi) { updatedDocsDC = updatedDocsDC[0]; } return callback(null, numReplaced, updatedDocsDC); } }); }); }]); }; Datastore.prototype.update = function () { this.executor.push({ this: this, fn: this._update, arguments: arguments }); }; /** * Remove all docs matching the query * For now very naive implementation (similar to update) * @param {Object} query * @param {Object} options Optional options * options.multi If true, can update multiple documents (defaults to false) * @param {Function} cb Optional callback, signature: err, numRemoved * * @api private Use Datastore.remove which has the same signature */ Datastore.prototype._remove = function (query, options, cb) { var callback , self = this, numRemoved = 0, removedDocs = [], multi ; if (typeof options === 'function') { cb = options; options = {}; } callback = cb || function () {}; multi = options.multi !== undefined ? options.multi : false; this.getCandidates(query, true, function (err, candidates) { if (err) { return callback(err); } try { candidates.forEach(function (d) { if (model.match(d, query) && (multi || numRemoved === 0)) { numRemoved += 1; removedDocs.push({ $$deleted: true, _id: d._id }); self.removeFromIndexes(d); } }); } catch (err) { return callback(err); } self.persistence.persistNewState(removedDocs, function (err) { if (err) { return callback(err); } return callback(null, numRemoved); }); }); }; Datastore.prototype.remove = function () { this.executor.push({ this: this, fn: this._remove, arguments: arguments }); }; module.exports = Datastore; },{"./cursor":5,"./customUtils":6,"./executor":8,"./indexes":9,"./model":10,"./persistence":11,"async":13,"events":1,"underscore":19,"util":3}],8:[function(require,module,exports){ var process=require("__browserify_process");/** * Responsible for sequentially executing actions on the database */ var async = require('async') ; function Executor () { this.buffer = []; this.ready = false; // This queue will execute all commands, one-by-one in order this.queue = async.queue(function (task, cb) { var newArguments = []; // task.arguments is an array-like object on which adding a new field doesn't work, so we transform it into a real array for (var i = 0; i < task.arguments.length; i += 1) { newArguments.push(task.arguments[i]); } var lastArg = task.arguments[task.arguments.length - 1]; // Always tell the queue task is complete. Execute callback if any was given. if (typeof lastArg === 'function') { // Callback was supplied newArguments[newArguments.length - 1] = function () { if (typeof setImmediate === 'function') { setImmediate(cb); } else { process.nextTick(cb); } lastArg.apply(null, arguments); }; } else if (!lastArg && task.arguments.length !== 0) { // false/undefined/null supplied as callbback newArguments[newArguments.length - 1] = function () { cb(); }; } else { // Nothing supplied as callback newArguments.push(function () { cb(); }); } task.fn.apply(task.this, newArguments); }, 1); } /** * If executor is ready, queue task (and process it immediately if executor was idle) * If not, buffer task for later processing * @param {Object} task * task.this - Object to use as this * task.fn - Function to execute * task.arguments - Array of arguments, IMPORTANT: only the last argument may be a function (the callback) * and the last argument cannot be false/undefined/null * @param {Boolean} forceQueuing Optional (defaults to false) force executor to queue task even if it is not ready */ Executor.prototype.push = function (task, forceQueuing) { if (this.ready || forceQueuing) { this.queue.push(task); } else { this.buffer.push(task); } }; /** * Queue all tasks in buffer (in the same order they came in) * Automatically sets executor as ready */ Executor.prototype.processBuffer = function () { var i; this.ready = true; for (i = 0; i < this.buffer.length; i += 1) { this.queue.push(this.buffer[i]); } this.buffer = []; }; // Interface module.exports = Executor; },{"__browserify_process":4,"async":13}],9:[function(require,module,exports){ var BinarySearchTree = require('binary-search-tree').AVLTree , model = require('./model') , _ = require('underscore') , util = require('util') ; /** * Two indexed pointers are equal iif they point to the same place */ function checkValueEquality (a, b) { return a === b; } /** * Type-aware projection */ function projectForUnique (elt) { if (elt === null) { return '$null'; } if (typeof elt === 'string') { return '$string' + elt; } if (typeof elt === 'boolean') { return '$boolean' + elt; } if (typeof elt === 'number') { return '$number' + elt; } if (util.isArray(elt)) { return '$date' + elt.getTime(); } return elt; // Arrays and objects, will check for pointer equality } /** * Create a new index * All methods on an index guarantee that either the whole operation was successful and the index changed * or the operation was unsuccessful and an error is thrown while the index is unchanged * @param {String} options.fieldName On which field should the index apply (can use dot notation to index on sub fields) * @param {Boolean} options.unique Optional, enforce a unique constraint (default: false) * @param {Boolean} options.sparse Optional, allow a sparse index (we can have documents for which fieldName is undefined) (default: false) */ function Index (options) { this.fieldName = options.fieldName; this.unique = options.unique || false; this.sparse = options.sparse || false; this.treeOptions = { unique: this.unique, compareKeys: model.compareThings, checkValueEquality: checkValueEquality }; this.reset(); // No data in the beginning } /** * Reset an index * @param {Document or Array of documents} newData Optional, data to initialize the index with * If an error is thrown during insertion, the index is not modified */ Index.prototype.reset = function (newData) { this.tree = new BinarySearchTree(this.treeOptions); if (newData) { this.insert(newData); } }; /** * Insert a new document in the index * If an array is passed, we insert all its elements (if one insertion fails the index is not modified) * O(log(n)) */ Index.prototype.insert = function (doc) { var key, self = this , keys, i, failingI, error ; if (util.isArray(doc)) { this.insertMultipleDocs(doc); return; } key = model.getDotValue(doc, this.fieldName); // We don't index documents that don't contain the field if the index is sparse if (key === undefined && this.sparse) { return; } if (!util.isArray(key)) { this.tree.insert(key, doc); } else { // If an insert fails due to a unique constraint, roll back all inserts before it keys = _.uniq(key, projectForUnique); for (i = 0; i < keys.length; i += 1) { try { this.tree.insert(keys[i], doc); } catch (e) { error = e; failingI = i; break; } } if (error) { for (i = 0; i < failingI; i += 1) { this.tree.delete(keys[i], doc); } throw error; } } }; /** * Insert an array of documents in the index * If a constraint is violated, the changes should be rolled back and an error thrown * * @API private */ Index.prototype.insertMultipleDocs = function (docs) { var i, error, failingI; for (i = 0; i < docs.length; i += 1) { try { this.insert(docs[i]); } catch (e) { error = e; failingI = i; break; } } if (error) { for (i = 0; i < failingI; i += 1) { this.remove(docs[i]); } throw error; } }; /** * Remove a document from the index * If an array is passed, we remove all its elements * The remove operation is safe with regards to the 'unique' constraint * O(log(n)) */ Index.prototype.remove = function (doc) { var key, self = this; if (util.isArray(doc)) { doc.forEach(function (d) { self.remove(d); }); return; } key = model.getDotValue(doc, this.fieldName); if (key === undefined && this.sparse) { return; } if (!util.isArray(key)) { this.tree.delete(key, doc); } else { _.uniq(key, projectForUnique).forEach(function (_key) { self.tree.delete(_key, doc); }); } }; /** * Update a document in the index * If a constraint is violated, changes are rolled back and an error thrown * Naive implementation, still in O(log(n)) */ Index.prototype.update = function (oldDoc, newDoc) { if (util.isArray(oldDoc)) { this.updateMultipleDocs(oldDoc); return; } this.remove(oldDoc); try { this.insert(newDoc); } catch (e) { this.insert(oldDoc); throw e; } }; /** * Update multiple documents in the index * If a constraint is violated, the changes need to be rolled back * and an error thrown * @param {Array of oldDoc, newDoc pairs} pairs * * @API private */ Index.prototype.updateMultipleDocs = function (pairs) { var i, failingI, error; for (i = 0; i < pairs.length; i += 1) { this.remove(pairs[i].oldDoc); } for (i = 0; i < pairs.length; i += 1) { try { this.insert(pairs[i].newDoc); } catch (e) { error = e; failingI = i; break; } } // If an error was raised, roll back changes in the inverse order if (error) { for (i = 0; i < failingI; i += 1) { this.remove(pairs[i].newDoc); } for (i = 0; i < pairs.length; i += 1) { this.insert(pairs[i].oldDoc); } throw error; } }; /** * Revert an update */ Index.prototype.revertUpdate = function (oldDoc, newDoc) { var revert = []; if (!util.isArray(oldDoc)) { this.update(newDoc, oldDoc); } else { oldDoc.forEach(function (pair) { revert.push({ oldDoc: pair.newDoc, newDoc: pair.oldDoc }); }); this.update(revert); } }; /** * Get all documents in index whose key match value (if it is a Thing) or one of the elements of value (if it is an array of Things) * @param {Thing} value Value to match the key against * @return {Array of documents} */ Index.prototype.getMatching = function (value) { var self = this; if (!util.isArray(value)) { return self.tree.search(value); } else { var _res = {}, res = []; value.forEach(function (v) { self.getMatching(v).forEach(function (doc) { _res[doc._id] = doc; }); }); Object.keys(_res).forEach(function (_id) { res.push(_res[_id]); }); return res; } }; /** * Get all documents in index whose key is between bounds are they are defined by query * Documents are sorted by key * @param {Query} query * @return {Array of documents} */ Index.prototype.getBetweenBounds = function (query) { return this.tree.betweenBounds(query); }; /** * Get all elements in the index * @return {Array of documents} */ Index.prototype.getAll = function () { var res = []; this.tree.executeOnEveryNode(function (node) { var i; for (i = 0; i < node.data.length; i += 1) { res.push(node.data[i]); } }); return res; }; // Interface module.exports = Index; },{"./model":10,"binary-search-tree":14,"underscore":19,"util":3}],10:[function(require,module,exports){ /** * Handle models (i.e. docs) * Serialization/deserialization * Copying * Querying, update */ var util = require('util') , _ = require('underscore') , modifierFunctions = {} , lastStepModifierFunctions = {} , comparisonFunctions = {} , logicalOperators = {} , arrayComparisonFunctions = {} ; /** * Check a key, throw an error if the key is non valid * @param {String} k key * @param {Model} v value, needed to treat the Date edge case * Non-treatable edge cases here: if part of the object if of the form { $$date: number } or { $$deleted: true } * Its serialized-then-deserialized version it will transformed into a Date object * But you really need to want it to trigger such behaviour, even when warned not to use '$' at the beginning of the field names... */ function checkKey (k, v) { if (typeof k === 'number') { k = k.toString(); } if (k[0] === '$' && !(k === '$$date' && typeof v === 'number') && !(k === '$$deleted' && v === true) && !(k === '$$indexCreated') && !(k === '$$indexRemoved')) { throw new Error('Field names cannot begin with the $ character'); } if (k.indexOf('.') !== -1) { throw new Error('Field names cannot contain a .'); } } /** * Check a DB object and throw an error if it's not valid * Works by applying the above checkKey function to all fields recursively */ function checkObject (obj) { if (util.isArray(obj)) { obj.forEach(function (o) { checkObject(o); }); } if (typeof obj === 'object' && obj !== null) { Object.keys(obj).forEach(function (k) { checkKey(k, obj[k]); checkObject(obj[k]); }); } } /** * Serialize an object to be persisted to a one-line string * For serialization/deserialization, we use the native JSON parser and not eval or Function * That gives us less freedom but data entered in the database may come from users * so eval and the like are not safe * Accepted primitive types: Number, String, Boolean, Date, null * Accepted secondary types: Objects, Arrays */ function serialize (obj) { var res; res = JSON.stringify(obj, function (k, v) { checkKey(k, v); if (v === undefined) { return undefined; } if (v === null) { return null; } // Hackish way of checking if object is Date (this way it works between execution contexts in node-webkit). // We can't use value directly because for dates it is already string in this function (date.toJSON was already called), so we use this if (typeof this[k].getTime === 'function') { return { $$date: this[k].getTime() }; } return v; }); return res; } /** * From a one-line representation of an object generate by the serialize function * Return the object itself */ function deserialize (rawData) { return JSON.parse(rawData, function (k, v) { if (k === '$$date') { return new Date(v); } if (typeof v === 'string' || typeof v === 'number' || typeof v === 'boolean' || v === null) { return v; } if (v && v.$$date) { return v.$$date; } return v; }); } /** * Deep copy a DB object * The optional strictKeys flag (defaulting to false) indicates whether to copy everything or only fields * where the keys are valid, i.e. don't begin with $ and don't contain a . */ function deepCopy (obj, strictKeys) { var res; if ( typeof obj === 'boolean' || typeof obj === 'number' || typeof obj === 'string' || obj === null || (util.isDate(obj)) ) { return obj; } if (util.isArray(obj)) { res = []; obj.forEach(function (o) { res.push(deepCopy(o, strictKeys)); }); return res; } if (typeof obj === 'object') { res = {}; Object.keys(obj).forEach(function (k) { if (!strictKeys || (k[0] !== '$' && k.indexOf('.') === -1)) { res[k] = deepCopy(obj[k], strictKeys); } }); return res; } return undefined; // For now everything else is undefined. We should probably throw an error instead } /** * Tells if an object is a primitive type or a "real" object * Arrays are considered primitive */ function isPrimitiveType (obj) { return ( typeof obj === 'boolean' || typeof obj === 'number' || typeof obj === 'string' || obj === null || util.isDate(obj) || util.isArray(obj)); } /** * Utility functions for comparing things * Assumes type checking was already done (a and b already have the same type) * compareNSB works for numbers, strings and booleans */ function compareNSB (a, b) { if (a < b) { return -1; } if (a > b) { return 1; } return 0; } function compareArrays (a, b) { var i, comp; for (i = 0; i < Math.min(a.length, b.length); i += 1) { comp = compareThings(a[i], b[i]); if (comp !== 0) { return comp; } } // Common section was identical, longest one wins return compareNSB(a.length, b.length); } /** * Compare { things U undefined } * Things are defined as any native types (string, number, boolean, null, date) and objects * We need to compare with undefined as it will be used in indexes * In the case of objects and arrays, we deep-compare * If two objects dont have the same type, the (arbitrary) type hierarchy is: undefined, null, number, strings, boolean, dates, arrays, objects * Return -1 if a < b, 1 if a > b and 0 if a = b (note that equality here is NOT the same as defined in areThingsEqual!) * * @param {Function} _compareStrings String comparing function, returning -1, 0 or 1, overriding default string comparison (useful for languages with accented letters) */ function compareThings (a, b, _compareStrings) { var aKeys, bKeys, comp, i , compareStrings = _compareStrings || compareNSB; // undefined if (a === undefined) { return b === undefined ? 0 : -1; } if (b === undefined) { return a === undefined ? 0 : 1; } // null if (a === null) { return b === null ? 0 : -1; } if (b === null) { return a === null ? 0 : 1; } // Numbers if (typeof a === 'number') { return typeof b === 'number' ? compareNSB(a, b) : -1; } if (typeof b === 'number') { return typeof a === 'number' ? compareNSB(a, b) : 1; } // Strings if (typeof a === 'string') { return typeof b === 'string' ? compareStrings(a, b) : -1; } if (typeof b === 'string') { return typeof a === 'string' ? compareStrings(a, b) : 1; } // Booleans if (typeof a === 'boolean') { return typeof b === 'boolean' ? compareNSB(a, b) : -1; } if (typeof b === 'boolean') { return typeof a === 'boolean' ? compareNSB(a, b) : 1; } // Dates if (util.isDate(a)) { return util.isDate(b) ? compareNSB(a.getTime(), b.getTime()) : -1; } if (util.isDate(b)) { return util.isDate(a) ? compareNSB(a.getTime(), b.getTime()) : 1; } // Arrays (first element is most significant and so on) if (util.isArray(a)) { return util.isArray(b) ? compareArrays(a, b) : -1; } if (util.isArray(b)) { return util.isArray(a) ? compareArrays(a, b) : 1; } // Objects aKeys = Object.keys(a).sort(); bKeys = Object.keys(b).sort(); for (i = 0; i < Math.min(aKeys.length, bKeys.length); i += 1) { comp = compareThings(a[aKeys[i]], b[bKeys[i]]); if (comp !== 0) { return comp; } } return compareNSB(aKeys.length, bKeys.length); } // ============================================================== // Updating documents // ============================================================== /** * The signature of modifier functions is as follows * Their structure is always the same: recursively follow the dot notation while creating * the nested documents if needed, then apply the "last step modifier" * @param {Object} obj The model to modify * @param {String} field Can contain dots, in that case that means we will set a subfield recursively * @param {Model} value */ /** * Set a field to a new value */ lastStepModifierFunctions.$set = function (obj, field, value) { obj[field] = value; }; /** * Unset a field */ lastStepModifierFunctions.$unset = function (obj, field, value) { delete obj[field]; }; /** * Push an element to the end of an array field * Optional modifier $each instead of value to push several values * Optional modifier $slice to slice the resulting array, see https://docs.mongodb.org/manual/reference/operator/update/slice/ * Différeence with MongoDB: if $slice is specified and not $each, we act as if value is an empty array */ lastStepModifierFunctions.$push = function (obj, field, value) { // Create the array if it doesn't exist if (!obj.hasOwnProperty(field)) { obj[field] = []; } if (!util.isArray(obj[field])) { throw new Error("Can't $push an element on non-array values"); } if (value !== null && typeof value === 'object' && value.$slice && value.$each === undefined) { value.$each = []; } if (value !== null && typeof value === 'object' && value.$each) { if (Object.keys(value).length >= 3 || (Object.keys(value).length === 2 && value.$slice === undefined)) { throw new Error("Can only use $slice in cunjunction with $each when $push to array"); } if (!util.isArray(value.$each)) { throw new Error("$each requires an array value"); } value.$each.forEach(function (v) { obj[field].push(v); }); if (value.$slice === undefined || typeof value.$slice !== 'number') { return; } if (value.$slice === 0) { obj[field] = []; } else { var start, end, n = obj[field].length; if (value.$slice < 0) { start = Math.max(0, n + value.$slice); end = n; } else if (value.$slice > 0) { start = 0; end = Math.min(n, value.$slice); } obj[field] = obj[field].slice(start, end); } } else { obj[field].push(value); } }; /** * Add an element to an array field only if it is not already in it * No modification if the element is already in the array * Note that it doesn't check whether the original array contains duplicates */ lastStepModifierFunctions.$addToSet = function (obj, field, value) { var addToSet = true; // Create the array if it doesn't exist if (!obj.hasOwnProperty(field)) { obj[field] = []; } if (!util.isArray(obj[field])) { throw new Error("Can't $addToSet an element on non-array values"); } if (value !== null && typeof value === 'object' && value.$each) { if (Object.keys(value).length > 1) { throw new Error("Can't use another field in conjunction with $each"); } if (!util.isArray(value.$each)) { throw new Error("$each requires an array value"); } value.$each.forEach(function (v) { lastStepModifierFunctions.$addToSet(obj, field, v); }); } else { obj[field].forEach(function (v) { if (compareThings(v, value) === 0) { addToSet = false; } }); if (addToSet) { obj[field].push(value); } } }; /** * Remove the first or last element of an array */ lastStepModifierFunctions.$pop = function (obj, field, value) { if (!util.isArray(obj[field])) { throw new Error("Can't $pop an element from non-array values"); } if (typeof value !== 'number') { throw new Error(value + " isn't an integer, can't use it with $pop"); } if (value === 0) { return; } if (value > 0) { obj[field] = obj[field].slice(0, obj[field].length - 1); } else { obj[field] = obj[field].slice(1); } }; /** * Removes all instances of a value from an existing array */ lastStepModifierFunctions.$pull = function (obj, field, value) { var arr, i; if (!util.isArray(obj[field])) { throw new Error("Can't $pull an element from non-array values"); } arr = obj[field]; for (i = arr.length - 1; i >= 0; i -= 1) { if (match(arr[i], value)) { arr.splice(i, 1); } } }; /** * Increment a numeric field's value */ lastStepModifierFunctions.$inc = function (obj, field, value) { if (typeof value !== 'number') { throw new Error(value + " must be a number"); } if (typeof obj[field] !== 'number') { if (!_.has(obj, field)) { obj[field] = value; } else { throw new Error("Don't use the $inc modifier on non-number fields"); } } else { obj[field] += value; } }; /** * Updates the value of the field, only if specified field is greater than the current value of the field */ lastStepModifierFunctions.$max = function (obj, field, value) { if (typeof obj[field] === 'undefined') { obj[field] = value; } else if (value > obj[field]) { obj[field] = value; } }; /** * Updates the value of the field, only if specified field is smaller than the current value of the field */ lastStepModifierFunctions.$min = function (obj, field, value) { if (typeof obj[field] === 'undefined') {  obj[field] = value; } else if (value < obj[field]) { obj[field] = value; } }; // Given its name, create the complete modifier function function createModifierFunction (modifier) { return function (obj, field, value) { var fieldParts = typeof field === 'string' ? field.split('.') : field; if (fieldParts.length === 1) { lastStepModifierFunctions[modifier](obj, field, value); } else { if (obj[fieldParts[0]] === undefined) { if (modifier === '$unset') { return; } // Bad looking specific fix, needs to be generalized modifiers that behave like $unset are implemented obj[fieldParts[0]] = {}; } modifierFunctions[modifier](obj[fieldParts[0]], fieldParts.slice(1), value); } }; } // Actually create all modifier functions Object.keys(lastStepModifierFunctions).forEach(function (modifier) { modifierFunctions[modifier] = createModifierFunction(modifier); }); /** * Modify a DB object according to an update query */ function modify (obj, updateQuery) { var keys = Object.keys(updateQuery) , firstChars = _.map(keys, function (item) { return item[0]; }) , dollarFirstChars = _.filter(firstChars, function (c) { return c === '$'; }) , newDoc, modifiers ; if (keys.indexOf('_id') !== -1 && updateQuery._id !== obj._id) { throw new Error("You cannot change a document's _id"); } if (dollarFirstChars.length !== 0 && dollarFirstChars.length !== firstChars.length) { throw new Error("You cannot mix modifiers and normal fields"); } if (dollarFirstChars.length === 0) { // Simply replace the object with the update query contents newDoc = deepCopy(updateQuery); newDoc._id = obj._id; } else { // Apply modifiers modifiers = _.uniq(keys); newDoc = deepCopy(obj); modifiers.forEach(function (m) { var keys; if (!modifierFunctions[m]) { throw new Error("Unknown modifier " + m); } // Can't rely on Object.keys throwing on non objects since ES6 // Not 100% satisfying as non objects can be interpreted as objects but no false negatives so we can live with it if (typeof updateQuery[m] !== 'object') { throw new Error("Modifier " + m + "'s argument must be an object"); } keys = Object.keys(updateQuery[m]); keys.forEach(function (k) { modifierFunctions[m](newDoc, k, updateQuery[m][k]); }); }); } // Check result is valid and return it checkObject(newDoc); if (obj._id !== newDoc._id) { throw new Error("You can't change a document's _id"); } return newDoc; }; // ============================================================== // Finding documents // ============================================================== /** * Get a value from object with dot notation * @param {Object} obj * @param {String} field */ function getDotValue (obj, field) { var fieldParts = typeof field === 'string' ? field.split('.') : field , i, objs; if (!obj) { return undefined; } // field cannot be empty so that means we should return undefined so that nothing can match if (fieldParts.length === 0) { return obj; } if (fieldParts.length === 1) { return obj[fieldParts[0]]; } if (util.isArray(obj[fieldParts[0]])) { // If the next field is an integer, return only this item of the array i = parseInt(fieldParts[1], 10); if (typeof i === 'number' && !isNaN(i)) { return getDotValue(obj[fieldParts[0]][i], fieldParts.slice(2)) } // Return the array of values objs = new Array(); for (i = 0; i < obj[fieldParts[0]].length; i += 1) { objs.push(getDotValue(obj[fieldParts[0]][i], fieldParts.slice(1))); } return objs; } else { return getDotValue(obj[fieldParts[0]], fieldParts.slice(1)); } } /** * Check whether 'things' are equal * Things are defined as any native types (string, number, boolean, null, date) and objects * In the case of object, we check deep equality * Returns true if they are, false otherwise */ function areThingsEqual (a, b) { var aKeys , bKeys , i; // Strings, booleans, numbers, null if (a === null || typeof a === 'string' || typeof a === 'boolean' || typeof a === 'number' || b === null || typeof b === 'string' || typeof b === 'boolean' || typeof b === 'number') { return a === b; } // Dates if (util.isDate(a) || util.isDate(b)) { return util.isDate(a) && util.isDate(b) && a.getTime() === b.getTime(); } // Arrays (no match since arrays are used as a $in) // undefined (no match since they mean field doesn't exist and can't be serialized) if ((!(util.isArray(a) && util.isArray(b)) && (util.isArray(a) || util.isArray(b))) || a === undefined || b === undefined) { return false; } // General objects (check for deep equality) // a and b should be objects at this point try { aKeys = Object.keys(a); bKeys = Object.keys(b); } catch (e) { return false; } if (aKeys.length !== bKeys.length) { return false; } for (i = 0; i < aKeys.length; i += 1) { if (bKeys.indexOf(aKeys[i]) === -1) { return false; } if (!areThingsEqual(a[aKeys[i]], b[aKeys[i]])) { return false; } } return true; } /** * Check that two values are comparable */ function areComparable (a, b) { if (typeof a !== 'string' && typeof a !== 'number' && !util.isDate(a) && typeof b !== 'string' && typeof b !== 'number' && !util.isDate(b)) { return false; } if (typeof a !== typeof b) { return false; } return true; } /** * Arithmetic and comparison operators * @param {Native value} a Value in the object * @param {Native value} b Value in the query */ comparisonFunctions.$lt = function (a, b) { return areComparable(a, b) && a < b; }; comparisonFunctions.$lte = function (a, b) { return areComparable(a, b) && a <= b; }; comparisonFunctions.$gt = function (a, b) { return areComparable(a, b) && a > b; }; comparisonFunctions.$gte = function (a, b) { return areComparable(a, b) && a >= b; }; comparisonFunctions.$ne = function (a, b) { if (a === undefined) { return true; } return !areThingsEqual(a, b); }; comparisonFunctions.$in = function (a, b) { var i; if (!util.isArray(b)) { throw new Error("$in operator called with a non-array"); } for (i = 0; i < b.length; i += 1) { if (areThingsEqual(a, b[i])) { return true; } } return false; }; comparisonFunctions.$nin = function (a, b) { if (!util.isArray(b)) { throw new Error("$nin operator called with a non-array"); } return !comparisonFunctions.$in(a, b); }; comparisonFunctions.$regex = function (a, b) { if (!util.isRegExp(b)) { throw new Error("$regex operator called with non regular expression"); } if (typeof a !== 'string') { return false } else { return b.test(a); } }; comparisonFunctions.$exists = function (value, exists) { if (exists || exists === '') { // This will be true for all values of exists except false, null, undefined and 0 exists = true; // That's strange behaviour (we should only use true/false) but that's the way Mongo does it... } else { exists = false; } if (value === undefined) { return !exists } else { return exists; } }; // Specific to arrays comparisonFunctions.$size = function (obj, value) { if (!util.isArray(obj)) { return false; } if (value % 1 !== 0) { throw new Error("$size operator called without an integer"); } return (obj.length == value); }; comparisonFunctions.$elemMatch = function (obj, value) { if (!util.isArray(obj)) { return false; } var i = obj.length; var result = false; // Initialize result while (i--) { if (match(obj[i], value)) { // If match for array element, return true result = true; break; } } return result; }; arrayComparisonFunctions.$size = true; arrayComparisonFunctions.$elemMatch = true; /** * Match any of the subqueries * @param {Model} obj * @param {Array of Queries} query */ logicalOperators.$or = function (obj, query) { var i; if (!util.isArray(query)) { throw new Error("$or operator used without an array"); } for (i = 0; i < query.length; i += 1) { if (match(obj, query[i])) { return true; } } return false; }; /** * Match all of the subqueries * @param {Model} obj * @param {Array of Queries} query */ logicalOperators.$and = function (obj, query) { var i; if (!util.isArray(query)) { throw new Error("$and operator used without an array"); } for (i = 0; i < query.length; i += 1) { if (!match(obj, query[i])) { return false; } } return true; }; /** * Inverted match of the query * @param {Model} obj * @param {Query} query */ logicalOperators.$not = function (obj, query) { return !match(obj, query); }; /** * Use a function to match * @param {Model} obj * @param {Query} query */ logicalOperators.$where = function (obj, fn) { var result; if (!_.isFunction(fn)) { throw new Error("$where operator used without a function"); } result = fn.call(obj); if (!_.isBoolean(result)) { throw new Error("$where function must return boolean"); } return result; }; /** * Tell if a given document matches a query * @param {Object} obj Document to check * @param {Object} query */ function match (obj, query) { var queryKeys, queryKey, queryValue, i; // Primitive query against a primitive type // This is a bit of a hack since we construct an object with an arbitrary key only to dereference it later // But I don't have time for a cleaner implementation now if (isPrimitiveType(obj) || isPrimitiveType(query)) { return matchQueryPart({ needAKey: obj }, 'needAKey', query); } // Normal query queryKeys = Object.keys(query); for (i = 0; i < queryKeys.length; i += 1) { queryKey = queryKeys[i]; queryValue = query[queryKey]; if (queryKey[0] === '$') { if (!logicalOperators[queryKey]) { throw new Error("Unknown logical operator " + queryKey); } if (!logicalOperators[queryKey](obj, queryValue)) { return false; } } else { if (!matchQueryPart(obj, queryKey, queryValue)) { return false; } } } return true; }; /** * Match an object against a specific { key: value } part of a query * if the treatObjAsValue flag is set, don't try to match every part separately, but the array as a whole */ function matchQueryPart (obj, queryKey, queryValue, treatObjAsValue) { var objValue = getDotValue(obj, queryKey) , i, keys, firstChars, dollarFirstChars; // Check if the value is an array if we don't force a treatment as value if (util.isArray(objValue) && !treatObjAsValue) { // If the queryValue is an array, try to perform an exact match if (util.isArray(queryValue)) { return matchQueryPart(obj, queryKey, queryValue, true); } // Check if we are using an array-specific comparison function if (queryValue !== null && typeof queryValue === 'object' && !util.isRegExp(queryValue)) { keys = Object.keys(queryValue); for (i = 0; i < keys.length; i += 1) { if (arrayComparisonFunctions[keys[i]]) { return matchQueryPart(obj, queryKey, queryValue, true); } } } // If not, treat it as an array of { obj, query } where there needs to be at least one match for (i = 0; i < objValue.length; i += 1) { if (matchQueryPart({ k: objValue[i] }, 'k', queryValue)) { return true; } // k here could be any string } return false; } // queryValue is an actual object. Determine whether it contains comparison operators // or only normal fields. Mixed objects are not allowed if (queryValue !== null && typeof queryValue === 'object' && !util.isRegExp(queryValue) && !util.isArray(queryValue)) { keys = Object.keys(queryValue); firstChars = _.map(keys, function (item) { return item[0]; }); dollarFirstChars = _.filter(firstChars, function (c) { return c === '$'; }); if (dollarFirstChars.length !== 0 && dollarFirstChars.length !== firstChars.length) { throw new Error("You cannot mix operators and normal fields"); } // queryValue is an object of this form: { $comparisonOperator1: value1, ... } if (dollarFirstChars.length > 0) { for (i = 0; i < keys.length; i += 1) { if (!comparisonFunctions[keys[i]]) { throw new Error("Unknown comparison function " + keys[i]); } if (!comparisonFunctions[keys[i]](objValue, queryValue[keys[i]])) { return false; } } return true; } } // Using regular expressions with basic querying if (util.isRegExp(queryValue)) { return comparisonFunctions.$regex(objValue, queryValue); } // queryValue is either a native value or a normal object // Basic matching is possible if (!areThingsEqual(objValue, queryValue)) { return false; } return true; } // Interface module.exports.serialize = serialize; module.exports.deserialize = deserialize; module.exports.deepCopy = deepCopy; module.exports.checkObject = checkObject; module.exports.isPrimitiveType = isPrimitiveType; module.exports.modify = modify; module.exports.getDotValue = getDotValue; module.exports.match = match; module.exports.areThingsEqual = areThingsEqual; module.exports.compareThings = compareThings; },{"underscore":19,"util":3}],11:[function(require,module,exports){ var process=require("__browserify_process");/** * Handle every persistence-related task * The interface Datastore expects to be implemented is * * Persistence.loadDatabase(callback) and callback has signature err * * Persistence.persistNewState(newDocs, callback) where newDocs is an array of documents and callback has signature err */ var storage = require('./storage') , path = require('path') , model = require('./model') , async = require('async') , customUtils = require('./customUtils') , Index = require('./indexes') ; /** * Create a new Persistence object for database options.db * @param {Datastore} options.db * @param {Boolean} options.nodeWebkitAppName Optional, specify the name of your NW app if you want options.filename to be relative to the directory where * Node Webkit stores application data such as cookies and local storage (the best place to store data in my opinion) */ function Persistence (options) { var i, j, randomString; this.db = options.db; this.inMemoryOnly = this.db.inMemoryOnly; this.filename = this.db.filename; this.corruptAlertThreshold = options.corruptAlertThreshold !== undefined ? options.corruptAlertThreshold : 0.1; if (!this.inMemoryOnly && this.filename && this.filename.charAt(this.filename.length - 1) === '~') { throw new Error("The datafile name can't end with a ~, which is reserved for crash safe backup files"); } // After serialization and before deserialization hooks with some basic sanity checks if (options.afterSerialization && !options.beforeDeserialization) { throw new Error("Serialization hook defined but deserialization hook undefined, cautiously refusing to start NeDB to prevent dataloss"); } if (!options.afterSerialization && options.beforeDeserialization) { throw new Error("Serialization hook undefined but deserialization hook defined, cautiously refusing to start NeDB to prevent dataloss"); } this.afterSerialization = options.afterSerialization || function (s) { return s; }; this.beforeDeserialization = options.beforeDeserialization || function (s) { return s; }; for (i = 1; i < 30; i += 1) { for (j = 0; j < 10; j += 1) { randomString = customUtils.uid(i); if (this.beforeDeserialization(this.afterSerialization(randomString)) !== randomString) { throw new Error("beforeDeserialization is not the reverse of afterSerialization, cautiously refusing to start NeDB to prevent dataloss"); } } } // For NW apps, store data in the same directory where NW stores application data if (this.filename && options.nodeWebkitAppName) { console.log("=================================================================="); console.log("WARNING: The nodeWebkitAppName option is deprecated"); console.log("To get the path to the directory where Node Webkit stores the data"); console.log("for your app, use the internal nw.gui module like this"); console.log("require('nw.gui').App.dataPath"); console.log("See https://github.com/rogerwang/node-webkit/issues/500"); console.log("=================================================================="); this.filename = Persistence.getNWAppFilename(options.nodeWebkitAppName, this.filename); } }; /** * Check if a directory exists and create it on the fly if it is not the case * cb is optional, signature: err */ Persistence.ensureDirectoryExists = function (dir, cb) { var callback = cb || function () {} ; storage.mkdirp(dir, function (err) { return callback(err); }); }; /** * Return the path the datafile if the given filename is relative to the directory where Node Webkit stores * data for this application. Probably the best place to store data */ Persistence.getNWAppFilename = function (appName, relativeFilename) { var home; switch (process.platform) { case 'win32': case 'win64': home = process.env.LOCALAPPDATA || process.env.APPDATA; if (!home) { throw new Error("Couldn't find the base application data folder"); } home = path.join(home, appName); break; case 'darwin': home = process.env.HOME; if (!home) { throw new Error("Couldn't find the base application data directory"); } home = path.join(home, 'Library', 'Application Support', appName); break; case 'linux': home = process.env.HOME; if (!home) { throw new Error("Couldn't find the base application data directory"); } home = path.join(home, '.config', appName); break; default: throw new Error("Can't use the Node Webkit relative path for platform " + process.platform); break; } return path.join(home, 'nedb-data', relativeFilename); } /** * Persist cached database * This serves as a compaction function since the cache always contains only the number of documents in the collection * while the data file is append-only so it may grow larger * @param {Function} cb Optional callback, signature: err */ Persistence.prototype.persistCachedDatabase = function (cb) { var callback = cb || function () {} , toPersist = '' , self = this ; if (this.inMemoryOnly) { return callback(null); } this.db.getAllData().forEach(function (doc) { toPersist += self.afterSerialization(model.serialize(doc)) + '\n'; }); Object.keys(this.db.indexes).forEach(function (fieldName) { if (fieldName != "_id") { // The special _id index is managed by datastore.js, the others need to be persisted toPersist += self.afterSerialization(model.serialize({ $$indexCreated: { fieldName: fieldName, unique: self.db.indexes[fieldName].unique, sparse: self.db.indexes[fieldName].sparse }})) + '\n'; } }); storage.crashSafeWriteFile(this.filename, toPersist, function (err) { if (err) { return callback(err); } self.db.emit('compaction.done'); return callback(null); }); }; /** * Queue a rewrite of the datafile */ Persistence.prototype.compactDatafile = function () { this.db.executor.push({ this: this, fn: this.persistCachedDatabase, arguments: [] }); }; /** * Set automatic compaction every interval ms * @param {Number} interval in milliseconds, with an enforced minimum of 5 seconds */ Persistence.prototype.setAutocompactionInterval = function (interval) { var self = this , minInterval = 5000 , realInterval = Math.max(interval || 0, minInterval) ; this.stopAutocompaction(); this.autocompactionIntervalId = setInterval(function () { self.compactDatafile(); }, realInterval); }; /** * Stop autocompaction (do nothing if autocompaction was not running) */ Persistence.prototype.stopAutocompaction = function () { if (this.autocompactionIntervalId) { clearInterval(this.autocompactionIntervalId); } }; /** * Persist new state for the given newDocs (can be insertion, update or removal) * Use an append-only format * @param {Array} newDocs Can be empty if no doc was updated/removed * @param {Function} cb Optional, signature: err */ Persistence.prototype.persistNewState = function (newDocs, cb) { var self = this , toPersist = '' , callback = cb || function () {} ; // In-memory only datastore if (self.inMemoryOnly) { return callback(null); } newDocs.forEach(function (doc) { toPersist += self.afterSerialization(model.serialize(doc)) + '\n'; }); if (toPersist.length === 0) { return callback(null); } storage.appendFile(self.filename, toPersist, 'utf8', function (err) { return callback(err); }); }; /** * From a database's raw data, return the corresponding * machine understandable collection */ Persistence.prototype.treatRawData = function (rawData) { var data = rawData.split('\n') , dataById = {} , tdata = [] , i , indexes = {} , corruptItems = -1 // Last line of every data file is usually blank so not really corrupt ; for (i = 0; i < data.length; i += 1) { var doc; try { doc = model.deserialize(this.beforeDeserialization(data[i])); if (doc._id) { if (doc.$$deleted === true) { delete dataById[doc._id]; } else { dataById[doc._id] = doc; } } else if (doc.$$indexCreated && doc.$$indexCreated.fieldName != undefined) { indexes[doc.$$indexCreated.fieldName] = doc.$$indexCreated; } else if (typeof doc.$$indexRemoved === "string") { delete indexes[doc.$$indexRemoved]; } } catch (e) { corruptItems += 1; } } // A bit lenient on corruption if (data.length > 0 && corruptItems / data.length > this.corruptAlertThreshold) { throw new Error("More than " + Math.floor(100 * this.corruptAlertThreshold) + "% of the data file is corrupt, the wrong beforeDeserialization hook may be used. Cautiously refusing to start NeDB to prevent dataloss"); } Object.keys(dataById).forEach(function (k) { tdata.push(dataById[k]); }); return { data: tdata, indexes: indexes }; }; /** * Load the database * 1) Create all indexes * 2) Insert all data * 3) Compact the database * This means pulling data out of the data file or creating it if it doesn't exist * Also, all data is persisted right away, which has the effect of compacting the database file * This operation is very quick at startup for a big collection (60ms for ~10k docs) * @param {Function} cb Optional callback, signature: err */ Persistence.prototype.loadDatabase = function (cb) { var callback = cb || function () {} , self = this ; self.db.resetIndexes(); // In-memory only datastore if (self.inMemoryOnly) { return callback(null); } async.waterfall([ function (cb) { Persistence.ensureDirectoryExists(path.dirname(self.filename), function (err) { storage.ensureDatafileIntegrity(self.filename, function (err) { storage.readFile(self.filename, 'utf8', function (err, rawData) { if (err) { return cb(err); } try { var treatedData = self.treatRawData(rawData); } catch (e) { return cb(e); } // Recreate all indexes in the datafile Object.keys(treatedData.indexes).forEach(function (key) { self.db.indexes[key] = new Index(treatedData.indexes[key]); }); // Fill cached database (i.e. all indexes) with data try { self.db.resetIndexes(treatedData.data); } catch (e) { self.db.resetIndexes(); // Rollback any index which didn't fail return cb(e); } self.db.persistence.persistCachedDatabase(cb); }); }); }); } ], function (err) { if (err) { return callback(err); } self.db.executor.processBuffer(); return callback(null); }); }; // Interface module.exports = Persistence; },{"./customUtils":6,"./indexes":9,"./model":10,"./storage":12,"__browserify_process":4,"async":13,"path":2}],12:[function(require,module,exports){ /** * Way data is stored for this database * For a Node.js/Node Webkit database it's the file system * For a browser-side database it's localforage, which uses the best backend available (IndexedDB then WebSQL then localStorage) * * This version is the browser version */ var localforage = require('localforage') // Configure localforage to display NeDB name for now. Would be a good idea to let user use his own app name localforage.config({ name: 'NeDB' , storeName: 'nedbdata' }); function exists (filename, callback) { localforage.getItem(filename, function (err, value) { if (value !== null) { // Even if value is undefined, localforage returns null return callback(true); } else { return callback(false); } }); } function rename (filename, newFilename, callback) { localforage.getItem(filename, function (err, value) { if (value === null) { localforage.removeItem(newFilename, function () { return callback(); }); } else { localforage.setItem(newFilename, value, function () { localforage.removeItem(filename, function () { return callback(); }); }); } }); } function writeFile (filename, contents, options, callback) { // Options do not matter in browser setup if (typeof options === 'function') { callback = options; } localforage.setItem(filename, contents, function () { return callback(); }); } function appendFile (filename, toAppend, options, callback) { // Options do not matter in browser setup if (typeof options === 'function') { callback = options; } localforage.getItem(filename, function (err, contents) { contents = contents || ''; contents += toAppend; localforage.setItem(filename, contents, function () { return callback(); }); }); } function readFile (filename, options, callback) { // Options do not matter in browser setup if (typeof options === 'function') { callback = options; } localforage.getItem(filename, function (err, contents) { return callback(null, contents || ''); }); } function unlink (filename, callback) { localforage.removeItem(filename, function () { return callback(); }); } // Nothing to do, no directories will be used on the browser function mkdirp (dir, callback) { return callback(); } // Nothing to do, no data corruption possible in the brower function ensureDatafileIntegrity (filename, callback) { return callback(null); } // Interface module.exports.exists = exists; module.exports.rename = rename; module.exports.writeFile = writeFile; module.exports.crashSafeWriteFile = writeFile; // No need for a crash safe function in the browser module.exports.appendFile = appendFile; module.exports.readFile = readFile; module.exports.unlink = unlink; module.exports.mkdirp = mkdirp; module.exports.ensureDatafileIntegrity = ensureDatafileIntegrity; },{"localforage":18}],13:[function(require,module,exports){ var process=require("__browserify_process");/*global setImmediate: false, setTimeout: false, console: false */ (function () { var async = {}; // global on the server, window in the browser var root, previous_async; root = this; if (root != null) { previous_async = root.async; } async.noConflict = function () { root.async = previous_async; return async; }; function only_once(fn) { var called = false; return function() { if (called) throw new Error("Callback was already called."); called = true; fn.apply(root, arguments); } } //// cross-browser compatiblity functions //// var _each = function (arr, iterator) { if (arr.forEach) { return arr.forEach(iterator); } for (var i = 0; i < arr.length; i += 1) { iterator(arr[i], i, arr); } }; var _map = function (arr, iterator) { if (arr.map) { return arr.map(iterator); } var results = []; _each(arr, function (x, i, a) { results.push(iterator(x, i, a)); }); return results; }; var _reduce = function (arr, iterator, memo) { if (arr.reduce) { return arr.reduce(iterator, memo); } _each(arr, function (x, i, a) { memo = iterator(memo, x, i, a); }); return memo; }; var _keys = function (obj) { if (Object.keys) { return Object.keys(obj); } var keys = []; for (var k in obj) { if (obj.hasOwnProperty(k)) { keys.push(k); } } return keys; }; //// exported async module functions //// //// nextTick implementation with browser-compatible fallback //// if (typeof process === 'undefined' || !(process.nextTick)) { if (typeof setImmediate === 'function') { async.nextTick = function (fn) { // not a direct alias for IE10 compatibility setImmediate(fn); }; async.setImmediate = async.nextTick; } else { async.nextTick = function (fn) { setTimeout(fn, 0); }; async.setImmediate = async.nextTick; } } else { async.nextTick = process.nextTick; if (typeof setImmediate !== 'undefined') { async.setImmediate = function (fn) { // not a direct alias for IE10 compatibility setImmediate(fn); }; } else { async.setImmediate = async.nextTick; } } async.each = function (arr, iterator, callback) { callback = callback || function () {}; if (!arr.length) { return callback(); } var completed = 0; _each(arr, function (x) { iterator(x, only_once(function (err) { if (err) { callback(err); callback = function () {}; } else { completed += 1; if (completed >= arr.length) { callback(null); } } })); }); }; async.forEach = async.each; async.eachSeries = function (arr, iterator, callback) { callback = callback || function () {}; if (!arr.length) { return callback(); } var completed = 0; var iterate = function () { iterator(arr[completed], function (err) { if (err) { callback(err); callback = function () {}; } else { completed += 1; if (completed >= arr.length) { callback(null); } else { iterate(); } } }); }; iterate(); }; async.forEachSeries = async.eachSeries; async.eachLimit = function (arr, limit, iterator, callback) { var fn = _eachLimit(limit); fn.apply(null, [arr, iterator, callback]); }; async.forEachLimit = async.eachLimit; var _eachLimit = function (limit) { return function (arr, iterator, callback) { callback = callback || function () {}; if (!arr.length || limit <= 0) { return callback(); } var completed = 0; var started = 0; var running = 0; (function replenish () { if (completed >= arr.length) { return callback(); } while (running < limit && started < arr.length) { started += 1; running += 1; iterator(arr[started - 1], function (err) { if (err) { callback(err); callback = function () {}; } else { completed += 1; running -= 1; if (completed >= arr.length) { callback(); } else { replenish(); } } }); } })(); }; }; var doParallel = function (fn) { return function () { var args = Array.prototype.slice.call(arguments); return fn.apply(null, [async.each].concat(args)); }; }; var doParallelLimit = function(limit, fn) { return function () { var args = Array.prototype.slice.call(arguments); return fn.apply(null, [_eachLimit(limit)].concat(args)); }; }; var doSeries = function (fn) { return function () { var args = Array.prototype.slice.call(arguments); return fn.apply(null, [async.eachSeries].concat(args)); }; }; var _asyncMap = function (eachfn, arr, iterator, callback) { var results = []; arr = _map(arr, function (x, i) { return {index: i, value: x}; }); eachfn(arr, function (x, callback) { iterator(x.value, function (err, v) { results[x.index] = v; callback(err); }); }, function (err) { callback(err, results); }); }; async.map = doParallel(_asyncMap); async.mapSeries = doSeries(_asyncMap); async.mapLimit = function (arr, limit, iterator, callback) { return _mapLimit(limit)(arr, iterator, callback); }; var _mapLimit = function(limit) { return doParallelLimit(limit, _asyncMap); }; // reduce only has a series version, as doing reduce in parallel won't // work in many situations. async.reduce = function (arr, memo, iterator, callback) { async.eachSeries(arr, function (x, callback) { iterator(memo, x, function (err, v) { memo = v; callback(err); }); }, function (err) { callback(err, memo); }); }; // inject alias async.inject = async.reduce; // foldl alias async.foldl = async.reduce; async.reduceRight = function (arr, memo, iterator, callback) { var reversed = _map(arr, function (x) { return x; }).reverse(); async.reduce(reversed, memo, iterator, callback); }; // foldr alias async.foldr = async.reduceRight; var _filter = function (eachfn, arr, iterator, callback) { var results = []; arr = _map(arr, function (x, i) { return {index: i, value: x}; }); eachfn(arr, function (x, callback) { iterator(x.value, function (v) { if (v) { results.push(x); } callback(); }); }, function (err) { callback(_map(results.sort(function (a, b) { return a.index - b.index; }), function (x) { return x.value; })); }); }; async.filter = doParallel(_filter); async.filterSeries = doSeries(_filter); // select alias async.select = async.filter; async.selectSeries = async.filterSeries; var _reject = function (eachfn, arr, iterator, callback) { var results = []; arr = _map(arr, function (x, i) { return {index: i, value: x}; }); eachfn(arr, function (x, callback) { iterator(x.value, function (v) { if (!v) { results.push(x); } callback(); }); }, function (err) { callback(_map(results.sort(function (a, b) { return a.index - b.index; }), function (x) { return x.value; })); }); }; async.reject = doParallel(_reject); async.rejectSeries = doSeries(_reject); var _detect = function (eachfn, arr, iterator, main_callback) { eachfn(arr, function (x, callback) { iterator(x, function (result) { if (result) { main_callback(x); main_callback = function () {}; } else { callback(); } }); }, function (err) { main_callback(); }); }; async.detect = doParallel(_detect); async.detectSeries = doSeries(_detect); async.some = function (arr, iterator, main_callback) { async.each(arr, function (x, callback) { iterator(x, function (v) { if (v) { main_callback(true); main_callback = function () {}; } callback(); }); }, function (err) { main_callback(false); }); }; // any alias async.any = async.some; async.every = function (arr, iterator, main_callback) { async.each(arr, function (x, callback) { iterator(x, function (v) { if (!v) { main_callback(false); main_callback = function () {}; } callback(); }); }, function (err) { main_callback(true); }); }; // all alias async.all = async.every; async.sortBy = function (arr, iterator, callback) { async.map(arr, function (x, callback) { iterator(x, function (err, criteria) { if (err) { callback(err); } else { callback(null, {value: x, criteria: criteria}); } }); }, function (err, results) { if (err) { return callback(err); } else { var fn = function (left, right) { var a = left.criteria, b = right.criteria; return a < b ? -1 : a > b ? 1 : 0; }; callback(null, _map(results.sort(fn), function (x) { return x.value; })); } }); }; async.auto = function (tasks, callback) { callback = callback || function () {}; var keys = _keys(tasks); if (!keys.length) { return callback(null); } var results = {}; var listeners = []; var addListener = function (fn) { listeners.unshift(fn); }; var removeListener = function (fn) { for (var i = 0; i < listeners.length; i += 1) { if (listeners[i] === fn) { listeners.splice(i, 1); return; } } }; var taskComplete = function () { _each(listeners.slice(0), function (fn) { fn(); }); }; addListener(function () { if (_keys(results).length === keys.length) { callback(null, results); callback = function () {}; } }); _each(keys, function (k) { var task = (tasks[k] instanceof Function) ? [tasks[k]]: tasks[k]; var taskCallback = function (err) { var args = Array.prototype.slice.call(arguments, 1); if (args.length <= 1) { args = args[0]; } if (err) { var safeResults = {}; _each(_keys(results), function(rkey) { safeResults[rkey] = results[rkey]; }); safeResults[k] = args; callback(err, safeResults); // stop subsequent errors hitting callback multiple times callback = function () {}; } else { results[k] = args; async.setImmediate(taskComplete); } }; var requires = task.slice(0, Math.abs(task.length - 1)) || []; var ready = function () { return _reduce(requires, function (a, x) { return (a && results.hasOwnProperty(x)); }, true) && !results.hasOwnProperty(k); }; if (ready()) { task[task.length - 1](taskCallback, results); } else { var listener = function () { if (ready()) { removeListener(listener); task[task.length - 1](taskCallback, results); } }; addListener(listener); } }); }; async.waterfall = function (tasks, callback) { callback = callback || function () {}; if (tasks.constructor !== Array) { var err = new Error('First argument to waterfall must be an array of functions'); return callback(err); } if (!tasks.length) { return callback(); } var wrapIterator = function (iterator) { return function (err) { if (err) { callback.apply(null, arguments); callback = function () {}; } else { var args = Array.prototype.slice.call(arguments, 1); var next = iterator.next(); if (next) { args.push(wrapIterator(next)); } else { args.push(callback); } async.setImmediate(function () { iterator.apply(null, args); }); } }; }; wrapIterator(async.iterator(tasks))(); }; var _parallel = function(eachfn, tasks, callback) { callback = callback || function () {}; if (tasks.constructor === Array) { eachfn.map(tasks, function (fn, callback) { if (fn) { fn(function (err) { var args = Array.prototype.slice.call(arguments, 1); if (args.length <= 1) { args = args[0]; } callback.call(null, err, args); }); } }, callback); } else { var results = {}; eachfn.each(_keys(tasks), function (k, callback) { tasks[k](function (err) { var args = Array.prototype.slice.call(arguments, 1); if (args.length <= 1) { args = args[0]; } results[k] = args; callback(err); }); }, function (err) { callback(err, results); }); } }; async.parallel = function (tasks, callback) { _parallel({ map: async.map, each: async.each }, tasks, callback); }; async.parallelLimit = function(tasks, limit, callback) { _parallel({ map: _mapLimit(limit), each: _eachLimit(limit) }, tasks, callback); }; async.series = function (tasks, callback) { callback = callback || function () {}; if (tasks.constructor === Array) { async.mapSeries(tasks, function (fn, callback) { if (fn) { fn(function (err) { var args = Array.prototype.slice.call(arguments, 1); if (args.length <= 1) { args = args[0]; } callback.call(null, err, args); }); } }, callback); } else { var results = {}; async.eachSeries(_keys(tasks), function (k, callback) { tasks[k](function (err) { var args = Array.prototype.slice.call(arguments, 1); if (args.length <= 1) { args = args[0]; } results[k] = args; callback(err); }); }, function (err) { callback(err, results); }); } }; async.iterator = function (tasks) { var makeCallback = function (index) { var fn = function () { if (tasks.length) { tasks[index].apply(null, arguments); } return fn.next(); }; fn.next = function () { return (index < tasks.length - 1) ? makeCallback(index + 1): null; }; return fn; }; return makeCallback(0); }; async.apply = function (fn) { var args = Array.prototype.slice.call(arguments, 1); return function () { return fn.apply( null, args.concat(Array.prototype.slice.call(arguments)) ); }; }; var _concat = function (eachfn, arr, fn, callback) { var r = []; eachfn(arr, function (x, cb) { fn(x, function (err, y) { r = r.concat(y || []); cb(err); }); }, function (err) { callback(err, r); }); }; async.concat = doParallel(_concat); async.concatSeries = doSeries(_concat); async.whilst = function (test, iterator, callback) { if (test()) { iterator(function (err) { if (err) { return callback(err); } async.whilst(test, iterator, callback); }); } else { callback(); } }; async.doWhilst = function (iterator, test, callback) { iterator(function (err) { if (err) { return callback(err); } if (test()) { async.doWhilst(iterator, test, callback); } else { callback(); } }); }; async.until = function (test, iterator, callback) { if (!test()) { iterator(function (err) { if (err) { return callback(err); } async.until(test, iterator, callback); }); } else { callback(); } }; async.doUntil = function (iterator, test, callback) { iterator(function (err) { if (err) { return callback(err); } if (!test()) { async.doUntil(iterator, test, callback); } else { callback(); } }); }; async.queue = function (worker, concurrency) { if (concurrency === undefined) { concurrency = 1; } function _insert(q, data, pos, callback) { if(data.constructor !== Array) { data = [data]; } _each(data, function(task) { var item = { data: task, callback: typeof callback === 'function' ? callback : null }; if (pos) { q.tasks.unshift(item); } else { q.tasks.push(item); } if (q.saturated && q.tasks.length === concurrency) { q.saturated(); } async.setImmediate(q.process); }); } var workers = 0; var q = { tasks: [], concurrency: concurrency, saturated: null, empty: null, drain: null, push: function (data, callback) { _insert(q, data, false, callback); }, unshift: function (data, callback) { _insert(q, data, true, callback); }, process: function () { if (workers < q.concurrency && q.tasks.length) { var task = q.tasks.shift(); if (q.empty && q.tasks.length === 0) { q.empty(); } workers += 1; var next = function () { workers -= 1; if (task.callback) { task.callback.apply(task, arguments); } if (q.drain && q.tasks.length + workers === 0) { q.drain(); } q.process(); }; var cb = only_once(next); worker(task.data, cb); } }, length: function () { return q.tasks.length; }, running: function () { return workers; } }; return q; }; async.cargo = function (worker, payload) { var working = false, tasks = []; var cargo = { tasks: tasks, payload: payload, saturated: null, empty: null, drain: null, push: function (data, callback) { if(data.constructor !== Array) { data = [data]; } _each(data, function(task) { tasks.push({ data: task, callback: typeof callback === 'function' ? callback : null }); if (cargo.saturated && tasks.length === payload) { cargo.saturated(); } }); async.setImmediate(cargo.process); }, process: function process() { if (working) return; if (tasks.length === 0) { if(cargo.drain) cargo.drain(); return; } var ts = typeof payload === 'number' ? tasks.splice(0, payload) : tasks.splice(0); var ds = _map(ts, function (task) { return task.data; }); if(cargo.empty) cargo.empty(); working = true; worker(ds, function () { working = false; var args = arguments; _each(ts, function (data) { if (data.callback) { data.callback.apply(null, args); } }); process(); }); }, length: function () { return tasks.length; }, running: function () { return working; } }; return cargo; }; var _console_fn = function (name) { return function (fn) { var args = Array.prototype.slice.call(arguments, 1); fn.apply(null, args.concat([function (err) { var args = Array.prototype.slice.call(arguments, 1); if (typeof console !== 'undefined') { if (err) { if (console.error) { console.error(err); } } else if (console[name]) { _each(args, function (x) { console[name](x); }); } } }])); }; }; async.log = _console_fn('log'); async.dir = _console_fn('dir'); /*async.info = _console_fn('info'); async.warn = _console_fn('warn'); async.error = _console_fn('error');*/ async.memoize = function (fn, hasher) { var memo = {}; var queues = {}; hasher = hasher || function (x) { return x; }; var memoized = function () { var args = Array.prototype.slice.call(arguments); var callback = args.pop(); var key = hasher.apply(null, args); if (key in memo) { callback.apply(null, memo[key]); } else if (key in queues) { queues[key].push(callback); } else { queues[key] = [callback]; fn.apply(null, args.concat([function () { memo[key] = arguments; var q = queues[key]; delete queues[key]; for (var i = 0, l = q.length; i < l; i++) { q[i].apply(null, arguments); } }])); } }; memoized.memo = memo; memoized.unmemoized = fn; return memoized; }; async.unmemoize = function (fn) { return function () { return (fn.unmemoized || fn).apply(null, arguments); }; }; async.times = function (count, iterator, callback) { var counter = []; for (var i = 0; i < count; i++) { counter.push(i); } return async.map(counter, iterator, callback); }; async.timesSeries = function (count, iterator, callback) { var counter = []; for (var i = 0; i < count; i++) { counter.push(i); } return async.mapSeries(counter, iterator, callback); }; async.compose = function (/* functions... */) { var fns = Array.prototype.reverse.call(arguments); return function () { var that = this; var args = Array.prototype.slice.call(arguments); var callback = args.pop(); async.reduce(fns, args, function (newargs, fn, cb) { fn.apply(that, newargs.concat([function () { var err = arguments[0]; var nextargs = Array.prototype.slice.call(arguments, 1); cb(err, nextargs); }])) }, function (err, results) { callback.apply(that, [err].concat(results)); }); }; }; var _applyEach = function (eachfn, fns /*args...*/) { var go = function () { var that = this; var args = Array.prototype.slice.call(arguments); var callback = args.pop(); return eachfn(fns, function (fn, cb) { fn.apply(that, args.concat([cb])); }, callback); }; if (arguments.length > 2) { var args = Array.prototype.slice.call(arguments, 2); return go.apply(this, args); } else { return go; } }; async.applyEach = doParallel(_applyEach); async.applyEachSeries = doSeries(_applyEach); async.forever = function (fn, callback) { function next(err) { if (err) { if (callback) { return callback(err); } throw err; } fn(next); } next(); }; // AMD / RequireJS if (typeof define !== 'undefined' && define.amd) { define([], function () { return async; }); } // Node.js else if (typeof module !== 'undefined' && module.exports) { module.exports = async; } // included directly via ================================================ FILE: browser-version/test/localforage.js ================================================ /*! localForage -- Offline Storage, Improved Version 1.3.0 https://mozilla.github.io/localForage (c) 2013-2015 Mozilla, Apache License 2.0 */ (function() { var define, requireModule, require, requirejs; (function() { var registry = {}, seen = {}; define = function(name, deps, callback) { registry[name] = { deps: deps, callback: callback }; }; requirejs = require = requireModule = function(name) { requirejs._eak_seen = registry; if (seen[name]) { return seen[name]; } seen[name] = {}; if (!registry[name]) { throw new Error("Could not find module " + name); } var mod = registry[name], deps = mod.deps, callback = mod.callback, reified = [], exports; for (var i=0, l=deps.length; i dbInfo.db.version; if (isDowngrade) { // If the version is not the default one // then warn for impossible downgrade. if (dbInfo.version !== defaultVersion) { globalObject.console.warn('The database "' + dbInfo.name + '"' + ' can\'t be downgraded from version ' + dbInfo.db.version + ' to version ' + dbInfo.version + '.'); } // Align the versions to prevent errors. dbInfo.version = dbInfo.db.version; } if (isUpgrade || isNewStore) { // If the store is new then increment the version (if needed). // This will trigger an "upgradeneeded" event which is required // for creating a store. if (isNewStore) { var incVersion = dbInfo.db.version + 1; if (incVersion > dbInfo.version) { dbInfo.version = incVersion; } } return true; } return false; } function getItem(key, callback) { var self = this; // Cast the key to a string, as that's all we can set as a key. if (typeof key !== 'string') { globalObject.console.warn(key + ' used as a key, but it is not a string.'); key = String(key); } var promise = new Promise(function (resolve, reject) { self.ready().then(function () { var dbInfo = self._dbInfo; var store = dbInfo.db.transaction(dbInfo.storeName, 'readonly').objectStore(dbInfo.storeName); var req = store.get(key); req.onsuccess = function () { var value = req.result; if (value === undefined) { value = null; } if (_isEncodedBlob(value)) { value = _decodeBlob(value); } resolve(value); }; req.onerror = function () { reject(req.error); }; })['catch'](reject); }); executeCallback(promise, callback); return promise; } // Iterate over all items stored in database. function iterate(iterator, callback) { var self = this; var promise = new Promise(function (resolve, reject) { self.ready().then(function () { var dbInfo = self._dbInfo; var store = dbInfo.db.transaction(dbInfo.storeName, 'readonly').objectStore(dbInfo.storeName); var req = store.openCursor(); var iterationNumber = 1; req.onsuccess = function () { var cursor = req.result; if (cursor) { var value = cursor.value; if (_isEncodedBlob(value)) { value = _decodeBlob(value); } var result = iterator(value, cursor.key, iterationNumber++); if (result !== void 0) { resolve(result); } else { cursor['continue'](); } } else { resolve(); } }; req.onerror = function () { reject(req.error); }; })['catch'](reject); }); executeCallback(promise, callback); return promise; } function setItem(key, value, callback) { var self = this; // Cast the key to a string, as that's all we can set as a key. if (typeof key !== 'string') { globalObject.console.warn(key + ' used as a key, but it is not a string.'); key = String(key); } var promise = new Promise(function (resolve, reject) { var dbInfo; self.ready().then(function () { dbInfo = self._dbInfo; return _checkBlobSupport(dbInfo.db); }).then(function (blobSupport) { if (!blobSupport && value instanceof Blob) { return _encodeBlob(value); } return value; }).then(function (value) { var transaction = dbInfo.db.transaction(dbInfo.storeName, 'readwrite'); var store = transaction.objectStore(dbInfo.storeName); // The reason we don't _save_ null is because IE 10 does // not support saving the `null` type in IndexedDB. How // ironic, given the bug below! // See: https://github.com/mozilla/localForage/issues/161 if (value === null) { value = undefined; } var req = store.put(value, key); transaction.oncomplete = function () { // Cast to undefined so the value passed to // callback/promise is the same as what one would get out // of `getItem()` later. This leads to some weirdness // (setItem('foo', undefined) will return `null`), but // it's not my fault localStorage is our baseline and that // it's weird. if (value === undefined) { value = null; } resolve(value); }; transaction.onabort = transaction.onerror = function () { var err = req.error ? req.error : req.transaction.error; reject(err); }; })['catch'](reject); }); executeCallback(promise, callback); return promise; } function removeItem(key, callback) { var self = this; // Cast the key to a string, as that's all we can set as a key. if (typeof key !== 'string') { globalObject.console.warn(key + ' used as a key, but it is not a string.'); key = String(key); } var promise = new Promise(function (resolve, reject) { self.ready().then(function () { var dbInfo = self._dbInfo; var transaction = dbInfo.db.transaction(dbInfo.storeName, 'readwrite'); var store = transaction.objectStore(dbInfo.storeName); // We use a Grunt task to make this safe for IE and some // versions of Android (including those used by Cordova). // Normally IE won't like `.delete()` and will insist on // using `['delete']()`, but we have a build step that // fixes this for us now. var req = store['delete'](key); transaction.oncomplete = function () { resolve(); }; transaction.onerror = function () { reject(req.error); }; // The request will be also be aborted if we've exceeded our storage // space. transaction.onabort = function () { var err = req.error ? req.error : req.transaction.error; reject(err); }; })['catch'](reject); }); executeCallback(promise, callback); return promise; } function clear(callback) { var self = this; var promise = new Promise(function (resolve, reject) { self.ready().then(function () { var dbInfo = self._dbInfo; var transaction = dbInfo.db.transaction(dbInfo.storeName, 'readwrite'); var store = transaction.objectStore(dbInfo.storeName); var req = store.clear(); transaction.oncomplete = function () { resolve(); }; transaction.onabort = transaction.onerror = function () { var err = req.error ? req.error : req.transaction.error; reject(err); }; })['catch'](reject); }); executeCallback(promise, callback); return promise; } function length(callback) { var self = this; var promise = new Promise(function (resolve, reject) { self.ready().then(function () { var dbInfo = self._dbInfo; var store = dbInfo.db.transaction(dbInfo.storeName, 'readonly').objectStore(dbInfo.storeName); var req = store.count(); req.onsuccess = function () { resolve(req.result); }; req.onerror = function () { reject(req.error); }; })['catch'](reject); }); executeCallback(promise, callback); return promise; } function key(n, callback) { var self = this; var promise = new Promise(function (resolve, reject) { if (n < 0) { resolve(null); return; } self.ready().then(function () { var dbInfo = self._dbInfo; var store = dbInfo.db.transaction(dbInfo.storeName, 'readonly').objectStore(dbInfo.storeName); var advanced = false; var req = store.openCursor(); req.onsuccess = function () { var cursor = req.result; if (!cursor) { // this means there weren't enough keys resolve(null); return; } if (n === 0) { // We have the first key, return it if that's what they // wanted. resolve(cursor.key); } else { if (!advanced) { // Otherwise, ask the cursor to skip ahead n // records. advanced = true; cursor.advance(n); } else { // When we get here, we've got the nth key. resolve(cursor.key); } } }; req.onerror = function () { reject(req.error); }; })['catch'](reject); }); executeCallback(promise, callback); return promise; } function keys(callback) { var self = this; var promise = new Promise(function (resolve, reject) { self.ready().then(function () { var dbInfo = self._dbInfo; var store = dbInfo.db.transaction(dbInfo.storeName, 'readonly').objectStore(dbInfo.storeName); var req = store.openCursor(); var keys = []; req.onsuccess = function () { var cursor = req.result; if (!cursor) { resolve(keys); return; } keys.push(cursor.key); cursor['continue'](); }; req.onerror = function () { reject(req.error); }; })['catch'](reject); }); executeCallback(promise, callback); return promise; } function executeCallback(promise, callback) { if (callback) { promise.then(function (result) { callback(null, result); }, function (error) { callback(error); }); } } var asyncStorage = { _driver: 'asyncStorage', _initStorage: _initStorage, iterate: iterate, getItem: getItem, setItem: setItem, removeItem: removeItem, clear: clear, length: length, key: key, keys: keys }; exports['default'] = asyncStorage; }).call(typeof window !== 'undefined' ? window : self); module.exports = exports['default']; /***/ }, /* 2 */ /***/ function(module, exports, __webpack_require__) { // If IndexedDB isn't available, we'll fall back to localStorage. // Note that this will have considerable performance and storage // side-effects (all data will be serialized on save and only data that // can be converted to a string via `JSON.stringify()` will be saved). 'use strict'; exports.__esModule = true; (function () { 'use strict'; var globalObject = this; var localStorage = null; // If the app is running inside a Google Chrome packaged webapp, or some // other context where localStorage isn't available, we don't use // localStorage. This feature detection is preferred over the old // `if (window.chrome && window.chrome.runtime)` code. // See: https://github.com/mozilla/localForage/issues/68 try { // If localStorage isn't available, we get outta here! // This should be inside a try catch if (!this.localStorage || !('setItem' in this.localStorage)) { return; } // Initialize localStorage and create a variable to use throughout // the code. localStorage = this.localStorage; } catch (e) { return; } // Config the localStorage backend, using options set in the config. function _initStorage(options) { var self = this; var dbInfo = {}; if (options) { for (var i in options) { dbInfo[i] = options[i]; } } dbInfo.keyPrefix = dbInfo.name + '/'; if (dbInfo.storeName !== self._defaultConfig.storeName) { dbInfo.keyPrefix += dbInfo.storeName + '/'; } self._dbInfo = dbInfo; return new Promise(function (resolve, reject) { resolve(__webpack_require__(3)); }).then(function (lib) { dbInfo.serializer = lib; return Promise.resolve(); }); } // Remove all keys from the datastore, effectively destroying all data in // the app's key/value store! function clear(callback) { var self = this; var promise = self.ready().then(function () { var keyPrefix = self._dbInfo.keyPrefix; for (var i = localStorage.length - 1; i >= 0; i--) { var key = localStorage.key(i); if (key.indexOf(keyPrefix) === 0) { localStorage.removeItem(key); } } }); executeCallback(promise, callback); return promise; } // Retrieve an item from the store. Unlike the original async_storage // library in Gaia, we don't modify return values at all. If a key's value // is `undefined`, we pass that value to the callback function. function getItem(key, callback) { var self = this; // Cast the key to a string, as that's all we can set as a key. if (typeof key !== 'string') { globalObject.console.warn(key + ' used as a key, but it is not a string.'); key = String(key); } var promise = self.ready().then(function () { var dbInfo = self._dbInfo; var result = localStorage.getItem(dbInfo.keyPrefix + key); // If a result was found, parse it from the serialized // string into a JS object. If result isn't truthy, the key // is likely undefined and we'll pass it straight to the // callback. if (result) { result = dbInfo.serializer.deserialize(result); } return result; }); executeCallback(promise, callback); return promise; } // Iterate over all items in the store. function iterate(iterator, callback) { var self = this; var promise = self.ready().then(function () { var dbInfo = self._dbInfo; var keyPrefix = dbInfo.keyPrefix; var keyPrefixLength = keyPrefix.length; var length = localStorage.length; // We use a dedicated iterator instead of the `i` variable below // so other keys we fetch in localStorage aren't counted in // the `iterationNumber` argument passed to the `iterate()` // callback. // // See: github.com/mozilla/localForage/pull/435#discussion_r38061530 var iterationNumber = 1; for (var i = 0; i < length; i++) { var key = localStorage.key(i); if (key.indexOf(keyPrefix) !== 0) { continue; } var value = localStorage.getItem(key); // If a result was found, parse it from the serialized // string into a JS object. If result isn't truthy, the // key is likely undefined and we'll pass it straight // to the iterator. if (value) { value = dbInfo.serializer.deserialize(value); } value = iterator(value, key.substring(keyPrefixLength), iterationNumber++); if (value !== void 0) { return value; } } }); executeCallback(promise, callback); return promise; } // Same as localStorage's key() method, except takes a callback. function key(n, callback) { var self = this; var promise = self.ready().then(function () { var dbInfo = self._dbInfo; var result; try { result = localStorage.key(n); } catch (error) { result = null; } // Remove the prefix from the key, if a key is found. if (result) { result = result.substring(dbInfo.keyPrefix.length); } return result; }); executeCallback(promise, callback); return promise; } function keys(callback) { var self = this; var promise = self.ready().then(function () { var dbInfo = self._dbInfo; var length = localStorage.length; var keys = []; for (var i = 0; i < length; i++) { if (localStorage.key(i).indexOf(dbInfo.keyPrefix) === 0) { keys.push(localStorage.key(i).substring(dbInfo.keyPrefix.length)); } } return keys; }); executeCallback(promise, callback); return promise; } // Supply the number of keys in the datastore to the callback function. function length(callback) { var self = this; var promise = self.keys().then(function (keys) { return keys.length; }); executeCallback(promise, callback); return promise; } // Remove an item from the store, nice and simple. function removeItem(key, callback) { var self = this; // Cast the key to a string, as that's all we can set as a key. if (typeof key !== 'string') { globalObject.console.warn(key + ' used as a key, but it is not a string.'); key = String(key); } var promise = self.ready().then(function () { var dbInfo = self._dbInfo; localStorage.removeItem(dbInfo.keyPrefix + key); }); executeCallback(promise, callback); return promise; } // Set a key's value and run an optional callback once the value is set. // Unlike Gaia's implementation, the callback function is passed the value, // in case you want to operate on that value only after you're sure it // saved, or something like that. function setItem(key, value, callback) { var self = this; // Cast the key to a string, as that's all we can set as a key. if (typeof key !== 'string') { globalObject.console.warn(key + ' used as a key, but it is not a string.'); key = String(key); } var promise = self.ready().then(function () { // Convert undefined values to null. // https://github.com/mozilla/localForage/pull/42 if (value === undefined) { value = null; } // Save the original value to pass to the callback. var originalValue = value; return new Promise(function (resolve, reject) { var dbInfo = self._dbInfo; dbInfo.serializer.serialize(value, function (value, error) { if (error) { reject(error); } else { try { localStorage.setItem(dbInfo.keyPrefix + key, value); resolve(originalValue); } catch (e) { // localStorage capacity exceeded. // TODO: Make this a specific error/event. if (e.name === 'QuotaExceededError' || e.name === 'NS_ERROR_DOM_QUOTA_REACHED') { reject(e); } reject(e); } } }); }); }); executeCallback(promise, callback); return promise; } function executeCallback(promise, callback) { if (callback) { promise.then(function (result) { callback(null, result); }, function (error) { callback(error); }); } } var localStorageWrapper = { _driver: 'localStorageWrapper', _initStorage: _initStorage, // Default API, from Gaia/localStorage. iterate: iterate, getItem: getItem, setItem: setItem, removeItem: removeItem, clear: clear, length: length, key: key, keys: keys }; exports['default'] = localStorageWrapper; }).call(typeof window !== 'undefined' ? window : self); module.exports = exports['default']; /***/ }, /* 3 */ /***/ function(module, exports) { 'use strict'; exports.__esModule = true; (function () { 'use strict'; // Sadly, the best way to save binary data in WebSQL/localStorage is serializing // it to Base64, so this is how we store it to prevent very strange errors with less // verbose ways of binary <-> string data storage. var BASE_CHARS = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'; var BLOB_TYPE_PREFIX = '~~local_forage_type~'; var BLOB_TYPE_PREFIX_REGEX = /^~~local_forage_type~([^~]+)~/; var SERIALIZED_MARKER = '__lfsc__:'; var SERIALIZED_MARKER_LENGTH = SERIALIZED_MARKER.length; // OMG the serializations! var TYPE_ARRAYBUFFER = 'arbf'; var TYPE_BLOB = 'blob'; var TYPE_INT8ARRAY = 'si08'; var TYPE_UINT8ARRAY = 'ui08'; var TYPE_UINT8CLAMPEDARRAY = 'uic8'; var TYPE_INT16ARRAY = 'si16'; var TYPE_INT32ARRAY = 'si32'; var TYPE_UINT16ARRAY = 'ur16'; var TYPE_UINT32ARRAY = 'ui32'; var TYPE_FLOAT32ARRAY = 'fl32'; var TYPE_FLOAT64ARRAY = 'fl64'; var TYPE_SERIALIZED_MARKER_LENGTH = SERIALIZED_MARKER_LENGTH + TYPE_ARRAYBUFFER.length; // Get out of our habit of using `window` inline, at least. var globalObject = this; // Abstracts constructing a Blob object, so it also works in older // browsers that don't support the native Blob constructor. (i.e. // old QtWebKit versions, at least). function _createBlob(parts, properties) { parts = parts || []; properties = properties || {}; try { return new Blob(parts, properties); } catch (err) { if (err.name !== 'TypeError') { throw err; } var BlobBuilder = globalObject.BlobBuilder || globalObject.MSBlobBuilder || globalObject.MozBlobBuilder || globalObject.WebKitBlobBuilder; var builder = new BlobBuilder(); for (var i = 0; i < parts.length; i += 1) { builder.append(parts[i]); } return builder.getBlob(properties.type); } } // Serialize a value, afterwards executing a callback (which usually // instructs the `setItem()` callback/promise to be executed). This is how // we store binary data with localStorage. function serialize(value, callback) { var valueString = ''; if (value) { valueString = value.toString(); } // Cannot use `value instanceof ArrayBuffer` or such here, as these // checks fail when running the tests using casper.js... // // TODO: See why those tests fail and use a better solution. if (value && (value.toString() === '[object ArrayBuffer]' || value.buffer && value.buffer.toString() === '[object ArrayBuffer]')) { // Convert binary arrays to a string and prefix the string with // a special marker. var buffer; var marker = SERIALIZED_MARKER; if (value instanceof ArrayBuffer) { buffer = value; marker += TYPE_ARRAYBUFFER; } else { buffer = value.buffer; if (valueString === '[object Int8Array]') { marker += TYPE_INT8ARRAY; } else if (valueString === '[object Uint8Array]') { marker += TYPE_UINT8ARRAY; } else if (valueString === '[object Uint8ClampedArray]') { marker += TYPE_UINT8CLAMPEDARRAY; } else if (valueString === '[object Int16Array]') { marker += TYPE_INT16ARRAY; } else if (valueString === '[object Uint16Array]') { marker += TYPE_UINT16ARRAY; } else if (valueString === '[object Int32Array]') { marker += TYPE_INT32ARRAY; } else if (valueString === '[object Uint32Array]') { marker += TYPE_UINT32ARRAY; } else if (valueString === '[object Float32Array]') { marker += TYPE_FLOAT32ARRAY; } else if (valueString === '[object Float64Array]') { marker += TYPE_FLOAT64ARRAY; } else { callback(new Error('Failed to get type for BinaryArray')); } } callback(marker + bufferToString(buffer)); } else if (valueString === '[object Blob]') { // Conver the blob to a binaryArray and then to a string. var fileReader = new FileReader(); fileReader.onload = function () { // Backwards-compatible prefix for the blob type. var str = BLOB_TYPE_PREFIX + value.type + '~' + bufferToString(this.result); callback(SERIALIZED_MARKER + TYPE_BLOB + str); }; fileReader.readAsArrayBuffer(value); } else { try { callback(JSON.stringify(value)); } catch (e) { console.error("Couldn't convert value into a JSON string: ", value); callback(null, e); } } } // Deserialize data we've inserted into a value column/field. We place // special markers into our strings to mark them as encoded; this isn't // as nice as a meta field, but it's the only sane thing we can do whilst // keeping localStorage support intact. // // Oftentimes this will just deserialize JSON content, but if we have a // special marker (SERIALIZED_MARKER, defined above), we will extract // some kind of arraybuffer/binary data/typed array out of the string. function deserialize(value) { // If we haven't marked this string as being specially serialized (i.e. // something other than serialized JSON), we can just return it and be // done with it. if (value.substring(0, SERIALIZED_MARKER_LENGTH) !== SERIALIZED_MARKER) { return JSON.parse(value); } // The following code deals with deserializing some kind of Blob or // TypedArray. First we separate out the type of data we're dealing // with from the data itself. var serializedString = value.substring(TYPE_SERIALIZED_MARKER_LENGTH); var type = value.substring(SERIALIZED_MARKER_LENGTH, TYPE_SERIALIZED_MARKER_LENGTH); var blobType; // Backwards-compatible blob type serialization strategy. // DBs created with older versions of localForage will simply not have the blob type. if (type === TYPE_BLOB && BLOB_TYPE_PREFIX_REGEX.test(serializedString)) { var matcher = serializedString.match(BLOB_TYPE_PREFIX_REGEX); blobType = matcher[1]; serializedString = serializedString.substring(matcher[0].length); } var buffer = stringToBuffer(serializedString); // Return the right type based on the code/type set during // serialization. switch (type) { case TYPE_ARRAYBUFFER: return buffer; case TYPE_BLOB: return _createBlob([buffer], { type: blobType }); case TYPE_INT8ARRAY: return new Int8Array(buffer); case TYPE_UINT8ARRAY: return new Uint8Array(buffer); case TYPE_UINT8CLAMPEDARRAY: return new Uint8ClampedArray(buffer); case TYPE_INT16ARRAY: return new Int16Array(buffer); case TYPE_UINT16ARRAY: return new Uint16Array(buffer); case TYPE_INT32ARRAY: return new Int32Array(buffer); case TYPE_UINT32ARRAY: return new Uint32Array(buffer); case TYPE_FLOAT32ARRAY: return new Float32Array(buffer); case TYPE_FLOAT64ARRAY: return new Float64Array(buffer); default: throw new Error('Unkown type: ' + type); } } function stringToBuffer(serializedString) { // Fill the string into a ArrayBuffer. var bufferLength = serializedString.length * 0.75; var len = serializedString.length; var i; var p = 0; var encoded1, encoded2, encoded3, encoded4; if (serializedString[serializedString.length - 1] === '=') { bufferLength--; if (serializedString[serializedString.length - 2] === '=') { bufferLength--; } } var buffer = new ArrayBuffer(bufferLength); var bytes = new Uint8Array(buffer); for (i = 0; i < len; i += 4) { encoded1 = BASE_CHARS.indexOf(serializedString[i]); encoded2 = BASE_CHARS.indexOf(serializedString[i + 1]); encoded3 = BASE_CHARS.indexOf(serializedString[i + 2]); encoded4 = BASE_CHARS.indexOf(serializedString[i + 3]); /*jslint bitwise: true */ bytes[p++] = encoded1 << 2 | encoded2 >> 4; bytes[p++] = (encoded2 & 15) << 4 | encoded3 >> 2; bytes[p++] = (encoded3 & 3) << 6 | encoded4 & 63; } return buffer; } // Converts a buffer to a string to store, serialized, in the backend // storage library. function bufferToString(buffer) { // base64-arraybuffer var bytes = new Uint8Array(buffer); var base64String = ''; var i; for (i = 0; i < bytes.length; i += 3) { /*jslint bitwise: true */ base64String += BASE_CHARS[bytes[i] >> 2]; base64String += BASE_CHARS[(bytes[i] & 3) << 4 | bytes[i + 1] >> 4]; base64String += BASE_CHARS[(bytes[i + 1] & 15) << 2 | bytes[i + 2] >> 6]; base64String += BASE_CHARS[bytes[i + 2] & 63]; } if (bytes.length % 3 === 2) { base64String = base64String.substring(0, base64String.length - 1) + '='; } else if (bytes.length % 3 === 1) { base64String = base64String.substring(0, base64String.length - 2) + '=='; } return base64String; } var localforageSerializer = { serialize: serialize, deserialize: deserialize, stringToBuffer: stringToBuffer, bufferToString: bufferToString }; exports['default'] = localforageSerializer; }).call(typeof window !== 'undefined' ? window : self); module.exports = exports['default']; /***/ }, /* 4 */ /***/ function(module, exports, __webpack_require__) { /* * Includes code from: * * base64-arraybuffer * https://github.com/niklasvh/base64-arraybuffer * * Copyright (c) 2012 Niklas von Hertzen * Licensed under the MIT license. */ 'use strict'; exports.__esModule = true; (function () { 'use strict'; var globalObject = this; var openDatabase = this.openDatabase; // If WebSQL methods aren't available, we can stop now. if (!openDatabase) { return; } // Open the WebSQL database (automatically creates one if one didn't // previously exist), using any options set in the config. function _initStorage(options) { var self = this; var dbInfo = { db: null }; if (options) { for (var i in options) { dbInfo[i] = typeof options[i] !== 'string' ? options[i].toString() : options[i]; } } var dbInfoPromise = new Promise(function (resolve, reject) { // Open the database; the openDatabase API will automatically // create it for us if it doesn't exist. try { dbInfo.db = openDatabase(dbInfo.name, String(dbInfo.version), dbInfo.description, dbInfo.size); } catch (e) { return self.setDriver(self.LOCALSTORAGE).then(function () { return self._initStorage(options); }).then(resolve)['catch'](reject); } // Create our key/value table if it doesn't exist. dbInfo.db.transaction(function (t) { t.executeSql('CREATE TABLE IF NOT EXISTS ' + dbInfo.storeName + ' (id INTEGER PRIMARY KEY, key unique, value)', [], function () { self._dbInfo = dbInfo; resolve(); }, function (t, error) { reject(error); }); }); }); return new Promise(function (resolve, reject) { resolve(__webpack_require__(3)); }).then(function (lib) { dbInfo.serializer = lib; return dbInfoPromise; }); } function getItem(key, callback) { var self = this; // Cast the key to a string, as that's all we can set as a key. if (typeof key !== 'string') { globalObject.console.warn(key + ' used as a key, but it is not a string.'); key = String(key); } var promise = new Promise(function (resolve, reject) { self.ready().then(function () { var dbInfo = self._dbInfo; dbInfo.db.transaction(function (t) { t.executeSql('SELECT * FROM ' + dbInfo.storeName + ' WHERE key = ? LIMIT 1', [key], function (t, results) { var result = results.rows.length ? results.rows.item(0).value : null; // Check to see if this is serialized content we need to // unpack. if (result) { result = dbInfo.serializer.deserialize(result); } resolve(result); }, function (t, error) { reject(error); }); }); })['catch'](reject); }); executeCallback(promise, callback); return promise; } function iterate(iterator, callback) { var self = this; var promise = new Promise(function (resolve, reject) { self.ready().then(function () { var dbInfo = self._dbInfo; dbInfo.db.transaction(function (t) { t.executeSql('SELECT * FROM ' + dbInfo.storeName, [], function (t, results) { var rows = results.rows; var length = rows.length; for (var i = 0; i < length; i++) { var item = rows.item(i); var result = item.value; // Check to see if this is serialized content // we need to unpack. if (result) { result = dbInfo.serializer.deserialize(result); } result = iterator(result, item.key, i + 1); // void(0) prevents problems with redefinition // of `undefined`. if (result !== void 0) { resolve(result); return; } } resolve(); }, function (t, error) { reject(error); }); }); })['catch'](reject); }); executeCallback(promise, callback); return promise; } function setItem(key, value, callback) { var self = this; // Cast the key to a string, as that's all we can set as a key. if (typeof key !== 'string') { globalObject.console.warn(key + ' used as a key, but it is not a string.'); key = String(key); } var promise = new Promise(function (resolve, reject) { self.ready().then(function () { // The localStorage API doesn't return undefined values in an // "expected" way, so undefined is always cast to null in all // drivers. See: https://github.com/mozilla/localForage/pull/42 if (value === undefined) { value = null; } // Save the original value to pass to the callback. var originalValue = value; var dbInfo = self._dbInfo; dbInfo.serializer.serialize(value, function (value, error) { if (error) { reject(error); } else { dbInfo.db.transaction(function (t) { t.executeSql('INSERT OR REPLACE INTO ' + dbInfo.storeName + ' (key, value) VALUES (?, ?)', [key, value], function () { resolve(originalValue); }, function (t, error) { reject(error); }); }, function (sqlError) { // The transaction failed; check // to see if it's a quota error. if (sqlError.code === sqlError.QUOTA_ERR) { // We reject the callback outright for now, but // it's worth trying to re-run the transaction. // Even if the user accepts the prompt to use // more storage on Safari, this error will // be called. // // TODO: Try to re-run the transaction. reject(sqlError); } }); } }); })['catch'](reject); }); executeCallback(promise, callback); return promise; } function removeItem(key, callback) { var self = this; // Cast the key to a string, as that's all we can set as a key. if (typeof key !== 'string') { globalObject.console.warn(key + ' used as a key, but it is not a string.'); key = String(key); } var promise = new Promise(function (resolve, reject) { self.ready().then(function () { var dbInfo = self._dbInfo; dbInfo.db.transaction(function (t) { t.executeSql('DELETE FROM ' + dbInfo.storeName + ' WHERE key = ?', [key], function () { resolve(); }, function (t, error) { reject(error); }); }); })['catch'](reject); }); executeCallback(promise, callback); return promise; } // Deletes every item in the table. // TODO: Find out if this resets the AUTO_INCREMENT number. function clear(callback) { var self = this; var promise = new Promise(function (resolve, reject) { self.ready().then(function () { var dbInfo = self._dbInfo; dbInfo.db.transaction(function (t) { t.executeSql('DELETE FROM ' + dbInfo.storeName, [], function () { resolve(); }, function (t, error) { reject(error); }); }); })['catch'](reject); }); executeCallback(promise, callback); return promise; } // Does a simple `COUNT(key)` to get the number of items stored in // localForage. function length(callback) { var self = this; var promise = new Promise(function (resolve, reject) { self.ready().then(function () { var dbInfo = self._dbInfo; dbInfo.db.transaction(function (t) { // Ahhh, SQL makes this one soooooo easy. t.executeSql('SELECT COUNT(key) as c FROM ' + dbInfo.storeName, [], function (t, results) { var result = results.rows.item(0).c; resolve(result); }, function (t, error) { reject(error); }); }); })['catch'](reject); }); executeCallback(promise, callback); return promise; } // Return the key located at key index X; essentially gets the key from a // `WHERE id = ?`. This is the most efficient way I can think to implement // this rarely-used (in my experience) part of the API, but it can seem // inconsistent, because we do `INSERT OR REPLACE INTO` on `setItem()`, so // the ID of each key will change every time it's updated. Perhaps a stored // procedure for the `setItem()` SQL would solve this problem? // TODO: Don't change ID on `setItem()`. function key(n, callback) { var self = this; var promise = new Promise(function (resolve, reject) { self.ready().then(function () { var dbInfo = self._dbInfo; dbInfo.db.transaction(function (t) { t.executeSql('SELECT key FROM ' + dbInfo.storeName + ' WHERE id = ? LIMIT 1', [n + 1], function (t, results) { var result = results.rows.length ? results.rows.item(0).key : null; resolve(result); }, function (t, error) { reject(error); }); }); })['catch'](reject); }); executeCallback(promise, callback); return promise; } function keys(callback) { var self = this; var promise = new Promise(function (resolve, reject) { self.ready().then(function () { var dbInfo = self._dbInfo; dbInfo.db.transaction(function (t) { t.executeSql('SELECT key FROM ' + dbInfo.storeName, [], function (t, results) { var keys = []; for (var i = 0; i < results.rows.length; i++) { keys.push(results.rows.item(i).key); } resolve(keys); }, function (t, error) { reject(error); }); }); })['catch'](reject); }); executeCallback(promise, callback); return promise; } function executeCallback(promise, callback) { if (callback) { promise.then(function (result) { callback(null, result); }, function (error) { callback(error); }); } } var webSQLStorage = { _driver: 'webSQLStorage', _initStorage: _initStorage, iterate: iterate, getItem: getItem, setItem: setItem, removeItem: removeItem, clear: clear, length: length, key: key, keys: keys }; exports['default'] = webSQLStorage; }).call(typeof window !== 'undefined' ? window : self); module.exports = exports['default']; /***/ } /******/ ]) }); ; ================================================ FILE: browser-version/test/mocha.css ================================================ @charset "UTF-8"; body { font: 20px/1.5 "Helvetica Neue", Helvetica, Arial, sans-serif; padding: 60px 50px; } #mocha ul, #mocha li { margin: 0; padding: 0; } #mocha ul { list-style: none; } #mocha h1, #mocha h2 { margin: 0; } #mocha h1 { margin-top: 15px; font-size: 1em; font-weight: 200; } #mocha h1 a { text-decoration: none; color: inherit; } #mocha h1 a:hover { text-decoration: underline; } #mocha .suite .suite h1 { margin-top: 0; font-size: .8em; } #mocha h2 { font-size: 12px; font-weight: normal; cursor: pointer; } #mocha .suite { margin-left: 15px; } #mocha .test { margin-left: 15px; } #mocha .test:hover h2::after { position: relative; top: 0; right: -10px; content: '(view source)'; font-size: 12px; font-family: arial; color: #888; } #mocha .test.pending:hover h2::after { content: '(pending)'; font-family: arial; } #mocha .test.pass.medium .duration { background: #C09853; } #mocha .test.pass.slow .duration { background: #B94A48; } #mocha .test.pass::before { content: '✓'; font-size: 12px; display: block; float: left; margin-right: 5px; color: #00d6b2; } #mocha .test.pass .duration { font-size: 9px; margin-left: 5px; padding: 2px 5px; color: white; -webkit-box-shadow: inset 0 1px 1px rgba(0,0,0,.2); -moz-box-shadow: inset 0 1px 1px rgba(0,0,0,.2); box-shadow: inset 0 1px 1px rgba(0,0,0,.2); -webkit-border-radius: 5px; -moz-border-radius: 5px; -ms-border-radius: 5px; -o-border-radius: 5px; border-radius: 5px; } #mocha .test.pass.fast .duration { display: none; } #mocha .test.pending { color: #0b97c4; } #mocha .test.pending::before { content: '◦'; color: #0b97c4; } #mocha .test.fail { color: #c00; } #mocha .test.fail pre { color: black; } #mocha .test.fail::before { content: '✖'; font-size: 12px; display: block; float: left; margin-right: 5px; color: #c00; } #mocha .test pre.error { color: #c00; } #mocha .test pre { display: inline-block; font: 12px/1.5 monaco, monospace; margin: 5px; padding: 15px; border: 1px solid #eee; border-bottom-color: #ddd; -webkit-border-radius: 3px; -webkit-box-shadow: 0 1px 3px #eee; } #report.pass .test.fail { display: none; } #report.fail .test.pass { display: none; } #error { color: #c00; font-size: 1.5 em; font-weight: 100; letter-spacing: 1px; } #stats { position: fixed; top: 15px; right: 10px; font-size: 12px; margin: 0; color: #888; } #stats .progress { float: right; padding-top: 0; } #stats em { color: black; } #stats a { text-decoration: none; color: inherit; } #stats a:hover { border-bottom: 1px solid #eee; } #stats li { display: inline-block; margin: 0 5px; list-style: none; padding-top: 11px; } code .comment { color: #ddd } code .init { color: #2F6FAD } code .string { color: #5890AD } code .keyword { color: #8A6343 } code .number { color: #2F6FAD } ================================================ FILE: browser-version/test/mocha.js ================================================ ;(function(){ // CommonJS require() function require(p){ var path = require.resolve(p) , mod = require.modules[path]; if (!mod) throw new Error('failed to require "' + p + '"'); if (!mod.exports) { mod.exports = {}; mod.call(mod.exports, mod, mod.exports, require.relative(path)); } return mod.exports; } require.modules = {}; require.resolve = function (path){ var orig = path , reg = path + '.js' , index = path + '/index.js'; return require.modules[reg] && reg || require.modules[index] && index || orig; }; require.register = function (path, fn){ require.modules[path] = fn; }; require.relative = function (parent) { return function(p){ if ('.' != p.charAt(0)) return require(p); var path = parent.split('/') , segs = p.split('/'); path.pop(); for (var i = 0; i < segs.length; i++) { var seg = segs[i]; if ('..' == seg) path.pop(); else if ('.' != seg) path.push(seg); } return require(path.join('/')); }; }; require.register("browser/debug.js", function(module, exports, require){ module.exports = function(type){ return function(){ } }; }); // module: browser/debug.js require.register("browser/diff.js", function(module, exports, require){ }); // module: browser/diff.js require.register("browser/events.js", function(module, exports, require){ /** * Module exports. */ exports.EventEmitter = EventEmitter; /** * Check if `obj` is an array. */ function isArray(obj) { return '[object Array]' == {}.toString.call(obj); } /** * Event emitter constructor. * * @api public */ function EventEmitter(){}; /** * Adds a listener. * * @api public */ EventEmitter.prototype.on = function (name, fn) { if (!this.$events) { this.$events = {}; } if (!this.$events[name]) { this.$events[name] = fn; } else if (isArray(this.$events[name])) { this.$events[name].push(fn); } else { this.$events[name] = [this.$events[name], fn]; } return this; }; EventEmitter.prototype.addListener = EventEmitter.prototype.on; /** * Adds a volatile listener. * * @api public */ EventEmitter.prototype.once = function (name, fn) { var self = this; function on () { self.removeListener(name, on); fn.apply(this, arguments); }; on.listener = fn; this.on(name, on); return this; }; /** * Removes a listener. * * @api public */ EventEmitter.prototype.removeListener = function (name, fn) { if (this.$events && this.$events[name]) { var list = this.$events[name]; if (isArray(list)) { var pos = -1; for (var i = 0, l = list.length; i < l; i++) { if (list[i] === fn || (list[i].listener && list[i].listener === fn)) { pos = i; break; } } if (pos < 0) { return this; } list.splice(pos, 1); if (!list.length) { delete this.$events[name]; } } else if (list === fn || (list.listener && list.listener === fn)) { delete this.$events[name]; } } return this; }; /** * Removes all listeners for an event. * * @api public */ EventEmitter.prototype.removeAllListeners = function (name) { if (name === undefined) { this.$events = {}; return this; } if (this.$events && this.$events[name]) { this.$events[name] = null; } return this; }; /** * Gets all listeners for a certain event. * * @api public */ EventEmitter.prototype.listeners = function (name) { if (!this.$events) { this.$events = {}; } if (!this.$events[name]) { this.$events[name] = []; } if (!isArray(this.$events[name])) { this.$events[name] = [this.$events[name]]; } return this.$events[name]; }; /** * Emits an event. * * @api public */ EventEmitter.prototype.emit = function (name) { if (!this.$events) { return false; } var handler = this.$events[name]; if (!handler) { return false; } var args = [].slice.call(arguments, 1); if ('function' == typeof handler) { handler.apply(this, args); } else if (isArray(handler)) { var listeners = handler.slice(); for (var i = 0, l = listeners.length; i < l; i++) { listeners[i].apply(this, args); } } else { return false; } return true; }; }); // module: browser/events.js require.register("browser/fs.js", function(module, exports, require){ }); // module: browser/fs.js require.register("browser/path.js", function(module, exports, require){ }); // module: browser/path.js require.register("browser/progress.js", function(module, exports, require){ /** * Expose `Progress`. */ module.exports = Progress; /** * Initialize a new `Progress` indicator. */ function Progress() { this.percent = 0; this.size(0); this.fontSize(11); this.font('helvetica, arial, sans-serif'); } /** * Set progress size to `n`. * * @param {Number} n * @return {Progress} for chaining * @api public */ Progress.prototype.size = function(n){ this._size = n; return this; }; /** * Set text to `str`. * * @param {String} str * @return {Progress} for chaining * @api public */ Progress.prototype.text = function(str){ this._text = str; return this; }; /** * Set font size to `n`. * * @param {Number} n * @return {Progress} for chaining * @api public */ Progress.prototype.fontSize = function(n){ this._fontSize = n; return this; }; /** * Set font `family`. * * @param {String} family * @return {Progress} for chaining */ Progress.prototype.font = function(family){ this._font = family; return this; }; /** * Update percentage to `n`. * * @param {Number} n * @return {Progress} for chaining */ Progress.prototype.update = function(n){ this.percent = n; return this; }; /** * Draw on `ctx`. * * @param {CanvasRenderingContext2d} ctx * @return {Progress} for chaining */ Progress.prototype.draw = function(ctx){ var percent = Math.min(this.percent, 100) , size = this._size , half = size / 2 , x = half , y = half , rad = half - 1 , fontSize = this._fontSize; ctx.font = fontSize + 'px ' + this._font; var angle = Math.PI * 2 * (percent / 100); ctx.clearRect(0, 0, size, size); // outer circle ctx.strokeStyle = '#9f9f9f'; ctx.beginPath(); ctx.arc(x, y, rad, 0, angle, false); ctx.stroke(); // inner circle ctx.strokeStyle = '#eee'; ctx.beginPath(); ctx.arc(x, y, rad - 1, 0, angle, true); ctx.stroke(); // text var text = this._text || (percent | 0) + '%' , w = ctx.measureText(text).width; ctx.fillText( text , x - w / 2 + 1 , y + fontSize / 2 - 1); return this; }; }); // module: browser/progress.js require.register("browser/tty.js", function(module, exports, require){ exports.isatty = function(){ return true; }; exports.getWindowSize = function(){ return [window.innerHeight, window.innerWidth]; }; }); // module: browser/tty.js require.register("context.js", function(module, exports, require){ /** * Expose `Context`. */ module.exports = Context; /** * Initialize a new `Context`. * * @api private */ function Context(){} /** * Set or get the context `Runnable` to `runnable`. * * @param {Runnable} runnable * @return {Context} * @api private */ Context.prototype.runnable = function(runnable){ if (0 == arguments.length) return this._runnable; this.test = this._runnable = runnable; return this; }; /** * Set test timeout `ms`. * * @param {Number} ms * @return {Context} self * @api private */ Context.prototype.timeout = function(ms){ this.runnable().timeout(ms); return this; }; /** * Set test slowness threshold `ms`. * * @param {Number} ms * @return {Context} self * @api private */ Context.prototype.slow = function(ms){ this.runnable().slow(ms); return this; }; /** * Inspect the context void of `._runnable`. * * @return {String} * @api private */ Context.prototype.inspect = function(){ return JSON.stringify(this, function(key, val){ if ('_runnable' == key) return; if ('test' == key) return; return val; }, 2); }; }); // module: context.js require.register("hook.js", function(module, exports, require){ /** * Module dependencies. */ var Runnable = require('./runnable'); /** * Expose `Hook`. */ module.exports = Hook; /** * Initialize a new `Hook` with the given `title` and callback `fn`. * * @param {String} title * @param {Function} fn * @api private */ function Hook(title, fn) { Runnable.call(this, title, fn); this.type = 'hook'; } /** * Inherit from `Runnable.prototype`. */ Hook.prototype = new Runnable; Hook.prototype.constructor = Hook; /** * Get or set the test `err`. * * @param {Error} err * @return {Error} * @api public */ Hook.prototype.error = function(err){ if (0 == arguments.length) { var err = this._error; this._error = null; return err; } this._error = err; }; }); // module: hook.js require.register("interfaces/bdd.js", function(module, exports, require){ /** * Module dependencies. */ var Suite = require('../suite') , Test = require('../test'); /** * BDD-style interface: * * describe('Array', function(){ * describe('#indexOf()', function(){ * it('should return -1 when not present', function(){ * * }); * * it('should return the index when present', function(){ * * }); * }); * }); * */ module.exports = function(suite){ var suites = [suite]; suite.on('pre-require', function(context, file, mocha){ /** * Execute before running tests. */ context.before = function(fn){ suites[0].beforeAll(fn); }; /** * Execute after running tests. */ context.after = function(fn){ suites[0].afterAll(fn); }; /** * Execute before each test case. */ context.beforeEach = function(fn){ suites[0].beforeEach(fn); }; /** * Execute after each test case. */ context.afterEach = function(fn){ suites[0].afterEach(fn); }; /** * Describe a "suite" with the given `title` * and callback `fn` containing nested suites * and/or tests. */ context.describe = context.context = function(title, fn){ var suite = Suite.create(suites[0], title); suites.unshift(suite); fn(); suites.shift(); return suite; }; /** * Pending describe. */ context.xdescribe = context.xcontext = context.describe.skip = function(title, fn){ var suite = Suite.create(suites[0], title); suite.pending = true; suites.unshift(suite); fn(); suites.shift(); }; /** * Exclusive suite. */ context.describe.only = function(title, fn){ var suite = context.describe(title, fn); mocha.grep(suite.fullTitle()); }; /** * Describe a specification or test-case * with the given `title` and callback `fn` * acting as a thunk. */ context.it = context.specify = function(title, fn){ var suite = suites[0]; if (suite.pending) var fn = null; var test = new Test(title, fn); suite.addTest(test); return test; }; /** * Exclusive test-case. */ context.it.only = function(title, fn){ var test = context.it(title, fn); mocha.grep(test.fullTitle()); }; /** * Pending test case. */ context.xit = context.xspecify = context.it.skip = function(title){ context.it(title); }; }); }; }); // module: interfaces/bdd.js require.register("interfaces/exports.js", function(module, exports, require){ /** * Module dependencies. */ var Suite = require('../suite') , Test = require('../test'); /** * TDD-style interface: * * exports.Array = { * '#indexOf()': { * 'should return -1 when the value is not present': function(){ * * }, * * 'should return the correct index when the value is present': function(){ * * } * } * }; * */ module.exports = function(suite){ var suites = [suite]; suite.on('require', visit); function visit(obj) { var suite; for (var key in obj) { if ('function' == typeof obj[key]) { var fn = obj[key]; switch (key) { case 'before': suites[0].beforeAll(fn); break; case 'after': suites[0].afterAll(fn); break; case 'beforeEach': suites[0].beforeEach(fn); break; case 'afterEach': suites[0].afterEach(fn); break; default: suites[0].addTest(new Test(key, fn)); } } else { var suite = Suite.create(suites[0], key); suites.unshift(suite); visit(obj[key]); suites.shift(); } } } }; }); // module: interfaces/exports.js require.register("interfaces/index.js", function(module, exports, require){ exports.bdd = require('./bdd'); exports.tdd = require('./tdd'); exports.qunit = require('./qunit'); exports.exports = require('./exports'); }); // module: interfaces/index.js require.register("interfaces/qunit.js", function(module, exports, require){ /** * Module dependencies. */ var Suite = require('../suite') , Test = require('../test'); /** * QUnit-style interface: * * suite('Array'); * * test('#length', function(){ * var arr = [1,2,3]; * ok(arr.length == 3); * }); * * test('#indexOf()', function(){ * var arr = [1,2,3]; * ok(arr.indexOf(1) == 0); * ok(arr.indexOf(2) == 1); * ok(arr.indexOf(3) == 2); * }); * * suite('String'); * * test('#length', function(){ * ok('foo'.length == 3); * }); * */ module.exports = function(suite){ var suites = [suite]; suite.on('pre-require', function(context){ /** * Execute before running tests. */ context.before = function(fn){ suites[0].beforeAll(fn); }; /** * Execute after running tests. */ context.after = function(fn){ suites[0].afterAll(fn); }; /** * Execute before each test case. */ context.beforeEach = function(fn){ suites[0].beforeEach(fn); }; /** * Execute after each test case. */ context.afterEach = function(fn){ suites[0].afterEach(fn); }; /** * Describe a "suite" with the given `title`. */ context.suite = function(title){ if (suites.length > 1) suites.shift(); var suite = Suite.create(suites[0], title); suites.unshift(suite); }; /** * Describe a specification or test-case * with the given `title` and callback `fn` * acting as a thunk. */ context.test = function(title, fn){ suites[0].addTest(new Test(title, fn)); }; }); }; }); // module: interfaces/qunit.js require.register("interfaces/tdd.js", function(module, exports, require){ /** * Module dependencies. */ var Suite = require('../suite') , Test = require('../test'); /** * TDD-style interface: * * suite('Array', function(){ * suite('#indexOf()', function(){ * suiteSetup(function(){ * * }); * * test('should return -1 when not present', function(){ * * }); * * test('should return the index when present', function(){ * * }); * * suiteTeardown(function(){ * * }); * }); * }); * */ module.exports = function(suite){ var suites = [suite]; suite.on('pre-require', function(context, file, mocha){ /** * Execute before each test case. */ context.setup = function(fn){ suites[0].beforeEach(fn); }; /** * Execute after each test case. */ context.teardown = function(fn){ suites[0].afterEach(fn); }; /** * Execute before the suite. */ context.suiteSetup = function(fn){ suites[0].beforeAll(fn); }; /** * Execute after the suite. */ context.suiteTeardown = function(fn){ suites[0].afterAll(fn); }; /** * Describe a "suite" with the given `title` * and callback `fn` containing nested suites * and/or tests. */ context.suite = function(title, fn){ var suite = Suite.create(suites[0], title); suites.unshift(suite); fn(); suites.shift(); return suite; }; /** * Exclusive test-case. */ context.suite.only = function(title, fn){ var suite = context.suite(title, fn); mocha.grep(suite.fullTitle()); }; /** * Describe a specification or test-case * with the given `title` and callback `fn` * acting as a thunk. */ context.test = function(title, fn){ var test = new Test(title, fn); suites[0].addTest(test); return test; }; /** * Exclusive test-case. */ context.test.only = function(title, fn){ var test = context.test(title, fn); mocha.grep(test.fullTitle()); }; }); }; }); // module: interfaces/tdd.js require.register("mocha.js", function(module, exports, require){ /*! * mocha * Copyright(c) 2011 TJ Holowaychuk * MIT Licensed */ /** * Module dependencies. */ var path = require('browser/path') , utils = require('./utils'); /** * Expose `Mocha`. */ exports = module.exports = Mocha; /** * Expose internals. */ exports.utils = utils; exports.interfaces = require('./interfaces'); exports.reporters = require('./reporters'); exports.Runnable = require('./runnable'); exports.Context = require('./context'); exports.Runner = require('./runner'); exports.Suite = require('./suite'); exports.Hook = require('./hook'); exports.Test = require('./test'); /** * Return image `name` path. * * @param {String} name * @return {String} * @api private */ function image(name) { return __dirname + '/../images/' + name + '.png'; } /** * Setup mocha with `options`. * * Options: * * - `ui` name "bdd", "tdd", "exports" etc * - `reporter` reporter instance, defaults to `mocha.reporters.Dot` * - `globals` array of accepted globals * - `timeout` timeout in milliseconds * - `slow` milliseconds to wait before considering a test slow * - `ignoreLeaks` ignore global leaks * - `grep` string or regexp to filter tests with * * @param {Object} options * @api public */ function Mocha(options) { options = options || {}; this.files = []; this.options = options; this.grep(options.grep); this.suite = new exports.Suite('', new exports.Context); this.ui(options.ui); this.reporter(options.reporter); if (options.timeout) this.timeout(options.timeout); if (options.slow) this.slow(options.slow); } /** * Add test `file`. * * @param {String} file * @api public */ Mocha.prototype.addFile = function(file){ this.files.push(file); return this; }; /** * Set reporter to `reporter`, defaults to "dot". * * @param {String|Function} reporter name of a reporter or a reporter constructor * @api public */ Mocha.prototype.reporter = function(reporter){ if ('function' == typeof reporter) { this._reporter = reporter; } else { reporter = reporter || 'dot'; try { this._reporter = require('./reporters/' + reporter); } catch (err) { this._reporter = require(reporter); } if (!this._reporter) throw new Error('invalid reporter "' + reporter + '"'); } return this; }; /** * Set test UI `name`, defaults to "bdd". * * @param {String} bdd * @api public */ Mocha.prototype.ui = function(name){ name = name || 'bdd'; this._ui = exports.interfaces[name]; if (!this._ui) throw new Error('invalid interface "' + name + '"'); this._ui = this._ui(this.suite); return this; }; /** * Load registered files. * * @api private */ Mocha.prototype.loadFiles = function(fn){ var self = this; var suite = this.suite; var pending = this.files.length; this.files.forEach(function(file){ file = path.resolve(file); suite.emit('pre-require', global, file, self); suite.emit('require', require(file), file, self); suite.emit('post-require', global, file, self); --pending || (fn && fn()); }); }; /** * Enable growl support. * * @api private */ Mocha.prototype._growl = function(runner, reporter) { var notify = require('growl'); runner.on('end', function(){ var stats = reporter.stats; if (stats.failures) { var msg = stats.failures + ' of ' + runner.total + ' tests failed'; notify(msg, { name: 'mocha', title: 'Failed', image: image('error') }); } else { notify(stats.passes + ' tests passed in ' + stats.duration + 'ms', { name: 'mocha' , title: 'Passed' , image: image('ok') }); } }); }; /** * Add regexp to grep, if `re` is a string it is escaped. * * @param {RegExp|String} re * @return {Mocha} * @api public */ Mocha.prototype.grep = function(re){ this.options.grep = 'string' == typeof re ? new RegExp(utils.escapeRegexp(re)) : re; return this; }; /** * Invert `.grep()` matches. * * @return {Mocha} * @api public */ Mocha.prototype.invert = function(){ this.options.invert = true; return this; }; /** * Ignore global leaks. * * @return {Mocha} * @api public */ Mocha.prototype.ignoreLeaks = function(){ this.options.ignoreLeaks = true; return this; }; /** * Enable global leak checking. * * @return {Mocha} * @api public */ Mocha.prototype.checkLeaks = function(){ this.options.ignoreLeaks = false; return this; }; /** * Enable growl support. * * @return {Mocha} * @api public */ Mocha.prototype.growl = function(){ this.options.growl = true; return this; }; /** * Ignore `globals` array or string. * * @param {Array|String} globals * @return {Mocha} * @api public */ Mocha.prototype.globals = function(globals){ this.options.globals = (this.options.globals || []).concat(globals); return this; }; /** * Set the timeout in milliseconds. * * @param {Number} timeout * @return {Mocha} * @api public */ Mocha.prototype.timeout = function(timeout){ this.suite.timeout(timeout); return this; }; /** * Set slowness threshold in milliseconds. * * @param {Number} slow * @return {Mocha} * @api public */ Mocha.prototype.slow = function(slow){ this.suite.slow(slow); return this; }; /** * Run tests and invoke `fn()` when complete. * * @param {Function} fn * @return {Runner} * @api public */ Mocha.prototype.run = function(fn){ if (this.files.length) this.loadFiles(); var suite = this.suite; var options = this.options; var runner = new exports.Runner(suite); var reporter = new this._reporter(runner); runner.ignoreLeaks = options.ignoreLeaks; if (options.grep) runner.grep(options.grep, options.invert); if (options.globals) runner.globals(options.globals); if (options.growl) this._growl(runner, reporter); return runner.run(fn); }; }); // module: mocha.js require.register("ms.js", function(module, exports, require){ /** * Helpers. */ var s = 1000; var m = s * 60; var h = m * 60; var d = h * 24; /** * Parse or format the given `val`. * * @param {String|Number} val * @return {String|Number} * @api public */ module.exports = function(val){ if ('string' == typeof val) return parse(val); return format(val); } /** * Parse the given `str` and return milliseconds. * * @param {String} str * @return {Number} * @api private */ function parse(str) { var m = /^((?:\d+)?\.?\d+) *(ms|seconds?|s|minutes?|m|hours?|h|days?|d|years?|y)?$/i.exec(str); if (!m) return; var n = parseFloat(m[1]); var type = (m[2] || 'ms').toLowerCase(); switch (type) { case 'years': case 'year': case 'y': return n * 31557600000; case 'days': case 'day': case 'd': return n * 86400000; case 'hours': case 'hour': case 'h': return n * 3600000; case 'minutes': case 'minute': case 'm': return n * 60000; case 'seconds': case 'second': case 's': return n * 1000; case 'ms': return n; } } /** * Format the given `ms`. * * @param {Number} ms * @return {String} * @api public */ function format(ms) { if (ms == d) return (ms / d) + ' day'; if (ms > d) return (ms / d) + ' days'; if (ms == h) return (ms / h) + ' hour'; if (ms > h) return (ms / h) + ' hours'; if (ms == m) return (ms / m) + ' minute'; if (ms > m) return (ms / m) + ' minutes'; if (ms == s) return (ms / s) + ' second'; if (ms > s) return (ms / s) + ' seconds'; return ms + ' ms'; } }); // module: ms.js require.register("reporters/base.js", function(module, exports, require){ /** * Module dependencies. */ var tty = require('browser/tty') , diff = require('browser/diff') , ms = require('../ms'); /** * Save timer references to avoid Sinon interfering (see GH-237). */ var Date = global.Date , setTimeout = global.setTimeout , setInterval = global.setInterval , clearTimeout = global.clearTimeout , clearInterval = global.clearInterval; /** * Check if both stdio streams are associated with a tty. */ var isatty = tty.isatty(1) && tty.isatty(2); /** * Expose `Base`. */ exports = module.exports = Base; /** * Enable coloring by default. */ exports.useColors = isatty; /** * Default color map. */ exports.colors = { 'pass': 90 , 'fail': 31 , 'bright pass': 92 , 'bright fail': 91 , 'bright yellow': 93 , 'pending': 36 , 'suite': 0 , 'error title': 0 , 'error message': 31 , 'error stack': 90 , 'checkmark': 32 , 'fast': 90 , 'medium': 33 , 'slow': 31 , 'green': 32 , 'light': 90 , 'diff gutter': 90 , 'diff added': 42 , 'diff removed': 41 }; /** * Color `str` with the given `type`, * allowing colors to be disabled, * as well as user-defined color * schemes. * * @param {String} type * @param {String} str * @return {String} * @api private */ var color = exports.color = function(type, str) { if (!exports.useColors) return str; return '\u001b[' + exports.colors[type] + 'm' + str + '\u001b[0m'; }; /** * Expose term window size, with some * defaults for when stderr is not a tty. */ exports.window = { width: isatty ? process.stdout.getWindowSize ? process.stdout.getWindowSize(1)[0] : tty.getWindowSize()[1] : 75 }; /** * Expose some basic cursor interactions * that are common among reporters. */ exports.cursor = { hide: function(){ process.stdout.write('\u001b[?25l'); }, show: function(){ process.stdout.write('\u001b[?25h'); }, deleteLine: function(){ process.stdout.write('\u001b[2K'); }, beginningOfLine: function(){ process.stdout.write('\u001b[0G'); }, CR: function(){ exports.cursor.deleteLine(); exports.cursor.beginningOfLine(); } }; /** * Outut the given `failures` as a list. * * @param {Array} failures * @api public */ exports.list = function(failures){ console.error(); failures.forEach(function(test, i){ // format var fmt = color('error title', ' %s) %s:\n') + color('error message', ' %s') + color('error stack', '\n%s\n'); // msg var err = test.err , message = err.message || '' , stack = err.stack || message , index = stack.indexOf(message) + message.length , msg = stack.slice(0, index) , actual = err.actual , expected = err.expected; // actual / expected diff if ('string' == typeof actual && 'string' == typeof expected) { var len = Math.max(actual.length, expected.length); if (len < 20) msg = errorDiff(err, 'Chars'); else msg = errorDiff(err, 'Words'); // linenos var lines = msg.split('\n'); if (lines.length > 4) { var width = String(lines.length).length; msg = lines.map(function(str, i){ return pad(++i, width) + ' |' + ' ' + str; }).join('\n'); } // legend msg = '\n' + color('diff removed', 'actual') + ' ' + color('diff added', 'expected') + '\n\n' + msg + '\n'; // indent msg = msg.replace(/^/gm, ' '); fmt = color('error title', ' %s) %s:\n%s') + color('error stack', '\n%s\n'); } // indent stack trace without msg stack = stack.slice(index ? index + 1 : index) .replace(/^/gm, ' '); console.error(fmt, (i + 1), test.fullTitle(), msg, stack); }); }; /** * Initialize a new `Base` reporter. * * All other reporters generally * inherit from this reporter, providing * stats such as test duration, number * of tests passed / failed etc. * * @param {Runner} runner * @api public */ function Base(runner) { var self = this , stats = this.stats = { suites: 0, tests: 0, passes: 0, pending: 0, failures: 0 } , failures = this.failures = []; if (!runner) return; this.runner = runner; runner.on('start', function(){ stats.start = new Date; }); runner.on('suite', function(suite){ stats.suites = stats.suites || 0; suite.root || stats.suites++; }); runner.on('test end', function(test){ stats.tests = stats.tests || 0; stats.tests++; }); runner.on('pass', function(test){ stats.passes = stats.passes || 0; var medium = test.slow() / 2; test.speed = test.duration > test.slow() ? 'slow' : test.duration > medium ? 'medium' : 'fast'; stats.passes++; }); runner.on('fail', function(test, err){ stats.failures = stats.failures || 0; stats.failures++; test.err = err; failures.push(test); }); runner.on('end', function(){ stats.end = new Date; stats.duration = new Date - stats.start; }); runner.on('pending', function(){ stats.pending++; }); } /** * Output common epilogue used by many of * the bundled reporters. * * @api public */ Base.prototype.epilogue = function(){ var stats = this.stats , fmt , tests; console.log(); function pluralize(n) { return 1 == n ? 'test' : 'tests'; } // failure if (stats.failures) { fmt = color('bright fail', ' ✖') + color('fail', ' %d of %d %s failed') + color('light', ':') console.error(fmt, stats.failures, this.runner.total, pluralize(this.runner.total)); Base.list(this.failures); console.error(); return; } // pass fmt = color('bright pass', ' ✔') + color('green', ' %d %s complete') + color('light', ' (%s)'); console.log(fmt, stats.tests || 0, pluralize(stats.tests), ms(stats.duration)); // pending if (stats.pending) { fmt = color('pending', ' •') + color('pending', ' %d %s pending'); console.log(fmt, stats.pending, pluralize(stats.pending)); } console.log(); }; /** * Pad the given `str` to `len`. * * @param {String} str * @param {String} len * @return {String} * @api private */ function pad(str, len) { str = String(str); return Array(len - str.length + 1).join(' ') + str; } /** * Return a character diff for `err`. * * @param {Error} err * @return {String} * @api private */ function errorDiff(err, type) { return diff['diff' + type](err.actual, err.expected).map(function(str){ str.value = str.value .replace(/\t/g, '') .replace(/\r/g, '') .replace(/\n/g, '\n'); if (str.added) return colorLines('diff added', str.value); if (str.removed) return colorLines('diff removed', str.value); return str.value; }).join(''); } /** * Color lines for `str`, using the color `name`. * * @param {String} name * @param {String} str * @return {String} * @api private */ function colorLines(name, str) { return str.split('\n').map(function(str){ return color(name, str); }).join('\n'); } }); // module: reporters/base.js require.register("reporters/doc.js", function(module, exports, require){ /** * Module dependencies. */ var Base = require('./base') , utils = require('../utils'); /** * Expose `Doc`. */ exports = module.exports = Doc; /** * Initialize a new `Doc` reporter. * * @param {Runner} runner * @api public */ function Doc(runner) { Base.call(this, runner); var self = this , stats = this.stats , total = runner.total , indents = 2; function indent() { return Array(indents).join(' '); } runner.on('suite', function(suite){ if (suite.root) return; ++indents; console.log('%s
', indent()); ++indents; console.log('%s

%s

', indent(), suite.title); console.log('%s
', indent()); }); runner.on('suite end', function(suite){ if (suite.root) return; console.log('%s
', indent()); --indents; console.log('%s
', indent()); --indents; }); runner.on('pass', function(test){ console.log('%s
%s
', indent(), test.title); var code = utils.escape(utils.clean(test.fn.toString())); console.log('%s
%s
', indent(), code); }); } }); // module: reporters/doc.js require.register("reporters/dot.js", function(module, exports, require){ /** * Module dependencies. */ var Base = require('./base') , color = Base.color; /** * Expose `Dot`. */ exports = module.exports = Dot; /** * Initialize a new `Dot` matrix test reporter. * * @param {Runner} runner * @api public */ function Dot(runner) { Base.call(this, runner); var self = this , stats = this.stats , width = Base.window.width * .75 | 0 , c = '․' , n = 0; runner.on('start', function(){ process.stdout.write('\n '); }); runner.on('pending', function(test){ process.stdout.write(color('pending', c)); }); runner.on('pass', function(test){ if (++n % width == 0) process.stdout.write('\n '); if ('slow' == test.speed) { process.stdout.write(color('bright yellow', c)); } else { process.stdout.write(color(test.speed, c)); } }); runner.on('fail', function(test, err){ if (++n % width == 0) process.stdout.write('\n '); process.stdout.write(color('fail', c)); }); runner.on('end', function(){ console.log(); self.epilogue(); }); } /** * Inherit from `Base.prototype`. */ Dot.prototype = new Base; Dot.prototype.constructor = Dot; }); // module: reporters/dot.js require.register("reporters/html-cov.js", function(module, exports, require){ /** * Module dependencies. */ var JSONCov = require('./json-cov') , fs = require('browser/fs'); /** * Expose `HTMLCov`. */ exports = module.exports = HTMLCov; /** * Initialize a new `JsCoverage` reporter. * * @param {Runner} runner * @api public */ function HTMLCov(runner) { var jade = require('jade') , file = __dirname + '/templates/coverage.jade' , str = fs.readFileSync(file, 'utf8') , fn = jade.compile(str, { filename: file }) , self = this; JSONCov.call(this, runner, false); runner.on('end', function(){ process.stdout.write(fn({ cov: self.cov , coverageClass: coverageClass })); }); } /** * Return coverage class for `n`. * * @return {String} * @api private */ function coverageClass(n) { if (n >= 75) return 'high'; if (n >= 50) return 'medium'; if (n >= 25) return 'low'; return 'terrible'; } }); // module: reporters/html-cov.js require.register("reporters/html.js", function(module, exports, require){ /** * Module dependencies. */ var Base = require('./base') , utils = require('../utils') , Progress = require('../browser/progress') , escape = utils.escape; /** * Save timer references to avoid Sinon interfering (see GH-237). */ var Date = global.Date , setTimeout = global.setTimeout , setInterval = global.setInterval , clearTimeout = global.clearTimeout , clearInterval = global.clearInterval; /** * Expose `Doc`. */ exports = module.exports = HTML; /** * Stats template. */ var statsTemplate = ''; /** * Initialize a new `Doc` reporter. * * @param {Runner} runner * @api public */ function HTML(runner, root) { Base.call(this, runner); var self = this , stats = this.stats , total = runner.total , stat = fragment(statsTemplate) , items = stat.getElementsByTagName('li') , passes = items[1].getElementsByTagName('em')[0] , passesLink = items[1].getElementsByTagName('a')[0] , failures = items[2].getElementsByTagName('em')[0] , failuresLink = items[2].getElementsByTagName('a')[0] , duration = items[3].getElementsByTagName('em')[0] , canvas = stat.getElementsByTagName('canvas')[0] , report = fragment('
    ') , stack = [report] , progress , ctx root = root || document.getElementById('mocha'); if (canvas.getContext) { var ratio = window.devicePixelRatio || 1; canvas.style.width = canvas.width; canvas.style.height = canvas.height; canvas.width *= ratio; canvas.height *= ratio; ctx = canvas.getContext('2d'); ctx.scale(ratio, ratio); progress = new Progress; } if (!root) return error('#mocha div missing, add it to your document'); // pass toggle on(passesLink, 'click', function () { var className = /pass/.test(report.className) ? '' : ' pass'; report.className = report.className.replace(/fail|pass/g, '') + className; }); // failure toggle on(failuresLink, 'click', function () { var className = /fail/.test(report.className) ? '' : ' fail'; report.className = report.className.replace(/fail|pass/g, '') + className; }); root.appendChild(stat); root.appendChild(report); if (progress) progress.size(40); runner.on('suite', function(suite){ if (suite.root) return; // suite var url = '?grep=' + encodeURIComponent(suite.fullTitle()); var el = fragment('
  • %s

  • ', url, escape(suite.title)); // container stack[0].appendChild(el); stack.unshift(document.createElement('ul')); el.appendChild(stack[0]); }); runner.on('suite end', function(suite){ if (suite.root) return; stack.shift(); }); runner.on('fail', function(test, err){ if ('hook' == test.type || err.uncaught) runner.emit('test end', test); }); runner.on('test end', function(test){ window.scrollTo(0, document.body.scrollHeight); // TODO: add to stats var percent = stats.tests / total * 100 | 0; if (progress) progress.update(percent).draw(ctx); // update stats var ms = new Date - stats.start; text(passes, stats.passes); text(failures, stats.failures); text(duration, (ms / 1000).toFixed(2)); // test if ('passed' == test.state) { var el = fragment('
  • %e%ems

  • ', test.speed, test.title, test.duration); } else if (test.pending) { var el = fragment('
  • %e

  • ', test.title); } else { var el = fragment('
  • %e

  • ', test.title); var str = test.err.stack || test.err.toString(); // FF / Opera do not add the message if (!~str.indexOf(test.err.message)) { str = test.err.message + '\n' + str; } // <=IE7 stringifies to [Object Error]. Since it can be overloaded, we // check for the result of the stringifying. if ('[object Error]' == str) str = test.err.message; // Safari doesn't give you a stack. Let's at least provide a source line. if (!test.err.stack && test.err.sourceURL && test.err.line !== undefined) { str += "\n(" + test.err.sourceURL + ":" + test.err.line + ")"; } el.appendChild(fragment('
    %e
    ', str)); } // toggle code // TODO: defer if (!test.pending) { var h2 = el.getElementsByTagName('h2')[0]; on(h2, 'click', function(){ pre.style.display = 'none' == pre.style.display ? 'inline-block' : 'none'; }); var pre = fragment('
    %e
    ', utils.clean(test.fn.toString())); el.appendChild(pre); pre.style.display = 'none'; } stack[0].appendChild(el); }); } /** * Display error `msg`. */ function error(msg) { document.body.appendChild(fragment('
    %s
    ', msg)); } /** * Return a DOM fragment from `html`. */ function fragment(html) { var args = arguments , div = document.createElement('div') , i = 1; div.innerHTML = html.replace(/%([se])/g, function(_, type){ switch (type) { case 's': return String(args[i++]); case 'e': return escape(args[i++]); } }); return div.firstChild; } /** * Set `el` text to `str`. */ function text(el, str) { if (el.textContent) { el.textContent = str; } else { el.innerText = str; } } /** * Listen on `event` with callback `fn`. */ function on(el, event, fn) { if (el.addEventListener) { el.addEventListener(event, fn, false); } else { el.attachEvent('on' + event, fn); } } }); // module: reporters/html.js require.register("reporters/index.js", function(module, exports, require){ exports.Base = require('./base'); exports.Dot = require('./dot'); exports.Doc = require('./doc'); exports.TAP = require('./tap'); exports.JSON = require('./json'); exports.HTML = require('./html'); exports.List = require('./list'); exports.Min = require('./min'); exports.Spec = require('./spec'); exports.Nyan = require('./nyan'); exports.XUnit = require('./xunit'); exports.Markdown = require('./markdown'); exports.Progress = require('./progress'); exports.Landing = require('./landing'); exports.JSONCov = require('./json-cov'); exports.HTMLCov = require('./html-cov'); exports.JSONStream = require('./json-stream'); exports.Teamcity = require('./teamcity'); }); // module: reporters/index.js require.register("reporters/json-cov.js", function(module, exports, require){ /** * Module dependencies. */ var Base = require('./base'); /** * Expose `JSONCov`. */ exports = module.exports = JSONCov; /** * Initialize a new `JsCoverage` reporter. * * @param {Runner} runner * @param {Boolean} output * @api public */ function JSONCov(runner, output) { var self = this , output = 1 == arguments.length ? true : output; Base.call(this, runner); var tests = [] , failures = [] , passes = []; runner.on('test end', function(test){ tests.push(test); }); runner.on('pass', function(test){ passes.push(test); }); runner.on('fail', function(test){ failures.push(test); }); runner.on('end', function(){ var cov = global._$jscoverage || {}; var result = self.cov = map(cov); result.stats = self.stats; result.tests = tests.map(clean); result.failures = failures.map(clean); result.passes = passes.map(clean); if (!output) return; process.stdout.write(JSON.stringify(result, null, 2 )); }); } /** * Map jscoverage data to a JSON structure * suitable for reporting. * * @param {Object} cov * @return {Object} * @api private */ function map(cov) { var ret = { instrumentation: 'node-jscoverage' , sloc: 0 , hits: 0 , misses: 0 , coverage: 0 , files: [] }; for (var filename in cov) { var data = coverage(filename, cov[filename]); ret.files.push(data); ret.hits += data.hits; ret.misses += data.misses; ret.sloc += data.sloc; } if (ret.sloc > 0) { ret.coverage = (ret.hits / ret.sloc) * 100; } return ret; }; /** * Map jscoverage data for a single source file * to a JSON structure suitable for reporting. * * @param {String} filename name of the source file * @param {Object} data jscoverage coverage data * @return {Object} * @api private */ function coverage(filename, data) { var ret = { filename: filename, coverage: 0, hits: 0, misses: 0, sloc: 0, source: {} }; data.source.forEach(function(line, num){ num++; if (data[num] === 0) { ret.misses++; ret.sloc++; } else if (data[num] !== undefined) { ret.hits++; ret.sloc++; } ret.source[num] = { source: line , coverage: data[num] === undefined ? '' : data[num] }; }); ret.coverage = ret.hits / ret.sloc * 100; return ret; } /** * Return a plain-object representation of `test` * free of cyclic properties etc. * * @param {Object} test * @return {Object} * @api private */ function clean(test) { return { title: test.title , fullTitle: test.fullTitle() , duration: test.duration } } }); // module: reporters/json-cov.js require.register("reporters/json-stream.js", function(module, exports, require){ /** * Module dependencies. */ var Base = require('./base') , color = Base.color; /** * Expose `List`. */ exports = module.exports = List; /** * Initialize a new `List` test reporter. * * @param {Runner} runner * @api public */ function List(runner) { Base.call(this, runner); var self = this , stats = this.stats , total = runner.total; runner.on('start', function(){ console.log(JSON.stringify(['start', { total: total }])); }); runner.on('pass', function(test){ console.log(JSON.stringify(['pass', clean(test)])); }); runner.on('fail', function(test, err){ console.log(JSON.stringify(['fail', clean(test)])); }); runner.on('end', function(){ process.stdout.write(JSON.stringify(['end', self.stats])); }); } /** * Return a plain-object representation of `test` * free of cyclic properties etc. * * @param {Object} test * @return {Object} * @api private */ function clean(test) { return { title: test.title , fullTitle: test.fullTitle() , duration: test.duration } } }); // module: reporters/json-stream.js require.register("reporters/json.js", function(module, exports, require){ /** * Module dependencies. */ var Base = require('./base') , cursor = Base.cursor , color = Base.color; /** * Expose `JSON`. */ exports = module.exports = JSONReporter; /** * Initialize a new `JSON` reporter. * * @param {Runner} runner * @api public */ function JSONReporter(runner) { var self = this; Base.call(this, runner); var tests = [] , failures = [] , passes = []; runner.on('test end', function(test){ tests.push(test); }); runner.on('pass', function(test){ passes.push(test); }); runner.on('fail', function(test){ failures.push(test); }); runner.on('end', function(){ var obj = { stats: self.stats , tests: tests.map(clean) , failures: failures.map(clean) , passes: passes.map(clean) }; process.stdout.write(JSON.stringify(obj, null, 2)); }); } /** * Return a plain-object representation of `test` * free of cyclic properties etc. * * @param {Object} test * @return {Object} * @api private */ function clean(test) { return { title: test.title , fullTitle: test.fullTitle() , duration: test.duration } } }); // module: reporters/json.js require.register("reporters/landing.js", function(module, exports, require){ /** * Module dependencies. */ var Base = require('./base') , cursor = Base.cursor , color = Base.color; /** * Expose `Landing`. */ exports = module.exports = Landing; /** * Airplane color. */ Base.colors.plane = 0; /** * Airplane crash color. */ Base.colors['plane crash'] = 31; /** * Runway color. */ Base.colors.runway = 90; /** * Initialize a new `Landing` reporter. * * @param {Runner} runner * @api public */ function Landing(runner) { Base.call(this, runner); var self = this , stats = this.stats , width = Base.window.width * .75 | 0 , total = runner.total , stream = process.stdout , plane = color('plane', '✈') , crashed = -1 , n = 0; function runway() { var buf = Array(width).join('-'); return ' ' + color('runway', buf); } runner.on('start', function(){ stream.write('\n '); cursor.hide(); }); runner.on('test end', function(test){ // check if the plane crashed var col = -1 == crashed ? width * ++n / total | 0 : crashed; // show the crash if ('failed' == test.state) { plane = color('plane crash', '✈'); crashed = col; } // render landing strip stream.write('\u001b[4F\n\n'); stream.write(runway()); stream.write('\n '); stream.write(color('runway', Array(col).join('⋅'))); stream.write(plane) stream.write(color('runway', Array(width - col).join('⋅') + '\n')); stream.write(runway()); stream.write('\u001b[0m'); }); runner.on('end', function(){ cursor.show(); console.log(); self.epilogue(); }); } /** * Inherit from `Base.prototype`. */ Landing.prototype = new Base; Landing.prototype.constructor = Landing; }); // module: reporters/landing.js require.register("reporters/list.js", function(module, exports, require){ /** * Module dependencies. */ var Base = require('./base') , cursor = Base.cursor , color = Base.color; /** * Expose `List`. */ exports = module.exports = List; /** * Initialize a new `List` test reporter. * * @param {Runner} runner * @api public */ function List(runner) { Base.call(this, runner); var self = this , stats = this.stats , n = 0; runner.on('start', function(){ console.log(); }); runner.on('test', function(test){ process.stdout.write(color('pass', ' ' + test.fullTitle() + ': ')); }); runner.on('pending', function(test){ var fmt = color('checkmark', ' -') + color('pending', ' %s'); console.log(fmt, test.fullTitle()); }); runner.on('pass', function(test){ var fmt = color('checkmark', ' ✓') + color('pass', ' %s: ') + color(test.speed, '%dms'); cursor.CR(); console.log(fmt, test.fullTitle(), test.duration); }); runner.on('fail', function(test, err){ cursor.CR(); console.log(color('fail', ' %d) %s'), ++n, test.fullTitle()); }); runner.on('end', self.epilogue.bind(self)); } /** * Inherit from `Base.prototype`. */ List.prototype = new Base; List.prototype.constructor = List; }); // module: reporters/list.js require.register("reporters/markdown.js", function(module, exports, require){ /** * Module dependencies. */ var Base = require('./base') , utils = require('../utils'); /** * Expose `Markdown`. */ exports = module.exports = Markdown; /** * Initialize a new `Markdown` reporter. * * @param {Runner} runner * @api public */ function Markdown(runner) { Base.call(this, runner); var self = this , stats = this.stats , total = runner.total , level = 0 , buf = ''; function title(str) { return Array(level).join('#') + ' ' + str; } function indent() { return Array(level).join(' '); } function mapTOC(suite, obj) { var ret = obj; obj = obj[suite.title] = obj[suite.title] || { suite: suite }; suite.suites.forEach(function(suite){ mapTOC(suite, obj); }); return ret; } function stringifyTOC(obj, level) { ++level; var buf = ''; var link; for (var key in obj) { if ('suite' == key) continue; if (key) link = ' - [' + key + '](#' + utils.slug(obj[key].suite.fullTitle()) + ')\n'; if (key) buf += Array(level).join(' ') + link; buf += stringifyTOC(obj[key], level); } --level; return buf; } function generateTOC(suite) { var obj = mapTOC(suite, {}); return stringifyTOC(obj, 0); } generateTOC(runner.suite); runner.on('suite', function(suite){ ++level; var slug = utils.slug(suite.fullTitle()); buf += '' + '\n'; buf += title(suite.title) + '\n'; }); runner.on('suite end', function(suite){ --level; }); runner.on('pass', function(test){ var code = utils.clean(test.fn.toString()); buf += test.title + '.\n'; buf += '\n```js\n'; buf += code + '\n'; buf += '```\n\n'; }); runner.on('end', function(){ process.stdout.write('# TOC\n'); process.stdout.write(generateTOC(runner.suite)); process.stdout.write(buf); }); } }); // module: reporters/markdown.js require.register("reporters/min.js", function(module, exports, require){ /** * Module dependencies. */ var Base = require('./base'); /** * Expose `Min`. */ exports = module.exports = Min; /** * Initialize a new `Min` minimal test reporter (best used with --watch). * * @param {Runner} runner * @api public */ function Min(runner) { Base.call(this, runner); runner.on('start', function(){ // clear screen process.stdout.write('\u001b[2J'); // set cursor position process.stdout.write('\u001b[1;3H'); }); runner.on('end', this.epilogue.bind(this)); } /** * Inherit from `Base.prototype`. */ Min.prototype = new Base; Min.prototype.constructor = Min; }); // module: reporters/min.js require.register("reporters/nyan.js", function(module, exports, require){ /** * Module dependencies. */ var Base = require('./base') , color = Base.color; /** * Expose `Dot`. */ exports = module.exports = NyanCat; /** * Initialize a new `Dot` matrix test reporter. * * @param {Runner} runner * @api public */ function NyanCat(runner) { Base.call(this, runner); var self = this , stats = this.stats , width = Base.window.width * .75 | 0 , rainbowColors = this.rainbowColors = self.generateColors() , colorIndex = this.colorIndex = 0 , numerOfLines = this.numberOfLines = 4 , trajectories = this.trajectories = [[], [], [], []] , nyanCatWidth = this.nyanCatWidth = 11 , trajectoryWidthMax = this.trajectoryWidthMax = (width - nyanCatWidth) , scoreboardWidth = this.scoreboardWidth = 5 , tick = this.tick = 0 , n = 0; runner.on('start', function(){ Base.cursor.hide(); self.draw('start'); }); runner.on('pending', function(test){ self.draw('pending'); }); runner.on('pass', function(test){ self.draw('pass'); }); runner.on('fail', function(test, err){ self.draw('fail'); }); runner.on('end', function(){ Base.cursor.show(); for (var i = 0; i < self.numberOfLines; i++) write('\n'); self.epilogue(); }); } /** * Draw the nyan cat with runner `status`. * * @param {String} status * @api private */ NyanCat.prototype.draw = function(status){ this.appendRainbow(); this.drawScoreboard(); this.drawRainbow(); this.drawNyanCat(status); this.tick = !this.tick; }; /** * Draw the "scoreboard" showing the number * of passes, failures and pending tests. * * @api private */ NyanCat.prototype.drawScoreboard = function(){ var stats = this.stats; var colors = Base.colors; function draw(color, n) { write(' '); write('\u001b[' + color + 'm' + n + '\u001b[0m'); write('\n'); } draw(colors.green, stats.passes); draw(colors.fail, stats.failures); draw(colors.pending, stats.pending); write('\n'); this.cursorUp(this.numberOfLines); }; /** * Append the rainbow. * * @api private */ NyanCat.prototype.appendRainbow = function(){ var segment = this.tick ? '_' : '-'; var rainbowified = this.rainbowify(segment); for (var index = 0; index < this.numberOfLines; index++) { var trajectory = this.trajectories[index]; if (trajectory.length >= this.trajectoryWidthMax) trajectory.shift(); trajectory.push(rainbowified); } }; /** * Draw the rainbow. * * @api private */ NyanCat.prototype.drawRainbow = function(){ var self = this; this.trajectories.forEach(function(line, index) { write('\u001b[' + self.scoreboardWidth + 'C'); write(line.join('')); write('\n'); }); this.cursorUp(this.numberOfLines); }; /** * Draw the nyan cat with `status`. * * @param {String} status * @api private */ NyanCat.prototype.drawNyanCat = function(status) { var self = this; var startWidth = this.scoreboardWidth + this.trajectories[0].length; [0, 1, 2, 3].forEach(function(index) { write('\u001b[' + startWidth + 'C'); switch (index) { case 0: write('_,------,'); write('\n'); break; case 1: var padding = self.tick ? ' ' : ' '; write('_|' + padding + '/\\_/\\ '); write('\n'); break; case 2: var padding = self.tick ? '_' : '__'; var tail = self.tick ? '~' : '^'; var face; switch (status) { case 'pass': face = '( ^ .^)'; break; case 'fail': face = '( o .o)'; break; default: face = '( - .-)'; } write(tail + '|' + padding + face + ' '); write('\n'); break; case 3: var padding = self.tick ? ' ' : ' '; write(padding + '"" "" '); write('\n'); break; } }); this.cursorUp(this.numberOfLines); }; /** * Move cursor up `n`. * * @param {Number} n * @api private */ NyanCat.prototype.cursorUp = function(n) { write('\u001b[' + n + 'A'); }; /** * Move cursor down `n`. * * @param {Number} n * @api private */ NyanCat.prototype.cursorDown = function(n) { write('\u001b[' + n + 'B'); }; /** * Generate rainbow colors. * * @return {Array} * @api private */ NyanCat.prototype.generateColors = function(){ var colors = []; for (var i = 0; i < (6 * 7); i++) { var pi3 = Math.floor(Math.PI / 3); var n = (i * (1.0 / 6)); var r = Math.floor(3 * Math.sin(n) + 3); var g = Math.floor(3 * Math.sin(n + 2 * pi3) + 3); var b = Math.floor(3 * Math.sin(n + 4 * pi3) + 3); colors.push(36 * r + 6 * g + b + 16); } return colors; }; /** * Apply rainbow to the given `str`. * * @param {String} str * @return {String} * @api private */ NyanCat.prototype.rainbowify = function(str){ var color = this.rainbowColors[this.colorIndex % this.rainbowColors.length]; this.colorIndex += 1; return '\u001b[38;5;' + color + 'm' + str + '\u001b[0m'; }; /** * Stdout helper. */ function write(string) { process.stdout.write(string); } /** * Inherit from `Base.prototype`. */ NyanCat.prototype = new Base; NyanCat.prototype.constructor = NyanCat; }); // module: reporters/nyan.js require.register("reporters/progress.js", function(module, exports, require){ /** * Module dependencies. */ var Base = require('./base') , cursor = Base.cursor , color = Base.color; /** * Expose `Progress`. */ exports = module.exports = Progress; /** * General progress bar color. */ Base.colors.progress = 90; /** * Initialize a new `Progress` bar test reporter. * * @param {Runner} runner * @param {Object} options * @api public */ function Progress(runner, options) { Base.call(this, runner); var self = this , options = options || {} , stats = this.stats , width = Base.window.width * .50 | 0 , total = runner.total , complete = 0 , max = Math.max; // default chars options.open = options.open || '['; options.complete = options.complete || '▬'; options.incomplete = options.incomplete || '⋅'; options.close = options.close || ']'; options.verbose = false; // tests started runner.on('start', function(){ console.log(); cursor.hide(); }); // tests complete runner.on('test end', function(){ complete++; var incomplete = total - complete , percent = complete / total , n = width * percent | 0 , i = width - n; cursor.CR(); process.stdout.write('\u001b[J'); process.stdout.write(color('progress', ' ' + options.open)); process.stdout.write(Array(n).join(options.complete)); process.stdout.write(Array(i).join(options.incomplete)); process.stdout.write(color('progress', options.close)); if (options.verbose) { process.stdout.write(color('progress', ' ' + complete + ' of ' + total)); } }); // tests are complete, output some stats // and the failures if any runner.on('end', function(){ cursor.show(); console.log(); self.epilogue(); }); } /** * Inherit from `Base.prototype`. */ Progress.prototype = new Base; Progress.prototype.constructor = Progress; }); // module: reporters/progress.js require.register("reporters/spec.js", function(module, exports, require){ /** * Module dependencies. */ var Base = require('./base') , cursor = Base.cursor , color = Base.color; /** * Expose `Spec`. */ exports = module.exports = Spec; /** * Initialize a new `Spec` test reporter. * * @param {Runner} runner * @api public */ function Spec(runner) { Base.call(this, runner); var self = this , stats = this.stats , indents = 0 , n = 0; function indent() { return Array(indents).join(' ') } runner.on('start', function(){ console.log(); }); runner.on('suite', function(suite){ ++indents; console.log(color('suite', '%s%s'), indent(), suite.title); }); runner.on('suite end', function(suite){ --indents; if (1 == indents) console.log(); }); runner.on('test', function(test){ process.stdout.write(indent() + color('pass', ' ◦ ' + test.title + ': ')); }); runner.on('pending', function(test){ var fmt = indent() + color('pending', ' - %s'); console.log(fmt, test.title); }); runner.on('pass', function(test){ if ('fast' == test.speed) { var fmt = indent() + color('checkmark', ' ✓') + color('pass', ' %s '); cursor.CR(); console.log(fmt, test.title); } else { var fmt = indent() + color('checkmark', ' ✓') + color('pass', ' %s ') + color(test.speed, '(%dms)'); cursor.CR(); console.log(fmt, test.title, test.duration); } }); runner.on('fail', function(test, err){ cursor.CR(); console.log(indent() + color('fail', ' %d) %s'), ++n, test.title); }); runner.on('end', self.epilogue.bind(self)); } /** * Inherit from `Base.prototype`. */ Spec.prototype = new Base; Spec.prototype.constructor = Spec; }); // module: reporters/spec.js require.register("reporters/tap.js", function(module, exports, require){ /** * Module dependencies. */ var Base = require('./base') , cursor = Base.cursor , color = Base.color; /** * Expose `TAP`. */ exports = module.exports = TAP; /** * Initialize a new `TAP` reporter. * * @param {Runner} runner * @api public */ function TAP(runner) { Base.call(this, runner); var self = this , stats = this.stats , n = 1; runner.on('start', function(){ var total = runner.grepTotal(runner.suite); console.log('%d..%d', 1, total); }); runner.on('test end', function(){ ++n; }); runner.on('pending', function(test){ console.log('ok %d %s # SKIP -', n, title(test)); }); runner.on('pass', function(test){ console.log('ok %d %s', n, title(test)); }); runner.on('fail', function(test, err){ console.log('not ok %d %s', n, title(test)); console.log(err.stack.replace(/^/gm, ' ')); }); } /** * Return a TAP-safe title of `test` * * @param {Object} test * @return {String} * @api private */ function title(test) { return test.fullTitle().replace(/#/g, ''); } }); // module: reporters/tap.js require.register("reporters/teamcity.js", function(module, exports, require){ /** * Module dependencies. */ var Base = require('./base'); /** * Expose `Teamcity`. */ exports = module.exports = Teamcity; /** * Initialize a new `Teamcity` reporter. * * @param {Runner} runner * @api public */ function Teamcity(runner) { Base.call(this, runner); var stats = this.stats; runner.on('start', function() { console.log("##teamcity[testSuiteStarted name='mocha.suite']"); }); runner.on('test', function(test) { console.log("##teamcity[testStarted name='" + escape(test.fullTitle()) + "']"); }); runner.on('fail', function(test, err) { console.log("##teamcity[testFailed name='" + escape(test.fullTitle()) + "' message='" + escape(err.message) + "']"); }); runner.on('pending', function(test) { console.log("##teamcity[testIgnored name='" + escape(test.fullTitle()) + "' message='pending']"); }); runner.on('test end', function(test) { console.log("##teamcity[testFinished name='" + escape(test.fullTitle()) + "' duration='" + test.duration + "']"); }); runner.on('end', function() { console.log("##teamcity[testSuiteFinished name='mocha.suite' duration='" + stats.duration + "']"); }); } /** * Escape the given `str`. */ function escape(str) { return str .replace(/\|/g, "||") .replace(/\n/g, "|n") .replace(/\r/g, "|r") .replace(/\[/g, "|[") .replace(/\]/g, "|]") .replace(/\u0085/g, "|x") .replace(/\u2028/g, "|l") .replace(/\u2029/g, "|p") .replace(/'/g, "|'"); } }); // module: reporters/teamcity.js require.register("reporters/xunit.js", function(module, exports, require){ /** * Module dependencies. */ var Base = require('./base') , utils = require('../utils') , escape = utils.escape; /** * Save timer references to avoid Sinon interfering (see GH-237). */ var Date = global.Date , setTimeout = global.setTimeout , setInterval = global.setInterval , clearTimeout = global.clearTimeout , clearInterval = global.clearInterval; /** * Expose `XUnit`. */ exports = module.exports = XUnit; /** * Initialize a new `XUnit` reporter. * * @param {Runner} runner * @api public */ function XUnit(runner) { Base.call(this, runner); var stats = this.stats , tests = [] , self = this; runner.on('pass', function(test){ tests.push(test); }); runner.on('fail', function(test){ tests.push(test); }); runner.on('end', function(){ console.log(tag('testsuite', { name: 'Mocha Tests' , tests: stats.tests , failures: stats.failures , errors: stats.failures , skip: stats.tests - stats.failures - stats.passes , timestamp: (new Date).toUTCString() , time: stats.duration / 1000 }, false)); tests.forEach(test); console.log(''); }); } /** * Inherit from `Base.prototype`. */ XUnit.prototype = new Base; XUnit.prototype.constructor = XUnit; /** * Output tag for the given `test.` */ function test(test) { var attrs = { classname: test.parent.fullTitle() , name: test.title , time: test.duration / 1000 }; if ('failed' == test.state) { var err = test.err; attrs.message = escape(err.message); console.log(tag('testcase', attrs, false, tag('failure', attrs, false, cdata(err.stack)))); } else if (test.pending) { console.log(tag('testcase', attrs, false, tag('skipped', {}, true))); } else { console.log(tag('testcase', attrs, true) ); } } /** * HTML tag helper. */ function tag(name, attrs, close, content) { var end = close ? '/>' : '>' , pairs = [] , tag; for (var key in attrs) { pairs.push(key + '="' + escape(attrs[key]) + '"'); } tag = '<' + name + (pairs.length ? ' ' + pairs.join(' ') : '') + end; if (content) tag += content + ''; } }); // module: reporters/xunit.js require.register("runnable.js", function(module, exports, require){ /** * Module dependencies. */ var EventEmitter = require('browser/events').EventEmitter , debug = require('browser/debug')('mocha:runnable'); /** * Save timer references to avoid Sinon interfering (see GH-237). */ var Date = global.Date , setTimeout = global.setTimeout , setInterval = global.setInterval , clearTimeout = global.clearTimeout , clearInterval = global.clearInterval; /** * Expose `Runnable`. */ module.exports = Runnable; /** * Initialize a new `Runnable` with the given `title` and callback `fn`. * * @param {String} title * @param {Function} fn * @api private */ function Runnable(title, fn) { this.title = title; this.fn = fn; this.async = fn && fn.length; this.sync = ! this.async; this._timeout = 2000; this._slow = 75; this.timedOut = false; } /** * Inherit from `EventEmitter.prototype`. */ Runnable.prototype = new EventEmitter; Runnable.prototype.constructor = Runnable; /** * Set & get timeout `ms`. * * @param {Number} ms * @return {Runnable|Number} ms or self * @api private */ Runnable.prototype.timeout = function(ms){ if (0 == arguments.length) return this._timeout; debug('timeout %d', ms); this._timeout = ms; if (this.timer) this.resetTimeout(); return this; }; /** * Set & get slow `ms`. * * @param {Number} ms * @return {Runnable|Number} ms or self * @api private */ Runnable.prototype.slow = function(ms){ if (0 === arguments.length) return this._slow; debug('timeout %d', ms); this._slow = ms; return this; }; /** * Return the full title generated by recursively * concatenating the parent's full title. * * @return {String} * @api public */ Runnable.prototype.fullTitle = function(){ return this.parent.fullTitle() + ' ' + this.title; }; /** * Clear the timeout. * * @api private */ Runnable.prototype.clearTimeout = function(){ clearTimeout(this.timer); }; /** * Inspect the runnable void of private properties. * * @return {String} * @api private */ Runnable.prototype.inspect = function(){ return JSON.stringify(this, function(key, val){ if ('_' == key[0]) return; if ('parent' == key) return '#'; if ('ctx' == key) return '#'; return val; }, 2); }; /** * Reset the timeout. * * @api private */ Runnable.prototype.resetTimeout = function(){ var self = this , ms = this.timeout(); this.clearTimeout(); if (ms) { this.timer = setTimeout(function(){ self.callback(new Error('timeout of ' + ms + 'ms exceeded')); self.timedOut = true; }, ms); } }; /** * Run the test and invoke `fn(err)`. * * @param {Function} fn * @api private */ Runnable.prototype.run = function(fn){ var self = this , ms = this.timeout() , start = new Date , ctx = this.ctx , finished , emitted; if (ctx) ctx.runnable(this); // timeout if (this.async) { if (ms) { this.timer = setTimeout(function(){ done(new Error('timeout of ' + ms + 'ms exceeded')); self.timedOut = true; }, ms); } } // called multiple times function multiple(err) { if (emitted) return; emitted = true; self.emit('error', err || new Error('done() called multiple times')); } // finished function done(err) { if (self.timedOut) return; if (finished) return multiple(err); self.clearTimeout(); self.duration = new Date - start; finished = true; fn(err); } // for .resetTimeout() this.callback = done; // async if (this.async) { try { this.fn.call(ctx, function(err){ if (err instanceof Error) return done(err); if (null != err) return done(new Error('done() invoked with non-Error: ' + err)); done(); }); } catch (err) { done(err); } return; } // sync try { if (!this.pending) this.fn.call(ctx); this.duration = new Date - start; fn(); } catch (err) { fn(err); } }; }); // module: runnable.js require.register("runner.js", function(module, exports, require){ /** * Module dependencies. */ var EventEmitter = require('browser/events').EventEmitter , debug = require('browser/debug')('mocha:runner') , Test = require('./test') , utils = require('./utils') , filter = utils.filter , keys = utils.keys , noop = function(){}; /** * Expose `Runner`. */ module.exports = Runner; /** * Initialize a `Runner` for the given `suite`. * * Events: * * - `start` execution started * - `end` execution complete * - `suite` (suite) test suite execution started * - `suite end` (suite) all tests (and sub-suites) have finished * - `test` (test) test execution started * - `test end` (test) test completed * - `hook` (hook) hook execution started * - `hook end` (hook) hook complete * - `pass` (test) test passed * - `fail` (test, err) test failed * * @api public */ function Runner(suite) { var self = this; this._globals = []; this.suite = suite; this.total = suite.total(); this.failures = 0; this.on('test end', function(test){ self.checkGlobals(test); }); this.on('hook end', function(hook){ self.checkGlobals(hook); }); this.grep(/.*/); this.globals(utils.keys(global).concat(['errno'])); } /** * Inherit from `EventEmitter.prototype`. */ Runner.prototype = new EventEmitter; Runner.prototype.constructor = Runner; /** * Run tests with full titles matching `re`. Updates runner.total * with number of tests matched. * * @param {RegExp} re * @param {Boolean} invert * @return {Runner} for chaining * @api public */ Runner.prototype.grep = function(re, invert){ debug('grep %s', re); this._grep = re; this._invert = invert; this.total = this.grepTotal(this.suite); return this; }; /** * Returns the number of tests matching the grep search for the * given suite. * * @param {Suite} suite * @return {Number} * @api public */ Runner.prototype.grepTotal = function(suite) { var self = this; var total = 0; suite.eachTest(function(test){ var match = self._grep.test(test.fullTitle()); if (self._invert) match = !match; if (match) total++; }); return total; }; /** * Allow the given `arr` of globals. * * @param {Array} arr * @return {Runner} for chaining * @api public */ Runner.prototype.globals = function(arr){ if (0 == arguments.length) return this._globals; debug('globals %j', arr); utils.forEach(arr, function(arr){ this._globals.push(arr); }, this); return this; }; /** * Check for global variable leaks. * * @api private */ Runner.prototype.checkGlobals = function(test){ if (this.ignoreLeaks) return; var leaks = filterLeaks(this._globals); this._globals = this._globals.concat(leaks); if (leaks.length > 1) { this.fail(test, new Error('global leaks detected: ' + leaks.join(', ') + '')); } else if (leaks.length) { this.fail(test, new Error('global leak detected: ' + leaks[0])); } }; /** * Fail the given `test`. * * @param {Test} test * @param {Error} err * @api private */ Runner.prototype.fail = function(test, err){ ++this.failures; test.state = 'failed'; if ('string' == typeof err) { err = new Error('the string "' + err + '" was thrown, throw an Error :)'); } this.emit('fail', test, err); }; /** * Fail the given `hook` with `err`. * * Hook failures (currently) hard-end due * to that fact that a failing hook will * surely cause subsequent tests to fail, * causing jumbled reporting. * * @param {Hook} hook * @param {Error} err * @api private */ Runner.prototype.failHook = function(hook, err){ this.fail(hook, err); this.emit('end'); }; /** * Run hook `name` callbacks and then invoke `fn()`. * * @param {String} name * @param {Function} function * @api private */ Runner.prototype.hook = function(name, fn){ var suite = this.suite , hooks = suite['_' + name] , self = this , timer; function next(i) { var hook = hooks[i]; if (!hook) return fn(); self.currentRunnable = hook; self.emit('hook', hook); hook.on('error', function(err){ self.failHook(hook, err); }); hook.run(function(err){ hook.removeAllListeners('error'); var testError = hook.error(); if (testError) self.fail(self.test, testError); if (err) return self.failHook(hook, err); self.emit('hook end', hook); next(++i); }); } process.nextTick(function(){ next(0); }); }; /** * Run hook `name` for the given array of `suites` * in order, and callback `fn(err)`. * * @param {String} name * @param {Array} suites * @param {Function} fn * @api private */ Runner.prototype.hooks = function(name, suites, fn){ var self = this , orig = this.suite; function next(suite) { self.suite = suite; if (!suite) { self.suite = orig; return fn(); } self.hook(name, function(err){ if (err) { self.suite = orig; return fn(err); } next(suites.pop()); }); } next(suites.pop()); }; /** * Run hooks from the top level down. * * @param {String} name * @param {Function} fn * @api private */ Runner.prototype.hookUp = function(name, fn){ var suites = [this.suite].concat(this.parents()).reverse(); this.hooks(name, suites, fn); }; /** * Run hooks from the bottom up. * * @param {String} name * @param {Function} fn * @api private */ Runner.prototype.hookDown = function(name, fn){ var suites = [this.suite].concat(this.parents()); this.hooks(name, suites, fn); }; /** * Return an array of parent Suites from * closest to furthest. * * @return {Array} * @api private */ Runner.prototype.parents = function(){ var suite = this.suite , suites = []; while (suite = suite.parent) suites.push(suite); return suites; }; /** * Run the current test and callback `fn(err)`. * * @param {Function} fn * @api private */ Runner.prototype.runTest = function(fn){ var test = this.test , self = this; try { test.on('error', function(err){ self.fail(test, err); }); test.run(fn); } catch (err) { fn(err); } }; /** * Run tests in the given `suite` and invoke * the callback `fn()` when complete. * * @param {Suite} suite * @param {Function} fn * @api private */ Runner.prototype.runTests = function(suite, fn){ var self = this , tests = suite.tests , test; function next(err) { // if we bail after first err if (self.failures && suite._bail) return fn(); // next test test = tests.shift(); // all done if (!test) return fn(); // grep var match = self._grep.test(test.fullTitle()); if (self._invert) match = !match; if (!match) return next(); // pending if (test.pending) { self.emit('pending', test); self.emit('test end', test); return next(); } // execute test and hook(s) self.emit('test', self.test = test); self.hookDown('beforeEach', function(){ self.currentRunnable = self.test; self.runTest(function(err){ test = self.test; if (err) { self.fail(test, err); self.emit('test end', test); return self.hookUp('afterEach', next); } test.state = 'passed'; self.emit('pass', test); self.emit('test end', test); self.hookUp('afterEach', next); }); }); } this.next = next; next(); }; /** * Run the given `suite` and invoke the * callback `fn()` when complete. * * @param {Suite} suite * @param {Function} fn * @api private */ Runner.prototype.runSuite = function(suite, fn){ var total = this.grepTotal(suite) , self = this , i = 0; debug('run suite %s', suite.fullTitle()); if (!total) return fn(); this.emit('suite', this.suite = suite); function next() { var curr = suite.suites[i++]; if (!curr) return done(); self.runSuite(curr, next); } function done() { self.suite = suite; self.hook('afterAll', function(){ self.emit('suite end', suite); fn(); }); } this.hook('beforeAll', function(){ self.runTests(suite, next); }); }; /** * Handle uncaught exceptions. * * @param {Error} err * @api private */ Runner.prototype.uncaught = function(err){ debug('uncaught exception %s', err.message); var runnable = this.currentRunnable; if (!runnable || 'failed' == runnable.state) return; runnable.clearTimeout(); err.uncaught = true; this.fail(runnable, err); // recover from test if ('test' == runnable.type) { this.emit('test end', runnable); this.hookUp('afterEach', this.next); return; } // bail on hooks this.emit('end'); }; /** * Run the root suite and invoke `fn(failures)` * on completion. * * @param {Function} fn * @return {Runner} for chaining * @api public */ Runner.prototype.run = function(fn){ var self = this , fn = fn || function(){}; debug('start'); // uncaught callback function uncaught(err) { self.uncaught(err); } // callback this.on('end', function(){ debug('end'); process.removeListener('uncaughtException', uncaught); fn(self.failures); }); // run suites this.emit('start'); this.runSuite(this.suite, function(){ debug('finished running'); self.emit('end'); }); // uncaught exception process.on('uncaughtException', uncaught); return this; }; /** * Filter leaks with the given globals flagged as `ok`. * * @param {Array} ok * @return {Array} * @api private */ function filterLeaks(ok) { return filter(keys(global), function(key){ var matched = filter(ok, function(ok){ if (~ok.indexOf('*')) return 0 == key.indexOf(ok.split('*')[0]); return key == ok; }); return matched.length == 0 && (!global.navigator || 'onerror' !== key); }); } }); // module: runner.js require.register("suite.js", function(module, exports, require){ /** * Module dependencies. */ var EventEmitter = require('browser/events').EventEmitter , debug = require('browser/debug')('mocha:suite') , milliseconds = require('./ms') , utils = require('./utils') , Hook = require('./hook'); /** * Expose `Suite`. */ exports = module.exports = Suite; /** * Create a new `Suite` with the given `title` * and parent `Suite`. When a suite with the * same title is already present, that suite * is returned to provide nicer reporter * and more flexible meta-testing. * * @param {Suite} parent * @param {String} title * @return {Suite} * @api public */ exports.create = function(parent, title){ var suite = new Suite(title, parent.ctx); suite.parent = parent; if (parent.pending) suite.pending = true; title = suite.fullTitle(); parent.addSuite(suite); return suite; }; /** * Initialize a new `Suite` with the given * `title` and `ctx`. * * @param {String} title * @param {Context} ctx * @api private */ function Suite(title, ctx) { this.title = title; this.ctx = ctx; this.suites = []; this.tests = []; this.pending = false; this._beforeEach = []; this._beforeAll = []; this._afterEach = []; this._afterAll = []; this.root = !title; this._timeout = 2000; this._slow = 75; this._bail = false; } /** * Inherit from `EventEmitter.prototype`. */ Suite.prototype = new EventEmitter; Suite.prototype.constructor = Suite; /** * Return a clone of this `Suite`. * * @return {Suite} * @api private */ Suite.prototype.clone = function(){ var suite = new Suite(this.title); debug('clone'); suite.ctx = this.ctx; suite.timeout(this.timeout()); suite.slow(this.slow()); suite.bail(this.bail()); return suite; }; /** * Set timeout `ms` or short-hand such as "2s". * * @param {Number|String} ms * @return {Suite|Number} for chaining * @api private */ Suite.prototype.timeout = function(ms){ if (0 == arguments.length) return this._timeout; if ('string' == typeof ms) ms = milliseconds(ms); debug('timeout %d', ms); this._timeout = parseInt(ms, 10); return this; }; /** * Set slow `ms` or short-hand such as "2s". * * @param {Number|String} ms * @return {Suite|Number} for chaining * @api private */ Suite.prototype.slow = function(ms){ if (0 === arguments.length) return this._slow; if ('string' == typeof ms) ms = milliseconds(ms); debug('slow %d', ms); this._slow = ms; return this; }; /** * Sets whether to bail after first error. * * @parma {Boolean} bail * @return {Suite|Number} for chaining * @api private */ Suite.prototype.bail = function(bail){ if (0 == arguments.length) return this._bail; debug('bail %s', bail); this._bail = bail; return this; }; /** * Run `fn(test[, done])` before running tests. * * @param {Function} fn * @return {Suite} for chaining * @api private */ Suite.prototype.beforeAll = function(fn){ if (this.pending) return this; var hook = new Hook('"before all" hook', fn); hook.parent = this; hook.timeout(this.timeout()); hook.slow(this.slow()); hook.ctx = this.ctx; this._beforeAll.push(hook); this.emit('beforeAll', hook); return this; }; /** * Run `fn(test[, done])` after running tests. * * @param {Function} fn * @return {Suite} for chaining * @api private */ Suite.prototype.afterAll = function(fn){ if (this.pending) return this; var hook = new Hook('"after all" hook', fn); hook.parent = this; hook.timeout(this.timeout()); hook.slow(this.slow()); hook.ctx = this.ctx; this._afterAll.push(hook); this.emit('afterAll', hook); return this; }; /** * Run `fn(test[, done])` before each test case. * * @param {Function} fn * @return {Suite} for chaining * @api private */ Suite.prototype.beforeEach = function(fn){ if (this.pending) return this; var hook = new Hook('"before each" hook', fn); hook.parent = this; hook.timeout(this.timeout()); hook.slow(this.slow()); hook.ctx = this.ctx; this._beforeEach.push(hook); this.emit('beforeEach', hook); return this; }; /** * Run `fn(test[, done])` after each test case. * * @param {Function} fn * @return {Suite} for chaining * @api private */ Suite.prototype.afterEach = function(fn){ if (this.pending) return this; var hook = new Hook('"after each" hook', fn); hook.parent = this; hook.timeout(this.timeout()); hook.slow(this.slow()); hook.ctx = this.ctx; this._afterEach.push(hook); this.emit('afterEach', hook); return this; }; /** * Add a test `suite`. * * @param {Suite} suite * @return {Suite} for chaining * @api private */ Suite.prototype.addSuite = function(suite){ suite.parent = this; suite.timeout(this.timeout()); suite.slow(this.slow()); suite.bail(this.bail()); this.suites.push(suite); this.emit('suite', suite); return this; }; /** * Add a `test` to this suite. * * @param {Test} test * @return {Suite} for chaining * @api private */ Suite.prototype.addTest = function(test){ test.parent = this; test.timeout(this.timeout()); test.slow(this.slow()); test.ctx = this.ctx; this.tests.push(test); this.emit('test', test); return this; }; /** * Return the full title generated by recursively * concatenating the parent's full title. * * @return {String} * @api public */ Suite.prototype.fullTitle = function(){ if (this.parent) { var full = this.parent.fullTitle(); if (full) return full + ' ' + this.title; } return this.title; }; /** * Return the total number of tests. * * @return {Number} * @api public */ Suite.prototype.total = function(){ return utils.reduce(this.suites, function(sum, suite){ return sum + suite.total(); }, 0) + this.tests.length; }; /** * Iterates through each suite recursively to find * all tests. Applies a function in the format * `fn(test)`. * * @param {Function} fn * @return {Suite} * @api private */ Suite.prototype.eachTest = function(fn){ utils.forEach(this.tests, fn); utils.forEach(this.suites, function(suite){ suite.eachTest(fn); }); return this; }; }); // module: suite.js require.register("test.js", function(module, exports, require){ /** * Module dependencies. */ var Runnable = require('./runnable'); /** * Expose `Test`. */ module.exports = Test; /** * Initialize a new `Test` with the given `title` and callback `fn`. * * @param {String} title * @param {Function} fn * @api private */ function Test(title, fn) { Runnable.call(this, title, fn); this.pending = !fn; this.type = 'test'; } /** * Inherit from `Runnable.prototype`. */ Test.prototype = new Runnable; Test.prototype.constructor = Test; }); // module: test.js require.register("utils.js", function(module, exports, require){ /** * Module dependencies. */ var fs = require('browser/fs') , path = require('browser/path') , join = path.join , debug = require('browser/debug')('mocha:watch'); /** * Ignored directories. */ var ignore = ['node_modules', '.git']; /** * Escape special characters in the given string of html. * * @param {String} html * @return {String} * @api private */ exports.escape = function(html){ return String(html) .replace(/&/g, '&') .replace(/"/g, '"') .replace(//g, '>'); }; /** * Array#forEach (<=IE8) * * @param {Array} array * @param {Function} fn * @param {Object} scope * @api private */ exports.forEach = function(arr, fn, scope){ for (var i = 0, l = arr.length; i < l; i++) fn.call(scope, arr[i], i); }; /** * Array#indexOf (<=IE8) * * @parma {Array} arr * @param {Object} obj to find index of * @param {Number} start * @api private */ exports.indexOf = function(arr, obj, start){ for (var i = start || 0, l = arr.length; i < l; i++) { if (arr[i] === obj) return i; } return -1; }; /** * Array#reduce (<=IE8) * * @param {Array} array * @param {Function} fn * @param {Object} initial value * @api private */ exports.reduce = function(arr, fn, val){ var rval = val; for (var i = 0, l = arr.length; i < l; i++) { rval = fn(rval, arr[i], i, arr); } return rval; }; /** * Array#filter (<=IE8) * * @param {Array} array * @param {Function} fn * @api private */ exports.filter = function(arr, fn){ var ret = []; for (var i = 0, l = arr.length; i < l; i++) { var val = arr[i]; if (fn(val, i, arr)) ret.push(val); } return ret; }; /** * Object.keys (<=IE8) * * @param {Object} obj * @return {Array} keys * @api private */ exports.keys = Object.keys || function(obj) { var keys = [] , has = Object.prototype.hasOwnProperty // for `window` on <=IE8 for (var key in obj) { if (has.call(obj, key)) { keys.push(key); } } return keys; }; /** * Watch the given `files` for changes * and invoke `fn(file)` on modification. * * @param {Array} files * @param {Function} fn * @api private */ exports.watch = function(files, fn){ var options = { interval: 100 }; files.forEach(function(file){ debug('file %s', file); fs.watchFile(file, options, function(curr, prev){ if (prev.mtime < curr.mtime) fn(file); }); }); }; /** * Ignored files. */ function ignored(path){ return !~ignore.indexOf(path); } /** * Lookup files in the given `dir`. * * @return {Array} * @api private */ exports.files = function(dir, ret){ ret = ret || []; fs.readdirSync(dir) .filter(ignored) .forEach(function(path){ path = join(dir, path); if (fs.statSync(path).isDirectory()) { exports.files(path, ret); } else if (path.match(/\.(js|coffee)$/)) { ret.push(path); } }); return ret; }; /** * Compute a slug from the given `str`. * * @param {String} str * @return {String} * @api private */ exports.slug = function(str){ return str .toLowerCase() .replace(/ +/g, '-') .replace(/[^-\w]/g, ''); }; /** * Strip the function definition from `str`, * and re-indent for pre whitespace. */ exports.clean = function(str) { str = str .replace(/^function *\(.*\) *{/, '') .replace(/\s+\}$/, ''); var spaces = str.match(/^\n?( *)/)[1].length , re = new RegExp('^ {' + spaces + '}', 'gm'); str = str.replace(re, ''); return exports.trim(str); }; /** * Escape regular expression characters in `str`. * * @param {String} str * @return {String} * @api private */ exports.escapeRegexp = function(str){ return str.replace(/[-\\^$*+?.()|[\]{}]/g, "\\$&"); }; /** * Trim the given `str`. * * @param {String} str * @return {String} * @api private */ exports.trim = function(str){ return str.replace(/^\s+|\s+$/g, ''); }; /** * Parse the given `qs`. * * @param {String} qs * @return {Object} * @api private */ exports.parseQuery = function(qs){ return exports.reduce(qs.replace('?', '').split('&'), function(obj, pair){ var i = pair.indexOf('=') , key = pair.slice(0, i) , val = pair.slice(++i); obj[key] = decodeURIComponent(val); return obj; }, {}); }; /** * Highlight the given string of `js`. * * @param {String} js * @return {String} * @api private */ function highlight(js) { return js .replace(//g, '>') .replace(/\/\/(.*)/gm, '//$1') .replace(/('.*?')/gm, '$1') .replace(/(\d+\.\d+)/gm, '$1') .replace(/(\d+)/gm, '$1') .replace(/\bnew *(\w+)/gm, 'new $1') .replace(/\b(function|new|throw|return|var|if|else)\b/gm, '$1') } /** * Highlight the contents of tag `name`. * * @param {String} name * @api private */ exports.highlightTags = function(name) { var code = document.getElementsByTagName(name); for (var i = 0, len = code.length; i < len; ++i) { code[i].innerHTML = highlight(code[i].innerHTML); } }; }); // module: utils.js /** * Node shims. * * These are meant only to allow * mocha.js to run untouched, not * to allow running node code in * the browser. */ process = {}; process.exit = function(status){}; process.stdout = {}; global = window; /** * next tick implementation. */ process.nextTick = (function(){ // postMessage behaves badly on IE8 if (window.ActiveXObject || !window.postMessage) { return function(fn){ fn() }; } // based on setZeroTimeout by David Baron // - http://dbaron.org/log/20100309-faster-timeouts var timeouts = [] , name = 'mocha-zero-timeout' window.addEventListener('message', function(e){ if (e.source == window && e.data == name) { if (e.stopPropagation) e.stopPropagation(); if (timeouts.length) timeouts.shift()(); } }, true); return function(fn){ timeouts.push(fn); window.postMessage(name, '*'); } })(); /** * Remove uncaughtException listener. */ process.removeListener = function(e){ if ('uncaughtException' == e) { window.onerror = null; } }; /** * Implements uncaughtException listener. */ process.on = function(e, fn){ if ('uncaughtException' == e) { window.onerror = fn; } }; // boot ;(function(){ /** * Expose mocha. */ var Mocha = window.Mocha = require('mocha'), mocha = window.mocha = new Mocha({ reporter: 'html' }); /** * Override ui to ensure that the ui functions are initialized. * Normally this would happen in Mocha.prototype.loadFiles. */ mocha.ui = function(ui){ Mocha.prototype.ui.call(this, ui); this.suite.emit('pre-require', window, null, this); return this; }; /** * Setup mocha with the given setting options. */ mocha.setup = function(opts){ if ('string' == typeof opts) opts = { ui: opts }; for (var opt in opts) this[opt](opts[opt]); return this; }; /** * Run mocha, returning the Runner. */ mocha.run = function(fn){ var options = mocha.options; mocha.globals('location'); var query = Mocha.utils.parseQuery(window.location.search || ''); if (query.grep) mocha.grep(query.grep); return Mocha.prototype.run.call(mocha, function(){ Mocha.utils.highlightTags('code'); if (fn) fn(); }); }; })(); })(); ================================================ FILE: browser-version/test/nedb-browser.js ================================================ /** * Testing the browser version of NeDB * The goal of these tests is not to be exhaustive, we have the server-side NeDB tests for that * This is more of a sanity check which executes most of the code at least once and checks * it behaves as the server version does */ var assert = chai.assert; /** * Given a docs array and an id, return the document whose id matches, or null if none is found */ function findById (docs, id) { return _.find(docs, function (doc) { return doc._id === id; }) || null; } describe('Basic CRUD functionality', function () { it('Able to create a database object in the browser', function () { var db = new Nedb(); assert.equal(db.inMemoryOnly, true); assert.equal(db.persistence.inMemoryOnly, true); }); it('Insertion and querying', function (done) { var db = new Nedb(); db.insert({ a: 4 }, function (err, newDoc1) { assert.isNull(err); db.insert({ a: 40 }, function (err, newDoc2) { assert.isNull(err); db.insert({ a: 400 }, function (err, newDoc3) { assert.isNull(err); db.find({ a: { $gt: 36 } }, function (err, docs) { var doc2 = _.find(docs, function (doc) { return doc._id === newDoc2._id; }) , doc3 = _.find(docs, function (doc) { return doc._id === newDoc3._id; }) ; assert.isNull(err); assert.equal(docs.length, 2); assert.equal(doc2.a, 40); assert.equal(doc3.a, 400); db.find({ a: { $lt: 36 } }, function (err, docs) { assert.isNull(err); assert.equal(docs.length, 1); assert.equal(docs[0].a, 4); done(); }); }); }); }); }); }); it('Querying with regular expressions', function (done) { var db = new Nedb(); db.insert({ planet: 'Earth' }, function (err, newDoc1) { assert.isNull(err); db.insert({ planet: 'Mars' }, function (err, newDoc2) { assert.isNull(err); db.insert({ planet: 'Jupiter' }, function (err, newDoc3) { assert.isNull(err); db.insert({ planet: 'Eaaaaaarth' }, function (err, newDoc4) { assert.isNull(err); db.insert({ planet: 'Maaaars' }, function (err, newDoc5) { assert.isNull(err); db.find({ planet: /ar/ }, function (err, docs) { assert.isNull(err); assert.equal(docs.length, 4); assert.equal(_.find(docs, function (doc) { return doc._id === newDoc1._id; }).planet, 'Earth'); assert.equal(_.find(docs, function (doc) { return doc._id === newDoc2._id; }).planet, 'Mars'); assert.equal(_.find(docs, function (doc) { return doc._id === newDoc4._id; }).planet, 'Eaaaaaarth'); assert.equal(_.find(docs, function (doc) { return doc._id === newDoc5._id; }).planet, 'Maaaars'); db.find({ planet: /aa+r/ }, function (err, docs) { assert.isNull(err); assert.equal(docs.length, 2); assert.equal(_.find(docs, function (doc) { return doc._id === newDoc4._id; }).planet, 'Eaaaaaarth'); assert.equal(_.find(docs, function (doc) { return doc._id === newDoc5._id; }).planet, 'Maaaars'); done(); }); }); }); }); }); }); }); }); it('Updating documents', function (done) { var db = new Nedb(); db.insert({ planet: 'Eaaaaarth' }, function (err, newDoc1) { db.insert({ planet: 'Maaaaars' }, function (err, newDoc2) { // Simple update db.update({ _id: newDoc2._id }, { $set: { planet: 'Saturn' } }, {}, function (err, nr) { assert.isNull(err); assert.equal(nr, 1); db.find({}, function (err, docs) { assert.equal(docs.length, 2); assert.equal(findById(docs, newDoc1._id).planet, 'Eaaaaarth'); assert.equal(findById(docs, newDoc2._id).planet, 'Saturn'); // Failing update db.update({ _id: 'unknown' }, { $inc: { count: 1 } }, {}, function (err, nr) { assert.isNull(err); assert.equal(nr, 0); db.find({}, function (err, docs) { assert.equal(docs.length, 2); assert.equal(findById(docs, newDoc1._id).planet, 'Eaaaaarth'); assert.equal(findById(docs, newDoc2._id).planet, 'Saturn'); // Document replacement db.update({ planet: 'Eaaaaarth' }, { planet: 'Uranus' }, { multi: false }, function (err, nr) { assert.isNull(err); assert.equal(nr, 1); db.find({}, function (err, docs) { assert.equal(docs.length, 2); assert.equal(findById(docs, newDoc1._id).planet, 'Uranus'); assert.equal(findById(docs, newDoc2._id).planet, 'Saturn'); // Multi update db.update({}, { $inc: { count: 3 } }, { multi: true }, function (err, nr) { assert.isNull(err); assert.equal(nr, 2); db.find({}, function (err, docs) { assert.equal(docs.length, 2); assert.equal(findById(docs, newDoc1._id).planet, 'Uranus'); assert.equal(findById(docs, newDoc1._id).count, 3); assert.equal(findById(docs, newDoc2._id).planet, 'Saturn'); assert.equal(findById(docs, newDoc2._id).count, 3); done(); }); }); }); }); }); }); }); }); }); }); }); it('Updating documents: special modifiers', function (done) { var db = new Nedb(); db.insert({ planet: 'Earth' }, function (err, newDoc1) { // Pushing to an array db.update({}, { $push: { satellites: 'Phobos' } }, {}, function (err, nr) { assert.isNull(err); assert.equal(nr, 1); db.findOne({}, function (err, doc) { assert.deepEqual(doc, { planet: 'Earth', _id: newDoc1._id, satellites: ['Phobos'] }); db.update({}, { $push: { satellites: 'Deimos' } }, {}, function (err, nr) { assert.isNull(err); assert.equal(nr, 1); db.findOne({}, function (err, doc) { assert.deepEqual(doc, { planet: 'Earth', _id: newDoc1._id, satellites: ['Phobos', 'Deimos'] }); done(); }); }); }); }); }); }); it('Upserts', function (done) { var db = new Nedb(); db.update({ a: 4 }, { $inc: { b: 1 } }, { upsert: true }, function (err, nr, upsert) { assert.isNull(err); // Return upserted document assert.equal(upsert.a, 4); assert.equal(upsert.b, 1); assert.equal(nr, 1); db.find({}, function (err, docs) { assert.equal(docs.length, 1); assert.equal(docs[0].a, 4); assert.equal(docs[0].b, 1); done(); }); }); }); it('Removing documents', function (done) { var db = new Nedb(); db.insert({ a: 2 }); db.insert({ a: 5 }); db.insert({ a: 7 }); // Multi remove db.remove({ a: { $in: [ 5, 7 ] } }, { multi: true }, function (err, nr) { assert.isNull(err); assert.equal(nr, 2); db.find({}, function (err, docs) { assert.equal(docs.length, 1); assert.equal(docs[0].a, 2); // Remove with no match db.remove({ b: { $exists: true } }, { multi: true }, function (err, nr) { assert.isNull(err); assert.equal(nr, 0); db.find({}, function (err, docs) { assert.equal(docs.length, 1); assert.equal(docs[0].a, 2); // Simple remove db.remove({ a: { $exists: true } }, { multi: true }, function (err, nr) { assert.isNull(err); assert.equal(nr, 1); db.find({}, function (err, docs) { assert.equal(docs.length, 0); done(); }); }); }); }); }); }); }); }); // ==== End of 'Basic CRUD functionality' ==== // describe('Indexing', function () { it('getCandidates works as expected', function (done) { var db = new Nedb(); db.insert({ a: 4 }, function () { db.insert({ a: 6 }, function () { db.insert({ a: 7 }, function () { db.getCandidates({ a: 6 }, function (err, candidates) { console.log(candidates); assert.equal(candidates.length, 3); assert.isDefined(_.find(candidates, function (doc) { return doc.a === 4; })); assert.isDefined(_.find(candidates, function (doc) { return doc.a === 6; })); assert.isDefined(_.find(candidates, function (doc) { return doc.a === 7; })); db.ensureIndex({ fieldName: 'a' }); db.getCandidates({ a: 6 }, function (err, candidates) { assert.equal(candidates.length, 1); assert.isDefined(_.find(candidates, function (doc) { return doc.a === 6; })); done(); }); }); }); }); }); }); it('Can use indexes to enforce a unique constraint', function (done) { var db = new Nedb(); db.ensureIndex({ fieldName: 'u', unique: true }); db.insert({ u : 5 }, function (err) { assert.isNull(err); db.insert({ u : 98 }, function (err) { assert.isNull(err); db.insert({ u : 5 }, function (err) { assert.equal(err.errorType, 'uniqueViolated'); done(); }); }); }); }); }); // ==== End of 'Indexing' ==== // describe("Don't forget to launch persistence tests!", function () { it("See file testPersistence.html", function (done) { done(); }); }); // ===== End of 'persistent in-browser database' ===== ================================================ FILE: browser-version/test/playground.html ================================================ Playground for NeDB ================================================ FILE: browser-version/test/testLoad.html ================================================ Test NeDB persistence load in the browser
    ================================================ FILE: browser-version/test/testLoad.js ================================================ console.log('BEGINNING'); var N = 50000 , db = new Nedb({ filename: 'loadTest', autoload: true }) , t, i , sample = JSON.stringify({ data: Math.random(), _id: Math.random() }); ; // Some inserts in sequence, using the default storage mechanism (IndexedDB in my case) function someInserts (sn, N, callback) { var i = 0, beg = Date.now(); async.whilst( function () { return i < N; } , function (_cb) { db.insert({ data: Math.random() }, function (err) { i += 1; return _cb(err); }); } , function (err) { console.log("Inserts, series " + sn + " " + (Date.now() - beg)); return callback(err); }); } // Manually updating the localStorage on the same variable function someLS (sn, N, callback) { var i = 0, beg = Date.now(); for (i = 0; i < N; i += 1) { localStorage.setItem('loadTestLS', getItem('loadTestLS') + sample); } console.log("localStorage, series " + sn + " " + (Date.now() - beg)); return callback(); } // Manually updating the localStorage on different variables function someLSDiff (sn, N, callback) { var i = 0, beg = Date.now(); for (i = 0; i < N; i += 1) { localStorage.setItem('loadTestLS-' + i, sample); } console.log("localStorage, series " + sn + " " + (Date.now() - beg)); return callback(); } // Manually updating the localforage default on the same variable (IndexedDB on my machine) function someLF (sn, N, callback) { var i = 0, beg = Date.now(); async.whilst( function () { return i < N; } , function (_cb) { localforage.getItem('loadTestLF', function (err, value) { if (err) { return _cb(err); } localforage.setItem('loadTestLF', value + sample, function (err) { i += 1; return _cb(err); }); }); } , function (err) { console.log("localForage/IDB, series " + sn + " " + (Date.now() - beg)); return callback(err); }); } // Manually updating the localforage default on the different variables (IndexedDB on my machine) function someLFDiff (sn, N, callback) { var i = 0, beg = Date.now(); async.whilst( function () { return i < N; } , function (_cb) { localforage.setItem('loadTestLF-' + i, sample, function (err) { i += 1; return _cb(err); }); } , function (err) { console.log("localForage/IDB, series " + sn + " " + (Date.now() - beg)); return callback(err); }); } localStorage.setItem('loadTestLS', ''); async.waterfall([ function (cb) { db.remove({}, { multi: true }, function (err) { return cb(err); }); } // Slow and gets slower with database size //, async.apply(someInserts, "#1", N) // N=5000, 141s //, async.apply(someInserts, "#2", N) // N=5000, 208s //, async.apply(someInserts, "#3", N) // N=5000, 281s //, async.apply(someInserts, "#4", N) // N=5000, 350s // Slow and gets slower really fast with database size, then outright crashes //, async.apply(someLS, "#1", N) // N=4000, 2.5s //, async.apply(someLS, "#2", N) // N=4000, 8.0s //, async.apply(someLS, "#3", N) // N=4000, 26.5s //, async.apply(someLS, "#4", N) // N=4000, 47.8s then crash, can't get string (with N=5000 crash happens on second pass) // Much faster and more consistent //, async.apply(someLSDiff, "#1", N) // N=50000, 0.7s //, async.apply(someLSDiff, "#2", N) // N=50000, 0.5s //, async.apply(someLSDiff, "#3", N) // N=50000, 0.5s //, async.apply(someLSDiff, "#4", N) // N=50000, 0.5s // Slow and gets slower with database size //, function (cb) { localforage.setItem('loadTestLF', '', function (err) { return cb(err) }) } //, async.apply(someLF, "#1", N) // N=5000, 69s //, async.apply(someLF, "#2", N) // N=5000, 108s //, async.apply(someLF, "#3", N) // N=5000, 137s //, async.apply(someLF, "#4", N) // N=5000, 169s // Quite fast and speed doesn't change with database size (tested with N=10000 and N=50000, still no slow-down) //, async.apply(someLFDiff, "#1", N) // N=5000, 18s //, async.apply(someLFDiff, "#2", N) // N=5000, 18s //, async.apply(someLFDiff, "#3", N) // N=5000, 18s //, async.apply(someLFDiff, "#4", N) // N=5000, 18s ]); ================================================ FILE: browser-version/test/testPersistence.html ================================================ Test NeDB persistence in the browser
    ================================================ FILE: browser-version/test/testPersistence.js ================================================ console.log("Beginning tests"); console.log("Please note these tests work on Chrome latest, might not work on other browsers due to discrepancies in how local storage works for the file:// protocol"); function testsFailed () { document.getElementById("results").innerHTML = "TESTS FAILED"; } var filename = 'test'; var db = new Nedb({ filename: filename, autoload: true }); db.remove({}, { multi: true }, function () { db.insert({ hello: 'world' }, function (err) { if (err) { testsFailed(); return; } window.location = './testPersistence2.html'; }); }); ================================================ FILE: browser-version/test/testPersistence2.html ================================================ Test NeDB persistence in the browser - Results
    ================================================ FILE: browser-version/test/testPersistence2.js ================================================ // Capture F5 to reload the base page testPersistence.html not this one $(document).on('keydown', function (e) { if (e.keyCode === 116) { e.preventDefault(); window.location = 'testPersistence.html'; } }); console.log("Checking tests results"); console.log("Please note these tests work on Chrome latest, might not work on other browsers due to discrepancies in how local storage works for the file:// protocol"); function testsFailed () { document.getElementById("results").innerHTML = "TESTS FAILED"; } var filename = 'test'; var db = new Nedb({ filename: filename, autoload: true }); db.find({}, function (err, docs) { if (docs.length !== 1) { console.log(docs); console.log("Unexpected length of document database"); return testsFailed(); } if (Object.keys(docs[0]).length !== 2) { console.log("Unexpected length insert document in database"); return testsFailed(); } if (docs[0].hello !== 'world') { console.log("Unexpected document"); return testsFailed(); } document.getElementById("results").innerHTML = "BROWSER PERSISTENCE TEST PASSED"; }); ================================================ FILE: index.js ================================================ var Datastore = require('./lib/datastore'); module.exports = Datastore; ================================================ FILE: lib/cursor.js ================================================ /** * Manage access to data, be it to find, update or remove it */ var model = require('./model') , _ = require('underscore') ; /** * Create a new cursor for this collection * @param {Datastore} db - The datastore this cursor is bound to * @param {Query} query - The query this cursor will operate on * @param {Function} execFn - Handler to be executed after cursor has found the results and before the callback passed to find/findOne/update/remove */ function Cursor (db, query, execFn) { this.db = db; this.query = query || {}; if (execFn) { this.execFn = execFn; } } /** * Set a limit to the number of results */ Cursor.prototype.limit = function(limit) { this._limit = limit; return this; }; /** * Skip a the number of results */ Cursor.prototype.skip = function(skip) { this._skip = skip; return this; }; /** * Sort results of the query * @param {SortQuery} sortQuery - SortQuery is { field: order }, field can use the dot-notation, order is 1 for ascending and -1 for descending */ Cursor.prototype.sort = function(sortQuery) { this._sort = sortQuery; return this; }; /** * Add the use of a projection * @param {Object} projection - MongoDB-style projection. {} means take all fields. Then it's { key1: 1, key2: 1 } to take only key1 and key2 * { key1: 0, key2: 0 } to omit only key1 and key2. Except _id, you can't mix takes and omits */ Cursor.prototype.projection = function(projection) { this._projection = projection; return this; }; /** * Apply the projection */ Cursor.prototype.project = function (candidates) { var res = [], self = this , keepId, action, keys ; if (this._projection === undefined || Object.keys(this._projection).length === 0) { return candidates; } keepId = this._projection._id === 0 ? false : true; this._projection = _.omit(this._projection, '_id'); // Check for consistency keys = Object.keys(this._projection); keys.forEach(function (k) { if (action !== undefined && self._projection[k] !== action) { throw new Error("Can't both keep and omit fields except for _id"); } action = self._projection[k]; }); // Do the actual projection candidates.forEach(function (candidate) { var toPush; if (action === 1) { // pick-type projection toPush = { $set: {} }; keys.forEach(function (k) { toPush.$set[k] = model.getDotValue(candidate, k); if (toPush.$set[k] === undefined) { delete toPush.$set[k]; } }); toPush = model.modify({}, toPush); } else { // omit-type projection toPush = { $unset: {} }; keys.forEach(function (k) { toPush.$unset[k] = true }); toPush = model.modify(candidate, toPush); } if (keepId) { toPush._id = candidate._id; } else { delete toPush._id; } res.push(toPush); }); return res; }; /** * Get all matching elements * Will return pointers to matched elements (shallow copies), returning full copies is the role of find or findOne * This is an internal function, use exec which uses the executor * * @param {Function} callback - Signature: err, results */ Cursor.prototype._exec = function(_callback) { var res = [], added = 0, skipped = 0, self = this , error = null , i, keys, key ; function callback (error, res) { if (self.execFn) { return self.execFn(error, res, _callback); } else { return _callback(error, res); } } this.db.getCandidates(this.query, function (err, candidates) { if (err) { return callback(err); } try { for (i = 0; i < candidates.length; i += 1) { if (model.match(candidates[i], self.query)) { // If a sort is defined, wait for the results to be sorted before applying limit and skip if (!self._sort) { if (self._skip && self._skip > skipped) { skipped += 1; } else { res.push(candidates[i]); added += 1; if (self._limit && self._limit <= added) { break; } } } else { res.push(candidates[i]); } } } } catch (err) { return callback(err); } // Apply all sorts if (self._sort) { keys = Object.keys(self._sort); // Sorting var criteria = []; for (i = 0; i < keys.length; i++) { key = keys[i]; criteria.push({ key: key, direction: self._sort[key] }); } res.sort(function(a, b) { var criterion, compare, i; for (i = 0; i < criteria.length; i++) { criterion = criteria[i]; compare = criterion.direction * model.compareThings(model.getDotValue(a, criterion.key), model.getDotValue(b, criterion.key), self.db.compareStrings); if (compare !== 0) { return compare; } } return 0; }); // Applying limit and skip var limit = self._limit || res.length , skip = self._skip || 0; res = res.slice(skip, skip + limit); } // Apply projection try { res = self.project(res); } catch (e) { error = e; res = undefined; } return callback(error, res); }); }; Cursor.prototype.exec = function () { this.db.executor.push({ this: this, fn: this._exec, arguments: arguments }); }; // Interface module.exports = Cursor; ================================================ FILE: lib/customUtils.js ================================================ var crypto = require('crypto') ; /** * Return a random alphanumerical string of length len * There is a very small probability (less than 1/1,000,000) for the length to be less than len * (il the base64 conversion yields too many pluses and slashes) but * that's not an issue here * The probability of a collision is extremely small (need 3*10^12 documents to have one chance in a million of a collision) * See http://en.wikipedia.org/wiki/Birthday_problem */ function uid (len) { return crypto.randomBytes(Math.ceil(Math.max(8, len * 2))) .toString('base64') .replace(/[+\/]/g, '') .slice(0, len); } // Interface module.exports.uid = uid; ================================================ FILE: lib/datastore.js ================================================ var customUtils = require('./customUtils') , model = require('./model') , async = require('async') , Executor = require('./executor') , Index = require('./indexes') , util = require('util') , _ = require('underscore') , Persistence = require('./persistence') , Cursor = require('./cursor') ; /** * Create a new collection * @param {String} options.filename Optional, datastore will be in-memory only if not provided * @param {Boolean} options.timestampData Optional, defaults to false. If set to true, createdAt and updatedAt will be created and populated automatically (if not specified by user) * @param {Boolean} options.inMemoryOnly Optional, defaults to false * @param {String} options.nodeWebkitAppName Optional, specify the name of your NW app if you want options.filename to be relative to the directory where * Node Webkit stores application data such as cookies and local storage (the best place to store data in my opinion) * @param {Boolean} options.autoload Optional, defaults to false * @param {Function} options.onload Optional, if autoload is used this will be called after the load database with the error object as parameter. If you don't pass it the error will be thrown * @param {Function} options.afterSerialization/options.beforeDeserialization Optional, serialization hooks * @param {Number} options.corruptAlertThreshold Optional, threshold after which an alert is thrown if too much data is corrupt * @param {Function} options.compareStrings Optional, string comparison function that overrides default for sorting * * Event Emitter - Events * * compaction.done - Fired whenever a compaction operation was finished */ function Datastore (options) { var filename; // Retrocompatibility with v0.6 and before if (typeof options === 'string') { filename = options; this.inMemoryOnly = false; // Default } else { options = options || {}; filename = options.filename; this.inMemoryOnly = options.inMemoryOnly || false; this.autoload = options.autoload || false; this.timestampData = options.timestampData || false; } // Determine whether in memory or persistent if (!filename || typeof filename !== 'string' || filename.length === 0) { this.filename = null; this.inMemoryOnly = true; } else { this.filename = filename; } // String comparison function this.compareStrings = options.compareStrings; // Persistence handling this.persistence = new Persistence({ db: this, nodeWebkitAppName: options.nodeWebkitAppName , afterSerialization: options.afterSerialization , beforeDeserialization: options.beforeDeserialization , corruptAlertThreshold: options.corruptAlertThreshold }); // This new executor is ready if we don't use persistence // If we do, it will only be ready once loadDatabase is called this.executor = new Executor(); if (this.inMemoryOnly) { this.executor.ready = true; } // Indexed by field name, dot notation can be used // _id is always indexed and since _ids are generated randomly the underlying // binary is always well-balanced this.indexes = {}; this.indexes._id = new Index({ fieldName: '_id', unique: true }); this.ttlIndexes = {}; // Queue a load of the database right away and call the onload handler // By default (no onload handler), if there is an error there, no operation will be possible so warn the user by throwing an exception if (this.autoload) { this.loadDatabase(options.onload || function (err) { if (err) { throw err; } }); } } util.inherits(Datastore, require('events').EventEmitter); /** * Load the database from the datafile, and trigger the execution of buffered commands if any */ Datastore.prototype.loadDatabase = function () { this.executor.push({ this: this.persistence, fn: this.persistence.loadDatabase, arguments: arguments }, true); }; /** * Get an array of all the data in the database */ Datastore.prototype.getAllData = function () { return this.indexes._id.getAll(); }; /** * Reset all currently defined indexes */ Datastore.prototype.resetIndexes = function (newData) { var self = this; Object.keys(this.indexes).forEach(function (i) { self.indexes[i].reset(newData); }); }; /** * Ensure an index is kept for this field. Same parameters as lib/indexes * For now this function is synchronous, we need to test how much time it takes * We use an async API for consistency with the rest of the code * @param {String} options.fieldName * @param {Boolean} options.unique * @param {Boolean} options.sparse * @param {Number} options.expireAfterSeconds - Optional, if set this index becomes a TTL index (only works on Date fields, not arrays of Date) * @param {Function} cb Optional callback, signature: err */ Datastore.prototype.ensureIndex = function (options, cb) { var err , callback = cb || function () {}; options = options || {}; if (!options.fieldName) { err = new Error("Cannot create an index without a fieldName"); err.missingFieldName = true; return callback(err); } if (this.indexes[options.fieldName]) { return callback(null); } this.indexes[options.fieldName] = new Index(options); if (options.expireAfterSeconds !== undefined) { this.ttlIndexes[options.fieldName] = options.expireAfterSeconds; } // With this implementation index creation is not necessary to ensure TTL but we stick with MongoDB's API here try { this.indexes[options.fieldName].insert(this.getAllData()); } catch (e) { delete this.indexes[options.fieldName]; return callback(e); } // We may want to force all options to be persisted including defaults, not just the ones passed the index creation function this.persistence.persistNewState([{ $$indexCreated: options }], function (err) { if (err) { return callback(err); } return callback(null); }); }; /** * Remove an index * @param {String} fieldName * @param {Function} cb Optional callback, signature: err */ Datastore.prototype.removeIndex = function (fieldName, cb) { var callback = cb || function () {}; delete this.indexes[fieldName]; this.persistence.persistNewState([{ $$indexRemoved: fieldName }], function (err) { if (err) { return callback(err); } return callback(null); }); }; /** * Add one or several document(s) to all indexes */ Datastore.prototype.addToIndexes = function (doc) { var i, failingIndex, error , keys = Object.keys(this.indexes) ; for (i = 0; i < keys.length; i += 1) { try { this.indexes[keys[i]].insert(doc); } catch (e) { failingIndex = i; error = e; break; } } // If an error happened, we need to rollback the insert on all other indexes if (error) { for (i = 0; i < failingIndex; i += 1) { this.indexes[keys[i]].remove(doc); } throw error; } }; /** * Remove one or several document(s) from all indexes */ Datastore.prototype.removeFromIndexes = function (doc) { var self = this; Object.keys(this.indexes).forEach(function (i) { self.indexes[i].remove(doc); }); }; /** * Update one or several documents in all indexes * To update multiple documents, oldDoc must be an array of { oldDoc, newDoc } pairs * If one update violates a constraint, all changes are rolled back */ Datastore.prototype.updateIndexes = function (oldDoc, newDoc) { var i, failingIndex, error , keys = Object.keys(this.indexes) ; for (i = 0; i < keys.length; i += 1) { try { this.indexes[keys[i]].update(oldDoc, newDoc); } catch (e) { failingIndex = i; error = e; break; } } // If an error happened, we need to rollback the update on all other indexes if (error) { for (i = 0; i < failingIndex; i += 1) { this.indexes[keys[i]].revertUpdate(oldDoc, newDoc); } throw error; } }; /** * Return the list of candidates for a given query * Crude implementation for now, we return the candidates given by the first usable index if any * We try the following query types, in this order: basic match, $in match, comparison match * One way to make it better would be to enable the use of multiple indexes if the first usable index * returns too much data. I may do it in the future. * * Returned candidates will be scanned to find and remove all expired documents * * @param {Query} query * @param {Boolean} dontExpireStaleDocs Optional, defaults to false, if true don't remove stale docs. Useful for the remove function which shouldn't be impacted by expirations * @param {Function} callback Signature err, candidates */ Datastore.prototype.getCandidates = function (query, dontExpireStaleDocs, callback) { var indexNames = Object.keys(this.indexes) , self = this , usableQueryKeys; if (typeof dontExpireStaleDocs === 'function') { callback = dontExpireStaleDocs; dontExpireStaleDocs = false; } async.waterfall([ // STEP 1: get candidates list by checking indexes from most to least frequent usecase function (cb) { // For a basic match usableQueryKeys = []; Object.keys(query).forEach(function (k) { if (typeof query[k] === 'string' || typeof query[k] === 'number' || typeof query[k] === 'boolean' || util.isDate(query[k]) || query[k] === null) { usableQueryKeys.push(k); } }); usableQueryKeys = _.intersection(usableQueryKeys, indexNames); if (usableQueryKeys.length > 0) { return cb(null, self.indexes[usableQueryKeys[0]].getMatching(query[usableQueryKeys[0]])); } // For a $in match usableQueryKeys = []; Object.keys(query).forEach(function (k) { if (query[k] && query[k].hasOwnProperty('$in')) { usableQueryKeys.push(k); } }); usableQueryKeys = _.intersection(usableQueryKeys, indexNames); if (usableQueryKeys.length > 0) { return cb(null, self.indexes[usableQueryKeys[0]].getMatching(query[usableQueryKeys[0]].$in)); } // For a comparison match usableQueryKeys = []; Object.keys(query).forEach(function (k) { if (query[k] && (query[k].hasOwnProperty('$lt') || query[k].hasOwnProperty('$lte') || query[k].hasOwnProperty('$gt') || query[k].hasOwnProperty('$gte'))) { usableQueryKeys.push(k); } }); usableQueryKeys = _.intersection(usableQueryKeys, indexNames); if (usableQueryKeys.length > 0) { return cb(null, self.indexes[usableQueryKeys[0]].getBetweenBounds(query[usableQueryKeys[0]])); } // By default, return all the DB data return cb(null, self.getAllData()); } // STEP 2: remove all expired documents , function (docs) { if (dontExpireStaleDocs) { return callback(null, docs); } var expiredDocsIds = [], validDocs = [], ttlIndexesFieldNames = Object.keys(self.ttlIndexes); docs.forEach(function (doc) { var valid = true; ttlIndexesFieldNames.forEach(function (i) { if (doc[i] !== undefined && util.isDate(doc[i]) && Date.now() > doc[i].getTime() + self.ttlIndexes[i] * 1000) { valid = false; } }); if (valid) { validDocs.push(doc); } else { expiredDocsIds.push(doc._id); } }); async.eachSeries(expiredDocsIds, function (_id, cb) { self._remove({ _id: _id }, {}, function (err) { if (err) { return callback(err); } return cb(); }); }, function (err) { return callback(null, validDocs); }); }]); }; /** * Insert a new document * @param {Function} cb Optional callback, signature: err, insertedDoc * * @api private Use Datastore.insert which has the same signature */ Datastore.prototype._insert = function (newDoc, cb) { var callback = cb || function () {} , preparedDoc ; try { preparedDoc = this.prepareDocumentForInsertion(newDoc) this._insertInCache(preparedDoc); } catch (e) { return callback(e); } this.persistence.persistNewState(util.isArray(preparedDoc) ? preparedDoc : [preparedDoc], function (err) { if (err) { return callback(err); } return callback(null, model.deepCopy(preparedDoc)); }); }; /** * Create a new _id that's not already in use */ Datastore.prototype.createNewId = function () { var tentativeId = customUtils.uid(16); // Try as many times as needed to get an unused _id. As explained in customUtils, the probability of this ever happening is extremely small, so this is O(1) if (this.indexes._id.getMatching(tentativeId).length > 0) { tentativeId = this.createNewId(); } return tentativeId; }; /** * Prepare a document (or array of documents) to be inserted in a database * Meaning adds _id and timestamps if necessary on a copy of newDoc to avoid any side effect on user input * @api private */ Datastore.prototype.prepareDocumentForInsertion = function (newDoc) { var preparedDoc, self = this; if (util.isArray(newDoc)) { preparedDoc = []; newDoc.forEach(function (doc) { preparedDoc.push(self.prepareDocumentForInsertion(doc)); }); } else { preparedDoc = model.deepCopy(newDoc); if (preparedDoc._id === undefined) { preparedDoc._id = this.createNewId(); } var now = new Date(); if (this.timestampData && preparedDoc.createdAt === undefined) { preparedDoc.createdAt = now; } if (this.timestampData && preparedDoc.updatedAt === undefined) { preparedDoc.updatedAt = now; } model.checkObject(preparedDoc); } return preparedDoc; }; /** * If newDoc is an array of documents, this will insert all documents in the cache * @api private */ Datastore.prototype._insertInCache = function (preparedDoc) { if (util.isArray(preparedDoc)) { this._insertMultipleDocsInCache(preparedDoc); } else { this.addToIndexes(preparedDoc); } }; /** * If one insertion fails (e.g. because of a unique constraint), roll back all previous * inserts and throws the error * @api private */ Datastore.prototype._insertMultipleDocsInCache = function (preparedDocs) { var i, failingI, error; for (i = 0; i < preparedDocs.length; i += 1) { try { this.addToIndexes(preparedDocs[i]); } catch (e) { error = e; failingI = i; break; } } if (error) { for (i = 0; i < failingI; i += 1) { this.removeFromIndexes(preparedDocs[i]); } throw error; } }; Datastore.prototype.insert = function () { this.executor.push({ this: this, fn: this._insert, arguments: arguments }); }; /** * Count all documents matching the query * @param {Object} query MongoDB-style query */ Datastore.prototype.count = function(query, callback) { var cursor = new Cursor(this, query, function(err, docs, callback) { if (err) { return callback(err); } return callback(null, docs.length); }); if (typeof callback === 'function') { cursor.exec(callback); } else { return cursor; } }; /** * Find all documents matching the query * If no callback is passed, we return the cursor so that user can limit, skip and finally exec * @param {Object} query MongoDB-style query * @param {Object} projection MongoDB-style projection */ Datastore.prototype.find = function (query, projection, callback) { switch (arguments.length) { case 1: projection = {}; // callback is undefined, will return a cursor break; case 2: if (typeof projection === 'function') { callback = projection; projection = {}; } // If not assume projection is an object and callback undefined break; } var cursor = new Cursor(this, query, function(err, docs, callback) { var res = [], i; if (err) { return callback(err); } for (i = 0; i < docs.length; i += 1) { res.push(model.deepCopy(docs[i])); } return callback(null, res); }); cursor.projection(projection); if (typeof callback === 'function') { cursor.exec(callback); } else { return cursor; } }; /** * Find one document matching the query * @param {Object} query MongoDB-style query * @param {Object} projection MongoDB-style projection */ Datastore.prototype.findOne = function (query, projection, callback) { switch (arguments.length) { case 1: projection = {}; // callback is undefined, will return a cursor break; case 2: if (typeof projection === 'function') { callback = projection; projection = {}; } // If not assume projection is an object and callback undefined break; } var cursor = new Cursor(this, query, function(err, docs, callback) { if (err) { return callback(err); } if (docs.length === 1) { return callback(null, model.deepCopy(docs[0])); } else { return callback(null, null); } }); cursor.projection(projection).limit(1); if (typeof callback === 'function') { cursor.exec(callback); } else { return cursor; } }; /** * Update all docs matching query * @param {Object} query * @param {Object} updateQuery * @param {Object} options Optional options * options.multi If true, can update multiple documents (defaults to false) * options.upsert If true, document is inserted if the query doesn't match anything * options.returnUpdatedDocs Defaults to false, if true return as third argument the array of updated matched documents (even if no change actually took place) * @param {Function} cb Optional callback, signature: (err, numAffected, affectedDocuments, upsert) * If update was an upsert, upsert flag is set to true * affectedDocuments can be one of the following: * * For an upsert, the upserted document * * For an update with returnUpdatedDocs option false, null * * For an update with returnUpdatedDocs true and multi false, the updated document * * For an update with returnUpdatedDocs true and multi true, the array of updated documents * * WARNING: The API was changed between v1.7.4 and v1.8, for consistency and readability reasons. Prior and including to v1.7.4, * the callback signature was (err, numAffected, updated) where updated was the updated document in case of an upsert * or the array of updated documents for an update if the returnUpdatedDocs option was true. That meant that the type of * affectedDocuments in a non multi update depended on whether there was an upsert or not, leaving only two ways for the * user to check whether an upsert had occured: checking the type of affectedDocuments or running another find query on * the whole dataset to check its size. Both options being ugly, the breaking change was necessary. * * @api private Use Datastore.update which has the same signature */ Datastore.prototype._update = function (query, updateQuery, options, cb) { var callback , self = this , numReplaced = 0 , multi, upsert , i ; if (typeof options === 'function') { cb = options; options = {}; } callback = cb || function () {}; multi = options.multi !== undefined ? options.multi : false; upsert = options.upsert !== undefined ? options.upsert : false; async.waterfall([ function (cb) { // If upsert option is set, check whether we need to insert the doc if (!upsert) { return cb(); } // Need to use an internal function not tied to the executor to avoid deadlock var cursor = new Cursor(self, query); cursor.limit(1)._exec(function (err, docs) { if (err) { return callback(err); } if (docs.length === 1) { return cb(); } else { var toBeInserted; try { model.checkObject(updateQuery); // updateQuery is a simple object with no modifier, use it as the document to insert toBeInserted = updateQuery; } catch (e) { // updateQuery contains modifiers, use the find query as the base, // strip it from all operators and update it according to updateQuery try { toBeInserted = model.modify(model.deepCopy(query, true), updateQuery); } catch (err) { return callback(err); } } return self._insert(toBeInserted, function (err, newDoc) { if (err) { return callback(err); } return callback(null, 1, newDoc, true); }); } }); } , function () { // Perform the update var modifiedDoc , modifications = [], createdAt; self.getCandidates(query, function (err, candidates) { if (err) { return callback(err); } // Preparing update (if an error is thrown here neither the datafile nor // the in-memory indexes are affected) try { for (i = 0; i < candidates.length; i += 1) { if (model.match(candidates[i], query) && (multi || numReplaced === 0)) { numReplaced += 1; if (self.timestampData) { createdAt = candidates[i].createdAt; } modifiedDoc = model.modify(candidates[i], updateQuery); if (self.timestampData) { modifiedDoc.createdAt = createdAt; modifiedDoc.updatedAt = new Date(); } modifications.push({ oldDoc: candidates[i], newDoc: modifiedDoc }); } } } catch (err) { return callback(err); } // Change the docs in memory try { self.updateIndexes(modifications); } catch (err) { return callback(err); } // Update the datafile var updatedDocs = _.pluck(modifications, 'newDoc'); self.persistence.persistNewState(updatedDocs, function (err) { if (err) { return callback(err); } if (!options.returnUpdatedDocs) { return callback(null, numReplaced); } else { var updatedDocsDC = []; updatedDocs.forEach(function (doc) { updatedDocsDC.push(model.deepCopy(doc)); }); if (! multi) { updatedDocsDC = updatedDocsDC[0]; } return callback(null, numReplaced, updatedDocsDC); } }); }); }]); }; Datastore.prototype.update = function () { this.executor.push({ this: this, fn: this._update, arguments: arguments }); }; /** * Remove all docs matching the query * For now very naive implementation (similar to update) * @param {Object} query * @param {Object} options Optional options * options.multi If true, can update multiple documents (defaults to false) * @param {Function} cb Optional callback, signature: err, numRemoved * * @api private Use Datastore.remove which has the same signature */ Datastore.prototype._remove = function (query, options, cb) { var callback , self = this, numRemoved = 0, removedDocs = [], multi ; if (typeof options === 'function') { cb = options; options = {}; } callback = cb || function () {}; multi = options.multi !== undefined ? options.multi : false; this.getCandidates(query, true, function (err, candidates) { if (err) { return callback(err); } try { candidates.forEach(function (d) { if (model.match(d, query) && (multi || numRemoved === 0)) { numRemoved += 1; removedDocs.push({ $$deleted: true, _id: d._id }); self.removeFromIndexes(d); } }); } catch (err) { return callback(err); } self.persistence.persistNewState(removedDocs, function (err) { if (err) { return callback(err); } return callback(null, numRemoved); }); }); }; Datastore.prototype.remove = function () { this.executor.push({ this: this, fn: this._remove, arguments: arguments }); }; module.exports = Datastore; ================================================ FILE: lib/executor.js ================================================ /** * Responsible for sequentially executing actions on the database */ var async = require('async') ; function Executor () { this.buffer = []; this.ready = false; // This queue will execute all commands, one-by-one in order this.queue = async.queue(function (task, cb) { var newArguments = []; // task.arguments is an array-like object on which adding a new field doesn't work, so we transform it into a real array for (var i = 0; i < task.arguments.length; i += 1) { newArguments.push(task.arguments[i]); } var lastArg = task.arguments[task.arguments.length - 1]; // Always tell the queue task is complete. Execute callback if any was given. if (typeof lastArg === 'function') { // Callback was supplied newArguments[newArguments.length - 1] = function () { if (typeof setImmediate === 'function') { setImmediate(cb); } else { process.nextTick(cb); } lastArg.apply(null, arguments); }; } else if (!lastArg && task.arguments.length !== 0) { // false/undefined/null supplied as callbback newArguments[newArguments.length - 1] = function () { cb(); }; } else { // Nothing supplied as callback newArguments.push(function () { cb(); }); } task.fn.apply(task.this, newArguments); }, 1); } /** * If executor is ready, queue task (and process it immediately if executor was idle) * If not, buffer task for later processing * @param {Object} task * task.this - Object to use as this * task.fn - Function to execute * task.arguments - Array of arguments, IMPORTANT: only the last argument may be a function (the callback) * and the last argument cannot be false/undefined/null * @param {Boolean} forceQueuing Optional (defaults to false) force executor to queue task even if it is not ready */ Executor.prototype.push = function (task, forceQueuing) { if (this.ready || forceQueuing) { this.queue.push(task); } else { this.buffer.push(task); } }; /** * Queue all tasks in buffer (in the same order they came in) * Automatically sets executor as ready */ Executor.prototype.processBuffer = function () { var i; this.ready = true; for (i = 0; i < this.buffer.length; i += 1) { this.queue.push(this.buffer[i]); } this.buffer = []; }; // Interface module.exports = Executor; ================================================ FILE: lib/indexes.js ================================================ var BinarySearchTree = require('binary-search-tree').AVLTree , model = require('./model') , _ = require('underscore') , util = require('util') ; /** * Two indexed pointers are equal iif they point to the same place */ function checkValueEquality (a, b) { return a === b; } /** * Type-aware projection */ function projectForUnique (elt) { if (elt === null) { return '$null'; } if (typeof elt === 'string') { return '$string' + elt; } if (typeof elt === 'boolean') { return '$boolean' + elt; } if (typeof elt === 'number') { return '$number' + elt; } if (util.isArray(elt)) { return '$date' + elt.getTime(); } return elt; // Arrays and objects, will check for pointer equality } /** * Create a new index * All methods on an index guarantee that either the whole operation was successful and the index changed * or the operation was unsuccessful and an error is thrown while the index is unchanged * @param {String} options.fieldName On which field should the index apply (can use dot notation to index on sub fields) * @param {Boolean} options.unique Optional, enforce a unique constraint (default: false) * @param {Boolean} options.sparse Optional, allow a sparse index (we can have documents for which fieldName is undefined) (default: false) */ function Index (options) { this.fieldName = options.fieldName; this.unique = options.unique || false; this.sparse = options.sparse || false; this.treeOptions = { unique: this.unique, compareKeys: model.compareThings, checkValueEquality: checkValueEquality }; this.reset(); // No data in the beginning } /** * Reset an index * @param {Document or Array of documents} newData Optional, data to initialize the index with * If an error is thrown during insertion, the index is not modified */ Index.prototype.reset = function (newData) { this.tree = new BinarySearchTree(this.treeOptions); if (newData) { this.insert(newData); } }; /** * Insert a new document in the index * If an array is passed, we insert all its elements (if one insertion fails the index is not modified) * O(log(n)) */ Index.prototype.insert = function (doc) { var key, self = this , keys, i, failingI, error ; if (util.isArray(doc)) { this.insertMultipleDocs(doc); return; } key = model.getDotValue(doc, this.fieldName); // We don't index documents that don't contain the field if the index is sparse if (key === undefined && this.sparse) { return; } if (!util.isArray(key)) { this.tree.insert(key, doc); } else { // If an insert fails due to a unique constraint, roll back all inserts before it keys = _.uniq(key, projectForUnique); for (i = 0; i < keys.length; i += 1) { try { this.tree.insert(keys[i], doc); } catch (e) { error = e; failingI = i; break; } } if (error) { for (i = 0; i < failingI; i += 1) { this.tree.delete(keys[i], doc); } throw error; } } }; /** * Insert an array of documents in the index * If a constraint is violated, the changes should be rolled back and an error thrown * * @API private */ Index.prototype.insertMultipleDocs = function (docs) { var i, error, failingI; for (i = 0; i < docs.length; i += 1) { try { this.insert(docs[i]); } catch (e) { error = e; failingI = i; break; } } if (error) { for (i = 0; i < failingI; i += 1) { this.remove(docs[i]); } throw error; } }; /** * Remove a document from the index * If an array is passed, we remove all its elements * The remove operation is safe with regards to the 'unique' constraint * O(log(n)) */ Index.prototype.remove = function (doc) { var key, self = this; if (util.isArray(doc)) { doc.forEach(function (d) { self.remove(d); }); return; } key = model.getDotValue(doc, this.fieldName); if (key === undefined && this.sparse) { return; } if (!util.isArray(key)) { this.tree.delete(key, doc); } else { _.uniq(key, projectForUnique).forEach(function (_key) { self.tree.delete(_key, doc); }); } }; /** * Update a document in the index * If a constraint is violated, changes are rolled back and an error thrown * Naive implementation, still in O(log(n)) */ Index.prototype.update = function (oldDoc, newDoc) { if (util.isArray(oldDoc)) { this.updateMultipleDocs(oldDoc); return; } this.remove(oldDoc); try { this.insert(newDoc); } catch (e) { this.insert(oldDoc); throw e; } }; /** * Update multiple documents in the index * If a constraint is violated, the changes need to be rolled back * and an error thrown * @param {Array of oldDoc, newDoc pairs} pairs * * @API private */ Index.prototype.updateMultipleDocs = function (pairs) { var i, failingI, error; for (i = 0; i < pairs.length; i += 1) { this.remove(pairs[i].oldDoc); } for (i = 0; i < pairs.length; i += 1) { try { this.insert(pairs[i].newDoc); } catch (e) { error = e; failingI = i; break; } } // If an error was raised, roll back changes in the inverse order if (error) { for (i = 0; i < failingI; i += 1) { this.remove(pairs[i].newDoc); } for (i = 0; i < pairs.length; i += 1) { this.insert(pairs[i].oldDoc); } throw error; } }; /** * Revert an update */ Index.prototype.revertUpdate = function (oldDoc, newDoc) { var revert = []; if (!util.isArray(oldDoc)) { this.update(newDoc, oldDoc); } else { oldDoc.forEach(function (pair) { revert.push({ oldDoc: pair.newDoc, newDoc: pair.oldDoc }); }); this.update(revert); } }; /** * Get all documents in index whose key match value (if it is a Thing) or one of the elements of value (if it is an array of Things) * @param {Thing} value Value to match the key against * @return {Array of documents} */ Index.prototype.getMatching = function (value) { var self = this; if (!util.isArray(value)) { return self.tree.search(value); } else { var _res = {}, res = []; value.forEach(function (v) { self.getMatching(v).forEach(function (doc) { _res[doc._id] = doc; }); }); Object.keys(_res).forEach(function (_id) { res.push(_res[_id]); }); return res; } }; /** * Get all documents in index whose key is between bounds are they are defined by query * Documents are sorted by key * @param {Query} query * @return {Array of documents} */ Index.prototype.getBetweenBounds = function (query) { return this.tree.betweenBounds(query); }; /** * Get all elements in the index * @return {Array of documents} */ Index.prototype.getAll = function () { var res = []; this.tree.executeOnEveryNode(function (node) { var i; for (i = 0; i < node.data.length; i += 1) { res.push(node.data[i]); } }); return res; }; // Interface module.exports = Index; ================================================ FILE: lib/model.js ================================================ /** * Handle models (i.e. docs) * Serialization/deserialization * Copying * Querying, update */ var util = require('util') , _ = require('underscore') , modifierFunctions = {} , lastStepModifierFunctions = {} , comparisonFunctions = {} , logicalOperators = {} , arrayComparisonFunctions = {} ; /** * Check a key, throw an error if the key is non valid * @param {String} k key * @param {Model} v value, needed to treat the Date edge case * Non-treatable edge cases here: if part of the object if of the form { $$date: number } or { $$deleted: true } * Its serialized-then-deserialized version it will transformed into a Date object * But you really need to want it to trigger such behaviour, even when warned not to use '$' at the beginning of the field names... */ function checkKey (k, v) { if (typeof k === 'number') { k = k.toString(); } if (k[0] === '$' && !(k === '$$date' && typeof v === 'number') && !(k === '$$deleted' && v === true) && !(k === '$$indexCreated') && !(k === '$$indexRemoved')) { throw new Error('Field names cannot begin with the $ character'); } if (k.indexOf('.') !== -1) { throw new Error('Field names cannot contain a .'); } } /** * Check a DB object and throw an error if it's not valid * Works by applying the above checkKey function to all fields recursively */ function checkObject (obj) { if (util.isArray(obj)) { obj.forEach(function (o) { checkObject(o); }); } if (typeof obj === 'object' && obj !== null) { Object.keys(obj).forEach(function (k) { checkKey(k, obj[k]); checkObject(obj[k]); }); } } /** * Serialize an object to be persisted to a one-line string * For serialization/deserialization, we use the native JSON parser and not eval or Function * That gives us less freedom but data entered in the database may come from users * so eval and the like are not safe * Accepted primitive types: Number, String, Boolean, Date, null * Accepted secondary types: Objects, Arrays */ function serialize (obj) { var res; res = JSON.stringify(obj, function (k, v) { checkKey(k, v); if (v === undefined) { return undefined; } if (v === null) { return null; } // Hackish way of checking if object is Date (this way it works between execution contexts in node-webkit). // We can't use value directly because for dates it is already string in this function (date.toJSON was already called), so we use this if (typeof this[k].getTime === 'function') { return { $$date: this[k].getTime() }; } return v; }); return res; } /** * From a one-line representation of an object generate by the serialize function * Return the object itself */ function deserialize (rawData) { return JSON.parse(rawData, function (k, v) { if (k === '$$date') { return new Date(v); } if (typeof v === 'string' || typeof v === 'number' || typeof v === 'boolean' || v === null) { return v; } if (v && v.$$date) { return v.$$date; } return v; }); } /** * Deep copy a DB object * The optional strictKeys flag (defaulting to false) indicates whether to copy everything or only fields * where the keys are valid, i.e. don't begin with $ and don't contain a . */ function deepCopy (obj, strictKeys) { var res; if ( typeof obj === 'boolean' || typeof obj === 'number' || typeof obj === 'string' || obj === null || (util.isDate(obj)) ) { return obj; } if (util.isArray(obj)) { res = []; obj.forEach(function (o) { res.push(deepCopy(o, strictKeys)); }); return res; } if (typeof obj === 'object') { res = {}; Object.keys(obj).forEach(function (k) { if (!strictKeys || (k[0] !== '$' && k.indexOf('.') === -1)) { res[k] = deepCopy(obj[k], strictKeys); } }); return res; } return undefined; // For now everything else is undefined. We should probably throw an error instead } /** * Tells if an object is a primitive type or a "real" object * Arrays are considered primitive */ function isPrimitiveType (obj) { return ( typeof obj === 'boolean' || typeof obj === 'number' || typeof obj === 'string' || obj === null || util.isDate(obj) || util.isArray(obj)); } /** * Utility functions for comparing things * Assumes type checking was already done (a and b already have the same type) * compareNSB works for numbers, strings and booleans */ function compareNSB (a, b) { if (a < b) { return -1; } if (a > b) { return 1; } return 0; } function compareArrays (a, b) { var i, comp; for (i = 0; i < Math.min(a.length, b.length); i += 1) { comp = compareThings(a[i], b[i]); if (comp !== 0) { return comp; } } // Common section was identical, longest one wins return compareNSB(a.length, b.length); } /** * Compare { things U undefined } * Things are defined as any native types (string, number, boolean, null, date) and objects * We need to compare with undefined as it will be used in indexes * In the case of objects and arrays, we deep-compare * If two objects dont have the same type, the (arbitrary) type hierarchy is: undefined, null, number, strings, boolean, dates, arrays, objects * Return -1 if a < b, 1 if a > b and 0 if a = b (note that equality here is NOT the same as defined in areThingsEqual!) * * @param {Function} _compareStrings String comparing function, returning -1, 0 or 1, overriding default string comparison (useful for languages with accented letters) */ function compareThings (a, b, _compareStrings) { var aKeys, bKeys, comp, i , compareStrings = _compareStrings || compareNSB; // undefined if (a === undefined) { return b === undefined ? 0 : -1; } if (b === undefined) { return a === undefined ? 0 : 1; } // null if (a === null) { return b === null ? 0 : -1; } if (b === null) { return a === null ? 0 : 1; } // Numbers if (typeof a === 'number') { return typeof b === 'number' ? compareNSB(a, b) : -1; } if (typeof b === 'number') { return typeof a === 'number' ? compareNSB(a, b) : 1; } // Strings if (typeof a === 'string') { return typeof b === 'string' ? compareStrings(a, b) : -1; } if (typeof b === 'string') { return typeof a === 'string' ? compareStrings(a, b) : 1; } // Booleans if (typeof a === 'boolean') { return typeof b === 'boolean' ? compareNSB(a, b) : -1; } if (typeof b === 'boolean') { return typeof a === 'boolean' ? compareNSB(a, b) : 1; } // Dates if (util.isDate(a)) { return util.isDate(b) ? compareNSB(a.getTime(), b.getTime()) : -1; } if (util.isDate(b)) { return util.isDate(a) ? compareNSB(a.getTime(), b.getTime()) : 1; } // Arrays (first element is most significant and so on) if (util.isArray(a)) { return util.isArray(b) ? compareArrays(a, b) : -1; } if (util.isArray(b)) { return util.isArray(a) ? compareArrays(a, b) : 1; } // Objects aKeys = Object.keys(a).sort(); bKeys = Object.keys(b).sort(); for (i = 0; i < Math.min(aKeys.length, bKeys.length); i += 1) { comp = compareThings(a[aKeys[i]], b[bKeys[i]]); if (comp !== 0) { return comp; } } return compareNSB(aKeys.length, bKeys.length); } // ============================================================== // Updating documents // ============================================================== /** * The signature of modifier functions is as follows * Their structure is always the same: recursively follow the dot notation while creating * the nested documents if needed, then apply the "last step modifier" * @param {Object} obj The model to modify * @param {String} field Can contain dots, in that case that means we will set a subfield recursively * @param {Model} value */ /** * Set a field to a new value */ lastStepModifierFunctions.$set = function (obj, field, value) { obj[field] = value; }; /** * Unset a field */ lastStepModifierFunctions.$unset = function (obj, field, value) { delete obj[field]; }; /** * Push an element to the end of an array field * Optional modifier $each instead of value to push several values * Optional modifier $slice to slice the resulting array, see https://docs.mongodb.org/manual/reference/operator/update/slice/ * Différeence with MongoDB: if $slice is specified and not $each, we act as if value is an empty array */ lastStepModifierFunctions.$push = function (obj, field, value) { // Create the array if it doesn't exist if (!obj.hasOwnProperty(field)) { obj[field] = []; } if (!util.isArray(obj[field])) { throw new Error("Can't $push an element on non-array values"); } if (value !== null && typeof value === 'object' && value.$slice && value.$each === undefined) { value.$each = []; } if (value !== null && typeof value === 'object' && value.$each) { if (Object.keys(value).length >= 3 || (Object.keys(value).length === 2 && value.$slice === undefined)) { throw new Error("Can only use $slice in cunjunction with $each when $push to array"); } if (!util.isArray(value.$each)) { throw new Error("$each requires an array value"); } value.$each.forEach(function (v) { obj[field].push(v); }); if (value.$slice === undefined || typeof value.$slice !== 'number') { return; } if (value.$slice === 0) { obj[field] = []; } else { var start, end, n = obj[field].length; if (value.$slice < 0) { start = Math.max(0, n + value.$slice); end = n; } else if (value.$slice > 0) { start = 0; end = Math.min(n, value.$slice); } obj[field] = obj[field].slice(start, end); } } else { obj[field].push(value); } }; /** * Add an element to an array field only if it is not already in it * No modification if the element is already in the array * Note that it doesn't check whether the original array contains duplicates */ lastStepModifierFunctions.$addToSet = function (obj, field, value) { var addToSet = true; // Create the array if it doesn't exist if (!obj.hasOwnProperty(field)) { obj[field] = []; } if (!util.isArray(obj[field])) { throw new Error("Can't $addToSet an element on non-array values"); } if (value !== null && typeof value === 'object' && value.$each) { if (Object.keys(value).length > 1) { throw new Error("Can't use another field in conjunction with $each"); } if (!util.isArray(value.$each)) { throw new Error("$each requires an array value"); } value.$each.forEach(function (v) { lastStepModifierFunctions.$addToSet(obj, field, v); }); } else { obj[field].forEach(function (v) { if (compareThings(v, value) === 0) { addToSet = false; } }); if (addToSet) { obj[field].push(value); } } }; /** * Remove the first or last element of an array */ lastStepModifierFunctions.$pop = function (obj, field, value) { if (!util.isArray(obj[field])) { throw new Error("Can't $pop an element from non-array values"); } if (typeof value !== 'number') { throw new Error(value + " isn't an integer, can't use it with $pop"); } if (value === 0) { return; } if (value > 0) { obj[field] = obj[field].slice(0, obj[field].length - 1); } else { obj[field] = obj[field].slice(1); } }; /** * Removes all instances of a value from an existing array */ lastStepModifierFunctions.$pull = function (obj, field, value) { var arr, i; if (!util.isArray(obj[field])) { throw new Error("Can't $pull an element from non-array values"); } arr = obj[field]; for (i = arr.length - 1; i >= 0; i -= 1) { if (match(arr[i], value)) { arr.splice(i, 1); } } }; /** * Increment a numeric field's value */ lastStepModifierFunctions.$inc = function (obj, field, value) { if (typeof value !== 'number') { throw new Error(value + " must be a number"); } if (typeof obj[field] !== 'number') { if (!_.has(obj, field)) { obj[field] = value; } else { throw new Error("Don't use the $inc modifier on non-number fields"); } } else { obj[field] += value; } }; /** * Updates the value of the field, only if specified field is greater than the current value of the field */ lastStepModifierFunctions.$max = function (obj, field, value) { if (typeof obj[field] === 'undefined') { obj[field] = value; } else if (value > obj[field]) { obj[field] = value; } }; /** * Updates the value of the field, only if specified field is smaller than the current value of the field */ lastStepModifierFunctions.$min = function (obj, field, value) { if (typeof obj[field] === 'undefined') {  obj[field] = value; } else if (value < obj[field]) { obj[field] = value; } }; // Given its name, create the complete modifier function function createModifierFunction (modifier) { return function (obj, field, value) { var fieldParts = typeof field === 'string' ? field.split('.') : field; if (fieldParts.length === 1) { lastStepModifierFunctions[modifier](obj, field, value); } else { if (obj[fieldParts[0]] === undefined) { if (modifier === '$unset') { return; } // Bad looking specific fix, needs to be generalized modifiers that behave like $unset are implemented obj[fieldParts[0]] = {}; } modifierFunctions[modifier](obj[fieldParts[0]], fieldParts.slice(1), value); } }; } // Actually create all modifier functions Object.keys(lastStepModifierFunctions).forEach(function (modifier) { modifierFunctions[modifier] = createModifierFunction(modifier); }); /** * Modify a DB object according to an update query */ function modify (obj, updateQuery) { var keys = Object.keys(updateQuery) , firstChars = _.map(keys, function (item) { return item[0]; }) , dollarFirstChars = _.filter(firstChars, function (c) { return c === '$'; }) , newDoc, modifiers ; if (keys.indexOf('_id') !== -1 && updateQuery._id !== obj._id) { throw new Error("You cannot change a document's _id"); } if (dollarFirstChars.length !== 0 && dollarFirstChars.length !== firstChars.length) { throw new Error("You cannot mix modifiers and normal fields"); } if (dollarFirstChars.length === 0) { // Simply replace the object with the update query contents newDoc = deepCopy(updateQuery); newDoc._id = obj._id; } else { // Apply modifiers modifiers = _.uniq(keys); newDoc = deepCopy(obj); modifiers.forEach(function (m) { var keys; if (!modifierFunctions[m]) { throw new Error("Unknown modifier " + m); } // Can't rely on Object.keys throwing on non objects since ES6 // Not 100% satisfying as non objects can be interpreted as objects but no false negatives so we can live with it if (typeof updateQuery[m] !== 'object') { throw new Error("Modifier " + m + "'s argument must be an object"); } keys = Object.keys(updateQuery[m]); keys.forEach(function (k) { modifierFunctions[m](newDoc, k, updateQuery[m][k]); }); }); } // Check result is valid and return it checkObject(newDoc); if (obj._id !== newDoc._id) { throw new Error("You can't change a document's _id"); } return newDoc; }; // ============================================================== // Finding documents // ============================================================== /** * Get a value from object with dot notation * @param {Object} obj * @param {String} field */ function getDotValue (obj, field) { var fieldParts = typeof field === 'string' ? field.split('.') : field , i, objs; if (!obj) { return undefined; } // field cannot be empty so that means we should return undefined so that nothing can match if (fieldParts.length === 0) { return obj; } if (fieldParts.length === 1) { return obj[fieldParts[0]]; } if (util.isArray(obj[fieldParts[0]])) { // If the next field is an integer, return only this item of the array i = parseInt(fieldParts[1], 10); if (typeof i === 'number' && !isNaN(i)) { return getDotValue(obj[fieldParts[0]][i], fieldParts.slice(2)) } // Return the array of values objs = new Array(); for (i = 0; i < obj[fieldParts[0]].length; i += 1) { objs.push(getDotValue(obj[fieldParts[0]][i], fieldParts.slice(1))); } return objs; } else { return getDotValue(obj[fieldParts[0]], fieldParts.slice(1)); } } /** * Check whether 'things' are equal * Things are defined as any native types (string, number, boolean, null, date) and objects * In the case of object, we check deep equality * Returns true if they are, false otherwise */ function areThingsEqual (a, b) { var aKeys , bKeys , i; // Strings, booleans, numbers, null if (a === null || typeof a === 'string' || typeof a === 'boolean' || typeof a === 'number' || b === null || typeof b === 'string' || typeof b === 'boolean' || typeof b === 'number') { return a === b; } // Dates if (util.isDate(a) || util.isDate(b)) { return util.isDate(a) && util.isDate(b) && a.getTime() === b.getTime(); } // Arrays (no match since arrays are used as a $in) // undefined (no match since they mean field doesn't exist and can't be serialized) if ((!(util.isArray(a) && util.isArray(b)) && (util.isArray(a) || util.isArray(b))) || a === undefined || b === undefined) { return false; } // General objects (check for deep equality) // a and b should be objects at this point try { aKeys = Object.keys(a); bKeys = Object.keys(b); } catch (e) { return false; } if (aKeys.length !== bKeys.length) { return false; } for (i = 0; i < aKeys.length; i += 1) { if (bKeys.indexOf(aKeys[i]) === -1) { return false; } if (!areThingsEqual(a[aKeys[i]], b[aKeys[i]])) { return false; } } return true; } /** * Check that two values are comparable */ function areComparable (a, b) { if (typeof a !== 'string' && typeof a !== 'number' && !util.isDate(a) && typeof b !== 'string' && typeof b !== 'number' && !util.isDate(b)) { return false; } if (typeof a !== typeof b) { return false; } return true; } /** * Arithmetic and comparison operators * @param {Native value} a Value in the object * @param {Native value} b Value in the query */ comparisonFunctions.$lt = function (a, b) { return areComparable(a, b) && a < b; }; comparisonFunctions.$lte = function (a, b) { return areComparable(a, b) && a <= b; }; comparisonFunctions.$gt = function (a, b) { return areComparable(a, b) && a > b; }; comparisonFunctions.$gte = function (a, b) { return areComparable(a, b) && a >= b; }; comparisonFunctions.$ne = function (a, b) { if (a === undefined) { return true; } return !areThingsEqual(a, b); }; comparisonFunctions.$in = function (a, b) { var i; if (!util.isArray(b)) { throw new Error("$in operator called with a non-array"); } for (i = 0; i < b.length; i += 1) { if (areThingsEqual(a, b[i])) { return true; } } return false; }; comparisonFunctions.$nin = function (a, b) { if (!util.isArray(b)) { throw new Error("$nin operator called with a non-array"); } return !comparisonFunctions.$in(a, b); }; comparisonFunctions.$regex = function (a, b) { if (!util.isRegExp(b)) { throw new Error("$regex operator called with non regular expression"); } if (typeof a !== 'string') { return false } else { return b.test(a); } }; comparisonFunctions.$exists = function (value, exists) { if (exists || exists === '') { // This will be true for all values of exists except false, null, undefined and 0 exists = true; // That's strange behaviour (we should only use true/false) but that's the way Mongo does it... } else { exists = false; } if (value === undefined) { return !exists } else { return exists; } }; // Specific to arrays comparisonFunctions.$size = function (obj, value) { if (!util.isArray(obj)) { return false; } if (value % 1 !== 0) { throw new Error("$size operator called without an integer"); } return (obj.length == value); }; comparisonFunctions.$elemMatch = function (obj, value) { if (!util.isArray(obj)) { return false; } var i = obj.length; var result = false; // Initialize result while (i--) { if (match(obj[i], value)) { // If match for array element, return true result = true; break; } } return result; }; arrayComparisonFunctions.$size = true; arrayComparisonFunctions.$elemMatch = true; /** * Match any of the subqueries * @param {Model} obj * @param {Array of Queries} query */ logicalOperators.$or = function (obj, query) { var i; if (!util.isArray(query)) { throw new Error("$or operator used without an array"); } for (i = 0; i < query.length; i += 1) { if (match(obj, query[i])) { return true; } } return false; }; /** * Match all of the subqueries * @param {Model} obj * @param {Array of Queries} query */ logicalOperators.$and = function (obj, query) { var i; if (!util.isArray(query)) { throw new Error("$and operator used without an array"); } for (i = 0; i < query.length; i += 1) { if (!match(obj, query[i])) { return false; } } return true; }; /** * Inverted match of the query * @param {Model} obj * @param {Query} query */ logicalOperators.$not = function (obj, query) { return !match(obj, query); }; /** * Use a function to match * @param {Model} obj * @param {Query} query */ logicalOperators.$where = function (obj, fn) { var result; if (!_.isFunction(fn)) { throw new Error("$where operator used without a function"); } result = fn.call(obj); if (!_.isBoolean(result)) { throw new Error("$where function must return boolean"); } return result; }; /** * Tell if a given document matches a query * @param {Object} obj Document to check * @param {Object} query */ function match (obj, query) { var queryKeys, queryKey, queryValue, i; // Primitive query against a primitive type // This is a bit of a hack since we construct an object with an arbitrary key only to dereference it later // But I don't have time for a cleaner implementation now if (isPrimitiveType(obj) || isPrimitiveType(query)) { return matchQueryPart({ needAKey: obj }, 'needAKey', query); } // Normal query queryKeys = Object.keys(query); for (i = 0; i < queryKeys.length; i += 1) { queryKey = queryKeys[i]; queryValue = query[queryKey]; if (queryKey[0] === '$') { if (!logicalOperators[queryKey]) { throw new Error("Unknown logical operator " + queryKey); } if (!logicalOperators[queryKey](obj, queryValue)) { return false; } } else { if (!matchQueryPart(obj, queryKey, queryValue)) { return false; } } } return true; }; /** * Match an object against a specific { key: value } part of a query * if the treatObjAsValue flag is set, don't try to match every part separately, but the array as a whole */ function matchQueryPart (obj, queryKey, queryValue, treatObjAsValue) { var objValue = getDotValue(obj, queryKey) , i, keys, firstChars, dollarFirstChars; // Check if the value is an array if we don't force a treatment as value if (util.isArray(objValue) && !treatObjAsValue) { // If the queryValue is an array, try to perform an exact match if (util.isArray(queryValue)) { return matchQueryPart(obj, queryKey, queryValue, true); } // Check if we are using an array-specific comparison function if (queryValue !== null && typeof queryValue === 'object' && !util.isRegExp(queryValue)) { keys = Object.keys(queryValue); for (i = 0; i < keys.length; i += 1) { if (arrayComparisonFunctions[keys[i]]) { return matchQueryPart(obj, queryKey, queryValue, true); } } } // If not, treat it as an array of { obj, query } where there needs to be at least one match for (i = 0; i < objValue.length; i += 1) { if (matchQueryPart({ k: objValue[i] }, 'k', queryValue)) { return true; } // k here could be any string } return false; } // queryValue is an actual object. Determine whether it contains comparison operators // or only normal fields. Mixed objects are not allowed if (queryValue !== null && typeof queryValue === 'object' && !util.isRegExp(queryValue) && !util.isArray(queryValue)) { keys = Object.keys(queryValue); firstChars = _.map(keys, function (item) { return item[0]; }); dollarFirstChars = _.filter(firstChars, function (c) { return c === '$'; }); if (dollarFirstChars.length !== 0 && dollarFirstChars.length !== firstChars.length) { throw new Error("You cannot mix operators and normal fields"); } // queryValue is an object of this form: { $comparisonOperator1: value1, ... } if (dollarFirstChars.length > 0) { for (i = 0; i < keys.length; i += 1) { if (!comparisonFunctions[keys[i]]) { throw new Error("Unknown comparison function " + keys[i]); } if (!comparisonFunctions[keys[i]](objValue, queryValue[keys[i]])) { return false; } } return true; } } // Using regular expressions with basic querying if (util.isRegExp(queryValue)) { return comparisonFunctions.$regex(objValue, queryValue); } // queryValue is either a native value or a normal object // Basic matching is possible if (!areThingsEqual(objValue, queryValue)) { return false; } return true; } // Interface module.exports.serialize = serialize; module.exports.deserialize = deserialize; module.exports.deepCopy = deepCopy; module.exports.checkObject = checkObject; module.exports.isPrimitiveType = isPrimitiveType; module.exports.modify = modify; module.exports.getDotValue = getDotValue; module.exports.match = match; module.exports.areThingsEqual = areThingsEqual; module.exports.compareThings = compareThings; ================================================ FILE: lib/persistence.js ================================================ /** * Handle every persistence-related task * The interface Datastore expects to be implemented is * * Persistence.loadDatabase(callback) and callback has signature err * * Persistence.persistNewState(newDocs, callback) where newDocs is an array of documents and callback has signature err */ var storage = require('./storage') , path = require('path') , model = require('./model') , async = require('async') , customUtils = require('./customUtils') , Index = require('./indexes') ; /** * Create a new Persistence object for database options.db * @param {Datastore} options.db * @param {Boolean} options.nodeWebkitAppName Optional, specify the name of your NW app if you want options.filename to be relative to the directory where * Node Webkit stores application data such as cookies and local storage (the best place to store data in my opinion) */ function Persistence (options) { var i, j, randomString; this.db = options.db; this.inMemoryOnly = this.db.inMemoryOnly; this.filename = this.db.filename; this.corruptAlertThreshold = options.corruptAlertThreshold !== undefined ? options.corruptAlertThreshold : 0.1; if (!this.inMemoryOnly && this.filename && this.filename.charAt(this.filename.length - 1) === '~') { throw new Error("The datafile name can't end with a ~, which is reserved for crash safe backup files"); } // After serialization and before deserialization hooks with some basic sanity checks if (options.afterSerialization && !options.beforeDeserialization) { throw new Error("Serialization hook defined but deserialization hook undefined, cautiously refusing to start NeDB to prevent dataloss"); } if (!options.afterSerialization && options.beforeDeserialization) { throw new Error("Serialization hook undefined but deserialization hook defined, cautiously refusing to start NeDB to prevent dataloss"); } this.afterSerialization = options.afterSerialization || function (s) { return s; }; this.beforeDeserialization = options.beforeDeserialization || function (s) { return s; }; for (i = 1; i < 30; i += 1) { for (j = 0; j < 10; j += 1) { randomString = customUtils.uid(i); if (this.beforeDeserialization(this.afterSerialization(randomString)) !== randomString) { throw new Error("beforeDeserialization is not the reverse of afterSerialization, cautiously refusing to start NeDB to prevent dataloss"); } } } // For NW apps, store data in the same directory where NW stores application data if (this.filename && options.nodeWebkitAppName) { console.log("=================================================================="); console.log("WARNING: The nodeWebkitAppName option is deprecated"); console.log("To get the path to the directory where Node Webkit stores the data"); console.log("for your app, use the internal nw.gui module like this"); console.log("require('nw.gui').App.dataPath"); console.log("See https://github.com/rogerwang/node-webkit/issues/500"); console.log("=================================================================="); this.filename = Persistence.getNWAppFilename(options.nodeWebkitAppName, this.filename); } }; /** * Check if a directory exists and create it on the fly if it is not the case * cb is optional, signature: err */ Persistence.ensureDirectoryExists = function (dir, cb) { var callback = cb || function () {} ; storage.mkdirp(dir, function (err) { return callback(err); }); }; /** * Return the path the datafile if the given filename is relative to the directory where Node Webkit stores * data for this application. Probably the best place to store data */ Persistence.getNWAppFilename = function (appName, relativeFilename) { var home; switch (process.platform) { case 'win32': case 'win64': home = process.env.LOCALAPPDATA || process.env.APPDATA; if (!home) { throw new Error("Couldn't find the base application data folder"); } home = path.join(home, appName); break; case 'darwin': home = process.env.HOME; if (!home) { throw new Error("Couldn't find the base application data directory"); } home = path.join(home, 'Library', 'Application Support', appName); break; case 'linux': home = process.env.HOME; if (!home) { throw new Error("Couldn't find the base application data directory"); } home = path.join(home, '.config', appName); break; default: throw new Error("Can't use the Node Webkit relative path for platform " + process.platform); break; } return path.join(home, 'nedb-data', relativeFilename); } /** * Persist cached database * This serves as a compaction function since the cache always contains only the number of documents in the collection * while the data file is append-only so it may grow larger * @param {Function} cb Optional callback, signature: err */ Persistence.prototype.persistCachedDatabase = function (cb) { var callback = cb || function () {} , toPersist = '' , self = this ; if (this.inMemoryOnly) { return callback(null); } this.db.getAllData().forEach(function (doc) { toPersist += self.afterSerialization(model.serialize(doc)) + '\n'; }); Object.keys(this.db.indexes).forEach(function (fieldName) { if (fieldName != "_id") { // The special _id index is managed by datastore.js, the others need to be persisted toPersist += self.afterSerialization(model.serialize({ $$indexCreated: { fieldName: fieldName, unique: self.db.indexes[fieldName].unique, sparse: self.db.indexes[fieldName].sparse }})) + '\n'; } }); storage.crashSafeWriteFile(this.filename, toPersist, function (err) { if (err) { return callback(err); } self.db.emit('compaction.done'); return callback(null); }); }; /** * Queue a rewrite of the datafile */ Persistence.prototype.compactDatafile = function () { this.db.executor.push({ this: this, fn: this.persistCachedDatabase, arguments: [] }); }; /** * Set automatic compaction every interval ms * @param {Number} interval in milliseconds, with an enforced minimum of 5 seconds */ Persistence.prototype.setAutocompactionInterval = function (interval) { var self = this , minInterval = 5000 , realInterval = Math.max(interval || 0, minInterval) ; this.stopAutocompaction(); this.autocompactionIntervalId = setInterval(function () { self.compactDatafile(); }, realInterval); }; /** * Stop autocompaction (do nothing if autocompaction was not running) */ Persistence.prototype.stopAutocompaction = function () { if (this.autocompactionIntervalId) { clearInterval(this.autocompactionIntervalId); } }; /** * Persist new state for the given newDocs (can be insertion, update or removal) * Use an append-only format * @param {Array} newDocs Can be empty if no doc was updated/removed * @param {Function} cb Optional, signature: err */ Persistence.prototype.persistNewState = function (newDocs, cb) { var self = this , toPersist = '' , callback = cb || function () {} ; // In-memory only datastore if (self.inMemoryOnly) { return callback(null); } newDocs.forEach(function (doc) { toPersist += self.afterSerialization(model.serialize(doc)) + '\n'; }); if (toPersist.length === 0) { return callback(null); } storage.appendFile(self.filename, toPersist, 'utf8', function (err) { return callback(err); }); }; /** * From a database's raw data, return the corresponding * machine understandable collection */ Persistence.prototype.treatRawData = function (rawData) { var data = rawData.split('\n') , dataById = {} , tdata = [] , i , indexes = {} , corruptItems = -1 // Last line of every data file is usually blank so not really corrupt ; for (i = 0; i < data.length; i += 1) { var doc; try { doc = model.deserialize(this.beforeDeserialization(data[i])); if (doc._id) { if (doc.$$deleted === true) { delete dataById[doc._id]; } else { dataById[doc._id] = doc; } } else if (doc.$$indexCreated && doc.$$indexCreated.fieldName != undefined) { indexes[doc.$$indexCreated.fieldName] = doc.$$indexCreated; } else if (typeof doc.$$indexRemoved === "string") { delete indexes[doc.$$indexRemoved]; } } catch (e) { corruptItems += 1; } } // A bit lenient on corruption if (data.length > 0 && corruptItems / data.length > this.corruptAlertThreshold) { throw new Error("More than " + Math.floor(100 * this.corruptAlertThreshold) + "% of the data file is corrupt, the wrong beforeDeserialization hook may be used. Cautiously refusing to start NeDB to prevent dataloss"); } Object.keys(dataById).forEach(function (k) { tdata.push(dataById[k]); }); return { data: tdata, indexes: indexes }; }; /** * Load the database * 1) Create all indexes * 2) Insert all data * 3) Compact the database * This means pulling data out of the data file or creating it if it doesn't exist * Also, all data is persisted right away, which has the effect of compacting the database file * This operation is very quick at startup for a big collection (60ms for ~10k docs) * @param {Function} cb Optional callback, signature: err */ Persistence.prototype.loadDatabase = function (cb) { var callback = cb || function () {} , self = this ; self.db.resetIndexes(); // In-memory only datastore if (self.inMemoryOnly) { return callback(null); } async.waterfall([ function (cb) { Persistence.ensureDirectoryExists(path.dirname(self.filename), function (err) { storage.ensureDatafileIntegrity(self.filename, function (err) { storage.readFile(self.filename, 'utf8', function (err, rawData) { if (err) { return cb(err); } try { var treatedData = self.treatRawData(rawData); } catch (e) { return cb(e); } // Recreate all indexes in the datafile Object.keys(treatedData.indexes).forEach(function (key) { self.db.indexes[key] = new Index(treatedData.indexes[key]); }); // Fill cached database (i.e. all indexes) with data try { self.db.resetIndexes(treatedData.data); } catch (e) { self.db.resetIndexes(); // Rollback any index which didn't fail return cb(e); } self.db.persistence.persistCachedDatabase(cb); }); }); }); } ], function (err) { if (err) { return callback(err); } self.db.executor.processBuffer(); return callback(null); }); }; // Interface module.exports = Persistence; ================================================ FILE: lib/storage.js ================================================ /** * Way data is stored for this database * For a Node.js/Node Webkit database it's the file system * For a browser-side database it's localforage which chooses the best option depending on user browser (IndexedDB then WebSQL then localStorage) * * This version is the Node.js/Node Webkit version * It's essentially fs, mkdirp and crash safe write and read functions */ var fs = require('fs') , mkdirp = require('mkdirp') , async = require('async') , path = require('path') , storage = {} ; storage.exists = fs.exists; storage.rename = fs.rename; storage.writeFile = fs.writeFile; storage.unlink = fs.unlink; storage.appendFile = fs.appendFile; storage.readFile = fs.readFile; storage.mkdirp = mkdirp; /** * Explicit name ... */ storage.ensureFileDoesntExist = function (file, callback) { storage.exists(file, function (exists) { if (!exists) { return callback(null); } storage.unlink(file, function (err) { return callback(err); }); }); }; /** * Flush data in OS buffer to storage if corresponding option is set * @param {String} options.filename * @param {Boolean} options.isDir Optional, defaults to false * If options is a string, it is assumed that the flush of the file (not dir) called options was requested */ storage.flushToStorage = function (options, callback) { var filename, flags; if (typeof options === 'string') { filename = options; flags = 'r+'; } else { filename = options.filename; flags = options.isDir ? 'r' : 'r+'; } // Windows can't fsync (FlushFileBuffers) directories. We can live with this as it cannot cause 100% dataloss // except in the very rare event of the first time database is loaded and a crash happens if (flags === 'r' && (process.platform === 'win32' || process.platform === 'win64')) { return callback(null); } fs.open(filename, flags, function (err, fd) { if (err) { return callback(err); } fs.fsync(fd, function (errFS) { fs.close(fd, function (errC) { if (errFS || errC) { var e = new Error('Failed to flush to storage'); e.errorOnFsync = errFS; e.errorOnClose = errC; return callback(e); } else { return callback(null); } }); }); }); }; /** * Fully write or rewrite the datafile, immune to crashes during the write operation (data will not be lost) * @param {String} filename * @param {String} data * @param {Function} cb Optional callback, signature: err */ storage.crashSafeWriteFile = function (filename, data, cb) { var callback = cb || function () {} , tempFilename = filename + '~'; async.waterfall([ async.apply(storage.flushToStorage, { filename: path.dirname(filename), isDir: true }) , function (cb) { storage.exists(filename, function (exists) { if (exists) { storage.flushToStorage(filename, function (err) { return cb(err); }); } else { return cb(); } }); } , function (cb) { storage.writeFile(tempFilename, data, function (err) { return cb(err); }); } , async.apply(storage.flushToStorage, tempFilename) , function (cb) { storage.rename(tempFilename, filename, function (err) { return cb(err); }); } , async.apply(storage.flushToStorage, { filename: path.dirname(filename), isDir: true }) ], function (err) { return callback(err); }) }; /** * Ensure the datafile contains all the data, even if there was a crash during a full file write * @param {String} filename * @param {Function} callback signature: err */ storage.ensureDatafileIntegrity = function (filename, callback) { var tempFilename = filename + '~'; storage.exists(filename, function (filenameExists) { // Write was successful if (filenameExists) { return callback(null); } storage.exists(tempFilename, function (oldFilenameExists) { // New database if (!oldFilenameExists) { return storage.writeFile(filename, '', 'utf8', function (err) { callback(err); }); } // Write failed, use old version storage.rename(tempFilename, filename, function (err) { return callback(err); }); }); }); }; // Interface module.exports = storage; ================================================ FILE: package.json ================================================ { "name": "nedb", "version": "1.8.0", "author": { "name": "Louis Chatriot", "email": "louis.chatriot@gmail.com" }, "contributors": [ "Louis Chatriot" ], "description": "File-based embedded data store for node.js", "keywords": [ "database", "datastore", "embedded" ], "homepage": "https://github.com/louischatriot/nedb", "repository": { "type": "git", "url": "git@github.com:louischatriot/nedb.git" }, "dependencies": { "async": "0.2.10", "binary-search-tree": "0.2.5", "localforage": "^1.3.0", "mkdirp": "~0.5.1", "underscore": "~1.4.4" }, "devDependencies": { "chai": "^3.2.0", "mocha": "1.4.x", "request": "2.9.x", "sinon": "1.3.x", "exec-time": "0.0.2", "commander": "1.1.1" }, "scripts": { "test": "./node_modules/.bin/mocha --reporter spec --timeout 10000" }, "main": "index", "browser": { "./lib/customUtils.js": "./browser-version/browser-specific/lib/customUtils.js", "./lib/storage.js": "./browser-version/browser-specific/lib/storage.js" }, "license": "SEE LICENSE IN LICENSE" } ================================================ FILE: test/cursor.test.js ================================================ var should = require('chai').should() , assert = require('chai').assert , testDb = 'workspace/test.db' , fs = require('fs') , path = require('path') , _ = require('underscore') , async = require('async') , model = require('../lib/model') , Datastore = require('../lib/datastore') , Persistence = require('../lib/persistence') , Cursor = require('../lib/cursor') ; describe('Cursor', function () { var d; beforeEach(function (done) { d = new Datastore({ filename: testDb }); d.filename.should.equal(testDb); d.inMemoryOnly.should.equal(false); async.waterfall([ function (cb) { Persistence.ensureDirectoryExists(path.dirname(testDb), function () { fs.exists(testDb, function (exists) { if (exists) { fs.unlink(testDb, cb); } else { return cb(); } }); }); } , function (cb) { d.loadDatabase(function (err) { assert.isNull(err); d.getAllData().length.should.equal(0); return cb(); }); } ], done); }); describe('Without sorting', function () { beforeEach(function (done) { d.insert({ age: 5 }, function (err) { d.insert({ age: 57 }, function (err) { d.insert({ age: 52 }, function (err) { d.insert({ age: 23 }, function (err) { d.insert({ age: 89 }, function (err) { return done(); }); }); }); }); }); }); it('Without query, an empty query or a simple query and no skip or limit', function (done) { async.waterfall([ function (cb) { var cursor = new Cursor(d); cursor.exec(function (err, docs) { assert.isNull(err); docs.length.should.equal(5); _.filter(docs, function(doc) { return doc.age === 5; })[0].age.should.equal(5); _.filter(docs, function(doc) { return doc.age === 57; })[0].age.should.equal(57); _.filter(docs, function(doc) { return doc.age === 52; })[0].age.should.equal(52); _.filter(docs, function(doc) { return doc.age === 23; })[0].age.should.equal(23); _.filter(docs, function(doc) { return doc.age === 89; })[0].age.should.equal(89); cb(); }); } , function (cb) { var cursor = new Cursor(d, {}); cursor.exec(function (err, docs) { assert.isNull(err); docs.length.should.equal(5); _.filter(docs, function(doc) { return doc.age === 5; })[0].age.should.equal(5); _.filter(docs, function(doc) { return doc.age === 57; })[0].age.should.equal(57); _.filter(docs, function(doc) { return doc.age === 52; })[0].age.should.equal(52); _.filter(docs, function(doc) { return doc.age === 23; })[0].age.should.equal(23); _.filter(docs, function(doc) { return doc.age === 89; })[0].age.should.equal(89); cb(); }); } , function (cb) { var cursor = new Cursor(d, { age: { $gt: 23 } }); cursor.exec(function (err, docs) { assert.isNull(err); docs.length.should.equal(3); _.filter(docs, function(doc) { return doc.age === 57; })[0].age.should.equal(57); _.filter(docs, function(doc) { return doc.age === 52; })[0].age.should.equal(52); _.filter(docs, function(doc) { return doc.age === 89; })[0].age.should.equal(89); cb(); }); } ], done); }); it('With an empty collection', function (done) { async.waterfall([ function (cb) { d.remove({}, { multi: true }, function(err) { return cb(err); }) } , function (cb) { var cursor = new Cursor(d); cursor.exec(function (err, docs) { assert.isNull(err); docs.length.should.equal(0); cb(); }); } ], done); }); it('With a limit', function (done) { var cursor = new Cursor(d); cursor.limit(3); cursor.exec(function (err, docs) { assert.isNull(err); docs.length.should.equal(3); // No way to predict which results are returned of course ... done(); }); }); it('With a skip', function (done) { var cursor = new Cursor(d); cursor.skip(2).exec(function (err, docs) { assert.isNull(err); docs.length.should.equal(3); // No way to predict which results are returned of course ... done(); }); }); it('With a limit and a skip and method chaining', function (done) { var cursor = new Cursor(d); cursor.limit(4).skip(3); // Only way to know that the right number of results was skipped is if limit + skip > number of results cursor.exec(function (err, docs) { assert.isNull(err); docs.length.should.equal(2); // No way to predict which results are returned of course ... done(); }); }); }); // ===== End of 'Without sorting' ===== describe('Sorting of the results', function () { beforeEach(function (done) { // We don't know the order in which docs wil be inserted but we ensure correctness by testing both sort orders d.insert({ age: 5 }, function (err) { d.insert({ age: 57 }, function (err) { d.insert({ age: 52 }, function (err) { d.insert({ age: 23 }, function (err) { d.insert({ age: 89 }, function (err) { return done(); }); }); }); }); }); }); it('Using one sort', function (done) { var cursor, i; cursor = new Cursor(d, {}); cursor.sort({ age: 1 }); cursor.exec(function (err, docs) { assert.isNull(err); // Results are in ascending order for (i = 0; i < docs.length - 1; i += 1) { assert(docs[i].age < docs[i + 1].age) } cursor.sort({ age: -1 }); cursor.exec(function (err, docs) { assert.isNull(err); // Results are in descending order for (i = 0; i < docs.length - 1; i += 1) { assert(docs[i].age > docs[i + 1].age) } done(); }); }); }); it("Sorting strings with custom string comparison function", function (done) { var db = new Datastore({ inMemoryOnly: true, autoload: true , compareStrings: function (a, b) { return a.length - b.length; } }); db.insert({ name: 'alpha' }); db.insert({ name: 'charlie' }); db.insert({ name: 'zulu' }); db.find({}).sort({ name: 1 }).exec(function (err, docs) { _.pluck(docs, 'name')[0].should.equal('zulu'); _.pluck(docs, 'name')[1].should.equal('alpha'); _.pluck(docs, 'name')[2].should.equal('charlie'); delete db.compareStrings; db.find({}).sort({ name: 1 }).exec(function (err, docs) { _.pluck(docs, 'name')[0].should.equal('alpha'); _.pluck(docs, 'name')[1].should.equal('charlie'); _.pluck(docs, 'name')[2].should.equal('zulu'); done(); }); }); }); it('With an empty collection', function (done) { async.waterfall([ function (cb) { d.remove({}, { multi: true }, function(err) { return cb(err); }) } , function (cb) { var cursor = new Cursor(d); cursor.sort({ age: 1 }); cursor.exec(function (err, docs) { assert.isNull(err); docs.length.should.equal(0); cb(); }); } ], done); }); it('Ability to chain sorting and exec', function (done) { var i; async.waterfall([ function (cb) { var cursor = new Cursor(d); cursor.sort({ age: 1 }).exec(function (err, docs) { assert.isNull(err); // Results are in ascending order for (i = 0; i < docs.length - 1; i += 1) { assert(docs[i].age < docs[i + 1].age) } cb(); }); } , function (cb) { var cursor = new Cursor(d); cursor.sort({ age: -1 }).exec(function (err, docs) { assert.isNull(err); // Results are in descending order for (i = 0; i < docs.length - 1; i += 1) { assert(docs[i].age > docs[i + 1].age) } cb(); }); } ], done); }); it('Using limit and sort', function (done) { var i; async.waterfall([ function (cb) { var cursor = new Cursor(d); cursor.sort({ age: 1 }).limit(3).exec(function (err, docs) { assert.isNull(err); docs.length.should.equal(3); docs[0].age.should.equal(5); docs[1].age.should.equal(23); docs[2].age.should.equal(52); cb(); }); } , function (cb) { var cursor = new Cursor(d); cursor.sort({ age: -1 }).limit(2).exec(function (err, docs) { assert.isNull(err); docs.length.should.equal(2); docs[0].age.should.equal(89); docs[1].age.should.equal(57); cb(); }); } ], done); }); it('Using a limit higher than total number of docs shouldnt cause an error', function (done) { var i; async.waterfall([ function (cb) { var cursor = new Cursor(d); cursor.sort({ age: 1 }).limit(7).exec(function (err, docs) { assert.isNull(err); docs.length.should.equal(5); docs[0].age.should.equal(5); docs[1].age.should.equal(23); docs[2].age.should.equal(52); docs[3].age.should.equal(57); docs[4].age.should.equal(89); cb(); }); } ], done); }); it('Using limit and skip with sort', function (done) { var i; async.waterfall([ function (cb) { var cursor = new Cursor(d); cursor.sort({ age: 1 }).limit(1).skip(2).exec(function (err, docs) { assert.isNull(err); docs.length.should.equal(1); docs[0].age.should.equal(52); cb(); }); } , function (cb) { var cursor = new Cursor(d); cursor.sort({ age: 1 }).limit(3).skip(1).exec(function (err, docs) { assert.isNull(err); docs.length.should.equal(3); docs[0].age.should.equal(23); docs[1].age.should.equal(52); docs[2].age.should.equal(57); cb(); }); } , function (cb) { var cursor = new Cursor(d); cursor.sort({ age: -1 }).limit(2).skip(2).exec(function (err, docs) { assert.isNull(err); docs.length.should.equal(2); docs[0].age.should.equal(52); docs[1].age.should.equal(23); cb(); }); } ], done); }); it('Using too big a limit and a skip with sort', function (done) { var i; async.waterfall([ function (cb) { var cursor = new Cursor(d); cursor.sort({ age: 1 }).limit(8).skip(2).exec(function (err, docs) { assert.isNull(err); docs.length.should.equal(3); docs[0].age.should.equal(52); docs[1].age.should.equal(57); docs[2].age.should.equal(89); cb(); }); } ], done); }); it('Using too big a skip with sort should return no result', function (done) { var i; async.waterfall([ function (cb) { var cursor = new Cursor(d); cursor.sort({ age: 1 }).skip(5).exec(function (err, docs) { assert.isNull(err); docs.length.should.equal(0); cb(); }); } , function (cb) { var cursor = new Cursor(d); cursor.sort({ age: 1 }).skip(7).exec(function (err, docs) { assert.isNull(err); docs.length.should.equal(0); cb(); }); } , function (cb) { var cursor = new Cursor(d); cursor.sort({ age: 1 }).limit(3).skip(7).exec(function (err, docs) { assert.isNull(err); docs.length.should.equal(0); cb(); }); } , function (cb) { var cursor = new Cursor(d); cursor.sort({ age: 1 }).limit(6).skip(7).exec(function (err, docs) { assert.isNull(err); docs.length.should.equal(0); cb(); }); } ], done); }); it('Sorting strings', function (done) { async.waterfall([ function (cb) { d.remove({}, { multi: true }, function (err) { if (err) { return cb(err); } d.insert({ name: 'jako'}, function () { d.insert({ name: 'jakeb' }, function () { d.insert({ name: 'sue' }, function () { return cb(); }); }); }); }); } , function (cb) { var cursor = new Cursor(d, {}); cursor.sort({ name: 1 }).exec(function (err, docs) { docs.length.should.equal(3); docs[0].name.should.equal('jakeb'); docs[1].name.should.equal('jako'); docs[2].name.should.equal('sue'); return cb(); }); } , function (cb) { var cursor = new Cursor(d, {}); cursor.sort({ name: -1 }).exec(function (err, docs) { docs.length.should.equal(3); docs[0].name.should.equal('sue'); docs[1].name.should.equal('jako'); docs[2].name.should.equal('jakeb'); return cb(); }); } ], done); }); it('Sorting nested fields with dates', function (done) { var doc1, doc2, doc3; async.waterfall([ function (cb) { d.remove({}, { multi: true }, function (err) { if (err) { return cb(err); } d.insert({ event: { recorded: new Date(400) } }, function (err, _doc1) { doc1 = _doc1; d.insert({ event: { recorded: new Date(60000) } }, function (err, _doc2) { doc2 = _doc2; d.insert({ event: { recorded: new Date(32) } }, function (err, _doc3) { doc3 = _doc3; return cb(); }); }); }); }); } , function (cb) { var cursor = new Cursor(d, {}); cursor.sort({ "event.recorded": 1 }).exec(function (err, docs) { docs.length.should.equal(3); docs[0]._id.should.equal(doc3._id); docs[1]._id.should.equal(doc1._id); docs[2]._id.should.equal(doc2._id); return cb(); }); } , function (cb) { var cursor = new Cursor(d, {}); cursor.sort({ "event.recorded": -1 }).exec(function (err, docs) { docs.length.should.equal(3); docs[0]._id.should.equal(doc2._id); docs[1]._id.should.equal(doc1._id); docs[2]._id.should.equal(doc3._id); return cb(); }); } ], done); }); it('Sorting when some fields are undefined', function (done) { async.waterfall([ function (cb) { d.remove({}, { multi: true }, function (err) { if (err) { return cb(err); } d.insert({ name: 'jako', other: 2 }, function () { d.insert({ name: 'jakeb', other: 3 }, function () { d.insert({ name: 'sue' }, function () { d.insert({ name: 'henry', other: 4 }, function () { return cb(); }); }); }); }); }); } , function (cb) { var cursor = new Cursor(d, {}); cursor.sort({ other: 1 }).exec(function (err, docs) { docs.length.should.equal(4); docs[0].name.should.equal('sue'); assert.isUndefined(docs[0].other); docs[1].name.should.equal('jako'); docs[1].other.should.equal(2); docs[2].name.should.equal('jakeb'); docs[2].other.should.equal(3); docs[3].name.should.equal('henry'); docs[3].other.should.equal(4); return cb(); }); } , function (cb) { var cursor = new Cursor(d, { name: { $in: [ 'suzy', 'jakeb', 'jako' ] } }); cursor.sort({ other: -1 }).exec(function (err, docs) { docs.length.should.equal(2); docs[0].name.should.equal('jakeb'); docs[0].other.should.equal(3); docs[1].name.should.equal('jako'); docs[1].other.should.equal(2); return cb(); }); } ], done); }); it('Sorting when all fields are undefined', function (done) { async.waterfall([ function (cb) { d.remove({}, { multi: true }, function (err) { if (err) { return cb(err); } d.insert({ name: 'jako'}, function () { d.insert({ name: 'jakeb' }, function () { d.insert({ name: 'sue' }, function () { return cb(); }); }); }); }); } , function (cb) { var cursor = new Cursor(d, {}); cursor.sort({ other: 1 }).exec(function (err, docs) { docs.length.should.equal(3); return cb(); }); } , function (cb) { var cursor = new Cursor(d, { name: { $in: [ 'sue', 'jakeb', 'jakob' ] } }); cursor.sort({ other: -1 }).exec(function (err, docs) { docs.length.should.equal(2); return cb(); }); } ], done); }); it('Multiple consecutive sorts', function(done) { async.waterfall([ function (cb) { d.remove({}, { multi: true }, function (err) { if (err) { return cb(err); } d.insert({ name: 'jako', age: 43, nid: 1 }, function () { d.insert({ name: 'jakeb', age: 43, nid: 2 }, function () { d.insert({ name: 'sue', age: 12, nid: 3 }, function () { d.insert({ name: 'zoe', age: 23, nid: 4 }, function () { d.insert({ name: 'jako', age: 35, nid: 5 }, function () { return cb(); }); }); }); }); }); }); } , function (cb) { var cursor = new Cursor(d, {}); cursor.sort({ name: 1, age: -1 }).exec(function (err, docs) { docs.length.should.equal(5); docs[0].nid.should.equal(2); docs[1].nid.should.equal(1); docs[2].nid.should.equal(5); docs[3].nid.should.equal(3); docs[4].nid.should.equal(4); return cb(); }); } , function (cb) { var cursor = new Cursor(d, {}); cursor.sort({ name: 1, age: 1 }).exec(function (err, docs) { docs.length.should.equal(5); docs[0].nid.should.equal(2); docs[1].nid.should.equal(5); docs[2].nid.should.equal(1); docs[3].nid.should.equal(3); docs[4].nid.should.equal(4); return cb(); }); } , function (cb) { var cursor = new Cursor(d, {}); cursor.sort({ age: 1, name: 1 }).exec(function (err, docs) { docs.length.should.equal(5); docs[0].nid.should.equal(3); docs[1].nid.should.equal(4); docs[2].nid.should.equal(5); docs[3].nid.should.equal(2); docs[4].nid.should.equal(1); return cb(); }); } , function (cb) { var cursor = new Cursor(d, {}); cursor.sort({ age: 1, name: -1 }).exec(function (err, docs) { docs.length.should.equal(5); docs[0].nid.should.equal(3); docs[1].nid.should.equal(4); docs[2].nid.should.equal(5); docs[3].nid.should.equal(1); docs[4].nid.should.equal(2); return cb(); }); } ], done); }); it('Similar data, multiple consecutive sorts', function(done) { var i, j, id , companies = [ 'acme', 'milkman', 'zoinks' ] , entities = [] ; async.waterfall([ function (cb) { d.remove({}, { multi: true }, function (err) { if (err) { return cb(err); } id = 1; for (i = 0; i < companies.length; i++) { for (j = 5; j <= 100; j += 5) { entities.push({ company: companies[i], cost: j, nid: id }); id++; } } async.each(entities, function(entity, callback) { d.insert(entity, function() { callback(); }); }, function(err) { return cb(); }); }); } , function (cb) { var cursor = new Cursor(d, {}); cursor.sort({ company: 1, cost: 1 }).exec(function (err, docs) { docs.length.should.equal(60); for (var i = 0; i < docs.length; i++) { docs[i].nid.should.equal(i+1); }; return cb(); }); } ], done); }); }); // ===== End of 'Sorting' ===== describe('Projections', function () { var doc1, doc2, doc3, doc4, doc0; beforeEach(function (done) { // We don't know the order in which docs wil be inserted but we ensure correctness by testing both sort orders d.insert({ age: 5, name: 'Jo', planet: 'B', toys: { bebe: true, ballon: 'much' } }, function (err, _doc0) { doc0 = _doc0; d.insert({ age: 57, name: 'Louis', planet: 'R', toys: { ballon: 'yeah', bebe: false } }, function (err, _doc1) { doc1 = _doc1; d.insert({ age: 52, name: 'Grafitti', planet: 'C', toys: { bebe: 'kind of' } }, function (err, _doc2) { doc2 = _doc2; d.insert({ age: 23, name: 'LM', planet: 'S' }, function (err, _doc3) { doc3 = _doc3; d.insert({ age: 89, planet: 'Earth' }, function (err, _doc4) { doc4 = _doc4; return done(); }); }); }); }); }); }); it('Takes all results if no projection or empty object given', function (done) { var cursor = new Cursor(d, {}); cursor.sort({ age: 1 }); // For easier finding cursor.exec(function (err, docs) { assert.isNull(err); docs.length.should.equal(5); assert.deepEqual(docs[0], doc0); assert.deepEqual(docs[1], doc3); assert.deepEqual(docs[2], doc2); assert.deepEqual(docs[3], doc1); assert.deepEqual(docs[4], doc4); cursor.projection({}); cursor.exec(function (err, docs) { assert.isNull(err); docs.length.should.equal(5); assert.deepEqual(docs[0], doc0); assert.deepEqual(docs[1], doc3); assert.deepEqual(docs[2], doc2); assert.deepEqual(docs[3], doc1); assert.deepEqual(docs[4], doc4); done(); }); }); }); it('Can take only the expected fields', function (done) { var cursor = new Cursor(d, {}); cursor.sort({ age: 1 }); // For easier finding cursor.projection({ age: 1, name: 1 }); cursor.exec(function (err, docs) { assert.isNull(err); docs.length.should.equal(5); // Takes the _id by default assert.deepEqual(docs[0], { age: 5, name: 'Jo', _id: doc0._id }); assert.deepEqual(docs[1], { age: 23, name: 'LM', _id: doc3._id }); assert.deepEqual(docs[2], { age: 52, name: 'Grafitti', _id: doc2._id }); assert.deepEqual(docs[3], { age: 57, name: 'Louis', _id: doc1._id }); assert.deepEqual(docs[4], { age: 89, _id: doc4._id }); // No problems if one field to take doesn't exist cursor.projection({ age: 1, name: 1, _id: 0 }); cursor.exec(function (err, docs) { assert.isNull(err); docs.length.should.equal(5); assert.deepEqual(docs[0], { age: 5, name: 'Jo' }); assert.deepEqual(docs[1], { age: 23, name: 'LM' }); assert.deepEqual(docs[2], { age: 52, name: 'Grafitti' }); assert.deepEqual(docs[3], { age: 57, name: 'Louis' }); assert.deepEqual(docs[4], { age: 89 }); // No problems if one field to take doesn't exist done(); }); }); }); it('Can omit only the expected fields', function (done) { var cursor = new Cursor(d, {}); cursor.sort({ age: 1 }); // For easier finding cursor.projection({ age: 0, name: 0 }); cursor.exec(function (err, docs) { assert.isNull(err); docs.length.should.equal(5); // Takes the _id by default assert.deepEqual(docs[0], { planet: 'B', _id: doc0._id, toys: { bebe: true, ballon: 'much' } }); assert.deepEqual(docs[1], { planet: 'S', _id: doc3._id }); assert.deepEqual(docs[2], { planet: 'C', _id: doc2._id, toys: { bebe: 'kind of' } }); assert.deepEqual(docs[3], { planet: 'R', _id: doc1._id, toys: { bebe: false, ballon: 'yeah' } }); assert.deepEqual(docs[4], { planet: 'Earth', _id: doc4._id }); cursor.projection({ age: 0, name: 0, _id: 0 }); cursor.exec(function (err, docs) { assert.isNull(err); docs.length.should.equal(5); assert.deepEqual(docs[0], { planet: 'B', toys: { bebe: true, ballon: 'much' } }); assert.deepEqual(docs[1], { planet: 'S' }); assert.deepEqual(docs[2], { planet: 'C', toys: { bebe: 'kind of' } }); assert.deepEqual(docs[3], { planet: 'R', toys: { bebe: false, ballon: 'yeah' } }); assert.deepEqual(docs[4], { planet: 'Earth' }); done(); }); }); }); it('Cannot use both modes except for _id', function (done) { var cursor = new Cursor(d, {}); cursor.sort({ age: 1 }); // For easier finding cursor.projection({ age: 1, name: 0 }); cursor.exec(function (err, docs) { assert.isNotNull(err); assert.isUndefined(docs); cursor.projection({ age: 1, _id: 0 }); cursor.exec(function (err, docs) { assert.isNull(err); assert.deepEqual(docs[0], { age: 5 }); assert.deepEqual(docs[1], { age: 23 }); assert.deepEqual(docs[2], { age: 52 }); assert.deepEqual(docs[3], { age: 57 }); assert.deepEqual(docs[4], { age: 89 }); cursor.projection({ age: 0, toys: 0, planet: 0, _id: 1 }); cursor.exec(function (err, docs) { assert.isNull(err); assert.deepEqual(docs[0], { name: 'Jo', _id: doc0._id }); assert.deepEqual(docs[1], { name: 'LM', _id: doc3._id }); assert.deepEqual(docs[2], { name: 'Grafitti', _id: doc2._id }); assert.deepEqual(docs[3], { name: 'Louis', _id: doc1._id }); assert.deepEqual(docs[4], { _id: doc4._id }); done(); }); }); }); }); it("Projections on embedded documents - omit type", function (done) { var cursor = new Cursor(d, {}); cursor.sort({ age: 1 }); // For easier finding cursor.projection({ name: 0, planet: 0, 'toys.bebe': 0, _id: 0 }); cursor.exec(function (err, docs) { assert.deepEqual(docs[0], { age: 5, toys: { ballon: 'much' } }); assert.deepEqual(docs[1], { age: 23 }); assert.deepEqual(docs[2], { age: 52, toys: {} }); assert.deepEqual(docs[3], { age: 57, toys: { ballon: 'yeah' } }); assert.deepEqual(docs[4], { age: 89 }); done(); }); }); it("Projections on embedded documents - pick type", function (done) { var cursor = new Cursor(d, {}); cursor.sort({ age: 1 }); // For easier finding cursor.projection({ name: 1, 'toys.ballon': 1, _id: 0 }); cursor.exec(function (err, docs) { assert.deepEqual(docs[0], { name: 'Jo', toys: { ballon: 'much' } }); assert.deepEqual(docs[1], { name: 'LM' }); assert.deepEqual(docs[2], { name: 'Grafitti' }); assert.deepEqual(docs[3], { name: 'Louis', toys: { ballon: 'yeah' } }); assert.deepEqual(docs[4], {}); done(); }); }); }); // ==== End of 'Projections' ==== }); ================================================ FILE: test/customUtil.test.js ================================================ var should = require('chai').should() , assert = require('chai').assert , customUtils = require('../lib/customUtils') , fs = require('fs') ; describe('customUtils', function () { describe('uid', function () { it('Generates a string of the expected length', function () { customUtils.uid(3).length.should.equal(3); customUtils.uid(16).length.should.equal(16); customUtils.uid(42).length.should.equal(42); customUtils.uid(1000).length.should.equal(1000); }); // Very small probability of conflict it('Generated uids should not be the same', function () { customUtils.uid(56).should.not.equal(customUtils.uid(56)); }); }); }); ================================================ FILE: test/db.test.js ================================================ var should = require('chai').should() , assert = require('chai').assert , testDb = 'workspace/test.db' , fs = require('fs') , path = require('path') , _ = require('underscore') , async = require('async') , model = require('../lib/model') , Datastore = require('../lib/datastore') , Persistence = require('../lib/persistence') , reloadTimeUpperBound = 60; // In ms, an upper bound for the reload time used to check createdAt and updatedAt ; describe('Database', function () { var d; beforeEach(function (done) { d = new Datastore({ filename: testDb }); d.filename.should.equal(testDb); d.inMemoryOnly.should.equal(false); async.waterfall([ function (cb) { Persistence.ensureDirectoryExists(path.dirname(testDb), function () { fs.exists(testDb, function (exists) { if (exists) { fs.unlink(testDb, cb); } else { return cb(); } }); }); } , function (cb) { d.loadDatabase(function (err) { assert.isNull(err); d.getAllData().length.should.equal(0); return cb(); }); } ], done); }); it('Constructor compatibility with v0.6-', function () { var dbef = new Datastore('somefile'); dbef.filename.should.equal('somefile'); dbef.inMemoryOnly.should.equal(false); var dbef = new Datastore(''); assert.isNull(dbef.filename); dbef.inMemoryOnly.should.equal(true); var dbef = new Datastore(); assert.isNull(dbef.filename); dbef.inMemoryOnly.should.equal(true); }); describe('Autoloading', function () { it('Can autoload a database and query it right away', function (done) { var fileStr = model.serialize({ _id: '1', a: 5, planet: 'Earth' }) + '\n' + model.serialize({ _id: '2', a: 5, planet: 'Mars' }) + '\n' , autoDb = 'workspace/auto.db' , db ; fs.writeFileSync(autoDb, fileStr, 'utf8'); db = new Datastore({ filename: autoDb, autoload: true }) db.find({}, function (err, docs) { assert.isNull(err); docs.length.should.equal(2); done(); }); }); it('Throws if autoload fails', function (done) { var fileStr = model.serialize({ _id: '1', a: 5, planet: 'Earth' }) + '\n' + model.serialize({ _id: '2', a: 5, planet: 'Mars' }) + '\n' + '{"$$indexCreated":{"fieldName":"a","unique":true}}' , autoDb = 'workspace/auto.db' , db ; fs.writeFileSync(autoDb, fileStr, 'utf8'); // Check the loadDatabase generated an error function onload (err) { err.errorType.should.equal('uniqueViolated'); done(); } db = new Datastore({ filename: autoDb, autoload: true, onload: onload }) db.find({}, function (err, docs) { done(new Error("Find should not be executed since autoload failed")); }); }); }); describe('Insert', function () { it('Able to insert a document in the database, setting an _id if none provided, and retrieve it even after a reload', function (done) { d.find({}, function (err, docs) { docs.length.should.equal(0); d.insert({ somedata: 'ok' }, function (err) { // The data was correctly updated d.find({}, function (err, docs) { assert.isNull(err); docs.length.should.equal(1); Object.keys(docs[0]).length.should.equal(2); docs[0].somedata.should.equal('ok'); assert.isDefined(docs[0]._id); // After a reload the data has been correctly persisted d.loadDatabase(function (err) { d.find({}, function (err, docs) { assert.isNull(err); docs.length.should.equal(1); Object.keys(docs[0]).length.should.equal(2); docs[0].somedata.should.equal('ok'); assert.isDefined(docs[0]._id); done(); }); }); }); }); }); }); it('Can insert multiple documents in the database', function (done) { d.find({}, function (err, docs) { docs.length.should.equal(0); d.insert({ somedata: 'ok' }, function (err) { d.insert({ somedata: 'another' }, function (err) { d.insert({ somedata: 'again' }, function (err) { d.find({}, function (err, docs) { docs.length.should.equal(3); _.pluck(docs, 'somedata').should.contain('ok'); _.pluck(docs, 'somedata').should.contain('another'); _.pluck(docs, 'somedata').should.contain('again'); done(); }); }); }); }); }); }); it('Can insert and get back from DB complex objects with all primitive and secondary types', function (done) { var da = new Date() , obj = { a: ['ee', 'ff', 42], date: da, subobj: { a: 'b', b: 'c' } } ; d.insert(obj, function (err) { d.findOne({}, function (err, res) { assert.isNull(err); res.a.length.should.equal(3); res.a[0].should.equal('ee'); res.a[1].should.equal('ff'); res.a[2].should.equal(42); res.date.getTime().should.equal(da.getTime()); res.subobj.a.should.equal('b'); res.subobj.b.should.equal('c'); done(); }); }); }); it('If an object returned from the DB is modified and refetched, the original value should be found', function (done) { d.insert({ a: 'something' }, function () { d.findOne({}, function (err, doc) { doc.a.should.equal('something'); doc.a = 'another thing'; doc.a.should.equal('another thing'); // Re-fetching with findOne should yield the persisted value d.findOne({}, function (err, doc) { doc.a.should.equal('something'); doc.a = 'another thing'; doc.a.should.equal('another thing'); // Re-fetching with find should yield the persisted value d.find({}, function (err, docs) { docs[0].a.should.equal('something'); done(); }); }); }); }); }); it('Cannot insert a doc that has a field beginning with a $ sign', function (done) { d.insert({ $something: 'atest' }, function (err) { assert.isDefined(err); done(); }); }); it('If an _id is already given when we insert a document, use that instead of generating a random one', function (done) { d.insert({ _id: 'test', stuff: true }, function (err, newDoc) { if (err) { return done(err); } newDoc.stuff.should.equal(true); newDoc._id.should.equal('test'); d.insert({ _id: 'test', otherstuff: 42 }, function (err) { err.errorType.should.equal('uniqueViolated'); done(); }); }); }); it('Modifying the insertedDoc after an insert doesnt change the copy saved in the database', function (done) { d.insert({ a: 2, hello: 'world' }, function (err, newDoc) { newDoc.hello = 'changed'; d.findOne({ a: 2 }, function (err, doc) { doc.hello.should.equal('world'); done(); }); }); }); it('Can insert an array of documents at once', function (done) { var docs = [{ a: 5, b: 'hello' }, { a: 42, b: 'world' }]; d.insert(docs, function (err) { d.find({}, function (err, docs) { var data; docs.length.should.equal(2); _.find(docs, function (doc) { return doc.a === 5; }).b.should.equal('hello'); _.find(docs, function (doc) { return doc.a === 42; }).b.should.equal('world'); // The data has been persisted correctly data = _.filter(fs.readFileSync(testDb, 'utf8').split('\n'), function (line) { return line.length > 0; }); data.length.should.equal(2); model.deserialize(data[0]).a.should.equal(5); model.deserialize(data[0]).b.should.equal('hello'); model.deserialize(data[1]).a.should.equal(42); model.deserialize(data[1]).b.should.equal('world'); done(); }); }); }); it('If a bulk insert violates a constraint, all changes are rolled back', function (done) { var docs = [{ a: 5, b: 'hello' }, { a: 42, b: 'world' }, { a: 5, b: 'bloup' }, { a: 7 }]; d.ensureIndex({ fieldName: 'a', unique: true }, function () { // Important to specify callback here to make sure filesystem synced d.insert(docs, function (err) { err.errorType.should.equal('uniqueViolated'); d.find({}, function (err, docs) { // Datafile only contains index definition var datafileContents = model.deserialize(fs.readFileSync(testDb, 'utf8')); assert.deepEqual(datafileContents, { $$indexCreated: { fieldName: 'a', unique: true } }); docs.length.should.equal(0); done(); }); }); }); }); it("If timestampData option is set, a createdAt field is added and persisted", function (done) { var newDoc = { hello: 'world' }, beginning = Date.now(); d = new Datastore({ filename: testDb, timestampData: true, autoload: true }); d.find({}, function (err, docs) { assert.isNull(err); docs.length.should.equal(0); d.insert(newDoc, function (err, insertedDoc) { // No side effect on given input assert.deepEqual(newDoc, { hello: 'world' }); // Insert doc has two new fields, _id and createdAt insertedDoc.hello.should.equal('world'); assert.isDefined(insertedDoc.createdAt); assert.isDefined(insertedDoc.updatedAt); insertedDoc.createdAt.should.equal(insertedDoc.updatedAt); assert.isDefined(insertedDoc._id); Object.keys(insertedDoc).length.should.equal(4); assert.isBelow(Math.abs(insertedDoc.createdAt.getTime() - beginning), reloadTimeUpperBound); // No more than 30ms should have elapsed (worst case, if there is a flush) // Modifying results of insert doesn't change the cache insertedDoc.bloup = "another"; Object.keys(insertedDoc).length.should.equal(5); d.find({}, function (err, docs) { docs.length.should.equal(1); assert.deepEqual(newDoc, { hello: 'world' }); assert.deepEqual({ hello: 'world', _id: insertedDoc._id, createdAt: insertedDoc.createdAt, updatedAt: insertedDoc.updatedAt }, docs[0]); // All data correctly persisted on disk d.loadDatabase(function () { d.find({}, function (err, docs) { docs.length.should.equal(1); assert.deepEqual(newDoc, { hello: 'world' }); assert.deepEqual({ hello: 'world', _id: insertedDoc._id, createdAt: insertedDoc.createdAt, updatedAt: insertedDoc.updatedAt }, docs[0]); done(); }); }); }); }); }); }); it("If timestampData option not set, don't create a createdAt and a updatedAt field", function (done) { d.insert({ hello: 'world' }, function (err, insertedDoc) { Object.keys(insertedDoc).length.should.equal(2); assert.isUndefined(insertedDoc.createdAt); assert.isUndefined(insertedDoc.updatedAt); d.find({}, function (err, docs) { docs.length.should.equal(1); assert.deepEqual(docs[0], insertedDoc); done(); }); }); }); it("If timestampData is set but createdAt is specified by user, don't change it", function (done) { var newDoc = { hello: 'world', createdAt: new Date(234) }, beginning = Date.now(); d = new Datastore({ filename: testDb, timestampData: true, autoload: true }); d.insert(newDoc, function (err, insertedDoc) { Object.keys(insertedDoc).length.should.equal(4); insertedDoc.createdAt.getTime().should.equal(234); // Not modified assert.isBelow(insertedDoc.updatedAt.getTime() - beginning, reloadTimeUpperBound); // Created d.find({}, function (err, docs) { assert.deepEqual(insertedDoc, docs[0]); d.loadDatabase(function () { d.find({}, function (err, docs) { assert.deepEqual(insertedDoc, docs[0]); done(); }); }); }); }); }); it("If timestampData is set but updatedAt is specified by user, don't change it", function (done) { var newDoc = { hello: 'world', updatedAt: new Date(234) }, beginning = Date.now(); d = new Datastore({ filename: testDb, timestampData: true, autoload: true }); d.insert(newDoc, function (err, insertedDoc) { Object.keys(insertedDoc).length.should.equal(4); insertedDoc.updatedAt.getTime().should.equal(234); // Not modified assert.isBelow(insertedDoc.createdAt.getTime() - beginning, reloadTimeUpperBound); // Created d.find({}, function (err, docs) { assert.deepEqual(insertedDoc, docs[0]); d.loadDatabase(function () { d.find({}, function (err, docs) { assert.deepEqual(insertedDoc, docs[0]); done(); }); }); }); }); }); it('Can insert a doc with id 0', function (done) { d.insert({ _id: 0, hello: 'world' }, function (err, doc) { doc._id.should.equal(0); doc.hello.should.equal('world'); done(); }); }); /** * Complicated behavior here. Basically we need to test that when a user function throws an exception, it is not caught * in NeDB and the callback called again, transforming a user error into a NeDB error. * * So we need a way to check that the callback is called only once and the exception thrown is indeed the client exception * Mocha's exception handling mechanism interferes with this since it already registers a listener on uncaughtException * which we need to use since findOne is not called in the same turn of the event loop (so no try/catch) * So we remove all current listeners, put our own which when called will register the former listeners (incl. Mocha's) again. * * Note: maybe using an in-memory only NeDB would give us an easier solution */ it('If the callback throws an uncaught exception, do not catch it inside findOne, this is userspace concern', function (done) { var tryCount = 0 , currentUncaughtExceptionHandlers = process.listeners('uncaughtException') , i ; process.removeAllListeners('uncaughtException'); process.on('uncaughtException', function MINE (ex) { process.removeAllListeners('uncaughtException'); for (i = 0; i < currentUncaughtExceptionHandlers.length; i += 1) { process.on('uncaughtException', currentUncaughtExceptionHandlers[i]); } ex.message.should.equal('SOME EXCEPTION'); done(); }); d.insert({ a: 5 }, function () { d.findOne({ a : 5}, function (err, doc) { if (tryCount === 0) { tryCount += 1; throw new Error('SOME EXCEPTION'); } else { done(new Error('Callback was called twice')); } }); }); }); }); // ==== End of 'Insert' ==== // describe('#getCandidates', function () { it('Can use an index to get docs with a basic match', function (done) { d.ensureIndex({ fieldName: 'tf' }, function (err) { d.insert({ tf: 4 }, function (err, _doc1) { d.insert({ tf: 6 }, function () { d.insert({ tf: 4, an: 'other' }, function (err, _doc2) { d.insert({ tf: 9 }, function () { d.getCandidates({ r: 6, tf: 4 }, function (err, data) { var doc1 = _.find(data, function (d) { return d._id === _doc1._id; }) , doc2 = _.find(data, function (d) { return d._id === _doc2._id; }) ; data.length.should.equal(2); assert.deepEqual(doc1, { _id: doc1._id, tf: 4 }); assert.deepEqual(doc2, { _id: doc2._id, tf: 4, an: 'other' }); done(); }); }); }); }); }); }); }); it('Can use an index to get docs with a $in match', function (done) { d.ensureIndex({ fieldName: 'tf' }, function (err) { d.insert({ tf: 4 }, function (err) { d.insert({ tf: 6 }, function (err, _doc1) { d.insert({ tf: 4, an: 'other' }, function (err) { d.insert({ tf: 9 }, function (err, _doc2) { d.getCandidates({ r: 6, tf: { $in: [6, 9, 5] } }, function (err, data) { var doc1 = _.find(data, function (d) { return d._id === _doc1._id; }) , doc2 = _.find(data, function (d) { return d._id === _doc2._id; }) ; data.length.should.equal(2); assert.deepEqual(doc1, { _id: doc1._id, tf: 6 }); assert.deepEqual(doc2, { _id: doc2._id, tf: 9 }); done(); }); }); }); }); }); }); }); it('If no index can be used, return the whole database', function (done) { d.ensureIndex({ fieldName: 'tf' }, function (err) { d.insert({ tf: 4 }, function (err, _doc1) { d.insert({ tf: 6 }, function (err, _doc2) { d.insert({ tf: 4, an: 'other' }, function (err, _doc3) { d.insert({ tf: 9 }, function (err, _doc4) { d.getCandidates({ r: 6, notf: { $in: [6, 9, 5] } }, function (err, data) { var doc1 = _.find(data, function (d) { return d._id === _doc1._id; }) , doc2 = _.find(data, function (d) { return d._id === _doc2._id; }) , doc3 = _.find(data, function (d) { return d._id === _doc3._id; }) , doc4 = _.find(data, function (d) { return d._id === _doc4._id; }) ; data.length.should.equal(4); assert.deepEqual(doc1, { _id: doc1._id, tf: 4 }); assert.deepEqual(doc2, { _id: doc2._id, tf: 6 }); assert.deepEqual(doc3, { _id: doc3._id, tf: 4, an: 'other' }); assert.deepEqual(doc4, { _id: doc4._id, tf: 9 }); done(); }); }); }); }); }); }); }); it('Can use indexes for comparison matches', function (done) { d.ensureIndex({ fieldName: 'tf' }, function (err) { d.insert({ tf: 4 }, function (err, _doc1) { d.insert({ tf: 6 }, function (err, _doc2) { d.insert({ tf: 4, an: 'other' }, function (err, _doc3) { d.insert({ tf: 9 }, function (err, _doc4) { d.getCandidates({ r: 6, tf: { $lte: 9, $gte: 6 } }, function (err, data) { var doc2 = _.find(data, function (d) { return d._id === _doc2._id; }) , doc4 = _.find(data, function (d) { return d._id === _doc4._id; }) ; data.length.should.equal(2); assert.deepEqual(doc2, { _id: doc2._id, tf: 6 }); assert.deepEqual(doc4, { _id: doc4._id, tf: 9 }); done(); }); }); }); }); }); }); }); it("Can set a TTL index that expires documents", function (done) { d.ensureIndex({ fieldName: 'exp', expireAfterSeconds: 0.2 }, function () { d.insert({ hello: 'world', exp: new Date() }, function () { setTimeout(function () { d.findOne({}, function (err, doc) { assert.isNull(err); doc.hello.should.equal('world'); setTimeout(function () { d.findOne({}, function (err, doc) { assert.isNull(err); assert.isNull(doc); d.on('compaction.done', function () { // After compaction, no more mention of the document, correctly removed var datafileContents = fs.readFileSync(testDb, 'utf8'); datafileContents.split('\n').length.should.equal(2); assert.isNull(datafileContents.match(/world/)); // New datastore on same datafile is empty var d2 = new Datastore({ filename: testDb, autoload: true }); d2.findOne({}, function (err, doc) { assert.isNull(err); assert.isNull(doc); done(); }); }); d.persistence.compactDatafile(); }); }, 101); }); }, 100); }); }); }); it("TTL indexes can expire multiple documents and only what needs to be expired", function (done) { d.ensureIndex({ fieldName: 'exp', expireAfterSeconds: 0.2 }, function () { d.insert({ hello: 'world1', exp: new Date() }, function () { d.insert({ hello: 'world2', exp: new Date() }, function () { d.insert({ hello: 'world3', exp: new Date((new Date()).getTime() + 100) }, function () { setTimeout(function () { d.find({}, function (err, docs) { assert.isNull(err); docs.length.should.equal(3); setTimeout(function () { d.find({}, function (err, docs) { assert.isNull(err); docs.length.should.equal(1); docs[0].hello.should.equal('world3'); setTimeout(function () { d.find({}, function (err, docs) { assert.isNull(err); docs.length.should.equal(0); done(); }); }, 101); }); }, 101); }); }, 100); }); }); }); }); }); it("Document where indexed field is absent or not a date are ignored", function (done) { d.ensureIndex({ fieldName: 'exp', expireAfterSeconds: 0.2 }, function () { d.insert({ hello: 'world1', exp: new Date() }, function () { d.insert({ hello: 'world2', exp: "not a date" }, function () { d.insert({ hello: 'world3' }, function () { setTimeout(function () { d.find({}, function (err, docs) { assert.isNull(err); docs.length.should.equal(3); setTimeout(function () { d.find({}, function (err, docs) { assert.isNull(err); docs.length.should.equal(2); docs[0].hello.should.not.equal('world1'); docs[1].hello.should.not.equal('world1'); done(); }); }, 101); }); }, 100); }); }); }); }); }); }); // ==== End of '#getCandidates' ==== // describe('Find', function () { it('Can find all documents if an empty query is used', function (done) { async.waterfall([ function (cb) { d.insert({ somedata: 'ok' }, function (err) { d.insert({ somedata: 'another', plus: 'additional data' }, function (err) { d.insert({ somedata: 'again' }, function (err) { return cb(err); }); }); }); } , function (cb) { // Test with empty object d.find({}, function (err, docs) { assert.isNull(err); docs.length.should.equal(3); _.pluck(docs, 'somedata').should.contain('ok'); _.pluck(docs, 'somedata').should.contain('another'); _.find(docs, function (d) { return d.somedata === 'another' }).plus.should.equal('additional data'); _.pluck(docs, 'somedata').should.contain('again'); return cb(); }); } ], done); }); it('Can find all documents matching a basic query', function (done) { async.waterfall([ function (cb) { d.insert({ somedata: 'ok' }, function (err) { d.insert({ somedata: 'again', plus: 'additional data' }, function (err) { d.insert({ somedata: 'again' }, function (err) { return cb(err); }); }); }); } , function (cb) { // Test with query that will return docs d.find({ somedata: 'again' }, function (err, docs) { assert.isNull(err); docs.length.should.equal(2); _.pluck(docs, 'somedata').should.not.contain('ok'); return cb(); }); } , function (cb) { // Test with query that doesn't match anything d.find({ somedata: 'nope' }, function (err, docs) { assert.isNull(err); docs.length.should.equal(0); return cb(); }); } ], done); }); it('Can find one document matching a basic query and return null if none is found', function (done) { async.waterfall([ function (cb) { d.insert({ somedata: 'ok' }, function (err) { d.insert({ somedata: 'again', plus: 'additional data' }, function (err) { d.insert({ somedata: 'again' }, function (err) { return cb(err); }); }); }); } , function (cb) { // Test with query that will return docs d.findOne({ somedata: 'ok' }, function (err, doc) { assert.isNull(err); Object.keys(doc).length.should.equal(2); doc.somedata.should.equal('ok'); assert.isDefined(doc._id); return cb(); }); } , function (cb) { // Test with query that doesn't match anything d.findOne({ somedata: 'nope' }, function (err, doc) { assert.isNull(err); assert.isNull(doc); return cb(); }); } ], done); }); it('Can find dates and objects (non JS-native types)', function (done) { var date1 = new Date(1234543) , date2 = new Date(9999) ; d.insert({ now: date1, sth: { name: 'nedb' } }, function () { d.findOne({ now: date1 }, function (err, doc) { assert.isNull(err); doc.sth.name.should.equal('nedb'); d.findOne({ now: date2 }, function (err, doc) { assert.isNull(err); assert.isNull(doc); d.findOne({ sth: { name: 'nedb' } }, function (err, doc) { assert.isNull(err); doc.sth.name.should.equal('nedb'); d.findOne({ sth: { name: 'other' } }, function (err, doc) { assert.isNull(err); assert.isNull(doc); done(); }); }); }); }); }); }); it('Can use dot-notation to query subfields', function (done) { d.insert({ greeting: { english: 'hello' } }, function () { d.findOne({ "greeting.english": 'hello' }, function (err, doc) { assert.isNull(err); doc.greeting.english.should.equal('hello'); d.findOne({ "greeting.english": 'hellooo' }, function (err, doc) { assert.isNull(err); assert.isNull(doc); d.findOne({ "greeting.englis": 'hello' }, function (err, doc) { assert.isNull(err); assert.isNull(doc); done(); }); }); }); }); }); it('Array fields match if any element matches', function (done) { d.insert({ fruits: ['pear', 'apple', 'banana'] }, function (err, doc1) { d.insert({ fruits: ['coconut', 'orange', 'pear'] }, function (err, doc2) { d.insert({ fruits: ['banana'] }, function (err, doc3) { d.find({ fruits: 'pear' }, function (err, docs) { assert.isNull(err); docs.length.should.equal(2); _.pluck(docs, '_id').should.contain(doc1._id); _.pluck(docs, '_id').should.contain(doc2._id); d.find({ fruits: 'banana' }, function (err, docs) { assert.isNull(err); docs.length.should.equal(2); _.pluck(docs, '_id').should.contain(doc1._id); _.pluck(docs, '_id').should.contain(doc3._id); d.find({ fruits: 'doesntexist' }, function (err, docs) { assert.isNull(err); docs.length.should.equal(0); done(); }); }); }); }); }); }); }); it('Returns an error if the query is not well formed', function (done) { d.insert({ hello: 'world' }, function () { d.find({ $or: { hello: 'world' } }, function (err, docs) { assert.isDefined(err); assert.isUndefined(docs); d.findOne({ $or: { hello: 'world' } }, function (err, doc) { assert.isDefined(err); assert.isUndefined(doc); done(); }); }); }); }); it('Changing the documents returned by find or findOne do not change the database state', function (done) { d.insert({ a: 2, hello: 'world' }, function () { d.findOne({ a: 2 }, function (err, doc) { doc.hello = 'changed'; d.findOne({ a: 2 }, function (err, doc) { doc.hello.should.equal('world'); d.find({ a: 2 }, function (err, docs) { docs[0].hello = 'changed'; d.findOne({ a: 2 }, function (err, doc) { doc.hello.should.equal('world'); done(); }); }); }); }); }); }); it('Can use sort, skip and limit if the callback is not passed to find but to exec', function (done) { d.insert({ a: 2, hello: 'world' }, function () { d.insert({ a: 24, hello: 'earth' }, function () { d.insert({ a: 13, hello: 'blueplanet' }, function () { d.insert({ a: 15, hello: 'home' }, function () { d.find({}).sort({ a: 1 }).limit(2).exec(function (err, docs) { assert.isNull(err); docs.length.should.equal(2); docs[0].hello.should.equal('world'); docs[1].hello.should.equal('blueplanet'); done(); }); }); }); }); }); }); it('Can use sort and skip if the callback is not passed to findOne but to exec', function (done) { d.insert({ a: 2, hello: 'world' }, function () { d.insert({ a: 24, hello: 'earth' }, function () { d.insert({ a: 13, hello: 'blueplanet' }, function () { d.insert({ a: 15, hello: 'home' }, function () { // No skip no query d.findOne({}).sort({ a: 1 }).exec(function (err, doc) { assert.isNull(err); doc.hello.should.equal('world'); // A query d.findOne({ a: { $gt: 14 } }).sort({ a: 1 }).exec(function (err, doc) { assert.isNull(err); doc.hello.should.equal('home'); // And a skip d.findOne({ a: { $gt: 14 } }).sort({ a: 1 }).skip(1).exec(function (err, doc) { assert.isNull(err); doc.hello.should.equal('earth'); // No result d.findOne({ a: { $gt: 14 } }).sort({ a: 1 }).skip(2).exec(function (err, doc) { assert.isNull(err); assert.isNull(doc); done(); }); }); }); }); }); }); }); }); }); it('Can use projections in find, normal or cursor way', function (done) { d.insert({ a: 2, hello: 'world' }, function (err, doc0) { d.insert({ a: 24, hello: 'earth' }, function (err, doc1) { d.find({ a: 2 }, { a: 0, _id: 0 }, function (err, docs) { assert.isNull(err); docs.length.should.equal(1); assert.deepEqual(docs[0], { hello: 'world' }); d.find({ a: 2 }, { a: 0, _id: 0 }).exec(function (err, docs) { assert.isNull(err); docs.length.should.equal(1); assert.deepEqual(docs[0], { hello: 'world' }); // Can't use both modes at once if not _id d.find({ a: 2 }, { a: 0, hello: 1 }, function (err, docs) { assert.isNotNull(err); assert.isUndefined(docs); d.find({ a: 2 }, { a: 0, hello: 1 }).exec(function (err, docs) { assert.isNotNull(err); assert.isUndefined(docs); done(); }); }); }); }); }); }); }); it('Can use projections in findOne, normal or cursor way', function (done) { d.insert({ a: 2, hello: 'world' }, function (err, doc0) { d.insert({ a: 24, hello: 'earth' }, function (err, doc1) { d.findOne({ a: 2 }, { a: 0, _id: 0 }, function (err, doc) { assert.isNull(err); assert.deepEqual(doc, { hello: 'world' }); d.findOne({ a: 2 }, { a: 0, _id: 0 }).exec(function (err, doc) { assert.isNull(err); assert.deepEqual(doc, { hello: 'world' }); // Can't use both modes at once if not _id d.findOne({ a: 2 }, { a: 0, hello: 1 }, function (err, doc) { assert.isNotNull(err); assert.isUndefined(doc); d.findOne({ a: 2 }, { a: 0, hello: 1 }).exec(function (err, doc) { assert.isNotNull(err); assert.isUndefined(doc); done(); }); }); }); }); }); }); }); }); // ==== End of 'Find' ==== // describe('Count', function() { it('Count all documents if an empty query is used', function (done) { async.waterfall([ function (cb) { d.insert({ somedata: 'ok' }, function (err) { d.insert({ somedata: 'another', plus: 'additional data' }, function (err) { d.insert({ somedata: 'again' }, function (err) { return cb(err); }); }); }); } , function (cb) { // Test with empty object d.count({}, function (err, docs) { assert.isNull(err); docs.should.equal(3); return cb(); }); } ], done); }); it('Count all documents matching a basic query', function (done) { async.waterfall([ function (cb) { d.insert({ somedata: 'ok' }, function (err) { d.insert({ somedata: 'again', plus: 'additional data' }, function (err) { d.insert({ somedata: 'again' }, function (err) { return cb(err); }); }); }); } , function (cb) { // Test with query that will return docs d.count({ somedata: 'again' }, function (err, docs) { assert.isNull(err); docs.should.equal(2); return cb(); }); } , function (cb) { // Test with query that doesn't match anything d.count({ somedata: 'nope' }, function (err, docs) { assert.isNull(err); docs.should.equal(0); return cb(); }); } ], done); }); it('Array fields match if any element matches', function (done) { d.insert({ fruits: ['pear', 'apple', 'banana'] }, function (err, doc1) { d.insert({ fruits: ['coconut', 'orange', 'pear'] }, function (err, doc2) { d.insert({ fruits: ['banana'] }, function (err, doc3) { d.count({ fruits: 'pear' }, function (err, docs) { assert.isNull(err); docs.should.equal(2); d.count({ fruits: 'banana' }, function (err, docs) { assert.isNull(err); docs.should.equal(2); d.count({ fruits: 'doesntexist' }, function (err, docs) { assert.isNull(err); docs.should.equal(0); done(); }); }); }); }); }); }); }); it('Returns an error if the query is not well formed', function (done) { d.insert({ hello: 'world' }, function () { d.count({ $or: { hello: 'world' } }, function (err, docs) { assert.isDefined(err); assert.isUndefined(docs); done(); }); }); }); }); describe('Update', function () { it("If the query doesn't match anything, database is not modified", function (done) { async.waterfall([ function (cb) { d.insert({ somedata: 'ok' }, function (err) { d.insert({ somedata: 'again', plus: 'additional data' }, function (err) { d.insert({ somedata: 'another' }, function (err) { return cb(err); }); }); }); } , function (cb) { // Test with query that doesn't match anything d.update({ somedata: 'nope' }, { newDoc: 'yes' }, { multi: true }, function (err, n) { assert.isNull(err); n.should.equal(0); d.find({}, function (err, docs) { var doc1 = _.find(docs, function (d) { return d.somedata === 'ok'; }) , doc2 = _.find(docs, function (d) { return d.somedata === 'again'; }) , doc3 = _.find(docs, function (d) { return d.somedata === 'another'; }) ; docs.length.should.equal(3); assert.isUndefined(_.find(docs, function (d) { return d.newDoc === 'yes'; })); assert.deepEqual(doc1, { _id: doc1._id, somedata: 'ok' }); assert.deepEqual(doc2, { _id: doc2._id, somedata: 'again', plus: 'additional data' }); assert.deepEqual(doc3, { _id: doc3._id, somedata: 'another' }); return cb(); }); }); } ], done); }); it("If timestampData option is set, update the updatedAt field", function (done) { var beginning = Date.now(); d = new Datastore({ filename: testDb, autoload: true, timestampData: true }); d.insert({ hello: 'world' }, function (err, insertedDoc) { assert.isBelow(insertedDoc.updatedAt.getTime() - beginning, reloadTimeUpperBound); assert.isBelow(insertedDoc.createdAt.getTime() - beginning, reloadTimeUpperBound); Object.keys(insertedDoc).length.should.equal(4); // Wait 100ms before performing the update setTimeout(function () { var step1 = Date.now(); d.update({ _id: insertedDoc._id }, { $set: { hello: 'mars' } }, {}, function () { d.find({ _id: insertedDoc._id }, function (err, docs) { docs.length.should.equal(1); Object.keys(docs[0]).length.should.equal(4); docs[0]._id.should.equal(insertedDoc._id); docs[0].createdAt.should.equal(insertedDoc.createdAt); docs[0].hello.should.equal('mars'); assert.isAbove(docs[0].updatedAt.getTime() - beginning, 99); // updatedAt modified assert.isBelow(docs[0].updatedAt.getTime() - step1, reloadTimeUpperBound); // updatedAt modified done(); }); }) }, 100); }); }); it("Can update multiple documents matching the query", function (done) { var id1, id2, id3; // Test DB state after update and reload function testPostUpdateState (cb) { d.find({}, function (err, docs) { var doc1 = _.find(docs, function (d) { return d._id === id1; }) , doc2 = _.find(docs, function (d) { return d._id === id2; }) , doc3 = _.find(docs, function (d) { return d._id === id3; }) ; docs.length.should.equal(3); Object.keys(doc1).length.should.equal(2); doc1.somedata.should.equal('ok'); doc1._id.should.equal(id1); Object.keys(doc2).length.should.equal(2); doc2.newDoc.should.equal('yes'); doc2._id.should.equal(id2); Object.keys(doc3).length.should.equal(2); doc3.newDoc.should.equal('yes'); doc3._id.should.equal(id3); return cb(); }); } // Actually launch the tests async.waterfall([ function (cb) { d.insert({ somedata: 'ok' }, function (err, doc1) { id1 = doc1._id; d.insert({ somedata: 'again', plus: 'additional data' }, function (err, doc2) { id2 = doc2._id; d.insert({ somedata: 'again' }, function (err, doc3) { id3 = doc3._id; return cb(err); }); }); }); } , function (cb) { d.update({ somedata: 'again' }, { newDoc: 'yes' }, { multi: true }, function (err, n) { assert.isNull(err); n.should.equal(2); return cb(); }); } , async.apply(testPostUpdateState) , function (cb) { d.loadDatabase(function (err) { cb(err); }); } , async.apply(testPostUpdateState) ], done); }); it("Can update only one document matching the query", function (done) { var id1, id2, id3; // Test DB state after update and reload function testPostUpdateState (cb) { d.find({}, function (err, docs) { var doc1 = _.find(docs, function (d) { return d._id === id1; }) , doc2 = _.find(docs, function (d) { return d._id === id2; }) , doc3 = _.find(docs, function (d) { return d._id === id3; }) ; docs.length.should.equal(3); assert.deepEqual(doc1, { somedata: 'ok', _id: doc1._id }); // doc2 or doc3 was modified. Since we sort on _id and it is random // it can be either of two situations try { assert.deepEqual(doc2, { newDoc: 'yes', _id: doc2._id }); assert.deepEqual(doc3, { somedata: 'again', _id: doc3._id }); } catch (e) { assert.deepEqual(doc2, { somedata: 'again', plus: 'additional data', _id: doc2._id }); assert.deepEqual(doc3, { newDoc: 'yes', _id: doc3._id }); } return cb(); }); } // Actually launch the test async.waterfall([ function (cb) { d.insert({ somedata: 'ok' }, function (err, doc1) { id1 = doc1._id; d.insert({ somedata: 'again', plus: 'additional data' }, function (err, doc2) { id2 = doc2._id; d.insert({ somedata: 'again' }, function (err, doc3) { id3 = doc3._id; return cb(err); }); }); }); } , function (cb) { // Test with query that doesn't match anything d.update({ somedata: 'again' }, { newDoc: 'yes' }, { multi: false }, function (err, n) { assert.isNull(err); n.should.equal(1); return cb(); }); } , async.apply(testPostUpdateState) , function (cb) { d.loadDatabase(function (err) { return cb(err); }); } , async.apply(testPostUpdateState) // The persisted state has been updated ], done); }); describe('Upserts', function () { it('Can perform upserts if needed', function (done) { d.update({ impossible: 'db is empty anyway' }, { newDoc: true }, {}, function (err, nr, upsert) { assert.isNull(err); nr.should.equal(0); assert.isUndefined(upsert); d.find({}, function (err, docs) { docs.length.should.equal(0); // Default option for upsert is false d.update({ impossible: 'db is empty anyway' }, { something: "created ok" }, { upsert: true }, function (err, nr, newDoc) { assert.isNull(err); nr.should.equal(1); newDoc.something.should.equal("created ok"); assert.isDefined(newDoc._id); d.find({}, function (err, docs) { docs.length.should.equal(1); // Default option for upsert is false docs[0].something.should.equal("created ok"); // Modifying the returned upserted document doesn't modify the database newDoc.newField = true; d.find({}, function (err, docs) { docs[0].something.should.equal("created ok"); assert.isUndefined(docs[0].newField); done(); }); }); }); }); }); }); it('If the update query is a normal object with no modifiers, it is the doc that will be upserted', function (done) { d.update({ $or: [{ a: 4 }, { a: 5 }] }, { hello: 'world', bloup: 'blap' }, { upsert: true }, function (err) { d.find({}, function (err, docs) { assert.isNull(err); docs.length.should.equal(1); var doc = docs[0]; Object.keys(doc).length.should.equal(3); doc.hello.should.equal('world'); doc.bloup.should.equal('blap'); done(); }); }); }); it('If the update query contains modifiers, it is applied to the object resulting from removing all operators from the find query 1', function (done) { d.update({ $or: [{ a: 4 }, { a: 5 }] }, { $set: { hello: 'world' }, $inc: { bloup: 3 } }, { upsert: true }, function (err) { d.find({ hello: 'world' }, function (err, docs) { assert.isNull(err); docs.length.should.equal(1); var doc = docs[0]; Object.keys(doc).length.should.equal(3); doc.hello.should.equal('world'); doc.bloup.should.equal(3); done(); }); }); }); it('If the update query contains modifiers, it is applied to the object resulting from removing all operators from the find query 2', function (done) { d.update({ $or: [{ a: 4 }, { a: 5 }], cac: 'rrr' }, { $set: { hello: 'world' }, $inc: { bloup: 3 } }, { upsert: true }, function (err) { d.find({ hello: 'world' }, function (err, docs) { assert.isNull(err); docs.length.should.equal(1); var doc = docs[0]; Object.keys(doc).length.should.equal(4); doc.cac.should.equal('rrr'); doc.hello.should.equal('world'); doc.bloup.should.equal(3); done(); }); }); }); it('Performing upsert with badly formatted fields yields a standard error not an exception', function(done) { d.update({_id: '1234'}, { $set: { $$badfield: 5 }}, { upsert: true }, function(err, doc) { assert.isDefined(err); done(); }) }); }); // ==== End of 'Upserts' ==== // it('Cannot perform update if the update query is not either registered-modifiers-only or copy-only, or contain badly formatted fields', function (done) { d.insert({ something: 'yup' }, function () { d.update({}, { boom: { $badfield: 5 } }, { multi: false }, function (err) { assert.isDefined(err); d.update({}, { boom: { "bad.field": 5 } }, { multi: false }, function (err) { assert.isDefined(err); d.update({}, { $inc: { test: 5 }, mixed: 'rrr' }, { multi: false }, function (err) { assert.isDefined(err); d.update({}, { $inexistent: { test: 5 } }, { multi: false }, function (err) { assert.isDefined(err); done(); }); }); }); }); }); }); it('Can update documents using multiple modifiers', function (done) { var id; d.insert({ something: 'yup', other: 40 }, function (err, newDoc) { id = newDoc._id; d.update({}, { $set: { something: 'changed' }, $inc: { other: 10 } }, { multi: false }, function (err, nr) { assert.isNull(err); nr.should.equal(1); d.findOne({ _id: id }, function (err, doc) { Object.keys(doc).length.should.equal(3); doc._id.should.equal(id); doc.something.should.equal('changed'); doc.other.should.equal(50); done(); }); }); }); }); it('Can upsert a document even with modifiers', function (done) { d.update({ bloup: 'blap' }, { $set: { hello: 'world' } }, { upsert: true }, function (err, nr, newDoc) { assert.isNull(err); nr.should.equal(1); newDoc.bloup.should.equal('blap'); newDoc.hello.should.equal('world'); assert.isDefined(newDoc._id); d.find({}, function (err, docs) { docs.length.should.equal(1); Object.keys(docs[0]).length.should.equal(3); docs[0].hello.should.equal('world'); docs[0].bloup.should.equal('blap'); assert.isDefined(docs[0]._id); done(); }); }); }); it('When using modifiers, the only way to update subdocs is with the dot-notation', function (done) { d.insert({ bloup: { blip: "blap", other: true } }, function () { // Correct methos d.update({}, { $set: { "bloup.blip": "hello" } }, {}, function () { d.findOne({}, function (err, doc) { doc.bloup.blip.should.equal("hello"); doc.bloup.other.should.equal(true); // Wrong d.update({}, { $set: { bloup: { blip: "ola" } } }, {}, function () { d.findOne({}, function (err, doc) { doc.bloup.blip.should.equal("ola"); assert.isUndefined(doc.bloup.other); // This information was lost done(); }); }); }); }); }); }); it('Returns an error if the query is not well formed', function (done) { d.insert({ hello: 'world' }, function () { d.update({ $or: { hello: 'world' } }, { a: 1 }, {}, function (err, nr, upsert) { assert.isDefined(err); assert.isUndefined(nr); assert.isUndefined(upsert); done(); }); }); }); it('If an error is thrown by a modifier, the database state is not changed', function (done) { d.insert({ hello: 'world' }, function (err, newDoc) { d.update({}, { $inc: { hello: 4 } }, {}, function (err, nr) { assert.isDefined(err); assert.isUndefined(nr); d.find({}, function (err, docs) { assert.deepEqual(docs, [ { _id: newDoc._id, hello: 'world' } ]); done(); }); }); }); }); it('Cant change the _id of a document', function (done) { d.insert({ a: 2 }, function (err, newDoc) { d.update({ a: 2 }, { a: 2, _id: 'nope' }, {}, function (err) { assert.isDefined(err); d.find({}, function (err, docs) { docs.length.should.equal(1); Object.keys(docs[0]).length.should.equal(2); docs[0].a.should.equal(2); docs[0]._id.should.equal(newDoc._id); d.update({ a: 2 }, { $set: { _id: 'nope' } }, {}, function (err) { assert.isDefined(err); d.find({}, function (err, docs) { docs.length.should.equal(1); Object.keys(docs[0]).length.should.equal(2); docs[0].a.should.equal(2); docs[0]._id.should.equal(newDoc._id); done(); }); }); }); }); }); }); it('Non-multi updates are persistent', function (done) { d.insert({ a:1, hello: 'world' }, function (err, doc1) { d.insert({ a:2, hello: 'earth' }, function (err, doc2) { d.update({ a: 2 }, { $set: { hello: 'changed' } }, {}, function (err) { assert.isNull(err); d.find({}, function (err, docs) { docs.sort(function (a, b) { return a.a - b.a; }); docs.length.should.equal(2); _.isEqual(docs[0], { _id: doc1._id, a:1, hello: 'world' }).should.equal(true); _.isEqual(docs[1], { _id: doc2._id, a:2, hello: 'changed' }).should.equal(true); // Even after a reload the database state hasn't changed d.loadDatabase(function (err) { assert.isNull(err); d.find({}, function (err, docs) { docs.sort(function (a, b) { return a.a - b.a; }); docs.length.should.equal(2); _.isEqual(docs[0], { _id: doc1._id, a:1, hello: 'world' }).should.equal(true); _.isEqual(docs[1], { _id: doc2._id, a:2, hello: 'changed' }).should.equal(true); done(); }); }); }); }); }); }); }); it('Multi updates are persistent', function (done) { d.insert({ a:1, hello: 'world' }, function (err, doc1) { d.insert({ a:2, hello: 'earth' }, function (err, doc2) { d.insert({ a:5, hello: 'pluton' }, function (err, doc3) { d.update({ a: { $in: [1, 2] } }, { $set: { hello: 'changed' } }, { multi: true }, function (err) { assert.isNull(err); d.find({}, function (err, docs) { docs.sort(function (a, b) { return a.a - b.a; }); docs.length.should.equal(3); _.isEqual(docs[0], { _id: doc1._id, a:1, hello: 'changed' }).should.equal(true); _.isEqual(docs[1], { _id: doc2._id, a:2, hello: 'changed' }).should.equal(true); _.isEqual(docs[2], { _id: doc3._id, a:5, hello: 'pluton' }).should.equal(true); // Even after a reload the database state hasn't changed d.loadDatabase(function (err) { assert.isNull(err); d.find({}, function (err, docs) { docs.sort(function (a, b) { return a.a - b.a; }); docs.length.should.equal(3); _.isEqual(docs[0], { _id: doc1._id, a:1, hello: 'changed' }).should.equal(true); _.isEqual(docs[1], { _id: doc2._id, a:2, hello: 'changed' }).should.equal(true); _.isEqual(docs[2], { _id: doc3._id, a:5, hello: 'pluton' }).should.equal(true); done(); }); }); }); }); }); }); }); }); it('Can update without the options arg (will use defaults then)', function (done) { d.insert({ a:1, hello: 'world' }, function (err, doc1) { d.insert({ a:2, hello: 'earth' }, function (err, doc2) { d.insert({ a:5, hello: 'pluton' }, function (err, doc3) { d.update({ a: 2 }, { $inc: { a: 10 } }, function (err, nr) { assert.isNull(err); nr.should.equal(1); d.find({}, function (err, docs) { var d1 = _.find(docs, function (doc) { return doc._id === doc1._id }) , d2 = _.find(docs, function (doc) { return doc._id === doc2._id }) , d3 = _.find(docs, function (doc) { return doc._id === doc3._id }) ; d1.a.should.equal(1); d2.a.should.equal(12); d3.a.should.equal(5); done(); }); }); }); }); }); }); it('If a multi update fails on one document, previous updates should be rolled back', function (done) { d.ensureIndex({ fieldName: 'a' }); d.insert({ a: 4 }, function (err, doc1) { d.insert({ a: 5 }, function (err, doc2) { d.insert({ a: 'abc' }, function (err, doc3) { // With this query, candidates are always returned in the order 4, 5, 'abc' so it's always the last one which fails d.update({ a: { $in: [4, 5, 'abc'] } }, { $inc: { a: 10 } }, { multi: true }, function (err) { assert.isDefined(err); // No index modified _.each(d.indexes, function (index) { var docs = index.getAll() , d1 = _.find(docs, function (doc) { return doc._id === doc1._id }) , d2 = _.find(docs, function (doc) { return doc._id === doc2._id }) , d3 = _.find(docs, function (doc) { return doc._id === doc3._id }) ; // All changes rolled back, including those that didn't trigger an error d1.a.should.equal(4); d2.a.should.equal(5); d3.a.should.equal('abc'); }); done(); }); }); }); }); }); it('If an index constraint is violated by an update, all changes should be rolled back', function (done) { d.ensureIndex({ fieldName: 'a', unique: true }); d.insert({ a: 4 }, function (err, doc1) { d.insert({ a: 5 }, function (err, doc2) { // With this query, candidates are always returned in the order 4, 5, 'abc' so it's always the last one which fails d.update({ a: { $in: [4, 5, 'abc'] } }, { $set: { a: 10 } }, { multi: true }, function (err) { assert.isDefined(err); // Check that no index was modified _.each(d.indexes, function (index) { var docs = index.getAll() , d1 = _.find(docs, function (doc) { return doc._id === doc1._id }) , d2 = _.find(docs, function (doc) { return doc._id === doc2._id }) ; d1.a.should.equal(4); d2.a.should.equal(5); }); done(); }); }); }); }); it("If options.returnUpdatedDocs is true, return all matched docs", function (done) { d.insert([{ a: 4 }, { a: 5 }, { a: 6 }], function (err, docs) { docs.length.should.equal(3); d.update({ a: 7 }, { $set: { u: 1 } }, { multi: true, returnUpdatedDocs: true }, function (err, num, updatedDocs) { num.should.equal(0); updatedDocs.length.should.equal(0); d.update({ a: 5 }, { $set: { u: 2 } }, { multi: true, returnUpdatedDocs: true }, function (err, num, updatedDocs) { num.should.equal(1); updatedDocs.length.should.equal(1); updatedDocs[0].a.should.equal(5); updatedDocs[0].u.should.equal(2); d.update({ a: { $in: [4, 6] } }, { $set: { u: 3 } }, { multi: true, returnUpdatedDocs: true }, function (err, num, updatedDocs) { num.should.equal(2); updatedDocs.length.should.equal(2); updatedDocs[0].u.should.equal(3); updatedDocs[1].u.should.equal(3); if (updatedDocs[0].a === 4) { updatedDocs[0].a.should.equal(4); updatedDocs[1].a.should.equal(6); } else { updatedDocs[0].a.should.equal(6); updatedDocs[1].a.should.equal(4); } done(); }); }); }); }); }); it("createdAt property is unchanged and updatedAt correct after an update, even a complete document replacement", function (done) { var d2 = new Datastore({ inMemoryOnly: true, timestampData: true }); d2.insert({ a: 1 }); d2.findOne({ a: 1 }, function (err, doc) { var createdAt = doc.createdAt.getTime(); // Modifying update setTimeout(function () { d2.update({ a: 1 }, { $set: { b: 2 } }, {}); d2.findOne({ a: 1 }, function (err, doc) { doc.createdAt.getTime().should.equal(createdAt); assert.isBelow(Date.now() - doc.updatedAt.getTime(), 5); // Complete replacement setTimeout(function () { d2.update({ a: 1 }, { c: 3 }, {}); d2.findOne({ c: 3 }, function (err, doc) { doc.createdAt.getTime().should.equal(createdAt); assert.isBelow(Date.now() - doc.updatedAt.getTime(), 5); done(); }); }, 20); }); }, 20); }); }); describe("Callback signature", function () { it("Regular update, multi false", function (done) { d.insert({ a: 1 }); d.insert({ a: 2 }); // returnUpdatedDocs set to false d.update({ a: 1 }, { $set: { b: 20 } }, {}, function (err, numAffected, affectedDocuments, upsert) { assert.isNull(err); numAffected.should.equal(1); assert.isUndefined(affectedDocuments); assert.isUndefined(upsert); // returnUpdatedDocs set to true d.update({ a: 1 }, { $set: { b: 21 } }, { returnUpdatedDocs: true }, function (err, numAffected, affectedDocuments, upsert) { assert.isNull(err); numAffected.should.equal(1); affectedDocuments.a.should.equal(1); affectedDocuments.b.should.equal(21); assert.isUndefined(upsert); done(); }); }); }); it("Regular update, multi true", function (done) { d.insert({ a: 1 }); d.insert({ a: 2 }); // returnUpdatedDocs set to false d.update({}, { $set: { b: 20 } }, { multi: true }, function (err, numAffected, affectedDocuments, upsert) { assert.isNull(err); numAffected.should.equal(2); assert.isUndefined(affectedDocuments); assert.isUndefined(upsert); // returnUpdatedDocs set to true d.update({}, { $set: { b: 21 } }, { multi: true, returnUpdatedDocs: true }, function (err, numAffected, affectedDocuments, upsert) { assert.isNull(err); numAffected.should.equal(2); affectedDocuments.length.should.equal(2); assert.isUndefined(upsert); done(); }); }); }); it("Upsert", function (done) { d.insert({ a: 1 }); d.insert({ a: 2 }); // Upsert flag not set d.update({ a: 3 }, { $set: { b: 20 } }, {}, function (err, numAffected, affectedDocuments, upsert) { assert.isNull(err); numAffected.should.equal(0); assert.isUndefined(affectedDocuments); assert.isUndefined(upsert); // Upsert flag set d.update({ a: 3 }, { $set: { b: 21 } }, { upsert: true }, function (err, numAffected, affectedDocuments, upsert) { assert.isNull(err); numAffected.should.equal(1); affectedDocuments.a.should.equal(3); affectedDocuments.b.should.equal(21); upsert.should.equal(true); d.find({}, function (err, docs) { docs.length.should.equal(3); done(); }); }); }); }); }); // ==== End of 'Update - Callback signature' ==== // }); // ==== End of 'Update' ==== // describe('Remove', function () { it('Can remove multiple documents', function (done) { var id1, id2, id3; // Test DB status function testPostUpdateState (cb) { d.find({}, function (err, docs) { docs.length.should.equal(1); Object.keys(docs[0]).length.should.equal(2); docs[0]._id.should.equal(id1); docs[0].somedata.should.equal('ok'); return cb(); }); } // Actually launch the test async.waterfall([ function (cb) { d.insert({ somedata: 'ok' }, function (err, doc1) { id1 = doc1._id; d.insert({ somedata: 'again', plus: 'additional data' }, function (err, doc2) { id2 = doc2._id; d.insert({ somedata: 'again' }, function (err, doc3) { id3 = doc3._id; return cb(err); }); }); }); } , function (cb) { // Test with query that doesn't match anything d.remove({ somedata: 'again' }, { multi: true }, function (err, n) { assert.isNull(err); n.should.equal(2); return cb(); }); } , async.apply(testPostUpdateState) , function (cb) { d.loadDatabase(function (err) { return cb(err); }); } , async.apply(testPostUpdateState) ], done); }); // This tests concurrency issues it('Remove can be called multiple times in parallel and everything that needs to be removed will be', function (done) { d.insert({ planet: 'Earth' }, function () { d.insert({ planet: 'Mars' }, function () { d.insert({ planet: 'Saturn' }, function () { d.find({}, function (err, docs) { docs.length.should.equal(3); // Remove two docs simultaneously var toRemove = ['Mars', 'Saturn']; async.each(toRemove, function(planet, cb) { d.remove({ planet: planet }, function (err) { return cb(err); }); }, function (err) { d.find({}, function (err, docs) { docs.length.should.equal(1); done(); }); }); }); }); }); }); }); it('Returns an error if the query is not well formed', function (done) { d.insert({ hello: 'world' }, function () { d.remove({ $or: { hello: 'world' } }, {}, function (err, nr, upsert) { assert.isDefined(err); assert.isUndefined(nr); assert.isUndefined(upsert); done(); }); }); }); it('Non-multi removes are persistent', function (done) { d.insert({ a:1, hello: 'world' }, function (err, doc1) { d.insert({ a:2, hello: 'earth' }, function (err, doc2) { d.insert({ a:3, hello: 'moto' }, function (err, doc3) { d.remove({ a: 2 }, {}, function (err) { assert.isNull(err); d.find({}, function (err, docs) { docs.sort(function (a, b) { return a.a - b.a; }); docs.length.should.equal(2); _.isEqual(docs[0], { _id: doc1._id, a:1, hello: 'world' }).should.equal(true); _.isEqual(docs[1], { _id: doc3._id, a:3, hello: 'moto' }).should.equal(true); // Even after a reload the database state hasn't changed d.loadDatabase(function (err) { assert.isNull(err); d.find({}, function (err, docs) { docs.sort(function (a, b) { return a.a - b.a; }); docs.length.should.equal(2); _.isEqual(docs[0], { _id: doc1._id, a:1, hello: 'world' }).should.equal(true); _.isEqual(docs[1], { _id: doc3._id, a:3, hello: 'moto' }).should.equal(true); done(); }); }); }); }); }); }); }); }); it('Multi removes are persistent', function (done) { d.insert({ a:1, hello: 'world' }, function (err, doc1) { d.insert({ a:2, hello: 'earth' }, function (err, doc2) { d.insert({ a:3, hello: 'moto' }, function (err, doc3) { d.remove({ a: { $in: [1, 3] } }, { multi: true }, function (err) { assert.isNull(err); d.find({}, function (err, docs) { docs.length.should.equal(1); _.isEqual(docs[0], { _id: doc2._id, a:2, hello: 'earth' }).should.equal(true); // Even after a reload the database state hasn't changed d.loadDatabase(function (err) { assert.isNull(err); d.find({}, function (err, docs) { docs.length.should.equal(1); _.isEqual(docs[0], { _id: doc2._id, a:2, hello: 'earth' }).should.equal(true); done(); }); }); }); }); }); }); }); }); it('Can remove without the options arg (will use defaults then)', function (done) { d.insert({ a:1, hello: 'world' }, function (err, doc1) { d.insert({ a:2, hello: 'earth' }, function (err, doc2) { d.insert({ a:5, hello: 'pluton' }, function (err, doc3) { d.remove({ a: 2 }, function (err, nr) { assert.isNull(err); nr.should.equal(1); d.find({}, function (err, docs) { var d1 = _.find(docs, function (doc) { return doc._id === doc1._id }) , d2 = _.find(docs, function (doc) { return doc._id === doc2._id }) , d3 = _.find(docs, function (doc) { return doc._id === doc3._id }) ; d1.a.should.equal(1); assert.isUndefined(d2); d3.a.should.equal(5); done(); }); }); }); }); }); }); }); // ==== End of 'Remove' ==== // describe('Using indexes', function () { describe('ensureIndex and index initialization in database loading', function () { it('ensureIndex can be called right after a loadDatabase and be initialized and filled correctly', function (done) { var now = new Date() , rawData = model.serialize({ _id: "aaa", z: "1", a: 2, ages: [1, 5, 12] }) + '\n' + model.serialize({ _id: "bbb", z: "2", hello: 'world' }) + '\n' + model.serialize({ _id: "ccc", z: "3", nested: { today: now } }) ; d.getAllData().length.should.equal(0); fs.writeFile(testDb, rawData, 'utf8', function () { d.loadDatabase(function () { d.getAllData().length.should.equal(3); assert.deepEqual(Object.keys(d.indexes), ['_id']); d.ensureIndex({ fieldName: 'z' }); d.indexes.z.fieldName.should.equal('z'); d.indexes.z.unique.should.equal(false); d.indexes.z.sparse.should.equal(false); d.indexes.z.tree.getNumberOfKeys().should.equal(3); d.indexes.z.tree.search('1')[0].should.equal(d.getAllData()[0]); d.indexes.z.tree.search('2')[0].should.equal(d.getAllData()[1]); d.indexes.z.tree.search('3')[0].should.equal(d.getAllData()[2]); done(); }); }); }); it('ensureIndex can be called twice on the same field, the second call will ahve no effect', function (done) { Object.keys(d.indexes).length.should.equal(1); Object.keys(d.indexes)[0].should.equal("_id"); d.insert({ planet: "Earth" }, function () { d.insert({ planet: "Mars" }, function () { d.find({}, function (err, docs) { docs.length.should.equal(2); d.ensureIndex({ fieldName: "planet" }, function (err) { assert.isNull(err); Object.keys(d.indexes).length.should.equal(2); Object.keys(d.indexes)[0].should.equal("_id"); Object.keys(d.indexes)[1].should.equal("planet"); d.indexes.planet.getAll().length.should.equal(2); // This second call has no effect, documents don't get inserted twice in the index d.ensureIndex({ fieldName: "planet" }, function (err) { assert.isNull(err); Object.keys(d.indexes).length.should.equal(2); Object.keys(d.indexes)[0].should.equal("_id"); Object.keys(d.indexes)[1].should.equal("planet"); d.indexes.planet.getAll().length.should.equal(2); done(); }); }); }); }); }); }); it('ensureIndex can be called after the data set was modified and the index still be correct', function (done) { var rawData = model.serialize({ _id: "aaa", z: "1", a: 2, ages: [1, 5, 12] }) + '\n' + model.serialize({ _id: "bbb", z: "2", hello: 'world' }) ; d.getAllData().length.should.equal(0); fs.writeFile(testDb, rawData, 'utf8', function () { d.loadDatabase(function () { d.getAllData().length.should.equal(2); assert.deepEqual(Object.keys(d.indexes), ['_id']); d.insert({ z: "12", yes: 'yes' }, function (err, newDoc1) { d.insert({ z: "14", nope: 'nope' }, function (err, newDoc2) { d.remove({ z: "2" }, {}, function () { d.update({ z: "1" }, { $set: { 'yes': 'yep' } }, {}, function () { assert.deepEqual(Object.keys(d.indexes), ['_id']); d.ensureIndex({ fieldName: 'z' }); d.indexes.z.fieldName.should.equal('z'); d.indexes.z.unique.should.equal(false); d.indexes.z.sparse.should.equal(false); d.indexes.z.tree.getNumberOfKeys().should.equal(3); // The pointers in the _id and z indexes are the same d.indexes.z.tree.search('1')[0].should.equal(d.indexes._id.getMatching('aaa')[0]); d.indexes.z.tree.search('12')[0].should.equal(d.indexes._id.getMatching(newDoc1._id)[0]); d.indexes.z.tree.search('14')[0].should.equal(d.indexes._id.getMatching(newDoc2._id)[0]); // The data in the z index is correct d.find({}, function (err, docs) { var doc0 = _.find(docs, function (doc) { return doc._id === 'aaa'; }) , doc1 = _.find(docs, function (doc) { return doc._id === newDoc1._id; }) , doc2 = _.find(docs, function (doc) { return doc._id === newDoc2._id; }) ; docs.length.should.equal(3); assert.deepEqual(doc0, { _id: "aaa", z: "1", a: 2, ages: [1, 5, 12], yes: 'yep' }); assert.deepEqual(doc1, { _id: newDoc1._id, z: "12", yes: 'yes' }); assert.deepEqual(doc2, { _id: newDoc2._id, z: "14", nope: 'nope' }); done(); }); }); }); }); }); }); }); }); it('ensureIndex can be called before a loadDatabase and still be initialized and filled correctly', function (done) { var now = new Date() , rawData = model.serialize({ _id: "aaa", z: "1", a: 2, ages: [1, 5, 12] }) + '\n' + model.serialize({ _id: "bbb", z: "2", hello: 'world' }) + '\n' + model.serialize({ _id: "ccc", z: "3", nested: { today: now } }) ; d.getAllData().length.should.equal(0); d.ensureIndex({ fieldName: 'z' }); d.indexes.z.fieldName.should.equal('z'); d.indexes.z.unique.should.equal(false); d.indexes.z.sparse.should.equal(false); d.indexes.z.tree.getNumberOfKeys().should.equal(0); fs.writeFile(testDb, rawData, 'utf8', function () { d.loadDatabase(function () { var doc1 = _.find(d.getAllData(), function (doc) { return doc.z === "1"; }) , doc2 = _.find(d.getAllData(), function (doc) { return doc.z === "2"; }) , doc3 = _.find(d.getAllData(), function (doc) { return doc.z === "3"; }) ; d.getAllData().length.should.equal(3); d.indexes.z.tree.getNumberOfKeys().should.equal(3); d.indexes.z.tree.search('1')[0].should.equal(doc1); d.indexes.z.tree.search('2')[0].should.equal(doc2); d.indexes.z.tree.search('3')[0].should.equal(doc3); done(); }); }); }); it('Can initialize multiple indexes on a database load', function (done) { var now = new Date() , rawData = model.serialize({ _id: "aaa", z: "1", a: 2, ages: [1, 5, 12] }) + '\n' + model.serialize({ _id: "bbb", z: "2", a: 'world' }) + '\n' + model.serialize({ _id: "ccc", z: "3", a: { today: now } }) ; d.getAllData().length.should.equal(0); d.ensureIndex({ fieldName: 'z' }, function () { d.ensureIndex({ fieldName: 'a' }, function () { d.indexes.a.tree.getNumberOfKeys().should.equal(0); d.indexes.z.tree.getNumberOfKeys().should.equal(0); fs.writeFile(testDb, rawData, 'utf8', function () { d.loadDatabase(function (err) { var doc1 = _.find(d.getAllData(), function (doc) { return doc.z === "1"; }) , doc2 = _.find(d.getAllData(), function (doc) { return doc.z === "2"; }) , doc3 = _.find(d.getAllData(), function (doc) { return doc.z === "3"; }) ; assert.isNull(err); d.getAllData().length.should.equal(3); d.indexes.z.tree.getNumberOfKeys().should.equal(3); d.indexes.z.tree.search('1')[0].should.equal(doc1); d.indexes.z.tree.search('2')[0].should.equal(doc2); d.indexes.z.tree.search('3')[0].should.equal(doc3); d.indexes.a.tree.getNumberOfKeys().should.equal(3); d.indexes.a.tree.search(2)[0].should.equal(doc1); d.indexes.a.tree.search('world')[0].should.equal(doc2); d.indexes.a.tree.search({ today: now })[0].should.equal(doc3); done(); }); }); }); }); }); it('If a unique constraint is not respected, database loading will not work and no data will be inserted', function (done) { var now = new Date() , rawData = model.serialize({ _id: "aaa", z: "1", a: 2, ages: [1, 5, 12] }) + '\n' + model.serialize({ _id: "bbb", z: "2", a: 'world' }) + '\n' + model.serialize({ _id: "ccc", z: "1", a: { today: now } }) ; d.getAllData().length.should.equal(0); d.ensureIndex({ fieldName: 'z', unique: true }); d.indexes.z.tree.getNumberOfKeys().should.equal(0); fs.writeFile(testDb, rawData, 'utf8', function () { d.loadDatabase(function (err) { err.errorType.should.equal('uniqueViolated'); err.key.should.equal("1"); d.getAllData().length.should.equal(0); d.indexes.z.tree.getNumberOfKeys().should.equal(0); done(); }); }); }); it('If a unique constraint is not respected, ensureIndex will return an error and not create an index', function (done) { d.insert({ a: 1, b: 4 }, function () { d.insert({ a: 2, b: 45 }, function () { d.insert({ a: 1, b: 3 }, function () { d.ensureIndex({ fieldName: 'b' }, function (err) { assert.isNull(err); d.ensureIndex({ fieldName: 'a', unique: true }, function (err) { err.errorType.should.equal('uniqueViolated'); assert.deepEqual(Object.keys(d.indexes), ['_id', 'b']); done(); }); }); }); }); }); }); it('Can remove an index', function (done) { d.ensureIndex({ fieldName: 'e' }, function (err) { assert.isNull(err); Object.keys(d.indexes).length.should.equal(2); assert.isNotNull(d.indexes.e); d.removeIndex("e", function (err) { assert.isNull(err); Object.keys(d.indexes).length.should.equal(1); assert.isUndefined(d.indexes.e); done(); }); }); }); }); // ==== End of 'ensureIndex and index initialization in database loading' ==== // describe('Indexing newly inserted documents', function () { it('Newly inserted documents are indexed', function (done) { d.ensureIndex({ fieldName: 'z' }); d.indexes.z.tree.getNumberOfKeys().should.equal(0); d.insert({ a: 2, z: 'yes' }, function (err, newDoc) { d.indexes.z.tree.getNumberOfKeys().should.equal(1); assert.deepEqual(d.indexes.z.getMatching('yes'), [newDoc]); d.insert({ a: 5, z: 'nope' }, function (err, newDoc) { d.indexes.z.tree.getNumberOfKeys().should.equal(2); assert.deepEqual(d.indexes.z.getMatching('nope'), [newDoc]); done(); }); }); }); it('If multiple indexes are defined, the document is inserted in all of them', function (done) { d.ensureIndex({ fieldName: 'z' }); d.ensureIndex({ fieldName: 'ya' }); d.indexes.z.tree.getNumberOfKeys().should.equal(0); d.insert({ a: 2, z: 'yes', ya: 'indeed' }, function (err, newDoc) { d.indexes.z.tree.getNumberOfKeys().should.equal(1); d.indexes.ya.tree.getNumberOfKeys().should.equal(1); assert.deepEqual(d.indexes.z.getMatching('yes'), [newDoc]); assert.deepEqual(d.indexes.ya.getMatching('indeed'), [newDoc]); d.insert({ a: 5, z: 'nope', ya: 'sure' }, function (err, newDoc2) { d.indexes.z.tree.getNumberOfKeys().should.equal(2); d.indexes.ya.tree.getNumberOfKeys().should.equal(2); assert.deepEqual(d.indexes.z.getMatching('nope'), [newDoc2]); assert.deepEqual(d.indexes.ya.getMatching('sure'), [newDoc2]); done(); }); }); }); it('Can insert two docs at the same key for a non unique index', function (done) { d.ensureIndex({ fieldName: 'z' }); d.indexes.z.tree.getNumberOfKeys().should.equal(0); d.insert({ a: 2, z: 'yes' }, function (err, newDoc) { d.indexes.z.tree.getNumberOfKeys().should.equal(1); assert.deepEqual(d.indexes.z.getMatching('yes'), [newDoc]); d.insert({ a: 5, z: 'yes' }, function (err, newDoc2) { d.indexes.z.tree.getNumberOfKeys().should.equal(1); assert.deepEqual(d.indexes.z.getMatching('yes'), [newDoc, newDoc2]); done(); }); }); }); it('If the index has a unique constraint, an error is thrown if it is violated and the data is not modified', function (done) { d.ensureIndex({ fieldName: 'z', unique: true }); d.indexes.z.tree.getNumberOfKeys().should.equal(0); d.insert({ a: 2, z: 'yes' }, function (err, newDoc) { d.indexes.z.tree.getNumberOfKeys().should.equal(1); assert.deepEqual(d.indexes.z.getMatching('yes'), [newDoc]); d.insert({ a: 5, z: 'yes' }, function (err) { err.errorType.should.equal('uniqueViolated'); err.key.should.equal('yes'); // Index didn't change d.indexes.z.tree.getNumberOfKeys().should.equal(1); assert.deepEqual(d.indexes.z.getMatching('yes'), [newDoc]); // Data didn't change assert.deepEqual(d.getAllData(), [newDoc]); d.loadDatabase(function () { d.getAllData().length.should.equal(1); assert.deepEqual(d.getAllData()[0], newDoc); done(); }); }); }); }); it('If an index has a unique constraint, other indexes cannot be modified when it raises an error', function (done) { d.ensureIndex({ fieldName: 'nonu1' }); d.ensureIndex({ fieldName: 'uni', unique: true }); d.ensureIndex({ fieldName: 'nonu2' }); d.insert({ nonu1: 'yes', nonu2: 'yes2', uni: 'willfail' }, function (err, newDoc) { assert.isNull(err); d.indexes.nonu1.tree.getNumberOfKeys().should.equal(1); d.indexes.uni.tree.getNumberOfKeys().should.equal(1); d.indexes.nonu2.tree.getNumberOfKeys().should.equal(1); d.insert({ nonu1: 'no', nonu2: 'no2', uni: 'willfail' }, function (err) { err.errorType.should.equal('uniqueViolated'); // No index was modified d.indexes.nonu1.tree.getNumberOfKeys().should.equal(1); d.indexes.uni.tree.getNumberOfKeys().should.equal(1); d.indexes.nonu2.tree.getNumberOfKeys().should.equal(1); assert.deepEqual(d.indexes.nonu1.getMatching('yes'), [newDoc]); assert.deepEqual(d.indexes.uni.getMatching('willfail'), [newDoc]); assert.deepEqual(d.indexes.nonu2.getMatching('yes2'), [newDoc]); done(); }); }); }); it('Unique indexes prevent you from inserting two docs where the field is undefined except if theyre sparse', function (done) { d.ensureIndex({ fieldName: 'zzz', unique: true }); d.indexes.zzz.tree.getNumberOfKeys().should.equal(0); d.insert({ a: 2, z: 'yes' }, function (err, newDoc) { d.indexes.zzz.tree.getNumberOfKeys().should.equal(1); assert.deepEqual(d.indexes.zzz.getMatching(undefined), [newDoc]); d.insert({ a: 5, z: 'other' }, function (err) { err.errorType.should.equal('uniqueViolated'); assert.isUndefined(err.key); d.ensureIndex({ fieldName: 'yyy', unique: true, sparse: true }); d.insert({ a: 5, z: 'other', zzz: 'set' }, function (err) { assert.isNull(err); d.indexes.yyy.getAll().length.should.equal(0); // Nothing indexed d.indexes.zzz.getAll().length.should.equal(2); done(); }); }); }); }); it('Insertion still works as before with indexing', function (done) { d.ensureIndex({ fieldName: 'a' }); d.ensureIndex({ fieldName: 'b' }); d.insert({ a: 1, b: 'hello' }, function (err, doc1) { d.insert({ a: 2, b: 'si' }, function (err, doc2) { d.find({}, function (err, docs) { assert.deepEqual(doc1, _.find(docs, function (d) { return d._id === doc1._id; })); assert.deepEqual(doc2, _.find(docs, function (d) { return d._id === doc2._id; })); done(); }); }); }); }); it('All indexes point to the same data as the main index on _id', function (done) { d.ensureIndex({ fieldName: 'a' }); d.insert({ a: 1, b: 'hello' }, function (err, doc1) { d.insert({ a: 2, b: 'si' }, function (err, doc2) { d.find({}, function (err, docs) { docs.length.should.equal(2); d.getAllData().length.should.equal(2); d.indexes._id.getMatching(doc1._id).length.should.equal(1); d.indexes.a.getMatching(1).length.should.equal(1); d.indexes._id.getMatching(doc1._id)[0].should.equal(d.indexes.a.getMatching(1)[0]); d.indexes._id.getMatching(doc2._id).length.should.equal(1); d.indexes.a.getMatching(2).length.should.equal(1); d.indexes._id.getMatching(doc2._id)[0].should.equal(d.indexes.a.getMatching(2)[0]); done(); }); }); }); }); it('If a unique constraint is violated, no index is changed, including the main one', function (done) { d.ensureIndex({ fieldName: 'a', unique: true }); d.insert({ a: 1, b: 'hello' }, function (err, doc1) { d.insert({ a: 1, b: 'si' }, function (err) { assert.isDefined(err); d.find({}, function (err, docs) { docs.length.should.equal(1); d.getAllData().length.should.equal(1); d.indexes._id.getMatching(doc1._id).length.should.equal(1); d.indexes.a.getMatching(1).length.should.equal(1); d.indexes._id.getMatching(doc1._id)[0].should.equal(d.indexes.a.getMatching(1)[0]); d.indexes.a.getMatching(2).length.should.equal(0); done(); }); }); }); }); }); // ==== End of 'Indexing newly inserted documents' ==== // describe('Updating indexes upon document update', function () { it('Updating docs still works as before with indexing', function (done) { d.ensureIndex({ fieldName: 'a' }); d.insert({ a: 1, b: 'hello' }, function (err, _doc1) { d.insert({ a: 2, b: 'si' }, function (err, _doc2) { d.update({ a: 1 }, { $set: { a: 456, b: 'no' } }, {}, function (err, nr) { var data = d.getAllData() , doc1 = _.find(data, function (doc) { return doc._id === _doc1._id; }) , doc2 = _.find(data, function (doc) { return doc._id === _doc2._id; }) ; assert.isNull(err); nr.should.equal(1); data.length.should.equal(2); assert.deepEqual(doc1, { a: 456, b: 'no', _id: _doc1._id }); assert.deepEqual(doc2, { a: 2, b: 'si', _id: _doc2._id }); d.update({}, { $inc: { a: 10 }, $set: { b: 'same' } }, { multi: true }, function (err, nr) { var data = d.getAllData() , doc1 = _.find(data, function (doc) { return doc._id === _doc1._id; }) , doc2 = _.find(data, function (doc) { return doc._id === _doc2._id; }) ; assert.isNull(err); nr.should.equal(2); data.length.should.equal(2); assert.deepEqual(doc1, { a: 466, b: 'same', _id: _doc1._id }); assert.deepEqual(doc2, { a: 12, b: 'same', _id: _doc2._id }); done(); }); }); }); }); }); it('Indexes get updated when a document (or multiple documents) is updated', function (done) { d.ensureIndex({ fieldName: 'a' }); d.ensureIndex({ fieldName: 'b' }); d.insert({ a: 1, b: 'hello' }, function (err, doc1) { d.insert({ a: 2, b: 'si' }, function (err, doc2) { // Simple update d.update({ a: 1 }, { $set: { a: 456, b: 'no' } }, {}, function (err, nr) { assert.isNull(err); nr.should.equal(1); d.indexes.a.tree.getNumberOfKeys().should.equal(2); d.indexes.a.getMatching(456)[0]._id.should.equal(doc1._id); d.indexes.a.getMatching(2)[0]._id.should.equal(doc2._id); d.indexes.b.tree.getNumberOfKeys().should.equal(2); d.indexes.b.getMatching('no')[0]._id.should.equal(doc1._id); d.indexes.b.getMatching('si')[0]._id.should.equal(doc2._id); // The same pointers are shared between all indexes d.indexes.a.tree.getNumberOfKeys().should.equal(2); d.indexes.b.tree.getNumberOfKeys().should.equal(2); d.indexes._id.tree.getNumberOfKeys().should.equal(2); d.indexes.a.getMatching(456)[0].should.equal(d.indexes._id.getMatching(doc1._id)[0]); d.indexes.b.getMatching('no')[0].should.equal(d.indexes._id.getMatching(doc1._id)[0]); d.indexes.a.getMatching(2)[0].should.equal(d.indexes._id.getMatching(doc2._id)[0]); d.indexes.b.getMatching('si')[0].should.equal(d.indexes._id.getMatching(doc2._id)[0]); // Multi update d.update({}, { $inc: { a: 10 }, $set: { b: 'same' } }, { multi: true }, function (err, nr) { assert.isNull(err); nr.should.equal(2); d.indexes.a.tree.getNumberOfKeys().should.equal(2); d.indexes.a.getMatching(466)[0]._id.should.equal(doc1._id); d.indexes.a.getMatching(12)[0]._id.should.equal(doc2._id); d.indexes.b.tree.getNumberOfKeys().should.equal(1); d.indexes.b.getMatching('same').length.should.equal(2); _.pluck(d.indexes.b.getMatching('same'), '_id').should.contain(doc1._id); _.pluck(d.indexes.b.getMatching('same'), '_id').should.contain(doc2._id); // The same pointers are shared between all indexes d.indexes.a.tree.getNumberOfKeys().should.equal(2); d.indexes.b.tree.getNumberOfKeys().should.equal(1); d.indexes.b.getAll().length.should.equal(2); d.indexes._id.tree.getNumberOfKeys().should.equal(2); d.indexes.a.getMatching(466)[0].should.equal(d.indexes._id.getMatching(doc1._id)[0]); d.indexes.a.getMatching(12)[0].should.equal(d.indexes._id.getMatching(doc2._id)[0]); // Can't test the pointers in b as their order is randomized, but it is the same as with a done(); }); }); }); }); }); it('If a simple update violates a contraint, all changes are rolled back and an error is thrown', function (done) { d.ensureIndex({ fieldName: 'a', unique: true }); d.ensureIndex({ fieldName: 'b', unique: true }); d.ensureIndex({ fieldName: 'c', unique: true }); d.insert({ a: 1, b: 10, c: 100 }, function (err, _doc1) { d.insert({ a: 2, b: 20, c: 200 }, function (err, _doc2) { d.insert({ a: 3, b: 30, c: 300 }, function (err, _doc3) { // Will conflict with doc3 d.update({ a: 2 }, { $inc: { a: 10, c: 1000 }, $set: { b: 30 } }, {}, function (err) { var data = d.getAllData() , doc1 = _.find(data, function (doc) { return doc._id === _doc1._id; }) , doc2 = _.find(data, function (doc) { return doc._id === _doc2._id; }) , doc3 = _.find(data, function (doc) { return doc._id === _doc3._id; }) ; err.errorType.should.equal('uniqueViolated'); // Data left unchanged data.length.should.equal(3); assert.deepEqual(doc1, { a: 1, b: 10, c: 100, _id: _doc1._id }); assert.deepEqual(doc2, { a: 2, b: 20, c: 200, _id: _doc2._id }); assert.deepEqual(doc3, { a: 3, b: 30, c: 300, _id: _doc3._id }); // All indexes left unchanged and pointing to the same docs d.indexes.a.tree.getNumberOfKeys().should.equal(3); d.indexes.a.getMatching(1)[0].should.equal(doc1); d.indexes.a.getMatching(2)[0].should.equal(doc2); d.indexes.a.getMatching(3)[0].should.equal(doc3); d.indexes.b.tree.getNumberOfKeys().should.equal(3); d.indexes.b.getMatching(10)[0].should.equal(doc1); d.indexes.b.getMatching(20)[0].should.equal(doc2); d.indexes.b.getMatching(30)[0].should.equal(doc3); d.indexes.c.tree.getNumberOfKeys().should.equal(3); d.indexes.c.getMatching(100)[0].should.equal(doc1); d.indexes.c.getMatching(200)[0].should.equal(doc2); d.indexes.c.getMatching(300)[0].should.equal(doc3); done(); }); }); }); }); }); it('If a multi update violates a contraint, all changes are rolled back and an error is thrown', function (done) { d.ensureIndex({ fieldName: 'a', unique: true }); d.ensureIndex({ fieldName: 'b', unique: true }); d.ensureIndex({ fieldName: 'c', unique: true }); d.insert({ a: 1, b: 10, c: 100 }, function (err, _doc1) { d.insert({ a: 2, b: 20, c: 200 }, function (err, _doc2) { d.insert({ a: 3, b: 30, c: 300 }, function (err, _doc3) { // Will conflict with doc3 d.update({ a: { $in: [1, 2] } }, { $inc: { a: 10, c: 1000 }, $set: { b: 30 } }, { multi: true }, function (err) { var data = d.getAllData() , doc1 = _.find(data, function (doc) { return doc._id === _doc1._id; }) , doc2 = _.find(data, function (doc) { return doc._id === _doc2._id; }) , doc3 = _.find(data, function (doc) { return doc._id === _doc3._id; }) ; err.errorType.should.equal('uniqueViolated'); // Data left unchanged data.length.should.equal(3); assert.deepEqual(doc1, { a: 1, b: 10, c: 100, _id: _doc1._id }); assert.deepEqual(doc2, { a: 2, b: 20, c: 200, _id: _doc2._id }); assert.deepEqual(doc3, { a: 3, b: 30, c: 300, _id: _doc3._id }); // All indexes left unchanged and pointing to the same docs d.indexes.a.tree.getNumberOfKeys().should.equal(3); d.indexes.a.getMatching(1)[0].should.equal(doc1); d.indexes.a.getMatching(2)[0].should.equal(doc2); d.indexes.a.getMatching(3)[0].should.equal(doc3); d.indexes.b.tree.getNumberOfKeys().should.equal(3); d.indexes.b.getMatching(10)[0].should.equal(doc1); d.indexes.b.getMatching(20)[0].should.equal(doc2); d.indexes.b.getMatching(30)[0].should.equal(doc3); d.indexes.c.tree.getNumberOfKeys().should.equal(3); d.indexes.c.getMatching(100)[0].should.equal(doc1); d.indexes.c.getMatching(200)[0].should.equal(doc2); d.indexes.c.getMatching(300)[0].should.equal(doc3); done(); }); }); }); }); }); }); // ==== End of 'Updating indexes upon document update' ==== // describe('Updating indexes upon document remove', function () { it('Removing docs still works as before with indexing', function (done) { d.ensureIndex({ fieldName: 'a' }); d.insert({ a: 1, b: 'hello' }, function (err, _doc1) { d.insert({ a: 2, b: 'si' }, function (err, _doc2) { d.insert({ a: 3, b: 'coin' }, function (err, _doc3) { d.remove({ a: 1 }, {}, function (err, nr) { var data = d.getAllData() , doc2 = _.find(data, function (doc) { return doc._id === _doc2._id; }) , doc3 = _.find(data, function (doc) { return doc._id === _doc3._id; }) ; assert.isNull(err); nr.should.equal(1); data.length.should.equal(2); assert.deepEqual(doc2, { a: 2, b: 'si', _id: _doc2._id }); assert.deepEqual(doc3, { a: 3, b: 'coin', _id: _doc3._id }); d.remove({ a: { $in: [2, 3] } }, { multi: true }, function (err, nr) { var data = d.getAllData() ; assert.isNull(err); nr.should.equal(2); data.length.should.equal(0); done(); }); }); }); }); }); }); it('Indexes get updated when a document (or multiple documents) is removed', function (done) { d.ensureIndex({ fieldName: 'a' }); d.ensureIndex({ fieldName: 'b' }); d.insert({ a: 1, b: 'hello' }, function (err, doc1) { d.insert({ a: 2, b: 'si' }, function (err, doc2) { d.insert({ a: 3, b: 'coin' }, function (err, doc3) { // Simple remove d.remove({ a: 1 }, {}, function (err, nr) { assert.isNull(err); nr.should.equal(1); d.indexes.a.tree.getNumberOfKeys().should.equal(2); d.indexes.a.getMatching(2)[0]._id.should.equal(doc2._id); d.indexes.a.getMatching(3)[0]._id.should.equal(doc3._id); d.indexes.b.tree.getNumberOfKeys().should.equal(2); d.indexes.b.getMatching('si')[0]._id.should.equal(doc2._id); d.indexes.b.getMatching('coin')[0]._id.should.equal(doc3._id); // The same pointers are shared between all indexes d.indexes.a.tree.getNumberOfKeys().should.equal(2); d.indexes.b.tree.getNumberOfKeys().should.equal(2); d.indexes._id.tree.getNumberOfKeys().should.equal(2); d.indexes.a.getMatching(2)[0].should.equal(d.indexes._id.getMatching(doc2._id)[0]); d.indexes.b.getMatching('si')[0].should.equal(d.indexes._id.getMatching(doc2._id)[0]); d.indexes.a.getMatching(3)[0].should.equal(d.indexes._id.getMatching(doc3._id)[0]); d.indexes.b.getMatching('coin')[0].should.equal(d.indexes._id.getMatching(doc3._id)[0]); // Multi remove d.remove({}, { multi: true }, function (err, nr) { assert.isNull(err); nr.should.equal(2); d.indexes.a.tree.getNumberOfKeys().should.equal(0); d.indexes.b.tree.getNumberOfKeys().should.equal(0); d.indexes._id.tree.getNumberOfKeys().should.equal(0); done(); }); }); }); }); }); }); }); // ==== End of 'Updating indexes upon document remove' ==== // describe('Persisting indexes', function () { it('Indexes are persisted to a separate file and recreated upon reload', function (done) { var persDb = "workspace/persistIndexes.db" , db ; if (fs.existsSync(persDb)) { fs.writeFileSync(persDb, '', 'utf8'); } db = new Datastore({ filename: persDb, autoload: true }); Object.keys(db.indexes).length.should.equal(1); Object.keys(db.indexes)[0].should.equal("_id"); db.insert({ planet: "Earth" }, function (err) { assert.isNull(err); db.insert({ planet: "Mars" }, function (err) { assert.isNull(err); db.ensureIndex({ fieldName: "planet" }, function (err) { Object.keys(db.indexes).length.should.equal(2); Object.keys(db.indexes)[0].should.equal("_id"); Object.keys(db.indexes)[1].should.equal("planet"); db.indexes._id.getAll().length.should.equal(2); db.indexes.planet.getAll().length.should.equal(2); db.indexes.planet.fieldName.should.equal("planet"); // After a reload the indexes are recreated db = new Datastore({ filename: persDb }); db.loadDatabase(function (err) { assert.isNull(err); Object.keys(db.indexes).length.should.equal(2); Object.keys(db.indexes)[0].should.equal("_id"); Object.keys(db.indexes)[1].should.equal("planet"); db.indexes._id.getAll().length.should.equal(2); db.indexes.planet.getAll().length.should.equal(2); db.indexes.planet.fieldName.should.equal("planet"); // After another reload the indexes are still there (i.e. they are preserved during autocompaction) db = new Datastore({ filename: persDb }); db.loadDatabase(function (err) { assert.isNull(err); Object.keys(db.indexes).length.should.equal(2); Object.keys(db.indexes)[0].should.equal("_id"); Object.keys(db.indexes)[1].should.equal("planet"); db.indexes._id.getAll().length.should.equal(2); db.indexes.planet.getAll().length.should.equal(2); db.indexes.planet.fieldName.should.equal("planet"); done(); }); }); }); }); }); }); it('Indexes are persisted with their options and recreated even if some db operation happen between loads', function (done) { var persDb = "workspace/persistIndexes.db" , db ; if (fs.existsSync(persDb)) { fs.writeFileSync(persDb, '', 'utf8'); } db = new Datastore({ filename: persDb, autoload: true }); Object.keys(db.indexes).length.should.equal(1); Object.keys(db.indexes)[0].should.equal("_id"); db.insert({ planet: "Earth" }, function (err) { assert.isNull(err); db.insert({ planet: "Mars" }, function (err) { assert.isNull(err); db.ensureIndex({ fieldName: "planet", unique: true, sparse: false }, function (err) { Object.keys(db.indexes).length.should.equal(2); Object.keys(db.indexes)[0].should.equal("_id"); Object.keys(db.indexes)[1].should.equal("planet"); db.indexes._id.getAll().length.should.equal(2); db.indexes.planet.getAll().length.should.equal(2); db.indexes.planet.unique.should.equal(true); db.indexes.planet.sparse.should.equal(false); db.insert({ planet: "Jupiter" }, function (err) { assert.isNull(err); // After a reload the indexes are recreated db = new Datastore({ filename: persDb }); db.loadDatabase(function (err) { assert.isNull(err); Object.keys(db.indexes).length.should.equal(2); Object.keys(db.indexes)[0].should.equal("_id"); Object.keys(db.indexes)[1].should.equal("planet"); db.indexes._id.getAll().length.should.equal(3); db.indexes.planet.getAll().length.should.equal(3); db.indexes.planet.unique.should.equal(true); db.indexes.planet.sparse.should.equal(false); db.ensureIndex({ fieldName: 'bloup', unique: false, sparse: true }, function (err) { assert.isNull(err); Object.keys(db.indexes).length.should.equal(3); Object.keys(db.indexes)[0].should.equal("_id"); Object.keys(db.indexes)[1].should.equal("planet"); Object.keys(db.indexes)[2].should.equal("bloup"); db.indexes._id.getAll().length.should.equal(3); db.indexes.planet.getAll().length.should.equal(3); db.indexes.bloup.getAll().length.should.equal(0); db.indexes.planet.unique.should.equal(true); db.indexes.planet.sparse.should.equal(false); db.indexes.bloup.unique.should.equal(false); db.indexes.bloup.sparse.should.equal(true); // After another reload the indexes are still there (i.e. they are preserved during autocompaction) db = new Datastore({ filename: persDb }); db.loadDatabase(function (err) { assert.isNull(err); Object.keys(db.indexes).length.should.equal(3); Object.keys(db.indexes)[0].should.equal("_id"); Object.keys(db.indexes)[1].should.equal("planet"); Object.keys(db.indexes)[2].should.equal("bloup"); db.indexes._id.getAll().length.should.equal(3); db.indexes.planet.getAll().length.should.equal(3); db.indexes.bloup.getAll().length.should.equal(0); db.indexes.planet.unique.should.equal(true); db.indexes.planet.sparse.should.equal(false); db.indexes.bloup.unique.should.equal(false); db.indexes.bloup.sparse.should.equal(true); done(); }); }); }); }); }); }); }); }); it('Indexes can also be removed and the remove persisted', function (done) { var persDb = "workspace/persistIndexes.db" , db ; if (fs.existsSync(persDb)) { fs.writeFileSync(persDb, '', 'utf8'); } db = new Datastore({ filename: persDb, autoload: true }); Object.keys(db.indexes).length.should.equal(1); Object.keys(db.indexes)[0].should.equal("_id"); db.insert({ planet: "Earth" }, function (err) { assert.isNull(err); db.insert({ planet: "Mars" }, function (err) { assert.isNull(err); db.ensureIndex({ fieldName: "planet" }, function (err) { assert.isNull(err); db.ensureIndex({ fieldName: "another" }, function (err) { assert.isNull(err); Object.keys(db.indexes).length.should.equal(3); Object.keys(db.indexes)[0].should.equal("_id"); Object.keys(db.indexes)[1].should.equal("planet"); Object.keys(db.indexes)[2].should.equal("another"); db.indexes._id.getAll().length.should.equal(2); db.indexes.planet.getAll().length.should.equal(2); db.indexes.planet.fieldName.should.equal("planet"); // After a reload the indexes are recreated db = new Datastore({ filename: persDb }); db.loadDatabase(function (err) { assert.isNull(err); Object.keys(db.indexes).length.should.equal(3); Object.keys(db.indexes)[0].should.equal("_id"); Object.keys(db.indexes)[1].should.equal("planet"); Object.keys(db.indexes)[2].should.equal("another"); db.indexes._id.getAll().length.should.equal(2); db.indexes.planet.getAll().length.should.equal(2); db.indexes.planet.fieldName.should.equal("planet"); // Index is removed db.removeIndex("planet", function (err) { assert.isNull(err); Object.keys(db.indexes).length.should.equal(2); Object.keys(db.indexes)[0].should.equal("_id"); Object.keys(db.indexes)[1].should.equal("another"); db.indexes._id.getAll().length.should.equal(2); // After a reload indexes are preserved db = new Datastore({ filename: persDb }); db.loadDatabase(function (err) { assert.isNull(err); Object.keys(db.indexes).length.should.equal(2); Object.keys(db.indexes)[0].should.equal("_id"); Object.keys(db.indexes)[1].should.equal("another"); db.indexes._id.getAll().length.should.equal(2); // After another reload the indexes are still there (i.e. they are preserved during autocompaction) db = new Datastore({ filename: persDb }); db.loadDatabase(function (err) { assert.isNull(err); Object.keys(db.indexes).length.should.equal(2); Object.keys(db.indexes)[0].should.equal("_id"); Object.keys(db.indexes)[1].should.equal("another"); db.indexes._id.getAll().length.should.equal(2); done(); }); }); }); }); }); }); }); }); }); }); // ==== End of 'Persisting indexes' ==== it('Results of getMatching should never contain duplicates', function (done) { d.ensureIndex({ fieldName: 'bad' }); d.insert({ bad: ['a', 'b'] }, function () { d.getCandidates({ bad: { $in: ['a', 'b'] } }, function (err, res) { res.length.should.equal(1); done(); }); }); }); }); // ==== End of 'Using indexes' ==== // }); ================================================ FILE: test/executor.test.js ================================================ var should = require('chai').should() , assert = require('chai').assert , testDb = 'workspace/test.db' , fs = require('fs') , path = require('path') , _ = require('underscore') , async = require('async') , model = require('../lib/model') , Datastore = require('../lib/datastore') , Persistence = require('../lib/persistence') ; // Test that even if a callback throws an exception, the next DB operations will still be executed // We prevent Mocha from catching the exception we throw on purpose by remembering all current handlers, remove them and register them back after test ends function testThrowInCallback (d, done) { var currentUncaughtExceptionHandlers = process.listeners('uncaughtException'); process.removeAllListeners('uncaughtException'); process.on('uncaughtException', function (err) { // Do nothing with the error which is only there to test we stay on track }); d.find({}, function (err) { process.nextTick(function () { d.insert({ bar: 1 }, function (err) { process.removeAllListeners('uncaughtException'); for (var i = 0; i < currentUncaughtExceptionHandlers.length; i += 1) { process.on('uncaughtException', currentUncaughtExceptionHandlers[i]); } done(); }); }); throw new Error('Some error'); }); } // Test that if the callback is falsy, the next DB operations will still be executed function testFalsyCallback (d, done) { d.insert({ a: 1 }, null); process.nextTick(function () { d.update({ a: 1 }, { a: 2 }, {}, null); process.nextTick(function () { d.update({ a: 2 }, { a: 1 }, null); process.nextTick(function () { d.remove({ a: 2 }, {}, null); process.nextTick(function () { d.remove({ a: 2 }, null); process.nextTick(function () { d.find({}, done); }); }); }); }); }); } // Test that operations are executed in the right order // We prevent Mocha from catching the exception we throw on purpose by remembering all current handlers, remove them and register them back after test ends function testRightOrder (d, done) { var currentUncaughtExceptionHandlers = process.listeners('uncaughtException'); process.removeAllListeners('uncaughtException'); process.on('uncaughtException', function (err) { // Do nothing with the error which is only there to test we stay on track }); d.find({}, function (err, docs) { docs.length.should.equal(0); d.insert({ a: 1 }, function () { d.update({ a: 1 }, { a: 2 }, {}, function () { d.find({}, function (err, docs) { docs[0].a.should.equal(2); process.nextTick(function () { d.update({ a: 2 }, { a: 3 }, {}, function () { d.find({}, function (err, docs) { docs[0].a.should.equal(3); process.removeAllListeners('uncaughtException'); for (var i = 0; i < currentUncaughtExceptionHandlers.length; i += 1) { process.on('uncaughtException', currentUncaughtExceptionHandlers[i]); } done(); }); }); }); throw new Error('Some error'); }); }); }); }); } // Note: The following test does not have any assertion because it // is meant to address the deprecation warning: // (node) warning: Recursive process.nextTick detected. This will break in the next version of node. Please use setImmediate for recursive deferral. // see var testEventLoopStarvation = function(d, done){ var times = 1001; var i = 0; while ( i b ? -1 : 1; }).should.equal(-1); }); }); // ==== End of 'Comparing things' ==== // describe('Querying', function () { describe('Comparing things', function () { it('Two things of different types cannot be equal, two identical native things are equal', function () { var toTest = [null, 'somestring', 42, true, new Date(72998322), { hello: 'world' }] , toTestAgainst = [null, 'somestring', 42, true, new Date(72998322), { hello: 'world' }] // Use another array so that we don't test pointer equality , i, j ; for (i = 0; i < toTest.length; i += 1) { for (j = 0; j < toTestAgainst.length; j += 1) { model.areThingsEqual(toTest[i], toTestAgainst[j]).should.equal(i === j); } } }); it('Can test native types null undefined string number boolean date equality', function () { var toTest = [null, undefined, 'somestring', 42, true, new Date(72998322), { hello: 'world' }] , toTestAgainst = [undefined, null, 'someotherstring', 5, false, new Date(111111), { hello: 'mars' }] , i ; for (i = 0; i < toTest.length; i += 1) { model.areThingsEqual(toTest[i], toTestAgainst[i]).should.equal(false); } }); it('If one side is an array or undefined, comparison fails', function () { var toTestAgainst = [null, undefined, 'somestring', 42, true, new Date(72998322), { hello: 'world' }] , i ; for (i = 0; i < toTestAgainst.length; i += 1) { model.areThingsEqual([1, 2, 3], toTestAgainst[i]).should.equal(false); model.areThingsEqual(toTestAgainst[i], []).should.equal(false); model.areThingsEqual(undefined, toTestAgainst[i]).should.equal(false); model.areThingsEqual(toTestAgainst[i], undefined).should.equal(false); } }); it('Can test objects equality', function () { model.areThingsEqual({ hello: 'world' }, {}).should.equal(false); model.areThingsEqual({ hello: 'world' }, { hello: 'mars' }).should.equal(false); model.areThingsEqual({ hello: 'world' }, { hello: 'world', temperature: 42 }).should.equal(false); model.areThingsEqual({ hello: 'world', other: { temperature: 42 }}, { hello: 'world', other: { temperature: 42 }}).should.equal(true); }); }); describe('Getting a fields value in dot notation', function () { it('Return first-level and nested values', function () { model.getDotValue({ hello: 'world' }, 'hello').should.equal('world'); model.getDotValue({ hello: 'world', type: { planet: true, blue: true } }, 'type.planet').should.equal(true); }); it('Return undefined if the field cannot be found in the object', function () { assert.isUndefined(model.getDotValue({ hello: 'world' }, 'helloo')); assert.isUndefined(model.getDotValue({ hello: 'world', type: { planet: true } }, 'type.plane')); }); it("Can navigate inside arrays with dot notation, and return the array of values in that case", function () { var dv; // Simple array of subdocuments dv = model.getDotValue({ planets: [ { name: 'Earth', number: 3 }, { name: 'Mars', number: 2 }, { name: 'Pluton', number: 9 } ] }, 'planets.name'); assert.deepEqual(dv, ['Earth', 'Mars', 'Pluton']); // Nested array of subdocuments dv = model.getDotValue({ nedb: true, data: { planets: [ { name: 'Earth', number: 3 }, { name: 'Mars', number: 2 }, { name: 'Pluton', number: 9 } ] } }, 'data.planets.number'); assert.deepEqual(dv, [3, 2, 9]); // Nested array in a subdocument of an array (yay, inception!) // TODO: make sure MongoDB doesn't flatten the array (it wouldn't make sense) dv = model.getDotValue({ nedb: true, data: { planets: [ { name: 'Earth', numbers: [ 1, 3 ] }, { name: 'Mars', numbers: [ 7 ] }, { name: 'Pluton', numbers: [ 9, 5, 1 ] } ] } }, 'data.planets.numbers'); assert.deepEqual(dv, [[ 1, 3 ], [ 7 ], [ 9, 5, 1 ]]); }); it("Can get a single value out of an array using its index", function () { var dv; // Simple index in dot notation dv = model.getDotValue({ planets: [ { name: 'Earth', number: 3 }, { name: 'Mars', number: 2 }, { name: 'Pluton', number: 9 } ] }, 'planets.1'); assert.deepEqual(dv, { name: 'Mars', number: 2 }); // Out of bounds index dv = model.getDotValue({ planets: [ { name: 'Earth', number: 3 }, { name: 'Mars', number: 2 }, { name: 'Pluton', number: 9 } ] }, 'planets.3'); assert.isUndefined(dv); // Index in nested array dv = model.getDotValue({ nedb: true, data: { planets: [ { name: 'Earth', number: 3 }, { name: 'Mars', number: 2 }, { name: 'Pluton', number: 9 } ] } }, 'data.planets.2'); assert.deepEqual(dv, { name: 'Pluton', number: 9 }); // Dot notation with index in the middle dv = model.getDotValue({ nedb: true, data: { planets: [ { name: 'Earth', number: 3 }, { name: 'Mars', number: 2 }, { name: 'Pluton', number: 9 } ] } }, 'data.planets.0.name'); dv.should.equal('Earth'); }); }); describe('Field equality', function () { it('Can find documents with simple fields', function () { model.match({ test: 'yeah' }, { test: 'yea' }).should.equal(false); model.match({ test: 'yeah' }, { test: 'yeahh' }).should.equal(false); model.match({ test: 'yeah' }, { test: 'yeah' }).should.equal(true); }); it('Can find documents with the dot-notation', function () { model.match({ test: { ooo: 'yeah' } }, { "test.ooo": 'yea' }).should.equal(false); model.match({ test: { ooo: 'yeah' } }, { "test.oo": 'yeah' }).should.equal(false); model.match({ test: { ooo: 'yeah' } }, { "tst.ooo": 'yeah' }).should.equal(false); model.match({ test: { ooo: 'yeah' } }, { "test.ooo": 'yeah' }).should.equal(true); }); it('Cannot find undefined', function () { model.match({ test: undefined }, { test: undefined }).should.equal(false); model.match({ test: { pp: undefined } }, { "test.pp": undefined }).should.equal(false); }); it('Nested objects are deep-equality matched and not treated as sub-queries', function () { model.match({ a: { b: 5 } }, { a: { b: 5 } }).should.equal(true); model.match({ a: { b: 5, c: 3 } }, { a: { b: 5 } }).should.equal(false); model.match({ a: { b: 5 } }, { a: { b: { $lt: 10 } } }).should.equal(false); (function () { model.match({ a: { b: 5 } }, { a: { $or: [ { b: 10 }, { b: 5 } ] } }) }).should.throw(); }); it("Can match for field equality inside an array with the dot notation", function () { model.match({ a: true, b: [ 'node', 'embedded', 'database' ] }, { 'b.1': 'node' }).should.equal(false); model.match({ a: true, b: [ 'node', 'embedded', 'database' ] }, { 'b.1': 'embedded' }).should.equal(true); model.match({ a: true, b: [ 'node', 'embedded', 'database' ] }, { 'b.1': 'database' }).should.equal(false); }) }); describe('Regular expression matching', function () { it('Matching a non-string to a regular expression always yields false', function () { var d = new Date() , r = new RegExp(d.getTime()); model.match({ test: true }, { test: /true/ }).should.equal(false); model.match({ test: null }, { test: /null/ }).should.equal(false); model.match({ test: 42 }, { test: /42/ }).should.equal(false); model.match({ test: d }, { test: r }).should.equal(false); }); it('Can match strings using basic querying', function () { model.match({ test: 'true' }, { test: /true/ }).should.equal(true); model.match({ test: 'babaaaar' }, { test: /aba+r/ }).should.equal(true); model.match({ test: 'babaaaar' }, { test: /^aba+r/ }).should.equal(false); model.match({ test: 'true' }, { test: /t[ru]e/ }).should.equal(false); }); it('Can match strings using the $regex operator', function () { model.match({ test: 'true' }, { test: { $regex: /true/ } }).should.equal(true); model.match({ test: 'babaaaar' }, { test: { $regex: /aba+r/ } }).should.equal(true); model.match({ test: 'babaaaar' }, { test: { $regex: /^aba+r/ } }).should.equal(false); model.match({ test: 'true' }, { test: { $regex: /t[ru]e/ } }).should.equal(false); }); it('Will throw if $regex operator is used with a non regex value', function () { (function () { model.match({ test: 'true' }, { test: { $regex: 42 } }) }).should.throw(); (function () { model.match({ test: 'true' }, { test: { $regex: 'true' } }) }).should.throw(); }); it('Can use the $regex operator in cunjunction with other operators', function () { model.match({ test: 'helLo' }, { test: { $regex: /ll/i, $nin: ['helL', 'helLop'] } }).should.equal(true); model.match({ test: 'helLo' }, { test: { $regex: /ll/i, $nin: ['helLo', 'helLop'] } }).should.equal(false); }); it('Can use dot-notation', function () { model.match({ test: { nested: 'true' } }, { 'test.nested': /true/ }).should.equal(true); model.match({ test: { nested: 'babaaaar' } }, { 'test.nested': /^aba+r/ }).should.equal(false); model.match({ test: { nested: 'true' } }, { 'test.nested': { $regex: /true/ } }).should.equal(true); model.match({ test: { nested: 'babaaaar' } }, { 'test.nested': { $regex: /^aba+r/ } }).should.equal(false); }); }); describe('$lt', function () { it('Cannot compare a field to an object, an array, null or a boolean, it will return false', function () { model.match({ a: 5 }, { a: { $lt: { a: 6 } } }).should.equal(false); model.match({ a: 5 }, { a: { $lt: [6, 7] } }).should.equal(false); model.match({ a: 5 }, { a: { $lt: null } }).should.equal(false); model.match({ a: 5 }, { a: { $lt: true } }).should.equal(false); }); it('Can compare numbers, with or without dot notation', function () { model.match({ a: 5 }, { a: { $lt: 6 } }).should.equal(true); model.match({ a: 5 }, { a: { $lt: 5 } }).should.equal(false); model.match({ a: 5 }, { a: { $lt: 4 } }).should.equal(false); model.match({ a: { b: 5 } }, { "a.b": { $lt: 6 } }).should.equal(true); model.match({ a: { b: 5 } }, { "a.b": { $lt: 3 } }).should.equal(false); }); it('Can compare strings, with or without dot notation', function () { model.match({ a: "nedb" }, { a: { $lt: "nedc" } }).should.equal(true); model.match({ a: "nedb" }, { a: { $lt: "neda" } }).should.equal(false); model.match({ a: { b: "nedb" } }, { "a.b": { $lt: "nedc" } }).should.equal(true); model.match({ a: { b: "nedb" } }, { "a.b": { $lt: "neda" } }).should.equal(false); }); it('If field is an array field, a match means a match on at least one element', function () { model.match({ a: [5, 10] }, { a: { $lt: 4 } }).should.equal(false); model.match({ a: [5, 10] }, { a: { $lt: 6 } }).should.equal(true); model.match({ a: [5, 10] }, { a: { $lt: 11 } }).should.equal(true); }); it('Works with dates too', function () { model.match({ a: new Date(1000) }, { a: { $gte: new Date(1001) } }).should.equal(false); model.match({ a: new Date(1000) }, { a: { $lt: new Date(1001) } }).should.equal(true); }); }); // General behaviour is tested in the block about $lt. Here we just test operators work describe('Other comparison operators: $lte, $gt, $gte, $ne, $in, $exists', function () { it('$lte', function () { model.match({ a: 5 }, { a: { $lte: 6 } }).should.equal(true); model.match({ a: 5 }, { a: { $lte: 5 } }).should.equal(true); model.match({ a: 5 }, { a: { $lte: 4 } }).should.equal(false); }); it('$gt', function () { model.match({ a: 5 }, { a: { $gt: 6 } }).should.equal(false); model.match({ a: 5 }, { a: { $gt: 5 } }).should.equal(false); model.match({ a: 5 }, { a: { $gt: 4 } }).should.equal(true); }); it('$gte', function () { model.match({ a: 5 }, { a: { $gte: 6 } }).should.equal(false); model.match({ a: 5 }, { a: { $gte: 5 } }).should.equal(true); model.match({ a: 5 }, { a: { $gte: 4 } }).should.equal(true); }); it('$ne', function () { model.match({ a: 5 }, { a: { $ne: 4 } }).should.equal(true); model.match({ a: 5 }, { a: { $ne: 5 } }).should.equal(false); model.match({ a: 5 }, { b: { $ne: 5 } }).should.equal(true); model.match({ a: false }, { a: { $ne: false } }).should.equal(false); }); it('$in', function () { model.match({ a: 5 }, { a: { $in: [6, 8, 9] } }).should.equal(false); model.match({ a: 6 }, { a: { $in: [6, 8, 9] } }).should.equal(true); model.match({ a: 7 }, { a: { $in: [6, 8, 9] } }).should.equal(false); model.match({ a: 8 }, { a: { $in: [6, 8, 9] } }).should.equal(true); model.match({ a: 9 }, { a: { $in: [6, 8, 9] } }).should.equal(true); (function () { model.match({ a: 5 }, { a: { $in: 5 } }); }).should.throw(); }); it('$nin', function () { model.match({ a: 5 }, { a: { $nin: [6, 8, 9] } }).should.equal(true); model.match({ a: 6 }, { a: { $nin: [6, 8, 9] } }).should.equal(false); model.match({ a: 7 }, { a: { $nin: [6, 8, 9] } }).should.equal(true); model.match({ a: 8 }, { a: { $nin: [6, 8, 9] } }).should.equal(false); model.match({ a: 9 }, { a: { $nin: [6, 8, 9] } }).should.equal(false); // Matches if field doesn't exist model.match({ a: 9 }, { b: { $nin: [6, 8, 9] } }).should.equal(true); (function () { model.match({ a: 5 }, { a: { $in: 5 } }); }).should.throw(); }); it('$exists', function () { model.match({ a: 5 }, { a: { $exists: 1 } }).should.equal(true); model.match({ a: 5 }, { a: { $exists: true } }).should.equal(true); model.match({ a: 5 }, { a: { $exists: new Date() } }).should.equal(true); model.match({ a: 5 }, { a: { $exists: '' } }).should.equal(true); model.match({ a: 5 }, { a: { $exists: [] } }).should.equal(true); model.match({ a: 5 }, { a: { $exists: {} } }).should.equal(true); model.match({ a: 5 }, { a: { $exists: 0 } }).should.equal(false); model.match({ a: 5 }, { a: { $exists: false } }).should.equal(false); model.match({ a: 5 }, { a: { $exists: null } }).should.equal(false); model.match({ a: 5 }, { a: { $exists: undefined } }).should.equal(false); model.match({ a: 5 }, { b: { $exists: true } }).should.equal(false); model.match({ a: 5 }, { b: { $exists: false } }).should.equal(true); }); }); describe('Comparing on arrays', function () { it("Can perform a direct array match", function () { model.match({ planets: ['Earth', 'Mars', 'Pluto'], something: 'else' }, { planets: ['Earth', 'Mars'] }).should.equal(false); model.match({ planets: ['Earth', 'Mars', 'Pluto'], something: 'else' }, { planets: ['Earth', 'Mars', 'Pluto'] }).should.equal(true); model.match({ planets: ['Earth', 'Mars', 'Pluto'], something: 'else' }, { planets: ['Earth', 'Pluto', 'Mars'] }).should.equal(false); }); it('Can query on the size of an array field', function () { // Non nested documents model.match({ childrens: [ { name: "Huey", age: 3 }, { name: "Dewey", age: 7 }, { name: "Louie", age: 12 } ] }, { "childrens": { $size: 0 } }).should.equal(false); model.match({ childrens: [ { name: "Huey", age: 3 }, { name: "Dewey", age: 7 }, { name: "Louie", age: 12 } ] }, { "childrens": { $size: 1 } }).should.equal(false); model.match({ childrens: [ { name: "Huey", age: 3 }, { name: "Dewey", age: 7 }, { name: "Louie", age: 12 } ] }, { "childrens": { $size: 2 } }).should.equal(false); model.match({ childrens: [ { name: "Huey", age: 3 }, { name: "Dewey", age: 7 }, { name: "Louie", age: 12 } ] }, { "childrens": { $size: 3 } }).should.equal(true); // Nested documents model.match({ hello: 'world', description: { satellites: ['Moon', 'Hubble'], diameter: 6300 } }, { "description.satellites": { $size: 0 } }).should.equal(false); model.match({ hello: 'world', description: { satellites: ['Moon', 'Hubble'], diameter: 6300 } }, { "description.satellites": { $size: 1 } }).should.equal(false); model.match({ hello: 'world', description: { satellites: ['Moon', 'Hubble'], diameter: 6300 } }, { "description.satellites": { $size: 2 } }).should.equal(true); model.match({ hello: 'world', description: { satellites: ['Moon', 'Hubble'], diameter: 6300 } }, { "description.satellites": { $size: 3 } }).should.equal(false); // Using a projected array model.match({ childrens: [ { name: "Huey", age: 3 }, { name: "Dewey", age: 7 }, { name: "Louie", age: 12 } ] }, { "childrens.names": { $size: 0 } }).should.equal(false); model.match({ childrens: [ { name: "Huey", age: 3 }, { name: "Dewey", age: 7 }, { name: "Louie", age: 12 } ] }, { "childrens.names": { $size: 1 } }).should.equal(false); model.match({ childrens: [ { name: "Huey", age: 3 }, { name: "Dewey", age: 7 }, { name: "Louie", age: 12 } ] }, { "childrens.names": { $size: 2 } }).should.equal(false); model.match({ childrens: [ { name: "Huey", age: 3 }, { name: "Dewey", age: 7 }, { name: "Louie", age: 12 } ] }, { "childrens.names": { $size: 3 } }).should.equal(true); }); it('$size operator works with empty arrays', function () { model.match({ childrens: [] }, { "childrens": { $size: 0 } }).should.equal(true); model.match({ childrens: [] }, { "childrens": { $size: 2 } }).should.equal(false); model.match({ childrens: [] }, { "childrens": { $size: 3 } }).should.equal(false); }); it('Should throw an error if a query operator is used without comparing to an integer', function () { (function () { model.match({ a: [1, 5] }, { a: { $size: 1.4 } }); }).should.throw(); (function () { model.match({ a: [1, 5] }, { a: { $size: 'fdf' } }); }).should.throw(); (function () { model.match({ a: [1, 5] }, { a: { $size: { $lt: 5 } } }); }).should.throw(); }); it('Using $size operator on a non-array field should prevent match but not throw', function () { model.match({ a: 5 }, { a: { $size: 1 } }).should.equal(false); }); it('Can use $size several times in the same matcher', function () { model.match({ childrens: [ 'Riri', 'Fifi', 'Loulou' ] }, { "childrens": { $size: 3, $size: 3 } }).should.equal(true); model.match({ childrens: [ 'Riri', 'Fifi', 'Loulou' ] }, { "childrens": { $size: 3, $size: 4 } }).should.equal(false); // Of course this can never be true }); it('Can query array documents with multiple simultaneous conditions', function () { // Non nested documents model.match({ childrens: [ { name: "Huey", age: 3 }, { name: "Dewey", age: 7 }, { name: "Louie", age: 12 } ] }, { "childrens": { $elemMatch: { name: "Dewey", age: 7 } } }).should.equal(true); model.match({ childrens: [ { name: "Huey", age: 3 }, { name: "Dewey", age: 7 }, { name: "Louie", age: 12 } ] }, { "childrens": { $elemMatch: { name: "Dewey", age: 12 } } }).should.equal(false); model.match({ childrens: [ { name: "Huey", age: 3 }, { name: "Dewey", age: 7 }, { name: "Louie", age: 12 } ] }, { "childrens": { $elemMatch: { name: "Louie", age: 3 } } }).should.equal(false); // Nested documents model.match({ outer: { childrens: [ { name: "Huey", age: 3 }, { name: "Dewey", age: 7 }, { name: "Louie", age: 12 } ] } }, { "outer.childrens": { $elemMatch: { name: "Dewey", age: 7 } } }).should.equal(true); model.match({ outer: { childrens: [ { name: "Huey", age: 3 }, { name: "Dewey", age: 7 }, { name: "Louie", age: 12 } ] } }, { "outer.childrens": { $elemMatch: { name: "Dewey", age: 12 } } }).should.equal(false); model.match({ outer: { childrens: [ { name: "Huey", age: 3 }, { name: "Dewey", age: 7 }, { name: "Louie", age: 12 } ] } }, { "outer.childrens": { $elemMatch: { name: "Louie", age: 3 } } }).should.equal(false); }); it('$elemMatch operator works with empty arrays', function () { model.match({ childrens: [] }, { "childrens": { $elemMatch: { name: "Mitsos" } } }).should.equal(false); model.match({ childrens: [] }, { "childrens": { $elemMatch: {} } }).should.equal(false); }); it('Can use more complex comparisons inside nested query documents', function () { model.match({ childrens: [ { name: "Huey", age: 3 }, { name: "Dewey", age: 7 }, { name: "Louie", age: 12 } ] }, { "childrens": { $elemMatch: { name: "Dewey", age: { $gt: 6, $lt: 8 } } } }).should.equal(true); model.match({ childrens: [ { name: "Huey", age: 3 }, { name: "Dewey", age: 7 }, { name: "Louie", age: 12 } ] }, { "childrens": { $elemMatch: { name: "Dewey", age: { $in: [ 6, 7, 8 ] } } } } ).should.equal(true); model.match({ childrens: [ { name: "Huey", age: 3 }, { name: "Dewey", age: 7 }, { name: "Louie", age: 12 } ] }, { "childrens": { $elemMatch: { name: "Dewey", age: { $gt: 6, $lt: 7 } } } }).should.equal(false); model.match({ childrens: [ { name: "Huey", age: 3 }, { name: "Dewey", age: 7 }, { name: "Louie", age: 12 } ] }, { "childrens": { $elemMatch: { name: "Louie", age: { $gt: 6, $lte: 7 } } } }).should.equal(false); }); }); describe('Logical operators $or, $and, $not', function () { it('Any of the subqueries should match for an $or to match', function () { model.match({ hello: 'world' }, { $or: [ { hello: 'pluton' }, { hello: 'world' } ] }).should.equal(true); model.match({ hello: 'pluton' }, { $or: [ { hello: 'pluton' }, { hello: 'world' } ] }).should.equal(true); model.match({ hello: 'nope' }, { $or: [ { hello: 'pluton' }, { hello: 'world' } ] }).should.equal(false); model.match({ hello: 'world', age: 15 }, { $or: [ { hello: 'pluton' }, { age: { $lt: 20 } } ] }).should.equal(true); model.match({ hello: 'world', age: 15 }, { $or: [ { hello: 'pluton' }, { age: { $lt: 10 } } ] }).should.equal(false); }); it('All of the subqueries should match for an $and to match', function () { model.match({ hello: 'world', age: 15 }, { $and: [ { age: 15 }, { hello: 'world' } ] }).should.equal(true); model.match({ hello: 'world', age: 15 }, { $and: [ { age: 16 }, { hello: 'world' } ] }).should.equal(false); model.match({ hello: 'world', age: 15 }, { $and: [ { hello: 'world' }, { age: { $lt: 20 } } ] }).should.equal(true); model.match({ hello: 'world', age: 15 }, { $and: [ { hello: 'pluton' }, { age: { $lt: 20 } } ] }).should.equal(false); }); it('Subquery should not match for a $not to match', function () { model.match({ a: 5, b: 10 }, { a: 5 }).should.equal(true); model.match({ a: 5, b: 10 }, { $not: { a: 5 } }).should.equal(false); }); it('Logical operators are all top-level, only other logical operators can be above', function () { (function () { model.match({ a: { b: 7 } }, { a: { $or: [ { b: 5 }, { b: 7 } ] } })}).should.throw(); model.match({ a: { b: 7 } }, { $or: [ { "a.b": 5 }, { "a.b": 7 } ] }).should.equal(true); }); it('Logical operators can be combined as long as they are on top of the decision tree', function () { model.match({ a: 5, b: 7, c: 12 }, { $or: [ { $and: [ { a: 5 }, { b: 8 } ] }, { $and: [{ a: 5 }, { c : { $lt: 40 } }] } ] }).should.equal(true); model.match({ a: 5, b: 7, c: 12 }, { $or: [ { $and: [ { a: 5 }, { b: 8 } ] }, { $and: [{ a: 5 }, { c : { $lt: 10 } }] } ] }).should.equal(false); }); it('Should throw an error if a logical operator is used without an array or if an unknown logical operator is used', function () { (function () { model.match({ a: 5 }, { $or: { a: 5, a: 6 } }); }).should.throw(); (function () { model.match({ a: 5 }, { $and: { a: 5, a: 6 } }); }).should.throw(); (function () { model.match({ a: 5 }, { $unknown: [ { a: 5 } ] }); }).should.throw(); }); }); describe('Comparison operator $where', function () { it('Function should match and not match correctly', function () { model.match({ a: 4}, { $where: function () { return this.a === 4; } }).should.equal(true); model.match({ a: 4}, { $where: function () { return this.a === 5; } }).should.equal(false); }); it('Should throw an error if the $where function is not, in fact, a function', function () { (function () { model.match({ a: 4 }, { $where: 'not a function' }); }).should.throw(); }); it('Should throw an error if the $where function returns a non-boolean', function () { (function () { model.match({ a: 4 }, { $where: function () { return 'not a boolean'; } }); }).should.throw(); }); it('Should be able to do the complex matching it must be used for', function () { var checkEmail = function() { if (!this.firstName || !this.lastName) { return false; } return this.firstName.toLowerCase() + "." + this.lastName.toLowerCase() + "@gmail.com" === this.email; }; model.match({ firstName: "John", lastName: "Doe", email: "john.doe@gmail.com" }, { $where: checkEmail }).should.equal(true); model.match({ firstName: "john", lastName: "doe", email: "john.doe@gmail.com" }, { $where: checkEmail }).should.equal(true); model.match({ firstName: "Jane", lastName: "Doe", email: "john.doe@gmail.com" }, { $where: checkEmail }).should.equal(false); model.match({ firstName: "John", lastName: "Deere", email: "john.doe@gmail.com" }, { $where: checkEmail }).should.equal(false); model.match({ lastName: "Doe", email: "john.doe@gmail.com" }, { $where: checkEmail }).should.equal(false); }); }); describe('Array fields', function () { it('Field equality', function () { model.match({ tags: ['node', 'js', 'db'] }, { tags: 'python' }).should.equal(false); model.match({ tags: ['node', 'js', 'db'] }, { tagss: 'js' }).should.equal(false); model.match({ tags: ['node', 'js', 'db'] }, { tags: 'js' }).should.equal(true); model.match({ tags: ['node', 'js', 'db'] }, { tags: 'js', tags: 'node' }).should.equal(true); // Mixed matching with array and non array model.match({ tags: ['node', 'js', 'db'], nedb: true }, { tags: 'js', nedb: true }).should.equal(true); // Nested matching model.match({ number: 5, data: { tags: ['node', 'js', 'db'] } }, { "data.tags": 'js' }).should.equal(true); model.match({ number: 5, data: { tags: ['node', 'js', 'db'] } }, { "data.tags": 'j' }).should.equal(false); }); it('With one comparison operator', function () { model.match({ ages: [3, 7, 12] }, { ages: { $lt: 2 } }).should.equal(false); model.match({ ages: [3, 7, 12] }, { ages: { $lt: 3 } }).should.equal(false); model.match({ ages: [3, 7, 12] }, { ages: { $lt: 4 } }).should.equal(true); model.match({ ages: [3, 7, 12] }, { ages: { $lt: 8 } }).should.equal(true); model.match({ ages: [3, 7, 12] }, { ages: { $lt: 13 } }).should.equal(true); }); it('Works with arrays that are in subdocuments', function () { model.match({ children: { ages: [3, 7, 12] } }, { "children.ages": { $lt: 2 } }).should.equal(false); model.match({ children: { ages: [3, 7, 12] } }, { "children.ages": { $lt: 3 } }).should.equal(false); model.match({ children: { ages: [3, 7, 12] } }, { "children.ages": { $lt: 4 } }).should.equal(true); model.match({ children: { ages: [3, 7, 12] } }, { "children.ages": { $lt: 8 } }).should.equal(true); model.match({ children: { ages: [3, 7, 12] } }, { "children.ages": { $lt: 13 } }).should.equal(true); }); it('Can query inside arrays thanks to dot notation', function () { model.match({ childrens: [ { name: "Huey", age: 3 }, { name: "Dewey", age: 7 }, { name: "Louie", age: 12 } ] }, { "childrens.age": { $lt: 2 } }).should.equal(false); model.match({ childrens: [ { name: "Huey", age: 3 }, { name: "Dewey", age: 7 }, { name: "Louie", age: 12 } ] }, { "childrens.age": { $lt: 3 } }).should.equal(false); model.match({ childrens: [ { name: "Huey", age: 3 }, { name: "Dewey", age: 7 }, { name: "Louie", age: 12 } ] }, { "childrens.age": { $lt: 4 } }).should.equal(true); model.match({ childrens: [ { name: "Huey", age: 3 }, { name: "Dewey", age: 7 }, { name: "Louie", age: 12 } ] }, { "childrens.age": { $lt: 8 } }).should.equal(true); model.match({ childrens: [ { name: "Huey", age: 3 }, { name: "Dewey", age: 7 }, { name: "Louie", age: 12 } ] }, { "childrens.age": { $lt: 13 } }).should.equal(true); model.match({ childrens: [ { name: "Huey", age: 3 }, { name: "Dewey", age: 7 }, { name: "Louie", age: 12 } ] }, { "childrens.name": 'Louis' }).should.equal(false); model.match({ childrens: [ { name: "Huey", age: 3 }, { name: "Dewey", age: 7 }, { name: "Louie", age: 12 } ] }, { "childrens.name": 'Louie' }).should.equal(true); model.match({ childrens: [ { name: "Huey", age: 3 }, { name: "Dewey", age: 7 }, { name: "Louie", age: 12 } ] }, { "childrens.name": 'Lewi' }).should.equal(false); }); it('Can query for a specific element inside arrays thanks to dot notation', function () { model.match({ childrens: [ { name: "Huey", age: 3 }, { name: "Dewey", age: 7 }, { name: "Louie", age: 12 } ] }, { "childrens.0.name": 'Louie' }).should.equal(false); model.match({ childrens: [ { name: "Huey", age: 3 }, { name: "Dewey", age: 7 }, { name: "Louie", age: 12 } ] }, { "childrens.1.name": 'Louie' }).should.equal(false); model.match({ childrens: [ { name: "Huey", age: 3 }, { name: "Dewey", age: 7 }, { name: "Louie", age: 12 } ] }, { "childrens.2.name": 'Louie' }).should.equal(true); model.match({ childrens: [ { name: "Huey", age: 3 }, { name: "Dewey", age: 7 }, { name: "Louie", age: 12 } ] }, { "childrens.3.name": 'Louie' }).should.equal(false); }); it('A single array-specific operator and the query is treated as array specific', function () { (function () { model.match({ childrens: [ 'Riri', 'Fifi', 'Loulou' ] }, { "childrens": { "Fifi": true, $size: 3 } })}).should.throw(); }); it('Can mix queries on array fields and non array filds with array specific operators', function () { model.match({ uncle: 'Donald', nephews: [ 'Riri', 'Fifi', 'Loulou' ] }, { nephews: { $size: 2 }, uncle: 'Donald' }).should.equal(false); model.match({ uncle: 'Donald', nephews: [ 'Riri', 'Fifi', 'Loulou' ] }, { nephews: { $size: 3 }, uncle: 'Donald' }).should.equal(true); model.match({ uncle: 'Donald', nephews: [ 'Riri', 'Fifi', 'Loulou' ] }, { nephews: { $size: 4 }, uncle: 'Donald' }).should.equal(false); model.match({ uncle: 'Donals', nephews: [ 'Riri', 'Fifi', 'Loulou' ] }, { nephews: { $size: 3 }, uncle: 'Picsou' }).should.equal(false); model.match({ uncle: 'Donald', nephews: [ 'Riri', 'Fifi', 'Loulou' ] }, { nephews: { $size: 3 }, uncle: 'Donald' }).should.equal(true); model.match({ uncle: 'Donald', nephews: [ 'Riri', 'Fifi', 'Loulou' ] }, { nephews: { $size: 3 }, uncle: 'Daisy' }).should.equal(false); }); }); }); // ==== End of 'Querying' ==== // }); ================================================ FILE: test/persistence.test.js ================================================ var should = require('chai').should() , assert = require('chai').assert , testDb = 'workspace/test.db' , fs = require('fs') , path = require('path') , _ = require('underscore') , async = require('async') , model = require('../lib/model') , customUtils = require('../lib/customUtils') , Datastore = require('../lib/datastore') , Persistence = require('../lib/persistence') , storage = require('../lib/storage') , child_process = require('child_process') ; describe('Persistence', function () { var d; beforeEach(function (done) { d = new Datastore({ filename: testDb }); d.filename.should.equal(testDb); d.inMemoryOnly.should.equal(false); async.waterfall([ function (cb) { Persistence.ensureDirectoryExists(path.dirname(testDb), function () { fs.exists(testDb, function (exists) { if (exists) { fs.unlink(testDb, cb); } else { return cb(); } }); }); } , function (cb) { d.loadDatabase(function (err) { assert.isNull(err); d.getAllData().length.should.equal(0); return cb(); }); } ], done); }); it('Every line represents a document', function () { var now = new Date() , rawData = model.serialize({ _id: "1", a: 2, ages: [1, 5, 12] }) + '\n' + model.serialize({ _id: "2", hello: 'world' }) + '\n' + model.serialize({ _id: "3", nested: { today: now } }) , treatedData = d.persistence.treatRawData(rawData).data ; treatedData.sort(function (a, b) { return a._id - b._id; }); treatedData.length.should.equal(3); _.isEqual(treatedData[0], { _id: "1", a: 2, ages: [1, 5, 12] }).should.equal(true); _.isEqual(treatedData[1], { _id: "2", hello: 'world' }).should.equal(true); _.isEqual(treatedData[2], { _id: "3", nested: { today: now } }).should.equal(true); }); it('Badly formatted lines have no impact on the treated data', function () { var now = new Date() , rawData = model.serialize({ _id: "1", a: 2, ages: [1, 5, 12] }) + '\n' + 'garbage\n' + model.serialize({ _id: "3", nested: { today: now } }) , treatedData = d.persistence.treatRawData(rawData).data ; treatedData.sort(function (a, b) { return a._id - b._id; }); treatedData.length.should.equal(2); _.isEqual(treatedData[0], { _id: "1", a: 2, ages: [1, 5, 12] }).should.equal(true); _.isEqual(treatedData[1], { _id: "3", nested: { today: now } }).should.equal(true); }); it('Well formatted lines that have no _id are not included in the data', function () { var now = new Date() , rawData = model.serialize({ _id: "1", a: 2, ages: [1, 5, 12] }) + '\n' + model.serialize({ _id: "2", hello: 'world' }) + '\n' + model.serialize({ nested: { today: now } }) , treatedData = d.persistence.treatRawData(rawData).data ; treatedData.sort(function (a, b) { return a._id - b._id; }); treatedData.length.should.equal(2); _.isEqual(treatedData[0], { _id: "1", a: 2, ages: [1, 5, 12] }).should.equal(true); _.isEqual(treatedData[1], { _id: "2", hello: 'world' }).should.equal(true); }); it('If two lines concern the same doc (= same _id), the last one is the good version', function () { var now = new Date() , rawData = model.serialize({ _id: "1", a: 2, ages: [1, 5, 12] }) + '\n' + model.serialize({ _id: "2", hello: 'world' }) + '\n' + model.serialize({ _id: "1", nested: { today: now } }) , treatedData = d.persistence.treatRawData(rawData).data ; treatedData.sort(function (a, b) { return a._id - b._id; }); treatedData.length.should.equal(2); _.isEqual(treatedData[0], { _id: "1", nested: { today: now } }).should.equal(true); _.isEqual(treatedData[1], { _id: "2", hello: 'world' }).should.equal(true); }); it('If a doc contains $$deleted: true, that means we need to remove it from the data', function () { var now = new Date() , rawData = model.serialize({ _id: "1", a: 2, ages: [1, 5, 12] }) + '\n' + model.serialize({ _id: "2", hello: 'world' }) + '\n' + model.serialize({ _id: "1", $$deleted: true }) + '\n' + model.serialize({ _id: "3", today: now }) , treatedData = d.persistence.treatRawData(rawData).data ; treatedData.sort(function (a, b) { return a._id - b._id; }); treatedData.length.should.equal(2); _.isEqual(treatedData[0], { _id: "2", hello: 'world' }).should.equal(true); _.isEqual(treatedData[1], { _id: "3", today: now }).should.equal(true); }); it('If a doc contains $$deleted: true, no error is thrown if the doc wasnt in the list before', function () { var now = new Date() , rawData = model.serialize({ _id: "1", a: 2, ages: [1, 5, 12] }) + '\n' + model.serialize({ _id: "2", $$deleted: true }) + '\n' + model.serialize({ _id: "3", today: now }) , treatedData = d.persistence.treatRawData(rawData).data ; treatedData.sort(function (a, b) { return a._id - b._id; }); treatedData.length.should.equal(2); _.isEqual(treatedData[0], { _id: "1", a: 2, ages: [1, 5, 12] }).should.equal(true); _.isEqual(treatedData[1], { _id: "3", today: now }).should.equal(true); }); it('If a doc contains $$indexCreated, no error is thrown during treatRawData and we can get the index options', function () { var now = new Date() , rawData = model.serialize({ _id: "1", a: 2, ages: [1, 5, 12] }) + '\n' + model.serialize({ $$indexCreated: { fieldName: "test", unique: true } }) + '\n' + model.serialize({ _id: "3", today: now }) , treatedData = d.persistence.treatRawData(rawData).data , indexes = d.persistence.treatRawData(rawData).indexes ; Object.keys(indexes).length.should.equal(1); assert.deepEqual(indexes.test, { fieldName: "test", unique: true }); treatedData.sort(function (a, b) { return a._id - b._id; }); treatedData.length.should.equal(2); _.isEqual(treatedData[0], { _id: "1", a: 2, ages: [1, 5, 12] }).should.equal(true); _.isEqual(treatedData[1], { _id: "3", today: now }).should.equal(true); }); it('Compact database on load', function (done) { d.insert({ a: 2 }, function () { d.insert({ a: 4 }, function () { d.remove({ a: 2 }, {}, function () { // Here, the underlying file is 3 lines long for only one document var data = fs.readFileSync(d.filename, 'utf8').split('\n') , filledCount = 0; data.forEach(function (item) { if (item.length > 0) { filledCount += 1; } }); filledCount.should.equal(3); d.loadDatabase(function (err) { assert.isNull(err); // Now, the file has been compacted and is only 1 line long var data = fs.readFileSync(d.filename, 'utf8').split('\n') , filledCount = 0; data.forEach(function (item) { if (item.length > 0) { filledCount += 1; } }); filledCount.should.equal(1); done(); }); }) }); }); }); it('Calling loadDatabase after the data was modified doesnt change its contents', function (done) { d.loadDatabase(function () { d.insert({ a: 1 }, function (err) { assert.isNull(err); d.insert({ a: 2 }, function (err) { var data = d.getAllData() , doc1 = _.find(data, function (doc) { return doc.a === 1; }) , doc2 = _.find(data, function (doc) { return doc.a === 2; }) ; assert.isNull(err); data.length.should.equal(2); doc1.a.should.equal(1); doc2.a.should.equal(2); d.loadDatabase(function (err) { var data = d.getAllData() , doc1 = _.find(data, function (doc) { return doc.a === 1; }) , doc2 = _.find(data, function (doc) { return doc.a === 2; }) ; assert.isNull(err); data.length.should.equal(2); doc1.a.should.equal(1); doc2.a.should.equal(2); done(); }); }); }); }); }); it('Calling loadDatabase after the datafile was removed will reset the database', function (done) { d.loadDatabase(function () { d.insert({ a: 1 }, function (err) { assert.isNull(err); d.insert({ a: 2 }, function (err) { var data = d.getAllData() , doc1 = _.find(data, function (doc) { return doc.a === 1; }) , doc2 = _.find(data, function (doc) { return doc.a === 2; }) ; assert.isNull(err); data.length.should.equal(2); doc1.a.should.equal(1); doc2.a.should.equal(2); fs.unlink(testDb, function (err) { assert.isNull(err); d.loadDatabase(function (err) { assert.isNull(err); d.getAllData().length.should.equal(0); done(); }); }); }); }); }); }); it('Calling loadDatabase after the datafile was modified loads the new data', function (done) { d.loadDatabase(function () { d.insert({ a: 1 }, function (err) { assert.isNull(err); d.insert({ a: 2 }, function (err) { var data = d.getAllData() , doc1 = _.find(data, function (doc) { return doc.a === 1; }) , doc2 = _.find(data, function (doc) { return doc.a === 2; }) ; assert.isNull(err); data.length.should.equal(2); doc1.a.should.equal(1); doc2.a.should.equal(2); fs.writeFile(testDb, '{"a":3,"_id":"aaa"}', 'utf8', function (err) { assert.isNull(err); d.loadDatabase(function (err) { var data = d.getAllData() , doc1 = _.find(data, function (doc) { return doc.a === 1; }) , doc2 = _.find(data, function (doc) { return doc.a === 2; }) , doc3 = _.find(data, function (doc) { return doc.a === 3; }) ; assert.isNull(err); data.length.should.equal(1); doc3.a.should.equal(3); assert.isUndefined(doc1); assert.isUndefined(doc2); done(); }); }); }); }); }); }); it("When treating raw data, refuse to proceed if too much data is corrupt, to avoid data loss", function (done) { var corruptTestFilename = 'workspace/corruptTest.db' , fakeData = '{"_id":"one","hello":"world"}\n' + 'Some corrupt data\n' + '{"_id":"two","hello":"earth"}\n' + '{"_id":"three","hello":"you"}\n' , d ; fs.writeFileSync(corruptTestFilename, fakeData, "utf8"); // Default corruptAlertThreshold d = new Datastore({ filename: corruptTestFilename }); d.loadDatabase(function (err) { assert.isDefined(err); assert.isNotNull(err); fs.writeFileSync(corruptTestFilename, fakeData, "utf8"); d = new Datastore({ filename: corruptTestFilename, corruptAlertThreshold: 1 }); d.loadDatabase(function (err) { assert.isNull(err); fs.writeFileSync(corruptTestFilename, fakeData, "utf8"); d = new Datastore({ filename: corruptTestFilename, corruptAlertThreshold: 0 }); d.loadDatabase(function (err) { assert.isDefined(err); assert.isNotNull(err); done(); }); }); }); }); it("Can listen to compaction events", function (done) { d.on('compaction.done', function () { d.removeAllListeners('compaction.done'); // Tidy up for next tests done(); }); d.persistence.compactDatafile(); }); describe('Serialization hooks', function () { var as = function (s) { return "before_" + s + "_after"; } , bd = function (s) { return s.substring(7, s.length - 6); } it("Declaring only one hook will throw an exception to prevent data loss", function (done) { var hookTestFilename = 'workspace/hookTest.db' storage.ensureFileDoesntExist(hookTestFilename, function () { fs.writeFileSync(hookTestFilename, "Some content", "utf8"); (function () { new Datastore({ filename: hookTestFilename, autoload: true , afterSerialization: as }); }).should.throw(); // Data file left untouched fs.readFileSync(hookTestFilename, "utf8").should.equal("Some content"); (function () { new Datastore({ filename: hookTestFilename, autoload: true , beforeDeserialization: bd }); }).should.throw(); // Data file left untouched fs.readFileSync(hookTestFilename, "utf8").should.equal("Some content"); done(); }); }); it("Declaring two hooks that are not reverse of one another will cause an exception to prevent data loss", function (done) { var hookTestFilename = 'workspace/hookTest.db' storage.ensureFileDoesntExist(hookTestFilename, function () { fs.writeFileSync(hookTestFilename, "Some content", "utf8"); (function () { new Datastore({ filename: hookTestFilename, autoload: true , afterSerialization: as , beforeDeserialization: function (s) { return s; } }); }).should.throw(); // Data file left untouched fs.readFileSync(hookTestFilename, "utf8").should.equal("Some content"); done(); }); }); it("A serialization hook can be used to transform data before writing new state to disk", function (done) { var hookTestFilename = 'workspace/hookTest.db' storage.ensureFileDoesntExist(hookTestFilename, function () { var d = new Datastore({ filename: hookTestFilename, autoload: true , afterSerialization: as , beforeDeserialization: bd }) ; d.insert({ hello: "world" }, function () { var _data = fs.readFileSync(hookTestFilename, 'utf8') , data = _data.split('\n') , doc0 = bd(data[0]) ; data.length.should.equal(2); data[0].substring(0, 7).should.equal('before_'); data[0].substring(data[0].length - 6).should.equal('_after'); doc0 = model.deserialize(doc0); Object.keys(doc0).length.should.equal(2); doc0.hello.should.equal('world'); d.insert({ p: 'Mars' }, function () { var _data = fs.readFileSync(hookTestFilename, 'utf8') , data = _data.split('\n') , doc0 = bd(data[0]) , doc1 = bd(data[1]) ; data.length.should.equal(3); data[0].substring(0, 7).should.equal('before_'); data[0].substring(data[0].length - 6).should.equal('_after'); data[1].substring(0, 7).should.equal('before_'); data[1].substring(data[1].length - 6).should.equal('_after'); doc0 = model.deserialize(doc0); Object.keys(doc0).length.should.equal(2); doc0.hello.should.equal('world'); doc1 = model.deserialize(doc1); Object.keys(doc1).length.should.equal(2); doc1.p.should.equal('Mars'); d.ensureIndex({ fieldName: 'idefix' }, function () { var _data = fs.readFileSync(hookTestFilename, 'utf8') , data = _data.split('\n') , doc0 = bd(data[0]) , doc1 = bd(data[1]) , idx = bd(data[2]) ; data.length.should.equal(4); data[0].substring(0, 7).should.equal('before_'); data[0].substring(data[0].length - 6).should.equal('_after'); data[1].substring(0, 7).should.equal('before_'); data[1].substring(data[1].length - 6).should.equal('_after'); doc0 = model.deserialize(doc0); Object.keys(doc0).length.should.equal(2); doc0.hello.should.equal('world'); doc1 = model.deserialize(doc1); Object.keys(doc1).length.should.equal(2); doc1.p.should.equal('Mars'); idx = model.deserialize(idx); assert.deepEqual(idx, { '$$indexCreated': { fieldName: 'idefix' } }); done(); }); }); }); }); }); it("Use serialization hook when persisting cached database or compacting", function (done) { var hookTestFilename = 'workspace/hookTest.db' storage.ensureFileDoesntExist(hookTestFilename, function () { var d = new Datastore({ filename: hookTestFilename, autoload: true , afterSerialization: as , beforeDeserialization: bd }) ; d.insert({ hello: "world" }, function () { d.update({ hello: "world" }, { $set: { hello: "earth" } }, {}, function () { d.ensureIndex({ fieldName: 'idefix' }, function () { var _data = fs.readFileSync(hookTestFilename, 'utf8') , data = _data.split('\n') , doc0 = bd(data[0]) , doc1 = bd(data[1]) , idx = bd(data[2]) , _id ; data.length.should.equal(4); doc0 = model.deserialize(doc0); Object.keys(doc0).length.should.equal(2); doc0.hello.should.equal('world'); doc1 = model.deserialize(doc1); Object.keys(doc1).length.should.equal(2); doc1.hello.should.equal('earth'); doc0._id.should.equal(doc1._id); _id = doc0._id; idx = model.deserialize(idx); assert.deepEqual(idx, { '$$indexCreated': { fieldName: 'idefix' } }); d.persistence.persistCachedDatabase(function () { var _data = fs.readFileSync(hookTestFilename, 'utf8') , data = _data.split('\n') , doc0 = bd(data[0]) , idx = bd(data[1]) ; data.length.should.equal(3); doc0 = model.deserialize(doc0); Object.keys(doc0).length.should.equal(2); doc0.hello.should.equal('earth'); doc0._id.should.equal(_id); idx = model.deserialize(idx); assert.deepEqual(idx, { '$$indexCreated': { fieldName: 'idefix', unique: false, sparse: false } }); done(); }); }); }); }); }); }); it("Deserialization hook is correctly used when loading data", function (done) { var hookTestFilename = 'workspace/hookTest.db' storage.ensureFileDoesntExist(hookTestFilename, function () { var d = new Datastore({ filename: hookTestFilename, autoload: true , afterSerialization: as , beforeDeserialization: bd }) ; d.insert({ hello: "world" }, function (err, doc) { var _id = doc._id; d.insert({ yo: "ya" }, function () { d.update({ hello: "world" }, { $set: { hello: "earth" } }, {}, function () { d.remove({ yo: "ya" }, {}, function () { d.ensureIndex({ fieldName: 'idefix' }, function () { var _data = fs.readFileSync(hookTestFilename, 'utf8') , data = _data.split('\n') ; data.length.should.equal(6); // Everything is deserialized correctly, including deletes and indexes var d = new Datastore({ filename: hookTestFilename , afterSerialization: as , beforeDeserialization: bd }) ; d.loadDatabase(function () { d.find({}, function (err, docs) { docs.length.should.equal(1); docs[0].hello.should.equal("earth"); docs[0]._id.should.equal(_id); Object.keys(d.indexes).length.should.equal(2); Object.keys(d.indexes).indexOf("idefix").should.not.equal(-1); done(); }); }); }); }); }); }); }); }); }); }); // ==== End of 'Serialization hooks' ==== // describe('Prevent dataloss when persisting data', function () { it('Creating a datastore with in memory as true and a bad filename wont cause an error', function () { new Datastore({ filename: 'workspace/bad.db~', inMemoryOnly: true }); }) it('Creating a persistent datastore with a bad filename will cause an error', function () { (function () { new Datastore({ filename: 'workspace/bad.db~' }); }).should.throw(); }) it('If no file exists, ensureDatafileIntegrity creates an empty datafile', function (done) { var p = new Persistence({ db: { inMemoryOnly: false, filename: 'workspace/it.db' } }); if (fs.existsSync('workspace/it.db')) { fs.unlinkSync('workspace/it.db'); } if (fs.existsSync('workspace/it.db~')) { fs.unlinkSync('workspace/it.db~'); } fs.existsSync('workspace/it.db').should.equal(false); fs.existsSync('workspace/it.db~').should.equal(false); storage.ensureDatafileIntegrity(p.filename, function (err) { assert.isNull(err); fs.existsSync('workspace/it.db').should.equal(true); fs.existsSync('workspace/it.db~').should.equal(false); fs.readFileSync('workspace/it.db', 'utf8').should.equal(''); done(); }); }); it('If only datafile exists, ensureDatafileIntegrity will use it', function (done) { var p = new Persistence({ db: { inMemoryOnly: false, filename: 'workspace/it.db' } }); if (fs.existsSync('workspace/it.db')) { fs.unlinkSync('workspace/it.db'); } if (fs.existsSync('workspace/it.db~')) { fs.unlinkSync('workspace/it.db~'); } fs.writeFileSync('workspace/it.db', 'something', 'utf8'); fs.existsSync('workspace/it.db').should.equal(true); fs.existsSync('workspace/it.db~').should.equal(false); storage.ensureDatafileIntegrity(p.filename, function (err) { assert.isNull(err); fs.existsSync('workspace/it.db').should.equal(true); fs.existsSync('workspace/it.db~').should.equal(false); fs.readFileSync('workspace/it.db', 'utf8').should.equal('something'); done(); }); }); it('If temp datafile exists and datafile doesnt, ensureDatafileIntegrity will use it (cannot happen except upon first use)', function (done) { var p = new Persistence({ db: { inMemoryOnly: false, filename: 'workspace/it.db' } }); if (fs.existsSync('workspace/it.db')) { fs.unlinkSync('workspace/it.db'); } if (fs.existsSync('workspace/it.db~')) { fs.unlinkSync('workspace/it.db~~'); } fs.writeFileSync('workspace/it.db~', 'something', 'utf8'); fs.existsSync('workspace/it.db').should.equal(false); fs.existsSync('workspace/it.db~').should.equal(true); storage.ensureDatafileIntegrity(p.filename, function (err) { assert.isNull(err); fs.existsSync('workspace/it.db').should.equal(true); fs.existsSync('workspace/it.db~').should.equal(false); fs.readFileSync('workspace/it.db', 'utf8').should.equal('something'); done(); }); }); // Technically it could also mean the write was successful but the rename wasn't, but there is in any case no guarantee that the data in the temp file is whole so we have to discard the whole file it('If both temp and current datafiles exist, ensureDatafileIntegrity will use the datafile, as it means that the write of the temp file failed', function (done) { var theDb = new Datastore({ filename: 'workspace/it.db' }); if (fs.existsSync('workspace/it.db')) { fs.unlinkSync('workspace/it.db'); } if (fs.existsSync('workspace/it.db~')) { fs.unlinkSync('workspace/it.db~'); } fs.writeFileSync('workspace/it.db', '{"_id":"0","hello":"world"}', 'utf8'); fs.writeFileSync('workspace/it.db~', '{"_id":"0","hello":"other"}', 'utf8'); fs.existsSync('workspace/it.db').should.equal(true); fs.existsSync('workspace/it.db~').should.equal(true); storage.ensureDatafileIntegrity(theDb.persistence.filename, function (err) { assert.isNull(err); fs.existsSync('workspace/it.db').should.equal(true); fs.existsSync('workspace/it.db~').should.equal(true); fs.readFileSync('workspace/it.db', 'utf8').should.equal('{"_id":"0","hello":"world"}'); theDb.loadDatabase(function (err) { assert.isNull(err); theDb.find({}, function (err, docs) { assert.isNull(err); docs.length.should.equal(1); docs[0].hello.should.equal("world"); fs.existsSync('workspace/it.db').should.equal(true); fs.existsSync('workspace/it.db~').should.equal(false); done(); }); }); }); }); it('persistCachedDatabase should update the contents of the datafile and leave a clean state', function (done) { d.insert({ hello: 'world' }, function () { d.find({}, function (err, docs) { docs.length.should.equal(1); if (fs.existsSync(testDb)) { fs.unlinkSync(testDb); } if (fs.existsSync(testDb + '~')) { fs.unlinkSync(testDb + '~'); } fs.existsSync(testDb).should.equal(false); fs.writeFileSync(testDb + '~', 'something', 'utf8'); fs.existsSync(testDb + '~').should.equal(true); d.persistence.persistCachedDatabase(function (err) { var contents = fs.readFileSync(testDb, 'utf8'); assert.isNull(err); fs.existsSync(testDb).should.equal(true); fs.existsSync(testDb + '~').should.equal(false); if (!contents.match(/^{"hello":"world","_id":"[0-9a-zA-Z]{16}"}\n$/)) { throw new Error("Datafile contents not as expected"); } done(); }); }); }); }); it('After a persistCachedDatabase, there should be no temp or old filename', function (done) { d.insert({ hello: 'world' }, function () { d.find({}, function (err, docs) { docs.length.should.equal(1); if (fs.existsSync(testDb)) { fs.unlinkSync(testDb); } if (fs.existsSync(testDb + '~')) { fs.unlinkSync(testDb + '~'); } fs.existsSync(testDb).should.equal(false); fs.existsSync(testDb + '~').should.equal(false); fs.writeFileSync(testDb + '~', 'bloup', 'utf8'); fs.existsSync(testDb + '~').should.equal(true); d.persistence.persistCachedDatabase(function (err) { var contents = fs.readFileSync(testDb, 'utf8'); assert.isNull(err); fs.existsSync(testDb).should.equal(true); fs.existsSync(testDb + '~').should.equal(false); if (!contents.match(/^{"hello":"world","_id":"[0-9a-zA-Z]{16}"}\n$/)) { throw new Error("Datafile contents not as expected"); } done(); }); }); }); }); it('persistCachedDatabase should update the contents of the datafile and leave a clean state even if there is a temp datafile', function (done) { d.insert({ hello: 'world' }, function () { d.find({}, function (err, docs) { docs.length.should.equal(1); if (fs.existsSync(testDb)) { fs.unlinkSync(testDb); } fs.writeFileSync(testDb + '~', 'blabla', 'utf8'); fs.existsSync(testDb).should.equal(false); fs.existsSync(testDb + '~').should.equal(true); d.persistence.persistCachedDatabase(function (err) { var contents = fs.readFileSync(testDb, 'utf8'); assert.isNull(err); fs.existsSync(testDb).should.equal(true); fs.existsSync(testDb + '~').should.equal(false); if (!contents.match(/^{"hello":"world","_id":"[0-9a-zA-Z]{16}"}\n$/)) { throw new Error("Datafile contents not as expected"); } done(); }); }); }); }); it('persistCachedDatabase should update the contents of the datafile and leave a clean state even if there is a temp datafile', function (done) { var dbFile = 'workspace/test2.db', theDb; if (fs.existsSync(dbFile)) { fs.unlinkSync(dbFile); } if (fs.existsSync(dbFile + '~')) { fs.unlinkSync(dbFile + '~'); } theDb = new Datastore({ filename: dbFile }); theDb.loadDatabase(function (err) { var contents = fs.readFileSync(dbFile, 'utf8'); assert.isNull(err); fs.existsSync(dbFile).should.equal(true); fs.existsSync(dbFile + '~').should.equal(false); if (contents != "") { throw new Error("Datafile contents not as expected"); } done(); }); }); it('Persistence works as expected when everything goes fine', function (done) { var dbFile = 'workspace/test2.db', theDb, theDb2, doc1, doc2; async.waterfall([ async.apply(storage.ensureFileDoesntExist, dbFile) , async.apply(storage.ensureFileDoesntExist, dbFile + '~') , function (cb) { theDb = new Datastore({ filename: dbFile }); theDb.loadDatabase(cb); } , function (cb) { theDb.find({}, function (err, docs) { assert.isNull(err); docs.length.should.equal(0); return cb(); }); } , function (cb) { theDb.insert({ a: 'hello' }, function (err, _doc1) { assert.isNull(err); doc1 = _doc1; theDb.insert({ a: 'world' }, function (err, _doc2) { assert.isNull(err); doc2 = _doc2; return cb(); }); }); } , function (cb) { theDb.find({}, function (err, docs) { assert.isNull(err); docs.length.should.equal(2); _.find(docs, function (item) { return item._id === doc1._id }).a.should.equal('hello'); _.find(docs, function (item) { return item._id === doc2._id }).a.should.equal('world'); return cb(); }); } , function (cb) { theDb.loadDatabase(cb); } , function (cb) { // No change theDb.find({}, function (err, docs) { assert.isNull(err); docs.length.should.equal(2); _.find(docs, function (item) { return item._id === doc1._id }).a.should.equal('hello'); _.find(docs, function (item) { return item._id === doc2._id }).a.should.equal('world'); return cb(); }); } , function (cb) { fs.existsSync(dbFile).should.equal(true); fs.existsSync(dbFile + '~').should.equal(false); return cb(); } , function (cb) { theDb2 = new Datastore({ filename: dbFile }); theDb2.loadDatabase(cb); } , function (cb) { // No change in second db theDb2.find({}, function (err, docs) { assert.isNull(err); docs.length.should.equal(2); _.find(docs, function (item) { return item._id === doc1._id }).a.should.equal('hello'); _.find(docs, function (item) { return item._id === doc2._id }).a.should.equal('world'); return cb(); }); } , function (cb) { fs.existsSync(dbFile).should.equal(true); fs.existsSync(dbFile + '~').should.equal(false); return cb(); } ], done); }); // The child process will load the database with the given datafile, but the fs.writeFile function // is rewritten to crash the process before it finished (after 5000 bytes), to ensure data was not lost it('If system crashes during a loadDatabase, the former version is not lost', function (done) { var N = 500, toWrite = "", i, doc_i; // Ensuring the state is clean if (fs.existsSync('workspace/lac.db')) { fs.unlinkSync('workspace/lac.db'); } if (fs.existsSync('workspace/lac.db~')) { fs.unlinkSync('workspace/lac.db~'); } // Creating a db file with 150k records (a bit long to load) for (i = 0; i < N; i += 1) { toWrite += model.serialize({ _id: 'anid_' + i, hello: 'world' }) + '\n'; } fs.writeFileSync('workspace/lac.db', toWrite, 'utf8'); var datafileLength = fs.readFileSync('workspace/lac.db', 'utf8').length; // Loading it in a separate process that we will crash before finishing the loadDatabase child_process.fork('test_lac/loadAndCrash.test').on('exit', function (code) { code.should.equal(1); // See test_lac/loadAndCrash.test.js fs.existsSync('workspace/lac.db').should.equal(true); fs.existsSync('workspace/lac.db~').should.equal(true); fs.readFileSync('workspace/lac.db', 'utf8').length.should.equal(datafileLength); fs.readFileSync('workspace/lac.db~', 'utf8').length.should.equal(5000); // Reload database without a crash, check that no data was lost and fs state is clean (no temp file) var db = new Datastore({ filename: 'workspace/lac.db' }); db.loadDatabase(function (err) { assert.isNull(err); fs.existsSync('workspace/lac.db').should.equal(true); fs.existsSync('workspace/lac.db~').should.equal(false); fs.readFileSync('workspace/lac.db', 'utf8').length.should.equal(datafileLength); db.find({}, function (err, docs) { docs.length.should.equal(N); for (i = 0; i < N; i += 1) { doc_i = _.find(docs, function (d) { return d._id === 'anid_' + i; }); assert.isDefined(doc_i); assert.deepEqual({ hello: 'world', _id: 'anid_' + i }, doc_i); } return done(); }); }); }); }); // Not run on Windows as there is no clean way to set maximum file descriptors. Not an issue as the code itself is tested. it("Cannot cause EMFILE errors by opening too many file descriptors", function (done) { if (process.platform === 'win32' || process.platform === 'win64') { return done(); } child_process.execFile('test_lac/openFdsLaunch.sh', function (err, stdout, stderr) { if (err) { return done(err); } // The subprocess will not output anything to stdout unless part of the test fails if (stdout.length !== 0) { return done(stdout); } else { return done(); } }); }); }); // ==== End of 'Prevent dataloss when persisting data' ==== describe('ensureFileDoesntExist', function () { it('Doesnt do anything if file already doesnt exist', function (done) { storage.ensureFileDoesntExist('workspace/nonexisting', function (err) { assert.isNull(err); fs.existsSync('workspace/nonexisting').should.equal(false); done(); }); }); it('Deletes file if it exists', function (done) { fs.writeFileSync('workspace/existing', 'hello world', 'utf8'); fs.existsSync('workspace/existing').should.equal(true); storage.ensureFileDoesntExist('workspace/existing', function (err) { assert.isNull(err); fs.existsSync('workspace/existing').should.equal(false); done(); }); }); }); // ==== End of 'ensureFileDoesntExist' ==== }); ================================================ FILE: test_lac/loadAndCrash.test.js ================================================ /** * Load and modify part of fs to ensure writeFile will crash after writing 5000 bytes */ var fs = require('fs'); function rethrow() { // Only enable in debug mode. A backtrace uses ~1000 bytes of heap space and // is fairly slow to generate. if (DEBUG) { var backtrace = new Error(); return function(err) { if (err) { backtrace.stack = err.name + ': ' + err.message + backtrace.stack.substr(backtrace.name.length); throw backtrace; } }; } return function(err) { if (err) { throw err; // Forgot a callback but don't know where? Use NODE_DEBUG=fs } }; } function maybeCallback(cb) { return typeof cb === 'function' ? cb : rethrow(); } function isFd(path) { return (path >>> 0) === path; } function assertEncoding(encoding) { if (encoding && !Buffer.isEncoding(encoding)) { throw new Error('Unknown encoding: ' + encoding); } } var onePassDone = false; function writeAll(fd, isUserFd, buffer, offset, length, position, callback_) { var callback = maybeCallback(arguments[arguments.length - 1]); if (onePassDone) { process.exit(1); } // Crash on purpose before rewrite done var l = Math.min(5000, length); // Force write by chunks of 5000 bytes to ensure data will be incomplete on crash // write(fd, buffer, offset, length, position, callback) fs.write(fd, buffer, offset, l, position, function(writeErr, written) { if (writeErr) { if (isUserFd) { if (callback) callback(writeErr); } else { fs.close(fd, function() { if (callback) callback(writeErr); }); } } else { onePassDone = true; if (written === length) { if (isUserFd) { if (callback) callback(null); } else { fs.close(fd, callback); } } else { offset += written; length -= written; if (position !== null) { position += written; } writeAll(fd, isUserFd, buffer, offset, length, position, callback); } } }); } fs.writeFile = function(path, data, options, callback_) { var callback = maybeCallback(arguments[arguments.length - 1]); if (!options || typeof options === 'function') { options = { encoding: 'utf8', mode: 438, flag: 'w' }; // Mode 438 == 0o666 (compatibility with older Node releases) } else if (typeof options === 'string') { options = { encoding: options, mode: 438, flag: 'w' }; // Mode 438 == 0o666 (compatibility with older Node releases) } else if (typeof options !== 'object') { throwOptionsError(options); } assertEncoding(options.encoding); var flag = options.flag || 'w'; if (isFd(path)) { writeFd(path, true); return; } fs.open(path, flag, options.mode, function(openErr, fd) { if (openErr) { if (callback) callback(openErr); } else { writeFd(fd, false); } }); function writeFd(fd, isUserFd) { var buffer = (data instanceof Buffer) ? data : new Buffer('' + data, options.encoding || 'utf8'); var position = /a/.test(flag) ? null : 0; writeAll(fd, isUserFd, buffer, 0, buffer.length, position, callback); } }; // End of fs modification var Nedb = require('../lib/datastore.js') , db = new Nedb({ filename: 'workspace/lac.db' }) ; db.loadDatabase(); ================================================ FILE: test_lac/openFds.test.js ================================================ var fs = require('fs') , child_process = require('child_process') , async = require('async') , Nedb = require('../lib/datastore') , db = new Nedb({ filename: './workspace/openfds.db', autoload: true }) , N = 64 // Half the allowed file descriptors , i, fds ; function multipleOpen (filename, N, callback) { async.whilst( function () { return i < N; } , function (cb) { fs.open(filename, 'r', function (err, fd) { i += 1; if (fd) { fds.push(fd); } return cb(err); }); } , callback); } async.waterfall([ // Check that ulimit has been set to the correct value function (cb) { i = 0; fds = []; multipleOpen('./test_lac/openFdsTestFile', 2 * N + 1, function (err) { if (!err) { console.log("No error occured while opening a file too many times"); } fds.forEach(function (fd) { fs.closeSync(fd); }); return cb(); }) } , function (cb) { i = 0; fds = []; multipleOpen('./test_lac/openFdsTestFile2', N, function (err) { if (err) { console.log('An unexpected error occured when opening file not too many times: ' + err); } fds.forEach(function (fd) { fs.closeSync(fd); }); return cb(); }) } // Then actually test NeDB persistence , function () { db.remove({}, { multi: true }, function (err) { if (err) { console.log(err); } db.insert({ hello: 'world' }, function (err) { if (err) { console.log(err); } i = 0; async.whilst( function () { return i < 2 * N + 1; } , function (cb) { db.persistence.persistCachedDatabase(function (err) { if (err) { return cb(err); } i += 1; return cb(); }); } , function (err) { if (err) { console.log("Got unexpected error during one peresistence operation: " + err); } } ); }); }); } ]); ================================================ FILE: test_lac/openFdsLaunch.sh ================================================ ulimit -n 128 node ./test_lac/openFds.test.js ================================================ FILE: test_lac/openFdsTestFile ================================================ Random stuff ================================================ FILE: test_lac/openFdsTestFile2 ================================================ Some other random stuff