Repository: vsivsi/meteor-file-collection
Branch: master
Commit: db1e0e133a35
Files: 17
Total size: 157.6 KB
Directory structure:
gitextract_kvam_bik/
├── .gitignore
├── .gitmodules
├── .travis.yml
├── .versions
├── HISTORY.md
├── LICENSE
├── README.md
├── package.js
├── packages/
│ └── .gitignore
├── src/
│ ├── gridFS.coffee
│ ├── gridFS_client.coffee
│ ├── gridFS_server.coffee
│ ├── http_access_server.coffee
│ ├── resumable_client.coffee
│ ├── resumable_server.coffee
│ └── server_shared.coffee
└── test/
└── file_collection_tests.coffee
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
.build*
.npm
sampleApp/packages
smart.lock
npm-debug.log
================================================
FILE: .gitmodules
================================================
[submodule "resumable"]
path = resumable
url = https://github.com/23/resumable.js.git
================================================
FILE: .travis.yml
================================================
language: node_js
node_js:
- "6.9.1"
before_install:
- "curl https://install.meteor.com | /bin/sh"
- "npm install -g coffee-script"
- "npm install -g spacejam"
# - "export PHANTOMJS_CDNURL=http://cnpmjs.org/downloads"
# - "npm install -g phantomjs-prebuilt"
- "export PATH=$PATH:$HOME/.meteor"
script: "spacejam test-packages ./"
================================================
FILE: .versions
================================================
allow-deny@1.0.5
babel-compiler@6.14.1
babel-runtime@1.0.1
base64@1.0.10
binary-heap@1.0.10
blaze@2.1.8
blaze-tools@1.0.9
boilerplate-generator@1.0.11
caching-compiler@1.1.9
callback-hook@1.0.10
check@1.2.5
coffeescript@1.12.3_1
ddp@1.2.5
ddp-client@1.3.3
ddp-common@1.2.8
ddp-server@1.3.13
deps@1.0.12
diff-sequence@1.0.7
ecmascript@0.6.3
ecmascript-runtime@0.3.15
ejson@1.0.13
geojson-utils@1.0.10
html-tools@1.0.10
htmljs@1.0.10
http@1.2.11
id-map@1.0.9
jquery@1.11.10
local-test:vsivsi:file-collection@1.3.8
logging@1.1.17
meteor@1.6.1
minimongo@1.0.20
modules@0.7.9
modules-runtime@0.7.9
mongo@1.1.15
mongo-id@1.0.6
npm-mongo@2.2.16_1
observe-sequence@1.0.15
ordered-dict@1.0.9
promise@0.8.8
random@1.0.10
reactive-var@1.0.11
retry@1.0.9
routepolicy@1.0.12
spacebars@1.0.12
spacebars-compiler@1.0.12
test-helpers@1.0.11
tinytest@1.0.12
tracker@1.1.2
ui@1.0.11
underscore@1.0.10
url@1.1.0
vsivsi:file-collection@1.3.8
webapp@1.3.13
webapp-hashing@1.0.9
================================================
FILE: HISTORY.md
================================================
## Revision history
### V.NEXT
* Improvements in the Cordova documentation. Thanks @rsmelo92.
### V1.3.8
* Added check to guard against catastrophic remote file deletion triggered by [this Meteor bug](https://github.com/vsivsi/meteor-file-collection/issues/152).
* Updated resumable.js to latest upstream version
* Removed jquery Atmosphere package as a dependency, as resumable.js no longer requires it.
* Documentation improvements
### V1.3.7
* Fixed server internal error when an HTTP requests matches the file-collection `baseURL` but does not match any of the defined HTTP interface definition paths. Also added unit test for this case. Thanks to @nathanbrizzee for reporting.
* Fixed (via gridfs-locking-stream npm package) an event memory leak when renewing locks on open files.
* Updated resumable.js to latest upstream version
* Updated npm and atmosphere dependencies for Meteor 1.4.2.3
### V1.3.6
* Fixed absolute URL for Cordova downloads. (thanks @crapthings)
* Updated npm and atmosphere dependencies for Meteor 1.3.4.4
### V1.3.5
* Added GET support for `Last-Modified-Since` HTTP header (thanks @edemaine)
* Fixed broken server tests when run on Windows (thanks @edemaine)
* Added `selector` parameter type check on function overriding default `remove` method (thanks @brucejo75)
* Updated Meteor package deps to 1.3.2.x versions
* Documentation updates for CORS, and cookie handling in sample code.
### V1.3.4
* Added `withCredentials: true` to the default Resumable object parameters to allow authentication with CORS
* Changed precedence of developer provided HTTP request handlers to come before default resumable.js route handlers, to permit adding headers to responses on that route.
* Updated resumable.js to latest `master`
* Updated npm dependencies.
### V1.3.3
* Use the internal MongoDB 2.1.x driver to create the indexes, not the Meteor provided 1.4.x driver, which is incompatible with MongoDB 3.2. Thanks to @snajjar for reporting.
* Updated package dependencies.
### v1.3.2
* Fixed bug when no HTTP options array is provided and resumable.js isn't used. Thanks to @ndarilek the PR.
* Updated npm dependencies.
* Documentation improvements.
### v1.3.1
* Fixed bugs affecting the built-in resumable.js support when application specified GET/POST routes also matched `/_resumable`. Fixes ensure that multipart parsing for POST requests only happens once, and that `_resumable` routes are fully handled before invoking application route handlers. Thanks to @lightpriest and @mzygmunt for reporting.
### v1.3.0
* Added ability to define custom HTTP OPTIONS request handlers, e.g. to support CORS
* The above feature can be used to add support for files in Apache Cordova apps. Thanks to @dnish for initial work on this.
* `maxUploadSize` option enables a configurable maximum file size for POST, PUT and resumable.js uploads. Thanks to @DanielDornhardt for this feature suggestion.
* Fixed improper ObjectID type in callback from `fc.upsertStream()`. Thanks @DrDanRyan for reporting.
* Updated to MongoDB 2.1.x driver
* Updated npm dependencies
* Added TravisCI support
### v1.2.2
* Fixed an uncaught throw in `importFile()` when the input file doesn't exist, thanks to @dpatte
* Updated resumable.js to latest master
* Updated dependencies
* Documentation improvements
* Fixed bug causing resumable HEAD requests to always return 404, even when a chunk was present in the gridFS store. Thanks to @dnish for help figuring this out.
* Fixed bug causing resumable GET requests to always return 204, even when a chunk was present in the gridFS store.
* Added unit tests for resumable.js GET/HEAD test request backend support
* Don't create unused readable streams for HTTP HEAD requests.
* Unit tests fixed for Meteor 1.2.x by adding many missing meteor packages to test build.
### v1.2.1
* This version number was accidentally skipped over, so there was no v1.2.1 release.
### v1.2.0
* Add the ability to perform limited local updates to the client minimongo collection using `fc.localUpdate()`
* Thanks to @Digital-Thor for his work on integrating [Sortable](https://github.com/RubaXa/Sortable/tree/master/meteor) and file-collection, demonstrating the usefulness of supporting local client-side update
* Fixed bug permitting `update` replacement of entire gridFS documents
* Added update related unit tests
* Defend against `Mongo.Collection !== Mongo.Collection.protype.constructor`
* Updated npm dependencies
* Updated resumable.js
* Switch to using HEAD requests for resumable.js chunk test requests
* Use `Meteor.Error` consistently
### v1.1.5
* Fixed issues related to lock timeouts and proper return values from `fc.remove()`
* Thanks to @timothyarmes for reporting these issues [issue 59](https://github.com/vsivsi/meteor-file-collection/issues/59)
* Added unit test coverage for basic `remove` functionality
* Documentation improvements
* Bumped npm dependencies
* Updated resumable.js to upstream master
### v1.1.4
* Resolving architecture specific build issues with newest mongodb driver
### v1.1.3
* Updated resumable.js and npm dependencies
### v1.1.2
* Checked in the `.versions` file
* Updated project directory structure, adding `src` and `test` subdirs.
* Updated npm dependencies.
* Documentation improvements.
### v1.1.1
* Added informative `throw` when server-only methods are erroneously called on the client
* Moved to current resumable.js master
* Bumped npm dependencies to latest versions
* Added explicit package version to `onTest()` call to work around a Meteor issue when running `meteor test-packages` within an app.
* Added ablity to set the resumable.js server side support MongoDB index name, via the `resumableIndexName` option to `new FileCollection()`. This fixes problems related to [issue 55](https://github.com/vsivsi/meteor-file-collection/issues/55). Thanks to @poojabansal for reporting.
### v1.1.0
* Changed the resumable.js server-side support to return status 204 for testChunk GET requests, rather than 404, which causes undesirable log entries in the client console.
* Fixed bug where received duplicate chunks could mistakenly both be written during resumable.js uploads
* Made POST body MIME/multipart parsing more resistent to malformed requests
* Added unit tests for resumable client and server-side support
* Automatic lock renewal support, can be controlled with `autoRenewLock` option on `fc.upsertStream()` and `fc.findOneStream()`
* `range` option to `fc.findOneStream()` now allows `start` or `end` to be safely omitted.
* Default `chunkSize` changed to 2MB - 1KB, matching the MongoDB recommendation for chunk sizes a little less than a power of 2.
* General performance improvements writing data to gridFS
* Updated mongodb, gridfs-locks and gridfs-locking-stream to newest versions
### v1.0.6
* Fixes #48, which caused unicode filenames to be corrupted in download SaveAs... dialogs. Thanks to @xurwxj for reporting.
* Bump versions of dependencies
### v1.0.5
* Version bump to enable publishing Windows platform build for Meteor 1.1
### v1.0.4
* Updated npm package dependencies
### v1.0.3
* Add automatic indexing for resumable.js queries, to improve uploading performance
* Bumped version of mongodb native driver
### v1.0.2
* Fixes failed unit test caused by null $set update query when using Meteor 1.0.4
* Update version of resumable.js
* Update mongodb npm package version
* Update meteor core package versions
### v1.0.1
* Fixes potential race condition in the underlying gridfs-locks package
* Updates npm package versions
### v1.0.0
* Added support for HTTP range requests (thanks to @riaan53!)
* Switched internally to using new style node.js streams for greatly improved flow-control when streaming large files
* HTTP access definitions may now include an optional custom express.js request handler function
* HTTP access file lookup functions may now access parsed MIME/multipart parameters for POST requests
* Updated all dependencies
* *BREAKING CHANGE:* `fc.upsertStream` may no longer append (mode 'w+') to existing files. This is a restriction added to the underlying node.js gridFS driver, and was a little used feature that was traded-off for node.js 0.10 new stream support
### v0.3.6
* Updated dependencies including resumable.js
### v0.3.5
* Rebuilt/published previous version using actual Meteor 1.0 instead of a checkout of Meteor 1.0
### v0.3.4
* Documentation improvements
* Dependent package version updates
### v0.3.3
* Added a polyfill for `Function.prototype.bind()` to enable compatibility with PhantomJS, which as of version 1.9.7 lacks support for `.bind()`
* Bumped mongodb npm package version.
### v0.3.2
* Bumped versions of npm dependencies, including a fix for a bson build error in the npm mongodb driver.
### v0.3.1
* Bumped versions of npm dependencies, including a fix for a rare gridfs file locking bug.
* Documentation fixes.
### v0.3.0
* Updated package name and information to conform with Meteor 0.9.0 package system. Thanks to @ryw for a PR that showed what needed to be done.
* Added versions.json file
* Documentation updates
* Added additional error checking when receiving a 'close' event.
* Don't automatically index the fileCollection.
* Updated express and mopngodb packages to latest versions
* All features deprecated in v0.2.0 are obsolete and removed
### v0.2.3
* Added additional checking that `_id` values in URLs are 24-digit hex strings before attempting to make them into ObjectIds
* Bumped express.js to latest version
### v0.2.2
* Fixed #15
* Updated README for new sample apps.
* Updated Resumable.js
### v0.2.1
* Added sanity checking of input to `fc.allow()` and `fc.deny()`
* Allow options to be truly optional w/ callback in `findOneStream()` and `upsertStream()`
* Fixed reversed/broken sort/skip options on `findOneStream()`
* Fixed an issue where `ObjectID`s in file metadata change type after `upsertStream()`
* Updated resumable.js
* Documentation improvements.
### v0.2.0
* `fc.allow` and `fc.deny` now support rules for the `'read'` operation, which secures HTTP GET/HEAD requests.
* `fc.allow` and `fc.deny` now support rules for the `'write'` operation, which impacts HTTP POST/PUT requests. `'write'` allow/deny rules are replacing the use of `'update'` rules, and work identically. The reason for the change is to avoid confusion with the `'update'` rules on Meteor collections and to better match the new `'read'` rules. `'update'` rules continue to work, but are now deprecated.
* HTTP GET requests now support the `?filename=somename.txt` query. This is similar to the `?download=true` option, except that the default filename used by the browser "Save As..." dialog is specified by the request URL.
* Added support for sending X-Auth-Token as an HTTP Cookie. This is safer than using the `?X-Auth-Token` URL query, which continues to work, but is now deprecated.
* When using the built-in Resumable.js upload support, if you create `'write'` allow/deny rules that depend on `userId` you must set an `X-Auth-Token` cookie. See the client [example code in README.md](https://github.com/vsivsi/meteor-file-collection#example) for an example of how to do this.
* Acceptance tests are now written in Coffeescript.
* Version updates for most Npm packages.
* Documentation improvements.
* The sample application has been moved to its own [GitHub repo](https://github.com/vsivsi/meteor-file-job-sample-app).
* Thanks to @elbowz for multiple feature suggestions.
### v0.1.18
* Allow/Deny rules are now called in the same order as in Meteor (deny rules go first).
### v0.1.17
* Fixed another issue when calling deprecated fileCollection object without new.
### v0.1.16
* Fixed issue when calling deprecated fileCollection object without new.
### v0.1.15
* Added FileCollection export
* Updated docs to use FileCollection and note that fileCollection is deprecated and will be removed in 0.2.0
* Added deprecation warning to console.warn for fileCollection use
* Bumped express version
### v0.1.14
* Improved documentation. Thanks to @renarl for suggestion.
* Updated express version.
### v0.1.13
* Updated versions of resumable, async, mongodb, gridfs-locks, gridfs-locking-stream and express
* Documentation improvements
### v0.1.12
* Fixed typos in documentation. Thanks to @dawjdh
### v0.1.11
* Fixed sample code in README.md. Thanks to @rcy
### v0.1.10
* Fixed resumable.js upload crash
### v0.1.9
* Fix missing filenames in resumable.js uploads caused by changes in mongodb 1.4.3
* upsertStream now correctly updates gridFS attributes when provided
### v0.1.8
* Updates for Meteor v0.8.1.1
* Documentation improvements
* Updated npm package versions
### v0.1.7
* Bumped package versions to fix more mongodb 2.4.x backwards compatility issues
### v0.1.6
* Bumped gridfs-locks version to fix a mongodb 2.4.x backwards compatility issue
### v0.1.0 - v0.1.5
* Initial revision and documentation improvements.
================================================
FILE: LICENSE
================================================
Copyright (C) 2014-2017 by Vaughn Iverson
file-collection is free software released under the MIT/X11 license:
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
================================================
FILE: README.md
================================================
# file-collection
[](https://travis-ci.org/vsivsi/meteor-file-collection)
## Notice!
#### **Effective 01-01-2018 this project will enter "maintenance mode". This means that I will no longer be implementing any new features or providing debugging help or "general support" via github issues.**
**I will (for some period of time) still consider *high quality* pull-requests implementing bug fixes and generally useful new minor features. ("High quality" is admittedly subjective, but some must-haves: documentation, tests, backward compatibility, low change complexity, harmony with existing design...)**
**If you would like to become a maintainer, Great!. The best way to get there is to make some high quality PRs! Eventually I will probably get tired of merging them and will just give you permission to push and publish.
If there is enough community interest, I will probably migrate all of my Meteor packages into a separate github org so that they can live on without my direct involvement.**
**Why? Simple: I'm no longer actively developing or maintaining any Meteor based applications (and have no forseeable plans to do so). Having moved on, the effort to keep up with changes in the Meteor/Atmosphere ecosystem no longer has any payoff for me personally.**
## Introduction
file-collection is a Meteor.js package that cleanly extends Meteor's Collection metaphor to efficiently manage collections of files and their data. File Collections are fully reactive, and if you know how to use Meteor Collections, you already know most of what you need to begin working with this package.
Major features:
* HTTP upload and download including support for Meteor authentication
* Client and Server integration of [resumable.js](http://resumablejs.com/) for robust chunked uploading
* Also compatible with traditional HTTP POST or PUT file uploading
* HTTP range requests support random access for resumable downloads, media seeking, etc.
* Robust file locking allows safe replacement and removal of files even on a busy server
* External changes to the underlying file store automatically synchronize with the Meteor collection
* Designed for efficient handling of millions of small files as well as huge files 10GB and above
These features (and more) are possible because file-collection tightly integrates MongoDB [gridFS](http://docs.mongodb.org/manual/reference/gridfs/) with Meteor Collections, without any intervening plumbing or unnecessary layers of abstraction.
#### Quick server-side example
```javascript
myFiles = new FileCollection('myFiles',
{ resumable: true, // Enable built-in resumable.js chunked upload support
http: [ // Define HTTP route
{ method: 'get', // Enable a GET endpoint
path: '/:md5', // this will be at route "/gridfs/myFiles/:md5"
lookup: function (params, query) { // uses express style url params
return { md5: params.md5 }; // a query mapping url to myFiles
}}]});
// You can add publications and allow/deny rules here to securely
// access myFiles from clients.
// On the server, you can access everything without limitation:
// Find a file document by name
thatFile = myFiles.findOne({ filename: 'lolcat.gif' });
// or get a file's data as a node.js Stream2
thatFileStream = myFiles.findOneStream({ filename: 'lolcat.gif' });
// Easily remove a file and its data
result = myFiles.remove(thatFile._id);
```
### Feature summary
Under the hood, file data is stored entirely within the Meteor MongoDB instance using a Mongo technology called [gridFS](http://docs.mongodb.org/manual/reference/gridfs/). Your file collection and the underlying gridFS collection remain perfectly in sync because they *are* the same collection; and file collections are automatically safe for concurrent read/write access to files via [MongoDB based locking](https://github.com/vsivsi/gridfs-locks). The file-collection package also provides a simple way to enable secure HTTP (GET, POST, PUT, DELETE) interfaces to your files, and additionally has built-in support for robust and resumable file uploads using the excellent [Resumable.js](http://www.resumablejs.com/) library.
### What's new in v1.3?
* CORS/Cordova support via the ability to define custom HTTP OPTIONS request handlers
* Global and per-request file upload size limits via the new `maxUploadSize` option
Additional changes are detailed in the HISTORY file.
### Design philosophy
**Update: CollectionFS appears to no longer be actively maintained, so caveat emptor.**
My goal in writing this package was to stay true to the spirit of Meteor and build something efficient and secure that "just works" with a minimum of fuss.
If you've been searching for ways to deal with file data on Meteor, you've probably also encountered [collectionFS](https://atmospherejs.com/cfs/standard-packages). If not, you should definitely check it out. It's a great set of packages written by smart people, and I even pitched in to help with a rewrite of their MongoDB gridFS support.
Here's the difference in a nutshell: collectionFS is a Ferrari, and file-collection is a Fiat.
They do approximately the same thing using some of the same technologies, but reflect different design priorities. file-collection is much simpler and somewhat less flexible; but if it meets your needs you'll find it has a lot fewer moving parts and may be significantly more efficient to work with and use.
If you're trying to quickly prototype an idea or you know that you just need a straightforward way of dealing with files, you should definitely try file-collection. Because it is so much simpler, you may also find that it is easier to understand and customize for the specific needs of your project.
## Example
Enough words, time for some more code...
The block below implements a `FileCollection` on server, including support for owner-secured HTTP file upload using `Resumable.js` and HTTP download. It also sets up the client to provide drag and drop chunked file uploads to the collection. The only things missing here are UI templates and some helper functions. See the [meteor-file-sample-app](https://github.com/vsivsi/meteor-file-sample-app) project for a complete working version written in [CoffeeScript](http://coffeescript.org/).
```javascript
// Create a file collection, and enable file upload and download using HTTP
myFiles = new FileCollection('myFiles',
{ resumable: true, // Enable built-in resumable.js upload support
http: [
{ method: 'get',
path: '/:md5', // this will be at route "/gridfs/myFiles/:md5"
lookup: function (params, query) { // uses express style url params
return { md5: params.md5 }; // a query mapping url to myFiles
}
}
]
}
);
if (Meteor.isServer) {
// Only publish files owned by this userId, and ignore
// file chunks being used by Resumable.js for current uploads
Meteor.publish('myData',
function (clientUserId) {
if (clientUserId === this.userId) {
return myFiles.find({ 'metadata._Resumable': { $exists: false },
'metadata.owner': this.userId });
} else { // Prevent client race condition:
return null; // This is triggered when publish is rerun with a new
// userId before client has resubscribed with that userId
}
}
);
// Allow rules for security. Should look familiar!
// Without these, no file writes would be allowed
myFiles.allow({
// The creator of a file owns it. UserId may be null.
insert: function (userId, file) {
// Assign the proper owner when a file is created
file.metadata = file.metadata || {};
file.metadata.owner = userId;
return true;
},
// Only owners can remove a file
remove: function (userId, file) {
// Only owners can delete
return (userId === file.metadata.owner);
},
// Only owners can retrieve a file via HTTP GET
read: function (userId, file) {
return (userId === file.metadata.owner);
},
// This rule secures the HTTP REST interfaces' PUT/POST
// Necessary to support Resumable.js
write: function (userId, file, fields) {
// Only owners can upload file data
return (userId === file.metadata.owner);
}
});
}
if (Meteor.isClient) {
Meteor.startup(function() {
// This assigns a file upload drop zone to some DOM node
myFiles.resumable.assignDrop($(".fileDrop"));
// This assigns a browse action to a DOM node
myFiles.resumable.assignBrowse($(".fileBrowse"));
// When a file is added via drag and drop
myFiles.resumable.on('fileAdded', function (file) {
// Create a new file in the file collection to upload
myFiles.insert({
_id: file.uniqueIdentifier, // This is the ID resumable will use
filename: file.fileName,
contentType: file.file.type
},
function (err, _id) { // Callback to .insert
if (err) { return console.error("File creation failed!", err); }
// Once the file exists on the server, start uploading
myFiles.resumable.upload();
}
);
});
// This autorun keeps a cookie up-to-date with the Meteor Auth token
// of the logged-in user. This is needed so that the read/write allow
// rules on the server can verify the userId of each HTTP request.
Deps.autorun(function () {
// Sending userId prevents a race condition
Meteor.subscribe('myData', Meteor.userId());
// $.cookie() assumes use of "jquery-cookie" Atmosphere package.
// You can use any other cookie package you may prefer...
$.cookie('X-Auth-Token', Accounts._storedLoginToken(), { path: '/' });
});
});
}
```
## Installation
I've only tested with Meteor v0.9.x and v1.x.x, and older versions run on Meteor v0.8 as well, but why would you want to do that?
To add to your project, run:
meteor add vsivsi:file-collection
The package exposes a global object `FileCollection` on both client and server.
If you'd like to try out the sample app, you can clone the repo from github:
```
git clone https://github.com/vsivsi/meteor-file-sample-app.git fcSample
```
Then go to the `fcSample` subdirectory and run meteor to launch:
```
cd fcSample
meteor
```
You should now be able to point your browser to `http://localhost:3000/` and play with the sample app.
A more advanced example that implements a basic image gallery with upload and download support and automatic thumbnail generation using the [job-collection package](https://atmospherejs.com/vsivsi/job-collection) is available here: https://github.com/vsivsi/meteor-file-job-sample-app
To run tests (using Meteor tiny-test):
```
git clone --recursive https://github.com/vsivsi/meteor-file-collection FileCollection
cd FileCollection
meteor test-packages ./
```
Load `http://localhost:3000/` and the tests should run in your browser and on the server.
## Use
Below you'll find the [MongoDB gridFS `files` data model](http://docs.mongodb.org/manual/reference/gridfs/#the-files-collection). This is also the schema used by file-collection because a FileCollection *is* a gridFS collection.
```javascript
{
"_id" : <ObjectId>,
"length" : <number>,
"chunkSize" : <number>
"uploadDate" : <Date>
"md5" : <string>
"filename" : <string>,
"contentType" : <string>,
"aliases" : <array of strings>,
"metadata" : <object>
}
```
Here are a few things to keep in mind about the gridFS file data model:
* Some of the attributes belong to gridFS, and you may **lose data** if you mess around with these.
* For this reason, `_id`, `length`, `chunkSize`, `uploadDate` and `md5` are read-only.
* Some of the attributes belong to you. Your application can do whatever you want with them.
* `filename`, `contentType`, `aliases` and `metadata` are yours. Go to town.
* `contentType` should probably be a valid [MIME Type](https://en.wikipedia.org/wiki/MIME_type)
* `filename` is *not* guaranteed unique. `_id` is a better bet if you want to be sure of what you're getting.
Sound complicated? It really isn't and file-collection is here to help.
First off, when you create a new file you use `myFiles.insert(...)` and just populate whatever attributes you care about. The file-collection package does the rest. You are guaranteed to get a valid gridFS file, even if you just do this: `id = myFiles.insert();`
Likewise, when you run `myFiles.update(...)` on the server, file-collection tries really hard to make sure that you aren't clobbering one of the "read-only" attributes with your update modifier. For safety, clients are never allowed to directly `update`, although you can selectively give them that power via `Meteor.methods()`.
### Limits and performance
There are essentially no hard limits on the number or size of files other than what your hardware will support.
At no point in normal operation is a file-sized data buffer ever in memory. All of the file data import/export mechanisms are [stream based](http://nodejs.org/api/stream.html#stream_stream), so even very active servers should not see much memory dedicated to file transfers.
File data is never copied within a collection. During chunked file uploading, file chunk references are changed, but the data itself is never copied. This makes file-collection particularly efficient when handling multi-gigabyte files.
file-collection uses robust multiple reader / exclusive writer file locking on top of gridFS, so essentially any number of readers and writers of shared files may peacefully coexist without risk of file corruption. Note that if you have other applications reading/writing directly to a gridFS collection (e.g. a node.js program, not using Meteor/file-collection), it will need to use the [`gridfs-locks`](https://www.npmjs.org/package/gridfs-locks) or [`gridfs-locking-stream`](https://www.npmjs.org/package/gridfs-locking-stream) npm packages to safely inter-operate with file-collection.
### Security
You may have noticed that the gridFS `files` data model says nothing about file ownership. That's your job. If you look again at the example code block above, you will see a bare bones `Meteor.userId` based ownership scheme implemented with the attribute `file.metadata.owner`. As with any Meteor Collection, allow/deny rules are needed to enforce and defend that document attribute, and file-collection implements that in *almost* the same way that ordinary Meteor Collections do. Here's how they're a little different:
* A file is always initially created as a valid zero-length gridFS file using `insert` on the client/server. When it takes place on the client, the `insert` allow/deny rules apply.
* The `remove` allow/deny rules work just as you would expect for client calls, and they also secure the HTTP DELETE method when it's used.
* The `read` allow/deny rules secure access to file data requested via HTTP GET. These rules have no effect on client `find()` or `findOne()` methods; these operations are secured by `Meteor.publish()` as with any meteor collection.
* The `write` allow/deny rules secure writing file *data* to a previously inserted file via HTTP methods. This means that an HTTP POST/PUT cannot create a new file by itself. It needs to have been inserted first, and only then can data be added to it using HTTP.
* There are no `update` allow/deny rules because clients are always prohibited from directly updating a file document's attributes.
* All HTTP methods are disabled by default. When enabled, they can be authenticated to a Meteor `userId` by using a currently valid authentication token passed either in the HTTP request header or using an HTTP Cookie.
## API
The `FileCollection` API is essentially an extension of the [Meteor Collection API](http://docs.meteor.com/#collections), with almost all of the same methods and a few new file specific ones mixed in.
The big loser is `upsert()`, it's gone in `FileCollection`. If you try to call it, you'll get an error. `update()` is also disabled on the client side, but it can be safely used on the server to implement `Meteor.Method()` calls for clients to use.
### fc = new FileCollection([name], [options])
#### Create a new `FileCollection` object - Server and Client
```javascript
// create a new FileCollection with all default values
fc = new FileCollection('fs', // base name of collection
{ resumable: false, // Disable resumable.js upload support
resumableIndexName: undefined, // Not used when resumable is false
chunkSize: 2*1024*1024 - 1024, // Use 2MB chunks for gridFS and resumable
baseURL: '\gridfs\fs', // Default base URL for all HTTP methods
locks: { // Parameters for gridfs-locks
timeOut: 360, // Seconds to wait for an unavailable lock
pollingInterval: 5, // Seconds to wait between lock attempts
lockExpiration: 90 // Seconds until a lock expires
}
http: [] // HTTP method definitions, none by default
}
);
```
**Note:** The same `FileCollection` call should be made on both the client and server.
`name` is the root name of the underlying MongoDB gridFS collection. If omitted, it defaults to `'fs'`, the default gridFS collection name. Internally, three collections are used for each `FileCollection` instance:
* `[name].files` - This is the collection you actually see when using file-collection
* `[name].chunks` - This collection contains the actual file data chunks. It is managed automatically.
* `[name].locks` - This collection is used by `gridfs-locks` to make concurrent reading/writing safe.
`FileCollection` is a subclass of `Meteor.Collection`, however it doesn't support the same `[options]`.
Meteor Collections support `connection`, `idGeneration` and `transform` options. Currently, file-collection only supports the default Meteor server connection, although this may change in the future. All `_id` values used by `FileCollection` are MongoDB style IDs. The Meteor Collection transform functionality is unsupported in `FileCollection`.
Here are the options `FileCollection` does support:
* `options.resumable` - `<boolean>` When `true`, exposes the [Resumable.js API](http://www.resumablejs.com/) on the client and the matching resumable HTTP support on the server.
* `options.resumableIndexName` - `<string>` When provided and `options.resumable` is `true`, this value will be the name of the internal-use MongoDB index that the server-side resumable.js support attempts to create. This is useful because the default index name MongoDB creates is long (94 chars out of a total maximum namespace length of 127 characters), which may create issues when combined with long collection and/or database names. If this collection already exists the first time an application runs using this setting, it will likely have no effect because an identical index will already exist (under a different name), causing MongoDB to ignore request to create a duplicate index with a different name. In this case, you must manually drop the old index and then restart your application to generate a new index with the requested name.
* `options.chunkSize` - `<integer>` Sets the gridFS and Resumable.js chunkSize in bytes. The default value of a little less than 2MB is probably a good compromise for most applications, with the maximum being 8MB - 1. Partial chunks are not padded, so there is no storage space benefit to using small chunk sizes. If you are uploading very large files over a fast network and upload spped matters, then a `chunkSize` of 8MB - 1KB (= 8387584) will likly optimize upload speed. However, if you elect to use such large `chunkSize` values, make sure that the replication oplog of your MongoDB instance is large enough to handle this, or you will risk having your client and server collections lose synchronization during uploads. Meteor's development mode only uses an oplog of 8 MB, which will almost certainly cause problems for high speed uploads to apps using a large `chunkSize`.
For more information on Meteor's use of the MongoDB oplog, see: [Meteor livequery](https://www.meteor.com/livequery).
* `options.baseURL` - `<string>` Sets the base route for all HTTP interfaces defined on this collection. Default value is `/gridfs/[name]`
* `options.locks` - `<object>` Locking parameters, the defaults should be fine and you shouldn't need to set this, but see the `gridfs-locks` [`LockCollection` docs](https://github.com/vsivsi/gridfs-locks#lockcollectiondb-options) for more information.
* `option.maxUploadSize` - `<integer>` Maximum number of bytes permitted for any HTTP POST, PUT or resumable.js file upload.
* `option.http` - <array of objects> HTTP interface configuration objects, described below:
#### Configuring HTTP methods
Each object in the `option.http` array defines one HTTP request interface on the server, and has these three attributes:
* `obj.method` - `<string>` The HTTP request method to define, one of `get`, `post`, `put`, `delete` (or `options` with a custom handler).
* `obj.path` - `<string>` An [express.js style](http://expressjs.com/4x/api.html#req.params) route path with parameters. This path will be added to the path specified by `options.baseURL`.
* `obj.lookup` - `<function>` A function that is called when an HTTP request matches the `method` and `path`. It is provided with the values of the route parameters and any URL query parameters, and it should return a mongoDB query object which can be used to find a file that matches those parameters. For POST requests, it is also provided any with MIME/multipart parameters and other file information from the multipart headers.
* `obj.handler` - `<function>` OPTIONAL! This is an advanced feature that allows the developer to provide a custom "express.js style" request handler to satisfy requests for this specific request interface. For an example of how this works, please see the resumable.js upload support implementation in the source file `resumable_server.coffee`.
When arranging http interface definition objects in the array provided to `options.http`, be sure to put more specific paths for a given HTTP method before more general ones. For example: `\hash\:md5` should come before `\:filename\:_id` because `"hash"` would match to filename, and so `\hash\:md5` would never match if it came second. Obviously this is a contrived example to demonstrate that order is significant.
Note that an authenticated userId is not provided to the `lookup` function. UserId based permissions should be managed using the allow/deny rules described later on.
Here are some example HTTP interface definition objects to get you started:
```javascript
// GET file data by md5 sum
{ method: 'get',
path: '/hash/:md5',
lookup: function (params, query) {
return { md5: params.md5 } } }
// DELETE a file by _id. Note that the URL parameter ":_id" is a special
// case, in that it will automatically be converted to a Meteor ObjectID
// in the passed params object.
{ method: 'delete',
path: '/:_id',
lookup: function (params, query) {
return { _id: params._id } } }
// GET a file based on a filename or alias name value
{ method: 'get',
path: '/name/:name',
lookup: function (params, query) {
return {$or: [ {filename: params.name },
{aliases: {$in: [ params.name ]}} ]} }}
// PUT data to a file based on _id and a secret value stored as metadata
// where the secret is supplied as a query parameter e.g. ?secret=sfkljs
{ method: 'put',
path: '/write/:_id',
lookup: function (params, query) {
return { _id: params._id, "metadata.secret": query.secret} }}
// POST data to a file based on _id and a secret value stored as metadata
// where the secret is supplied as a MIME/Multipart parameter
{ method: 'post',
path: '/post/:_id',
lookup: function (params, query, multipart) {
return { _id: params._id, "metadata.secret": multipart.params.secret} }}
// GET a file based on a query type and numeric coordinates metadata
{ method: 'get',
path: '/tile/:z/:x/:y',
lookup: function (params, query) {
return { "metadata.x": parseInt(params.x), // Note that all params
"metadata.y": parseInt(params.y), // (execept _id) are strings
"metadata.z": parseInt(params.z),
contentType: query.type} }}
```
#### CORS / Apache Cordova Support
The HTTP access in file-collection can be configured for compatibility with [Cross Origin Resource Sharing (CORS)](https://en.wikipedia.org/wiki/Cross-origin_resource_sharing) via use of a custom handler for the `'options'`
request method.
This provides a simple way to support accessing file-collection files in [Apache Cordova](https://github.com/meteor/meteor/wiki/Meteor-Cordova-integration) client applications:
```javascript
myFiles = new FileCollection('myFiles',
{ resumable: true, // Enable built-in resumable.js chunked upload support
http: [ // Define HTTP route
{ method: 'get', // Enable a GET endpoint
path: '/:md5', // this will be at route "/gridfs/myFiles/:md5"
lookup: function (params, query) { // uses express style url params
return { md5: params.md5 }; // a query mapping url to myFiles
},
handler: function (req, res, next) {
if (req.headers && req.headers.origin) {
res.setHeader('Access-Control-Allow-Origin', 'http://meteor.local'); // For Cordova
res.setHeader('Access-Control-Allow-Credentials', true);
}
next();
}
},
{ method: 'put', // Enable a PUT endpoint
path: '/:md5', // this will be at route "/gridfs/myFiles/:md5"
lookup: function (params, query) { // uses express style url params
return { md5: params.md5 }; // a query mapping url to myFiles
},
handler: function (req, res, next) {
if (req.headers && req.headers.origin) {
res.setHeader('Access-Control-Allow-Origin', 'http://meteor.local'); // For Cordova
res.setHeader('Access-Control-Allow-Credentials', true);
}
next();
}
},
{ method: 'options', // Enable an OPTIONS endpoint (for CORS)
path: '/:md5', // this will be at route "/gridfs/myFiles/:md5"
lookup: function (params, query) { // uses express style url params
return { md5: params.md5 }; // a query mapping url to myFiles
},
handler: function (req, res, next) { // Custom express.js handler for OPTIONS
res.writeHead(200, {
'Content-Type': 'text/plain',
'Access-Control-Allow-Origin': 'http://meteor.local', // For Cordova
'Access-Control-Allow-Credentials': true,
'Access-Control-Allow-Headers': 'x-auth-token, user-agent',
'Access-Control-Allow-Methods': 'GET, PUT'
});
res.end();
return;
}
}
]
}
);
```
If using resumable endpoint use this instead:
```javascript
myFiles = new FileCollection('myFiles',
{ resumable: true, // Enable built-in resumable.js chunked upload support
http: [ // Define HTTP route
{
method: 'POST', // Enable a POST endpoint
path: '/_resumable', // this will be at route "/gridfs/images/_resumable"
lookup: function (params, query) { // uses express style url params
return {}; // a dummy query
},
handler: function (req, res, next) {
if (req.headers && req.headers.origin) {
res.setHeader('Access-Control-Allow-Origin', req.headers.origin); // For Cordova
res.setHeader('Access-Control-Allow-Credentials', true);
}
next();
}
},
{
method: 'head', // Enable an HEAD endpoint (for CORS)
path: '/_resumable', // this will be at route "/gridfs/images/_resumable/"
lookup: function (params, query) { // uses express style url params
return { }; // a dummy query
},
handler: function (req, res, next) { // Custom express.js handler for HEAD
if (req.headers && req.headers.origin) {
res.setHeader('Access-Control-Allow-Origin', req.headers.origin); // For Cordova
res.setHeader('Access-Control-Allow-Credentials', true);
}
next();
}
},
{
method: 'options', // Enable an OPTIONS endpoint (for CORS)
path: '/_resumable', // this will be at route "/gridfs/images/_resumable/"
lookup: function (params, query) { // uses express style url params
return { }; // a dummy query
},
handler: function (req, res, next) { // Custom express.js handler for OPTIONS
res.writeHead(200, {
'Content-Type': 'text/plain',
'Access-Control-Allow-Origin': req.headers.origin, // For Cordova
'Access-Control-Allow-Credentials': true,
'Access-Control-Allow-Headers': 'x-auth-token, user-agent',
'Access-Control-Allow-Methods': 'GET, POST, HEAD, OPTIONS'
});
res.end();
return;
}
}
]
}
);
```
**Note!:** Reportedly due to a bug in Cordova, you need to add the following line into your mobile-config.js
```
App.accessRule("blob:*");
```
Please notice that this package will only work with "blob" types when using resumable on Cordova enviroment. If you are using a "file" type remember to convert it to blob before the upload.
#### HTTP authentication
Authentication of HTTP requests is performed using Meteor login tokens. When Meteor [Accounts](http://docs.meteor.com/#accounts_api) are used in an application, a logged in client can see its current token using `Accounts._storedLoginToken()`. Tokens are passed in HTTP requests using either the HTTP header `X-Auth-Token: [token]` or using an HTTP cookie named `X-Auth-Token=[token]`. If the token matches a valid logged in user, then that userId will be provided to any allow/deny rules that are called for permission for an action.
For non-Meteor clients that aren't logged-in humans using browsers, it is possible to authenticate with Meteor using the DDP protocol and programmatically obtain a token. See the [ddp-login](https://www.npmjs.org/package/ddp-login) npm package for a node.js library and command-line utility capable of logging into Meteor (similar libraries also exist for other languages such as Python).
#### HTTP request behaviors
URLs used to HTTP GET file data within a browser can be configured to automatically trigger a "File SaveAs..." download by using the `?download=true` query in the request URL. Similarly, if the `?filename=[filename.ext]` query is used, a "File SaveAs..." download will be invoked, but using the specified filename as the default, rather than the GridFS `filename` as is the case with `?download=true`.
To cache files in the browser use the `?cache=172800` query in the request URL, where 172800 (48h) is the time in seconds. This will set the header response information to `cache-control:max-age=172800, private`. Caching is useful when streaming videos or audio files to avoid unwanted calls to the server.
HTTP PUT requests write the data from the request body directly into the file. By contrast, HTTP POST requests assume that the body is formatted as MIME multipart/form-data (as an old-school browser form based file upload would generate), and the data written to the file is taken from the part named `"file"`. Below are example [cURL](`https://en.wikipedia.org/wiki/CURL#cURL`) commands that successfully invoke each of the four possible HTTP methods.
```sh
# This assumes a baseURL of '/gridfs/fs' and method definitions with a path
# of '/:_id' for each method, for example:
# { method: 'delete',
# path: '/:_id',
# lookup: function (params, query) {
# return { _id: params._id } } }
# The file with _id = 38a14c8fef2d6cef53c70792 must exist for these to succeed.
# The auth token should match a logged-in userId
# GET the file data
curl -X GET 'http://127.0.0.1:3000/gridfs/fs/38a14c8fef2d6cef53c70792' \
-H 'X-Auth-Token: 3pl5vbN_ZbKDJ1ko5JteO3ZSTrnQIl5g6fd8XW0U4NQ'
# POST with file in multipart/form-data
curl -X POST 'http://127.0.0.1:3000/gridfs/fs/38a14c8fef2d6cef53c70792' \
-F 'file=@"lolcat.gif";type=image/gif' \
-H 'X-Auth-Token: 3pl5vbN_ZbKDJ1ko5JteO3ZSTrnQIl5g6fd8XW0U4NQ'
# PUT with file in request body
curl -X PUT 'http://127.0.0.1:3000/gridfs/fs/38a14c8fef2d6cef53c70792' \
-H 'Content-Type: image/gif' \
-H 'X-Auth-Token: 3pl5vbN_ZbKDJ1ko5JteO3ZSTrnQIl5g6fd8XW0U4NQ' \
-T "lolcat.gif"
# DELETE the file
curl -X DELETE 'http://127.0.0.1:3000/gridfs/fs/38a14c8fef2d6cef53c70792' \
-H 'X-Auth-Token: 3pl5vbN_ZbKDJ1ko5JteO3ZSTrnQIl5g6fd8XW0U4NQ'
```
Below are the methods defined on the returned `FileCollection` object
### fc.resumable
#### Resumable.js API object - Client only
```javascript
fc.resumable.assignDrop($(".fileDrop")); // Assign a file drop target
// When a file is dropped on the target (or added some other way)
myData.resumable.on('fileAdded', function (file) {
// file contains a resumable,js file object, do something with it...
}
```
`fc.resumable` is a ready to use, preconfigured `Resumable` object that is available when a `FileCollection` is created with `options.resumable == true`. `fc.resumable` contains the results of calling `new Resumable([options])` where all of the options have been specified by file-collection to work with its server side support. See the [Resumable.js documentation](http://www.resumablejs.com/) for more details on how to use it.
### fc.find(selector, [options])
#### Find any number of files - Server and Client
```javascript
// Count the number of likely lolcats in collection, this is reactive
lols = fc.find({ 'contentType': 'image/gif'}).count();
```
`fc.find()` is identical to [Meteor's `Collection.find()`](http://docs.meteor.com/#find)
### fc.findOne(selector, [options])
#### Find a single file. - Server and Client
```javascript
// Grab the file document for a known lolcat
// This is not the file data, see fc.findOneStream() for that!
myLol = fc.findOne({ 'filename': 'lolcat.gif'});
```
`fc.findOne()` is identical to [Meteor's `Collection.findOne()`](http://docs.meteor.com/#findone)
### fc.insert([file], [callback])
#### Insert a new zero-length file. - Server and Client
```javascript
// Create a new zero-length file in the collection
// All fields are optional and will get defaults if omitted
_id = fc.insert({
_id: new Meteor.Collection.ObjectID(),
filename: 'nyancat.flv',
contentType: 'video/x-flv',
metadata: { owner: 'posterity' },
aliases: [ ]
}
// Callback here, if you really care...
);
```
`fc.insert()` is the same as [Meteor's `Collection.insert()`](http://docs.meteor.com/#insert), except that the document is forced to be a [gridFS `files` document](http://docs.mongodb.org/manual/reference/gridfs/#the-files-collection). All attributes not supplied get default values, non-gridFS attributes are silently dropped. Inserts from the client that do not conform to the gridFS data model will automatically be denied. Client inserts will additionally be subjected to any `'insert'` allow/deny rules (which default to deny all inserts).
### fc.remove(selector, [callback])
#### Remove a file and all of its data. - Server and Client
```javascript
// Make it go away, data and all
fc.remove(
{ filename: 'nyancat.flv' }
// Callback here, if you want to be absolultely sure it's really gone...
);
```
`fc.remove()` is nearly the same as [Meteor's `Collection.remove()`](http://docs.meteor.com/#remove), except that in addition to removing the file document, it also removes the file data chunks and locks from the gridFS store. For safety, undefined and empty selectors (`undefined`, `null` or `{}`) are all rejected. Client calls are subjected to any `'remove'` allow/deny rules (which default to deny all removes). Returns the number of documents actually removed on the server, except when invoked on the client without a callback. In that case it returns the simulated number of documents removed from the local mini-mongo store.
### fc.update(selector, modifier, [options], [callback])
#### Update application controlled gridFS file attributes. - Server only
Note: A local-only version of update is available on the client. See docs for `fc.localUpdate()` for details.
```javascript
// Update some attributes we own
fc.update(
{ filename: 'keyboardcat.mp4' },
{
$set: { 'metadata.comment': 'Play them off...' } },
$push: { aliases: 'Fatso.mp4' }
}
// Optional options here
// Optional callback here
);
```
`fc.update()` is nearly the same as [Meteor's `Collection.update()`](http://docs.meteor.com/#update), except that it is a server only method, and it will return an error if:
* any of the gridFS "read-only" attributes would be modified
* any standard gridFS document level attributes would be removed
* the `upsert` option is attempted
Since `fc.update()` only runs on the server, it is *not* subjected to any allow/deny rules.
### fc.localUpdate(selector, modifier, [options], [callback])
#### Update local minimongo file attributes. - Client only
**Warning!** Changes made using this function do not persist to the server! You must implement your own Meteor methods to perform persistent updates from a client. For example:
```javascript
// Implement latency compensated update using Meteor methods and localUpdate
Meteor.methods({
updateFileComment: function (fileId, comment) {
// Always check method params!
check(fileId, Mongo.ObjectID);
check(comment, Match.Where(function (x) {
check(x, String);
return x.length <= 140;
}));
// You'll probably want to do some kind of ownership check here...
var update = null;
// If desired you can avoid this by initializing fc.update
// on the client to be fc.localUpdate
if (this.isSimulation) {
update = fc.localUpdate; // Client stub updates locally for latency comp
} else { // isServer
update = fc.update; // Server actually persists the update
}
// Use whichever function the environment dictates
update({ _id: _id }, {
$set: { 'metadata.comment': comment }
}
// Optional options here
// Optional callback here
);
}
});
```
`fc.localUpdate()` is nearly the same as [Meteor's server-side `Collection.update()`](http://docs.meteor.com/#update), except that it is a client only method, and changes made using it do not propagate to the server. This call is useful for implementing latency compensation in the client UI when performing server updates using a Meteor method. This call can be invoked in the client Method stub to simulate what will be happening on the server. For this reason, this call can perform updates using complex selectors and the `multi` option, unlike client side updates on normal Mongo Collections.
It will return an error if:
* any of the gridFS "read-only" attributes would be modified
* any standard gridFS document level attributes would be removed
* the `upsert` option is attempted
Since `fc.localUpdate()` only changes data on the client, it is *not* subjected to any allow/deny rules.
### fc.allow(options)
#### Allow client insert and remove, and HTTP data accesses and updates, subject to your limitations. - Server only
`fc.allow(options)` is essentially the same as [Meteor's `Collection.allow()`](http://docs.meteor.com/#allow), except that the Meteor Collection `fetch` and `transform` options are not supported by `FileCollection`. In addition to returning true/false, rules may also return a (possibly empty) options object to indicate truth while affecting the behavior of the allowed request. See the `maxUploadSize` option on `'write'` allow rules as an example. Note that more than one allow rule may apply to a given request, but unlike deny rules, they are not all guaranteed to run. Allow rules are run in the order in which they are defined, and the first one to return a truthy value wins, which can be significant if they return options or otherwise modify state.
`insert` rules are essentially the same as for ordinary Meteor collections.
`remove` rules also apply to HTTP DELETE requests.
In addition to Meteor's `insert` and `remove` rules, file-collection also uses `read` and `write` rules. These are used to secure access to file data via HTTP GET and POST/PUT requests, respectively.
`read` rules apply only to HTTP GET/HEAD requests retrieving file data, and have the same parameters as all other rules.
`write` rules are analogous to `update` rules on Meteor collections, except that they apply only to HTTP PUT/POST requests modifying file data, and will only (and always) see changes to the `length` and `md5` fields. For that reason the `fieldNames` parameter is omitted. Similarly, because MongoDB updates are not directly involved, no `modifier` parameter is provided to the `write` function. Write rules may optionally return an object with a positive integer `maxUploadSize` attribute instead of `true`. This indicates the maximum allowable upload size for this request. If this max upload size is provided, it will override any value provided for the `maxUploadSize` option on the fileCollection as a whole. Nonpositive values of `maxUploadSize` mean there will be no upload size limit for this request.
The parameters for callback functions for all four types of allow/deny rules are the same:
```js
function (userId, file) {
// userId is Meteor account if authenticated
// file is the gridFS file record for the matching file
}
```
### fc.deny(options)
#### Override allow rules. - Server only
```javascript
fc.deny({
remove: function (userId, file) { return true; } // Nobody can remove, boo!
});
```
`fc.deny(options)` is the same as [Meteor's `Collection.deny()`](http://docs.meteor.com/#deny), except that the Meteor Collection `fetch` and `transform` options are not supported by `FileCollection`. See `fc.allow()` above for more deatils.
### fc.findOneStream(selector, [options], [callback])
#### Find a file collection file and return a readable stream for its data. - Server only
```javascript
// Get a readable data stream for a known lolcat
lolStream = fc.findOneStream({ 'filename': 'lolcat.gif'});
```
`fc.findOneStream()` is like `fc.findOne()` except instead of returning the `files` document for the found file, it returns a [Readable stream](http://nodejs.org/api/stream.html#stream_class_stream_readable) for the found file's data.
`options.range` -- To get partial data from the file, use the `range` option to specify an object with `start` and `end` attributes:
```javascript
stream = fc.findOneStream({ 'filename': 'lolcat.gif'}, { range: { start: 100, end: 200 }})
```
`options.autoRenewLock` -- When true, the read lock on the underlying gridFS file will automatically be renewed before it expires, potentially multiple times. If you need more control over lock expiration behavior in your application, set this option to `false`. Default: `true`
Other available options are `options.sort` and `options.skip` which have the same behavior as they do for Meteor's [`Collection.findOne()`](http://docs.meteor.com/#findone).
The returned stream is a gridfs-locking-stream `readStream`, which has some [special methods and events it emits](https://github.com/vsivsi/gridfs-locking-stream#locking-options). You probably won't need to use these, but the stream will emit `'expires-soon'` and `'expired'` events if its read lock is getting too old, and it has three methods that can be used to control locking:
* `stream.heldLock()` - Returns the gridfs-locks [`Lock` object](https://github.com/vsivsi/gridfs-locks#lock) held by the stream
* `stream.renewLock([callback])` - Renews the held lock for another expiration interval
* `stream.releaseLock([callback])` - Releases the held lock if you are done with the stream.
This last call, `stream.releaseLock()` may be useful if you use `file.findOneStream()` and then do not read the file to the end (which would cause the lock to release automatically). In this case, calling `stream.releaseLock()` is nice because it frees the lock before the expiration time is up. This would probably only matter for applications with lots of writers and readers contending for the same files, but it's good to know it exists. The values used for the locking parameters are set when the `FileCollection` is created via the `options.locks` option.
When the stream has ended, the `callback` is called with the gridFS file document.
### fc.upsertStream(file, [options], [callback])
#### Create/update a file collection file and return a writable stream to its data. - Server only
```javascript
// Get a writeable data stream to re-store all that is right and good
nyanStream = fc.upsertStream({ filename: 'nyancat.flv',
contentType: 'video/x-flv',
metadata: { caption: 'Not again!'}
});
```
`fc.upsertStream()` is a little bit like Meteor's `Collection.upsert()` only really not... If the `file` parameter contains an `_id` field, then the call will work on the file with that `_id`. If a file with that `_id` doesn't exist, or if no `_id` is provided, then a new file is `insert`ed into the file collection. Any application owned gridFS attributes (`filename`, `contentType`, `aliases`, `metadata`) that are present in the `file` parameter will be used for the file.
Once that is done, `fc.upsertStream()` returns a [writable stream](http://nodejs.org/api/stream.html#stream_class_stream_writable) for the file.
`options.autoRenewLock` -- Default: `true`. When true, the write lock on the underlying gridFS file will automatically be renewed before it expires, potentially multiple times. If you need more control over lock expiration behavior in your application, set this option to `false`.
*NOTE! Breaking Change*! Prior to file-collection v1.0, it was possible to specify `options.mode = 'w+'` and append to an existing file. This option is now ignored, and all calls to `fc.upsertStream()` will overwrite any existing data in the file.
The returned stream is a gridfs-locking-stream `writeStream`, which has some [special methods and events it emits](https://github.com/vsivsi/gridfs-locking-stream#locking-options). You probably won't need to use these, but the stream will emit `'expires-soon'` and `'expired'` events if its exclusive write lock is getting too old, and it has three methods that can be used to control locking:
* `stream.heldLock()` - Returns the gridfs-locks [`Lock` object](https://github.com/vsivsi/gridfs-locks#lock) held by the stream
* `stream.renewLock([callback])` - Renews the held lock for another expiration interval
* `stream.releaseLock([callback])` - Releases the held lock if you are done with the stream.
You probably won't need these, but it's good to know they're there. The values used for the locking parameters are set when the `FileCollection` is created via the `options.locks` option.
When the write stream has closed, the `callback` is called as `callback(error, file)`, where file is the gridFS file document following the write.
### fc.exportFile(selector, filePath, callback)
#### Export a file collection file to the local fileSystem. - Server only
```javascript
// Write a file to wherever it belongs in the filesystem
fc.exportFile({ 'filename': 'nyancat.flv'},
'/dev/null',
function(err) {
// Deal with it
});
```
`fc.exportFile()` is a convenience method that [pipes](http://nodejs.org/api/stream.html#stream_readable_pipe_destination_options) the readable stream produced by `fc.findOneStream()` into a local [file system writable stream](http://nodejs.org/api/fs.html#fs_fs_createwritestream_path_options).
The `selector` parameter works as it does with `fc.findOneStream()`. The `filePath` is the String directory path and filename in the local filesystem to write the file data to. The value of the `filename` attribute in the found gridFS file document is ignored. The callback is mandatory and will be called with a single parameter that will be either an `Error` object or `null` depending on the success of the operation.
### fc.importFile(filePath, file, callback)
#### Import a local filesystem file into a file collection file. - Server only
```javascript
// Read a file into the collection from the filesystem
fc.importFile('/funtimes/lolcat_183.gif',
{ filename: 'lolcat_183.gif',
contentType: 'image/gif'
},
function(err, file) {
// Deal with it
// Or file contains all of the details.
});
```
`fc.importFile()` is a convenience method that [pipes](http://nodejs.org/api/stream.html#stream_readable_pipe_destination_options) a local [file system readable stream](http://nodejs.org/api/fs.html#fs_fs_createreadstream_path_options) into the writable stream produced by a call to `fc.upsertStream()`.
The `file` parameter works as it does with `fc.upsertStream()`. The `filePath` is the String directory path and filename in the local filesystem of the file to open and copy into the gridFS file. The callback is mandatory and will be called with the same callback signature as `fc.upsertStream()`.
================================================
FILE: package.js
================================================
/***************************************************************************
### Copyright (C) 2014-2017 by Vaughn Iverson
### fileCollection is free software released under the MIT/X11 license.
### See included LICENSE file for details.
***************************************************************************/
var currentVersion = '1.3.8';
Package.describe({
summary: 'Collections that efficiently store files using MongoDB GridFS, with built-in HTTP support',
name: 'vsivsi:file-collection',
version: currentVersion,
git: 'https://github.com/vsivsi/meteor-file-collection.git'
});
Npm.depends({
// latest mongodb driver is 2.2.x, but early revs, currently seems broken
mongodb: '2.1.21',
'gridfs-locking-stream': '1.1.1',
'gridfs-locks': '1.3.4',
dicer: '0.2.5',
async: '2.1.4',
express: '4.14.1',
'cookie-parser': '1.4.3',
// Version 2.x of through2 is Streams3, so don't go there yet!
through2: '0.6.5'
});
Package.onUse(function(api) {
api.use('coffeescript@1.12.3_1', ['server','client']);
api.use('webapp@1.3.13', 'server');
api.use('mongo@1.1.15', ['server', 'client']);
api.use('minimongo@1.0.20', 'server');
api.use('check@1.2.5', ['server', 'client']);
api.addFiles('resumable/resumable.js', 'client');
api.addFiles('src/gridFS.coffee', ['server','client']);
api.addFiles('src/server_shared.coffee', 'server');
api.addFiles('src/gridFS_server.coffee', 'server');
api.addFiles('src/resumable_server.coffee', 'server');
api.addFiles('src/http_access_server.coffee', 'server');
api.addFiles('src/resumable_client.coffee', 'client');
api.addFiles('src/gridFS_client.coffee', 'client');
api.export('FileCollection');
});
Package.onTest(function (api) {
api.use('vsivsi:file-collection@' + currentVersion, ['server', 'client']);
api.use('coffeescript@1.12.3_1', ['server', 'client']);
api.use('tinytest@1.0.12', ['server', 'client']);
api.use('test-helpers@1.0.11', ['server','client']);
api.use('http@1.2.11', ['server','client']);
api.use('ejson@1.0.13',['server','client']);
api.use('mongo@1.1.15', ['server', 'client']);
api.use('check@1.2.5', ['server', 'client']);
api.use('tracker@1.1.2', 'client');
api.addFiles('test/file_collection_tests.coffee', ['server', 'client']);
});
================================================
FILE: packages/.gitignore
================================================
/fileCollection
================================================
FILE: src/gridFS.coffee
================================================
############################################################################
# Copyright (C) 2014-2017 by Vaughn Iverson
# fileCollection is free software released under the MIT/X11 license.
# See included LICENSE file for details.
############################################################################
share.defaultChunkSize = 2*1024*1024 - 1024
share.defaultRoot = 'fs'
share.resumableBase = '/_resumable'
share.insert_func = (file = {}, chunkSize) ->
try
id = new Mongo.ObjectID("#{file._id}")
catch
id = new Mongo.ObjectID()
subFile = {}
subFile._id = id
subFile.length = 0
subFile.md5 = 'd41d8cd98f00b204e9800998ecf8427e'
subFile.uploadDate = new Date()
subFile.chunkSize = chunkSize
subFile.filename = file.filename ? ''
subFile.metadata = file.metadata ? {}
subFile.aliases = file.aliases ? []
subFile.contentType = file.contentType ? 'application/octet-stream'
return subFile
share.reject_file_modifier = (modifier) ->
forbidden = Match.OneOf(
Match.ObjectIncluding({ _id: Match.Any })
Match.ObjectIncluding({ length: Match.Any })
Match.ObjectIncluding({ chunkSize: Match.Any })
Match.ObjectIncluding({ md5: Match.Any })
Match.ObjectIncluding({ uploadDate: Match.Any })
)
required = Match.OneOf(
Match.ObjectIncluding({ _id: Match.Any })
Match.ObjectIncluding({ length: Match.Any })
Match.ObjectIncluding({ chunkSize: Match.Any })
Match.ObjectIncluding({ md5: Match.Any })
Match.ObjectIncluding({ uploadDate: Match.Any })
Match.ObjectIncluding({ metadata: Match.Any })
Match.ObjectIncluding({ aliases: Match.Any })
Match.ObjectIncluding({ filename: Match.Any })
Match.ObjectIncluding({ contentType: Match.Any })
)
return Match.test modifier, Match.OneOf(
Match.ObjectIncluding({ $set: forbidden })
Match.ObjectIncluding({ $unset: required })
Match.ObjectIncluding({ $inc: forbidden })
Match.ObjectIncluding({ $mul: forbidden })
Match.ObjectIncluding({ $bit: forbidden })
Match.ObjectIncluding({ $min: forbidden })
Match.ObjectIncluding({ $max: forbidden })
Match.ObjectIncluding({ $rename: required })
Match.ObjectIncluding({ $currentDate: forbidden })
Match.Where (pat) -> # This requires that the update isn't a replacement
return not Match.test pat, Match.OneOf(
Match.ObjectIncluding({ $inc: Match.Any })
Match.ObjectIncluding({ $set: Match.Any })
Match.ObjectIncluding({ $unset: Match.Any })
Match.ObjectIncluding({ $addToSet: Match.Any })
Match.ObjectIncluding({ $pop: Match.Any })
Match.ObjectIncluding({ $pullAll: Match.Any })
Match.ObjectIncluding({ $pull: Match.Any })
Match.ObjectIncluding({ $pushAll: Match.Any })
Match.ObjectIncluding({ $push: Match.Any })
Match.ObjectIncluding({ $bit: Match.Any })
)
)
================================================
FILE: src/gridFS_client.coffee
================================================
############################################################################
# Copyright (C) 2014-2017 by Vaughn Iverson
# fileCollection is free software released under the MIT/X11 license.
# See included LICENSE file for details.
############################################################################
if Meteor.isClient
class FileCollection extends Mongo.Collection
constructor: (@root = share.defaultRoot, options = {}) ->
unless @ instanceof FileCollection
return new FileCollection(root, options)
unless @ instanceof Mongo.Collection
throw new Meteor.Error 'The global definition of Mongo.Collection has changed since the file-collection package was loaded. Please ensure that any packages that redefine Mongo.Collection are loaded before file-collection.'
unless Mongo.Collection is Mongo.Collection.prototype.constructor
throw new Meteor.Error 'The global definition of Mongo.Collection has been patched by another package, and the prototype constructor has been left in an inconsistent state. Please see this link for a workaround: https://github.com/vsivsi/meteor-file-sample-app/issues/2#issuecomment-120780592'
if typeof @root is 'object'
options = @root
@root = share.defaultRoot
@base = @root
@baseURL = options.baseURL ? "/gridfs/#{@root}"
@chunkSize = options.chunkSize ? share.defaultChunkSize
super @root + '.files', { idGeneration: 'MONGO' }
# This call sets up the optional support for resumable.js
# See the resumable.coffee file for more information
if options.resumable
share.setup_resumable.bind(@)()
# remove works as-is. No modifications necessary so it currently goes straight to super
# Insert only creates an empty (but valid) gridFS file. To put data into it from a client,
# you need to use an HTTP POST or PUT after the record is inserted. For security reasons,
# you shouldn't be able to POST or PUT to a file that hasn't been inserted.
insert: (file, callback = undefined) ->
# This call ensures that a full gridFS file document
# gets built from whatever is provided
file = share.insert_func file, @chunkSize
super file, callback
# This will only update the local client-side minimongo collection
# You can shadow update with this to enable latency compensation when
# updating the server-side collection using a Meteor method call
localUpdate: (selector, modifier, options = {}, callback = undefined) ->
if not callback? and typeof options is 'function'
callback = options
options = {}
if options.upsert?
err = new Meteor.Error "Update does not support the upsert option"
if callback?
return callback err
else
throw err
if share.reject_file_modifier(modifier)
err = new Meteor.Error "Modifying gridFS read-only document elements is a very bad idea!"
if callback?
return callback err
else
throw err
else
@find().collection.update selector, modifier, options, callback
allow: () ->
throw new Meteor.Error "File Collection Allow rules may not be set in client code."
deny: () ->
throw new Meteor.Error "File Collection Deny rules may not be set in client code."
upsert: () ->
throw new Meteor.Error "File Collections do not support 'upsert'"
update: () ->
throw new Meteor.Error "File Collections do not support 'update' on client, use method calls instead"
findOneStream: () ->
throw new Meteor.Error "File Collections do not support 'findOneStream' in client code."
upsertStream: () ->
throw new Meteor.Error "File Collections do not support 'upsertStream' in client code."
importFile: () ->
throw new Meteor.Error "File Collections do not support 'importFile' in client code."
exportFile: () ->
throw new Meteor.Error "File Collections do not support 'exportFile' in client code."
================================================
FILE: src/gridFS_server.coffee
================================================
############################################################################
# Copyright (C) 2014-2017 by Vaughn Iverson
# fileCollection is free software released under the MIT/X11 license.
# See included LICENSE file for details.
############################################################################
if Meteor.isServer
mongodb = Npm.require 'mongodb'
grid = Npm.require 'gridfs-locking-stream'
gridLocks = Npm.require 'gridfs-locks'
fs = Npm.require 'fs'
path = Npm.require 'path'
dicer = Npm.require 'dicer'
express = Npm.require 'express'
class FileCollection extends Mongo.Collection
constructor: (@root = share.defaultRoot, options = {}) ->
unless @ instanceof FileCollection
return new FileCollection(@root, options)
unless @ instanceof Mongo.Collection
throw new Meteor.Error 'The global definition of Mongo.Collection has changed since the file-collection package was loaded. Please ensure that any packages that redefine Mongo.Collection are loaded before file-collection.'
unless Mongo.Collection is Mongo.Collection.prototype.constructor
throw new Meteor.Error 'The global definition of Mongo.Collection has been patched by another package, and the prototype constructor has been left in an inconsistent state. Please see this link for a workaround: https://github.com/vsivsi/meteor-file-sample-app/issues/2#issuecomment-120780592'
if typeof @root is 'object'
options = @root
@root = share.defaultRoot
@chunkSize = options.chunkSize ? share.defaultChunkSize
@db = Meteor.wrapAsync(mongodb.MongoClient.connect)(process.env.MONGO_URL,{})
@lockOptions =
timeOut: options.locks?.timeOut ? 360
lockExpiration: options.locks?.lockExpiration ? 90
pollingInterval: options.locks?.pollingInterval ? 5
@locks = gridLocks.LockCollection @db,
root: @root
timeOut: @lockOptions.timeOut
lockExpiration: @lockOptions.lockExpiration
pollingInterval: @lockOptions.pollingInterval
@gfs = new grid(@db, mongodb, @root)
@baseURL = options.baseURL ? "/gridfs/#{@root}"
# if there are HTTP options, setup the express HTTP access point(s)
if options.resumable or options.http
share.setupHttpAccess.bind(@)(options)
# Default client allow/deny permissions
@allows = { read: [], insert: [], write: [], remove: [] }
@denys = { read: [], insert: [], write: [], remove: [] }
# Call super's constructor
super @root + '.files', { idGeneration: 'MONGO' }
# Default indexes
if options.resumable
indexOptions = {}
if typeof options.resumableIndexName is 'string'
indexOptions.name = options.resumableIndexName
@db.collection("#{@root}.files").ensureIndex({
'metadata._Resumable.resumableIdentifier': 1
'metadata._Resumable.resumableChunkNumber': 1
length: 1
}, indexOptions)
@maxUploadSize = options.maxUploadSize ? -1 # Negative is no limit...
## Delay this feature until demand is clear. Unit tests / documentation needed.
# unless options.additionalHTTPHeaders? and (typeof options.additionalHTTPHeaders is 'object')
# options.additionalHTTPHeaders = {}
#
# for h, v of options.additionalHTTPHeaders
# share.defaultResponseHeaders[h] = v
# Setup specific allow/deny rules for gridFS, and tie-in the application settings
FileCollection.__super__.allow.bind(@)
# Because allow rules are not guaranteed to run,
# all checking is done in the deny rules below
insert: (userId, file) => true
remove: (userId, file) => true
FileCollection.__super__.deny.bind(@)
insert: (userId, file) =>
# Make darn sure we're creating a valid gridFS .files document
check file,
_id: Mongo.ObjectID
length: Match.Where (x) =>
check x, Match.Integer
x is 0
md5: Match.Where (x) =>
check x, String
x is 'd41d8cd98f00b204e9800998ecf8427e' # The md5 of an empty file
uploadDate: Date
chunkSize: Match.Where (x) =>
check x, Match.Integer
x is @chunkSize
filename: String
contentType: String
aliases: [ String ]
metadata: Object
# Enforce a uniform chunkSize
unless file.chunkSize is @chunkSize
console.warn "Invalid chunksize"
return true
# call application rules
if share.check_allow_deny.bind(@) 'insert', userId, file
return false
return true
update: (userId, file, fields) =>
## Cowboy updates are not currently allowed from the client. Too much to screw up.
## For example, if you store file ownership info in a sub document under 'metadata'
## it will be complicated to guard against that being changed if you allow other parts
## of the metadata sub doc to be updated. Write specific Meteor methods instead to
## allow reasonable changes to the "metadata" parts of the gridFS file record.
return true
remove: (userId, file) =>
## Remove is now handled via the default method override below, so this should
## never be called.
return true
self = @ # Necessary in the method definition below
## Remove method override for this server-side collection
Meteor.server.method_handlers["#{@_prefix}remove"] = (selector) ->
check selector, Object
unless LocalCollection._selectorIsIdPerhapsAsObject(selector)
throw new Meteor.Error 403, "Not permitted. Untrusted code may only remove documents by ID."
cursor = self.find selector
if cursor.count() > 1
throw new Meteor.Error 500, "Remote remove selector targets multiple files.\nSee https://github.com/vsivsi/meteor-file-collection/issues/152#issuecomment-278824127"
[file] = cursor.fetch()
if file
if share.check_allow_deny.bind(self) 'remove', this.userId, file
return self.remove file
else
throw new Meteor.Error 403, "Access denied"
else
return 0
# Register application allow rules
allow: (allowOptions) ->
for type, func of allowOptions
unless type of @allows
throw new Meteor.Error "Unrecognized allow rule type '#{type}'."
unless typeof func is 'function'
throw new Meteor.Error "Allow rule #{type} must be a valid function."
@allows[type].push(func)
# Register application deny rules
deny: (denyOptions) ->
for type, func of denyOptions
unless type of @denys
throw new Meteor.Error "Unrecognized deny rule type '#{type}'."
unless typeof func is 'function'
throw new Meteor.Error "Deny rule #{type} must be a valid function."
@denys[type].push(func)
insert: (file = {}, callback = undefined) ->
file = share.insert_func file, @chunkSize
super file, callback
# Update is dangerous! The checks inside attempt to keep you out of
# trouble with gridFS. Clients can't update at all. Be careful!
# Only metadata, filename, aliases and contentType should ever be changed
# directly by a server.
update: (selector, modifier, options = {}, callback = undefined) ->
if not callback? and typeof options is 'function'
callback = options
options = {}
if options.upsert?
err = new Meteor.Error "Update does not support the upsert option"
if callback?
return callback err
else
throw err
if share.reject_file_modifier(modifier) and not options.force
err = new Meteor.Error "Modifying gridFS read-only document elements is a very bad idea!"
if callback?
return callback err
else
throw err
else
super selector, modifier, options, callback
upsert: (selector, modifier, options = {}, callback = undefined) ->
if not callback? and typeof options is 'function'
callback = options
err = new Meteor.Error "File Collections do not support 'upsert'"
if callback?
callback err
else
throw err
upsertStream: (file, options = {}, callback = undefined) ->
if not callback? and typeof options is 'function'
callback = options
options = {}
callback = share.bind_env callback
cbCalled = false
mods = {}
mods.filename = file.filename if file.filename?
mods.aliases = file.aliases if file.aliases?
mods.contentType = file.contentType if file.contentType?
mods.metadata = file.metadata if file.metadata?
options.autoRenewLock ?= true
if options.mode is 'w+'
throw new Meteor.Error "The ability to append file data in upsertStream() was removed in version 1.0.0"
# Make sure that we have an ID and it's valid
if file._id
found = @findOne {_id: file._id}
unless file._id and found
file._id = @insert mods
else if Object.keys(mods).length > 0
@update { _id: file._id }, { $set: mods }
writeStream = Meteor.wrapAsync(@gfs.createWriteStream.bind(@gfs))
root: @root
_id: mongodb.ObjectID("#{file._id}")
mode: 'w'
timeOut: @lockOptions.timeOut
lockExpiration: @lockOptions.lockExpiration
pollingInterval: @lockOptions.pollingInterval
if writeStream
if options.autoRenewLock
writeStream.on 'expires-soon', () =>
writeStream.renewLock (e, d) ->
if e or not d
console.warn "Automatic Write Lock Renewal Failed: #{file._id}", e
if callback?
writeStream.on 'close', (retFile) ->
if retFile
retFile._id = new Mongo.ObjectID retFile._id.toHexString()
callback(null, retFile)
writeStream.on 'error', (err) ->
callback(err)
return writeStream
return null
findOneStream: (selector, options = {}, callback = undefined) ->
if not callback? and typeof options is 'function'
callback = options
options = {}
callback = share.bind_env callback
opts = {}
opts.sort = options.sort if options.sort?
opts.skip = options.skip if options.skip?
file = @findOne selector, opts
if file
options.autoRenewLock ?= true
# Init the start and end range, default to full file or start/end as specified
range =
start: options.range?.start ? 0
end: options.range?.end ? file.length - 1
readStream = Meteor.wrapAsync(@gfs.createReadStream.bind(@gfs))
root: @root
_id: mongodb.ObjectID("#{file._id}")
timeOut: @lockOptions.timeOut
lockExpiration: @lockOptions.lockExpiration
pollingInterval: @lockOptions.pollingInterval
range:
startPos: range.start
endPos: range.end
if readStream
if options.autoRenewLock
readStream.on 'expires-soon', () =>
readStream.renewLock (e, d) ->
if e or not d
console.warn "Automatic Read Lock Renewal Failed: #{file._id}", e
if callback?
readStream.on 'close', () ->
callback(null, file)
readStream.on 'error', (err) ->
callback(err)
return readStream
return null
remove: (selector, callback = undefined) ->
callback = share.bind_env callback
if selector?
ret = 0
@find(selector).forEach (file) =>
res = Meteor.wrapAsync(@gfs.remove.bind(@gfs))
_id: mongodb.ObjectID("#{file._id}")
root: @root
timeOut: @lockOptions.timeOut
lockExpiration: @lockOptions.lockExpiration
pollingInterval: @lockOptions.pollingInterval
ret += if res then 1 else 0
callback? and callback null, ret
return ret
else
err = new Meteor.Error "Remove with an empty selector is not supported"
if callback?
callback err
return
else
throw err
importFile: (filePath, file, callback) ->
callback = share.bind_env callback
filePath = path.normalize filePath
file ?= {}
file.filename ?= path.basename filePath
readStream = fs.createReadStream filePath
readStream.on('error', share.bind_env(callback))
writeStream = @upsertStream file
readStream.pipe(share.streamChunker(@chunkSize)).pipe(writeStream)
.on('close', share.bind_env((d) -> callback(null, d)))
.on('error', share.bind_env(callback))
exportFile: (selector, filePath, callback) ->
callback = share.bind_env callback
filePath = path.normalize filePath
readStream = @findOneStream selector
writeStream = fs.createWriteStream filePath
readStream.pipe(writeStream)
.on('finish', share.bind_env(callback))
.on('error', share.bind_env(callback))
================================================
FILE: src/http_access_server.coffee
================================================
############################################################################
# Copyright (C) 2014-2017 by Vaughn Iverson
# fileCollection is free software released under the MIT/X11 license.
# See included LICENSE file for details.
############################################################################
if Meteor.isServer
express = Npm.require 'express'
cookieParser = Npm.require 'cookie-parser'
mongodb = Npm.require 'mongodb'
grid = Npm.require 'gridfs-locking-stream'
gridLocks = Npm.require 'gridfs-locks'
dicer = Npm.require 'dicer'
find_mime_boundary = (req) ->
RE_BOUNDARY = /^multipart\/.+?(?:; boundary=(?:(?:"(.+)")|(?:([^\s]+))))$/i
result = RE_BOUNDARY.exec req.headers['content-type']
result?[1] or result?[2]
# Fast MIME Multipart parsing of generic HTTP POST request bodies
dice_multipart = (req, res, next) ->
next = share.bind_env next
unless req.method is 'POST' and not req.diced
next()
return
req.diced = true # Don't reenter for the same request on multiple routes
responseSent = false
handleFailure = (msg, err = "", retCode = 500) ->
console.error "#{msg} \n", err
unless responseSent
responseSent = true
res.writeHead retCode, share.defaultResponseHeaders
res.end()
boundary = find_mime_boundary req
unless boundary
handleFailure "No MIME multipart boundary found for dicer"
return
params = {}
count = 0
fileStream = null
fileType = 'text/plain'
fileName = 'blob'
d = new dicer { boundary: boundary }
d.on 'part', (p) ->
p.on 'header', (header) ->
RE_FILE = /^form-data; name="file"; filename="([^"]+)"/
RE_PARAM = /^form-data; name="([^"]+)"/
for k, v of header
if k is 'content-type'
fileType = v
if k is 'content-disposition'
if re = RE_FILE.exec(v)
fileStream = p
fileName = re[1]
else if param = RE_PARAM.exec(v)?[1]
data = ''
count++
p.on 'data', (d) ->
data += d.toString()
p.on 'end', () ->
count--
params[param] = data
if count is 0 and fileStream
req.multipart =
fileStream: fileStream
fileName: fileName
fileType: fileType
params: params
responseSent = true
next()
else
console.warn "Dicer part", v
if count is 0 and fileStream
req.multipart =
fileStream: fileStream
fileName: fileName
fileType: fileType
params: params
responseSent = true
next()
p.on 'error', (err) ->
handleFailure 'Error in Dicer while parsing multipart:', err
d.on 'error', (err) ->
handleFailure 'Error in Dicer while parsing parts:', err
d.on 'finish', () ->
unless fileStream
handleFailure "Error in Dicer, no file found in POST"
req.pipe(d)
# Handle a generic HTTP POST file upload
# This curl command should be properly handled by this code:
# % curl -X POST 'http://127.0.0.1:3000/gridfs/fs/38a14c8fef2d6cef53c70792' \
# -F 'file=@"universe.png";type=image/png' -H 'X-Auth-Token: zrtrotHrDzwA4nC5'
post = (req, res, next) ->
# Handle filename or filetype data when included
req.gridFS.contentType = req.multipart.fileType if req.multipart.fileType
req.gridFS.filename = req.multipart.fileName if req.multipart.fileName
# Write the file data. No chunks here, this is the whole thing
stream = @upsertStream req.gridFS
if stream
req.multipart.fileStream.pipe(share.streamChunker(@chunkSize)).pipe(stream)
.on 'close', (retFile) ->
if retFile
res.writeHead(200, share.defaultResponseHeaders)
res.end()
.on 'error', (err) ->
res.writeHead(500, share.defaultResponseHeaders)
res.end()
else
res.writeHead(410, share.defaultResponseHeaders)
res.end()
# Handle a generic HTTP GET request
# This also handles HEAD requests
# If the request URL has a "?download=true" query, then a browser download response is triggered
get = (req, res, next) ->
headers = {}
for h, v of share.defaultResponseHeaders
headers[h] = v
## If If-Modified-Since header present, and parses to a date, then we
## return 304 (Not Modified Since) if the modification date is less than
## the specified date, or they both format to the same UTC string
## (which can deal with some sub-second rounding caused by formatting).
if req.headers['if-modified-since']
since = Date.parse req.headers['if-modified-since'] ## NaN if invaild
if since and req.gridFS.uploadDate and (req.headers['if-modified-since'] == req.gridFS.uploadDate.toUTCString() or since >= req.gridFS.uploadDate.getTime())
res.writeHead 304, headers
res.end()
return
# If range request in the header
if req.headers['range']
# Set status code to partial data
statusCode = 206
# Pick out the range required by the browser
parts = req.headers["range"].replace(/bytes=/, "").split("-")
start = parseInt(parts[0], 10)
end = (if parts[1] then parseInt(parts[1], 10) else req.gridFS.length - 1)
# Unable to handle range request - Send the valid range with status code 416
if (start < 0) or (end >= req.gridFS.length) or (start > end) or isNaN(start) or isNaN(end)
headers['Content-Range'] = 'bytes ' + '*/' + req.gridFS.length
res.writeHead 416, headers
res.end()
return
# Determine the chunk size
chunksize = (end - start) + 1
# Construct the range request header
headers['Content-Range'] = 'bytes ' + start + '-' + end + '/' + req.gridFS.length
headers['Accept-Ranges'] = 'bytes'
headers['Content-Type'] = req.gridFS.contentType
headers['Content-Length'] = chunksize
headers['Last-Modified'] = req.gridFS.uploadDate.toUTCString()
# Read the partial request from gridfs stream
unless req.method is 'HEAD'
stream = @findOneStream(
_id: req.gridFS._id
,
range:
start: start
end: end
)
# Otherwise prepare to stream the whole file
else
# Set default status code
statusCode = 200
# Set default headers
headers['Content-Type'] = req.gridFS.contentType
headers['Content-MD5'] = req.gridFS.md5
headers['Content-Length'] = req.gridFS.length
headers['Last-Modified'] = req.gridFS.uploadDate.toUTCString()
# Open file to stream
unless req.method is 'HEAD'
stream = @findOneStream { _id: req.gridFS._id }
# Trigger download in browser, optionally specify filename.
if (req.query.download and req.query.download.toLowerCase() == 'true') or req.query.filename
filename = encodeURIComponent(req.query.filename ? req.gridFS.filename)
headers['Content-Disposition'] = "attachment; filename=\"#{filename}\"; filename*=UTF-8''#{filename}"
# If specified in url query set cache to specified value, might want to add more options later.
if req.query.cache and not isNaN(parseInt(req.query.cache))
headers['Cache-Control'] = "max-age=" + parseInt(req.query.cache)+", private"
# HEADs don't have a body
if req.method is 'HEAD'
res.writeHead 204, headers
res.end()
return
# Stream file
if stream
res.writeHead statusCode, headers
stream.pipe(res)
.on 'close', () ->
res.end()
.on 'error', (err) ->
res.writeHead(500, share.defaultResponseHeaders)
res.end(err)
else
res.writeHead(410, share.defaultResponseHeaders)
res.end()
# Handle a generic HTTP PUT request
# This curl command should be properly handled by this code:
# % curl -X PUT 'http://127.0.0.1:3000/gridfs/fs/7868f3df8425ae68a572b334' \
# -T "universe.png" -H 'Content-Type: image/png' -H 'X-Auth-Token: tEPAwXbGwgfGiJL35'
put = (req, res, next) ->
# Handle content type if it's present
if req.headers['content-type']
req.gridFS.contentType = req.headers['content-type']
# Write the file
stream = @upsertStream req.gridFS
if stream
req.pipe(share.streamChunker(@chunkSize)).pipe(stream)
.on 'close', (retFile) ->
if retFile
res.writeHead(200, share.defaultResponseHeaders)
res.end()
else
.on 'error', (err) ->
res.writeHead(500, share.defaultResponseHeaders)
res.end(err)
else
res.writeHead(404, share.defaultResponseHeaders)
res.end("#{req.url} Not found!")
# Handle a generic HTTP DELETE request
# This curl command should be properly handled by this code:
# % curl -X DELETE 'http://127.0.0.1:3000/gridfs/fs/7868f3df8425ae68a572b334' \
# -H 'X-Auth-Token: tEPAwXbGwgfGiJL35'
del = (req, res, next) ->
@remove req.gridFS
res.writeHead(204, share.defaultResponseHeaders)
res.end()
# Setup all of the application specified paths and file lookups in express
# Also performs allow/deny permission checks for POST/PUT/DELETE
build_access_point = (http) ->
# Loop over the app supplied http paths
for r in http
if r.method.toUpperCase() is 'POST'
@router.post r.path, dice_multipart
# Add an express middleware for each application REST path
@router[r.method] r.path, do (r) =>
(req, res, next) =>
# params and queries literally named "_id" get converted to ObjectIDs automatically
req.params._id = share.safeObjectID(req.params._id) if req.params?._id?
req.query._id = share.safeObjectID(req.query._id) if req.query?._id?
# Build the path lookup mongoDB query object for the gridFS files collection
lookup = r.lookup?.bind(@)(req.params or {}, req.query or {}, req.multipart)
unless lookup?
# No lookup returned, so bailing
res.writeHead(500, share.defaultResponseHeaders)
res.end()
return
else
# Perform the collection query
req.gridFS = @findOne lookup
unless req.gridFS
res.writeHead(404, share.defaultResponseHeaders)
res.end()
return
# Make sure that the requested method is permitted for this file in the allow/deny rules
switch req.method
when 'HEAD', 'GET'
unless share.check_allow_deny.bind(@) 'read', req.meteorUserId, req.gridFS
res.writeHead(403, share.defaultResponseHeaders)
res.end()
return
when 'POST', 'PUT'
req.maxUploadSize = @maxUploadSize
unless opts = share.check_allow_deny.bind(@) 'write', req.meteorUserId, req.gridFS
res.writeHead(403, share.defaultResponseHeaders)
res.end()
return
if opts.maxUploadSize? and typeof opts.maxUploadSize is 'number'
req.maxUploadSize = opts.maxUploadSize
if req.maxUploadSize > 0
unless req.headers['content-length']?
res.writeHead(411, share.defaultResponseHeaders)
res.end()
return
unless parseInt(req.headers['content-length']) <= req.maxUploadSize
res.writeHead(413, share.defaultResponseHeaders)
res.end()
return
when 'DELETE'
unless share.check_allow_deny.bind(@) 'remove', req.meteorUserId, req.gridFS
res.writeHead(403, share.defaultResponseHeaders)
res.end()
return
when 'OPTIONS' # Should there be a permission for options?
unless (share.check_allow_deny.bind(@)('read', req.meteorUserId, req.gridFS) or
share.check_allow_deny.bind(@)('write', req.meteorUserId, req.gridFS) or
share.check_allow_deny.bind(@)('remove', req.meteorUserId, req.gridFS))
res.writeHead(403, share.defaultResponseHeaders)
res.end()
return
else
res.writeHead(500, share.defaultResponseHeaders)
res.end()
return
next()
# Add an express middleware for each custom request handler
if typeof r.handler is 'function'
@router[r.method] r.path, r.handler.bind(@)
# Add all of generic request handling methods to the express route
@router.route('/*')
.all (req, res, next) -> # There needs to be a valid req.gridFS object here
if req.gridFS?
next()
return
else
res.writeHead(404, share.defaultResponseHeaders)
res.end()
.head(get.bind(@)) # Generic HTTP method handlers
.get(get.bind(@))
.put(put.bind(@))
.post(post.bind(@))
.delete(del.bind(@))
.all (req, res, next) -> # Unkown methods are denied
res.writeHead(500, share.defaultResponseHeaders)
res.end()
# Performs a meteor userId lookup by hased access token
lookup_userId_by_token = (authToken) ->
userDoc = Meteor.users?.findOne
'services.resume.loginTokens':
$elemMatch:
hashedToken: Accounts?._hashLoginToken(authToken)
return userDoc?._id or null
# Express middleware to convert a Meteor access token provided in an HTTP request
# to a Meteor userId attached to the request object as req.meteorUserId
handle_auth = (req, res, next) ->
unless req.meteorUserId?
# Lookup userId if token is provided in HTTP heder
if req.headers?['x-auth-token']?
req.meteorUserId = lookup_userId_by_token req.headers['x-auth-token']
# Or as a URL query of the same name
else if req.cookies?['X-Auth-Token']?
req.meteorUserId = lookup_userId_by_token req.cookies['X-Auth-Token']
else
req.meteorUserId = null
next()
# Set up all of the middleware, including optional support for Resumable.js chunked uploads
share.setupHttpAccess = (options) ->
# Set up support for resumable.js if requested
if options.resumable
options.http = [] unless options.http?
resumableHandlers = []
otherHandlers = []
for h in options.http
if h.path is share.resumableBase
resumableHandlers.push h
else
otherHandlers.push h
resumableHandlers = resumableHandlers.concat share.resumablePaths
options.http = resumableHandlers.concat otherHandlers
# Don't setup any middleware unless there are routes defined
if options.http?.length > 0
r = express.Router()
r.use express.query() # Parse URL query strings
r.use cookieParser() # Parse cookies
r.use handle_auth # Turn x-auth-tokens into Meteor userIds
WebApp.rawConnectHandlers.use(@baseURL, share.bind_env(r))
# Setup application HTTP REST interface
@router = express.Router()
build_access_point.bind(@)(options.http, @router)
WebApp.rawConnectHandlers.use(@baseURL, share.bind_env(@router))
================================================
FILE: src/resumable_client.coffee
================================================
############################################################################
# Copyright (C) 2014-2017 by Vaughn Iverson
# fileCollection is free software released under the MIT/X11 license.
# See included LICENSE file for details.
############################################################################
if Meteor.isClient
# This is a polyfill for bind(), added to make phantomjs 1.9.7 work
unless Function.prototype.bind
Function.prototype.bind = (oThis) ->
if typeof this isnt "function"
# closest thing possible to the ECMAScript 5 internal IsCallable function
throw new TypeError("Function.prototype.bind - what is trying to be bound is not callable")
aArgs = Array.prototype.slice.call arguments, 1
fToBind = this
fNOP = () ->
fBound = () ->
func = if (this instanceof fNOP and oThis) then this else oThis
return fToBind.apply(func, aArgs.concat(Array.prototype.slice.call(arguments)))
fNOP.prototype = this.prototype
fBound.prototype = new fNOP()
return fBound
share.setup_resumable = () ->
url = "#{@baseURL}#{share.resumableBase}"
url = Meteor.absoluteUrl(url.replace /^\//, '') if Meteor.isCordova
r = new Resumable
target: url
generateUniqueIdentifier: (file) -> "#{new Mongo.ObjectID()}"
fileParameterName: 'file'
chunkSize: @chunkSize
testChunks: true
testMethod: 'HEAD'
permanentErrors: [204, 404, 415, 500, 501]
simultaneousUploads: 3
maxFiles: undefined
maxFilesErrorCallback: undefined
prioritizeFirstAndLastChunk: false
query: undefined
headers: {}
maxChunkRetries: 5
withCredentials: true
unless r.support
console.warn "resumable.js not supported by this Browser, uploads will be disabled"
@resumable = null
else
@resumable = r
================================================
FILE: src/resumable_server.coffee
================================================
############################################################################
# Copyright (C) 2014-2017 by Vaughn Iverson
# fileCollection is free software released under the MIT/X11 license.
# See included LICENSE file for details.
############################################################################
if Meteor.isServer
express = Npm.require 'express'
mongodb = Npm.require 'mongodb'
grid = Npm.require 'gridfs-locking-stream'
gridLocks = Npm.require 'gridfs-locks'
dicer = Npm.require 'dicer'
async = Npm.require 'async'
# This function checks to see if all of the parts of a Resumable.js uploaded file are now in the gridFS
# Collection. If so, it completes the file by moving all of the chunks to the correct file and cleans up
check_order = (file, callback) ->
fileId = mongodb.ObjectID("#{file.metadata._Resumable.resumableIdentifier}")
lock = gridLocks.Lock(fileId, @locks, {}).obtainWriteLock()
lock.on 'locked', () =>
files = @db.collection "#{@root}.files"
cursor = files.find(
{
'metadata._Resumable.resumableIdentifier': file.metadata._Resumable.resumableIdentifier
length:
$ne: 0
},
{
fields:
length: 1
metadata: 1
sort:
'metadata._Resumable.resumableChunkNumber': 1
}
)
cursor.count (err, count) =>
if err
lock.releaseLock()
return callback err
unless count >= 1
cursor.close()
lock.releaseLock()
return callback()
unless count is file.metadata._Resumable.resumableTotalChunks
cursor.close()
lock.releaseLock()
return callback()
# Manipulate the chunks and files collections directly under write lock
chunks = @db.collection "#{@root}.chunks"
cursor.batchSize file.metadata._Resumable.resumableTotalChunks + 1
cursor.toArray (err, parts) =>
if err
lock.releaseLock()
return callback err
async.eachLimit parts, 5,
(part, cb) =>
if err
console.error "Error from cursor.next()", err
cb err
return cb new Meteor.Error "Received null part" unless part
partId = mongodb.ObjectID("#{part._id}")
partlock = gridLocks.Lock(partId, @locks, {}).obtainWriteLock()
partlock.on 'locked', () =>
async.series [
# Move the chunks to the correct file
(cb) -> chunks.update { files_id: partId, n: 0 },
{ $set: { files_id: fileId, n: part.metadata._Resumable.resumableChunkNumber - 1 }}
cb
# Delete the temporary chunk file documents
(cb) -> files.remove { _id: partId }, cb
],
(err, res) =>
return cb err if err
unless part.metadata._Resumable.resumableChunkNumber is part.metadata._Resumable.resumableTotalChunks
partlock.removeLock()
cb()
else
# check for a final hanging gridfs chunk
chunks.update { files_id: partId, n: 1 },
{ $set: { files_id: fileId, n: part.metadata._Resumable.resumableChunkNumber }}
(err, res) ->
partlock.removeLock()
return cb err if err
cb()
partlock.on 'timed-out', () -> cb new Meteor.Error 'Partlock timed out!'
partlock.on 'expired', () -> cb new Meteor.Error 'Partlock expired!'
partlock.on 'error', (err) ->
console.error "Error obtaining partlock #{part._id}", err
cb err
(err) =>
if err
lock.releaseLock()
return callback err
# Build up the command for the md5 hash calculation
md5Command =
filemd5: fileId
root: "#{@root}"
# Send the command to calculate the md5 hash of the file
@db.command md5Command, (err, results) ->
if err
lock.releaseLock()
return callback err
# Update the size and md5 to the file data
files.update { _id: fileId }, { $set: { length: file.metadata._Resumable.resumableTotalSize, md5: results.md5 }},
(err, res) =>
lock.releaseLock()
callback err
lock.on 'expires-soon', () ->
lock.renewLock().once 'renewed', (ld) ->
unless ld
console.warn "Resumable upload lock renewal failed!"
lock.on 'expired', () -> callback new Meteor.Error "File Lock expired"
lock.on 'timed-out', () -> callback new Meteor.Error "File Lock timed out"
lock.on 'error', (err) -> callback err
# Handle HTTP POST requests from Resumable.js
resumable_post_lookup = (params, query, multipart) ->
return { _id: share.safeObjectID(multipart?.params?.resumableIdentifier) }
resumable_post_handler = (req, res, next) ->
# This has to be a resumable POST
unless req.multipart?.params?.resumableIdentifier
console.error "Missing resumable.js multipart information"
res.writeHead(501, share.defaultResponseHeaders)
res.end()
return
resumable = req.multipart.params
resumable.resumableTotalSize = parseInt resumable.resumableTotalSize
resumable.resumableTotalChunks = parseInt resumable.resumableTotalChunks
resumable.resumableChunkNumber = parseInt resumable.resumableChunkNumber
resumable.resumableChunkSize = parseInt resumable.resumableChunkSize
resumable.resumableCurrentChunkSize = parseInt resumable.resumableCurrentChunkSize
if req.maxUploadSize > 0
unless resumable.resumableTotalSize <= req.maxUploadSize
res.writeHead(413, share.defaultResponseHeaders)
res.end()
return
# Sanity check the chunk sizes that are critical to reassembling the file from parts
unless ((req.gridFS.chunkSize is resumable.resumableChunkSize) and
(resumable.resumableChunkNumber <= resumable.resumableTotalChunks) and
(resumable.resumableTotalSize/resumable.resumableChunkSize <= resumable.resumableTotalChunks+1) and
(resumable.resumableCurrentChunkSize is resumable.resumableChunkSize) or
((resumable.resumableChunkNumber is resumable.resumableTotalChunks) and
(resumable.resumableCurrentChunkSize < 2*resumable.resumableChunkSize)))
res.writeHead(501, share.defaultResponseHeaders)
res.end()
return
chunkQuery =
length: resumable.resumableCurrentChunkSize
'metadata._Resumable.resumableIdentifier': resumable.resumableIdentifier
'metadata._Resumable.resumableChunkNumber': resumable.resumableChunkNumber
# This is to handle duplicate chunk uploads in case of network weirdness
findResult = @findOne chunkQuery, { fields: { _id: 1 }}
if findResult
# Duplicate chunk... Don't rewrite it.
# console.warn "Duplicate chunk detected: #{resumable.resumableChunkNumber}, #{resumable.resumableIdentifier}"
res.writeHead(200, share.defaultResponseHeaders)
res.end()
else
# Everything looks good, so write this part
req.gridFS.metadata._Resumable = resumable
writeStream = @upsertStream
filename: "_Resumable_#{resumable.resumableIdentifier}_#{resumable.resumableChunkNumber}_#{resumable.resumableTotalChunks}"
metadata: req.gridFS.metadata
unless writeStream
res.writeHead(404, share.defaultResponseHeaders)
res.end()
return
req.multipart.fileStream.pipe(share.streamChunker(@chunkSize)).pipe(writeStream)
.on 'close', share.bind_env((retFile) =>
if retFile
# Check to see if all of the parts are now available and can be reassembled
check_order.bind(@)(req.gridFS, (err) ->
if err
console.error "Error reassembling chunks of resumable.js upload", err
res.writeHead(500, share.defaultResponseHeaders)
else
res.writeHead(200, share.defaultResponseHeaders)
res.end()
)
else
console.error "Missing retFile on pipe close"
res.writeHead(500, share.defaultResponseHeaders)
res.end()
)
.on 'error', share.bind_env((err) =>
console.error "Piping Error!", err
res.writeHead(500, share.defaultResponseHeaders)
res.end())
resumable_get_lookup = (params, query) ->
q = { _id: share.safeObjectID(query.resumableIdentifier) }
return q
# This handles Resumable.js "test GET" requests, that exist to determine
# if a part is already uploaded. It also handles HEAD requests, which
# should be a bit more efficient and resumable.js now supports
resumable_get_handler = (req, res, next) ->
query = req.query
chunkQuery =
$or: [
{
_id: share.safeObjectID(query.resumableIdentifier)
length: parseInt query.resumableTotalSize
}
{
length: parseInt query.resumableCurrentChunkSize
'metadata._Resumable.resumableIdentifier': query.resumableIdentifier
'metadata._Resumable.resumableChunkNumber': parseInt query.resumableChunkNumber
}
]
result = @findOne chunkQuery, { fields: { _id: 1 }}
if result
# Chunk is present
res.writeHead(200, share.defaultResponseHeaders)
else
# Chunk is missing
res.writeHead(204, share.defaultResponseHeaders)
res.end()
# Setup the GET and POST HTTP REST paths for Resumable.js in express
share.resumablePaths = [
{
method: 'head'
path: share.resumableBase
lookup: resumable_get_lookup
handler: resumable_get_handler
}
{
method: 'post'
path: share.resumableBase
lookup: resumable_post_lookup
handler: resumable_post_handler
}
{
method: 'get'
path: share.resumableBase
lookup: resumable_get_lookup
handler: resumable_get_handler
}
]
================================================
FILE: src/server_shared.coffee
================================================
############################################################################
# Copyright (C) 2014-2017 by Vaughn Iverson
# fileCollection is free software released under the MIT/X11 license.
# See included LICENSE file for details.
############################################################################
if Meteor.isServer
through2 = Npm.require 'through2'
share.defaultResponseHeaders =
'Content-Type': 'text/plain'
share.check_allow_deny = (type, userId, file, fields) ->
checkRules = (rules) ->
res = false
for func in rules[type] when not res
res = func(userId, file, fields)
return res
result = not checkRules(@denys) and checkRules(@allows)
return result
share.bind_env = (func) ->
if func?
return Meteor.bindEnvironment func, (err) -> throw err
else
return func
share.safeObjectID = (s) ->
if s?.match /^[0-9a-f]{24}$/i # Validate that _id is a 12 byte hex string
new Mongo.ObjectID s
else
null
share.streamChunker = (size = share.defaultChunkSize) ->
makeFuncs = (size) ->
bufferList = [ new Buffer(0) ]
total = 0
flush = (cb) ->
outSize = if total > size then size else total
if outSize > 0
outputBuffer = Buffer.concat bufferList, outSize
this.push outputBuffer
total -= outSize
lastBuffer = bufferList.pop()
newBuffer = lastBuffer.slice(lastBuffer.length - total)
bufferList = [ newBuffer ]
if total < size
cb()
else
flush.bind(this) cb
transform = (chunk, enc, cb) ->
bufferList.push chunk
total += chunk.length
if total < size
cb()
else
flush.bind(this) cb
return [transform, flush]
return through2.apply this, makeFuncs(size)
================================================
FILE: test/file_collection_tests.coffee
================================================
############################################################################
# Copyright (C) 2014-2017 by Vaughn Iverson
# fileCollection is free software released under the MIT/X11 license.
# See included LICENSE file for details.
############################################################################
if Meteor.isServer
os = Npm.require 'os'
path = Npm.require 'path'
bind_env = (func) ->
if typeof func is 'function'
return Meteor.bindEnvironment func, (err) -> throw err
else
return func
subWrapper = (sub, func) ->
(test, onComplete) ->
if Meteor.isClient
Tracker.autorun () ->
if sub.ready()
func test, onComplete
else
func test, onComplete
defaultColl = new FileCollection()
Tinytest.add 'FileCollection default constructor', (test) ->
test.instanceOf defaultColl, FileCollection, "FileCollection constructor failed"
test.equal defaultColl.root, 'fs', "default root isn't 'fs'"
test.equal defaultColl.chunkSize, 2*1024*1024 - 1024, "bad default chunksize"
test.equal defaultColl.baseURL, "/gridfs/fs", "bad default base URL"
idLookup = (params, query) ->
return { _id: params._id }
longString = ''
while longString.length < 4096
longString += '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz'
testColl = new FileCollection "test",
baseURL: "/test"
chunkSize: 16
resumable: true
maxUploadSize: 2048
http: [
{ method: 'get', path: '/:_id', lookup: idLookup}
{ method: 'post', path: '/:_id', lookup: idLookup}
{ method: 'put', path: '/:_id', lookup: idLookup}
{ method: 'delete', path: '/:_id', lookup: idLookup}
{ method: 'options', path: '/:_id', lookup: idLookup, handler: (req, res, next) ->
res.writeHead(200, { 'Content-Type': 'text/plain', 'Access-Control-Allow-Origin': 'http://meteor.local' })
res.end()
return
}
]
noReadColl = new FileCollection "noReadColl",
baseURL: "/noread"
chunkSize: 1024*1024
resumable: false
http: [
{ method: 'get', path: '/:_id', lookup: idLookup}
{ method: 'post', path: '/:_id', lookup: idLookup}
{ method: 'put', path: '/:_id', lookup: idLookup}
{ method: 'delete', path: '/:_id', lookup: idLookup}
]
noAllowColl = new FileCollection "noAllowColl"
denyColl = new FileCollection "denyColl"
Meteor.methods
updateTest: (id, reject) ->
check id, Mongo.ObjectID
check reject, Boolean
if this.isSimulation
testColl.localUpdate { _id: id }, { $set: { 'metadata.test2': true } }
else if reject
throw new Meteor.Error "Rejected by server"
else
testColl.update { _id: id }, { $set: { 'metadata.test2': false } }
if Meteor.isServer
Meteor.publish 'everything', () -> testColl.find {}
Meteor.publish 'noAllowCollPub', () -> noAllowColl.find {}
Meteor.publish 'noReadCollPub', () -> noReadColl.find {}
Meteor.publish 'denyCollPub', () -> denyColl.find {}
sub = null
testColl.allow
insert: () -> true
write: () -> true
remove: () -> true
read: () -> true
testColl.deny
insert: () -> false
write: () -> false
remove: () -> false
read: () -> false
noAllowColl.allow
read: () -> false
insert: () -> false
write: () -> false
remove: () -> false
noReadColl.allow
read: () -> false
insert: () -> true
write: () -> { maxUploadSize: 15 }
remove: () -> true
noReadColl.deny
read: () -> false
insert: () -> false
write: () -> false
remove: () -> false
denyColl.deny
read: () -> true
insert: () -> true
write: () -> true
remove: () -> true
Tinytest.add 'set allow/deny on FileCollection', (test) ->
test.equal testColl.allows.read[0](), true
test.equal testColl.allows.insert[0](), true
test.equal testColl.allows.remove[0](), true
test.equal testColl.allows.write[0](), true
test.equal testColl.denys.read[0](), false
test.equal testColl.denys.write[0](), false
test.equal testColl.denys.insert[0](), false
test.equal testColl.denys.remove[0](), false
Tinytest.add 'check server REST API', (test) ->
test.equal typeof testColl.router, 'function'
if Meteor.isClient
sub = Meteor.subscribe 'everything'
Tinytest.add 'Server-only methods throw on client', (test) ->
test.throws () -> testColl.allow({})
test.throws () -> testColl.deny({})
test.throws () -> testColl.upsert({})
test.throws () -> testColl.update({})
test.throws () -> testColl.findOneStream({})
test.throws () -> testColl.upsertStream({})
test.throws () -> testColl.importFile({})
test.throws () -> testColl.exportFile({})
Tinytest.add 'FileCollection constructor with options', (test) ->
test.instanceOf testColl, FileCollection, "FileCollection constructor failed"
test.equal testColl.root, 'test', "root collection not properly set"
test.equal testColl.chunkSize, 16, "chunkSize not set properly"
test.equal testColl.baseURL, "/test", "base URL not set properly"
Tinytest.add 'FileCollection insert, findOne and remove', (test) ->
_id = testColl.insert {}
test.isNotNull _id, "No _id returned by insert"
test.instanceOf _id, Mongo.ObjectID
file = testColl.findOne {_id : _id}
test.isNotNull file, "Invalid file returned by findOne"
test.equal typeof file, "object"
test.equal file.length, 0
test.equal file.md5, 'd41d8cd98f00b204e9800998ecf8427e'
test.instanceOf file.uploadDate, Date
test.equal file.chunkSize, 16
test.equal file.filename, ''
test.equal typeof file.metadata, "object"
test.instanceOf file.aliases, Array
test.equal file.contentType, 'application/octet-stream'
result = testColl.remove {_id : _id}
test.equal result, 1, "Incorrect number of files removed"
file = testColl.findOne {_id : _id}
test.isUndefined file, "File was not removed"
Tinytest.addAsync 'FileCollection insert, findOne and remove with callback', subWrapper(sub, (test, onComplete) ->
_id = testColl.insert {}, (err, retid) ->
test.fail(err) if err
test.isNotNull _id, "No _id returned by insert"
test.isNotNull retid, "No _id returned by insert callback"
test.instanceOf _id, Mongo.ObjectID, "_id is wrong type"
test.instanceOf retid, Mongo.ObjectID, "retid is wrong type"
test.equal _id, retid, "different ids returned in return and callback"
file = testColl.findOne {_id : retid}
test.isNotNull file, "Invalid file returned by findOne"
test.equal typeof file, "object"
test.equal file.length, 0
test.equal file.md5, 'd41d8cd98f00b204e9800998ecf8427e'
test.instanceOf file.uploadDate, Date
test.equal file.chunkSize, 16
test.equal file.filename, ''
test.equal typeof file.metadata, "object"
test.instanceOf file.aliases, Array
test.equal file.contentType, 'application/octet-stream'
finishCount = 0
finish = () ->
finishCount++
if finishCount > 1
onComplete()
obs = testColl.find({_id : _id}).observeChanges
removed: (id) ->
obs.stop()
test.ok EJSON.equals(id, _id), 'Incorrect file _id removed'
finish()
testColl.remove {_id : retid}, (err, result) ->
test.fail(err) if err
test.equal result, 1, "Incorrect number of files removed"
finish()
)
Tinytest.add 'FileCollection insert and find with options', (test) ->
_id = testColl.insert { filename: 'testfile', metadata: { x: 1 }, aliases: ["foo"], contentType: 'text/plain' }
test.isNotNull _id, "No _id returned by insert"
test.instanceOf _id, Meteor.Collection.ObjectID
file = testColl.findOne {_id : _id}
test.isNotNull file, "Invalid file returned by findOne"
test.equal typeof file, "object"
test.equal file.length, 0
test.equal file.md5, 'd41d8cd98f00b204e9800998ecf8427e'
test.instanceOf file.uploadDate, Date
test.equal file.chunkSize, 16
test.equal file.filename, 'testfile'
test.equal typeof file.metadata, "object"
test.equal file.metadata.x, 1
test.instanceOf file.aliases, Array
test.equal file.aliases[0], 'foo'
test.equal file.contentType, 'text/plain'
Tinytest.addAsync 'FileCollection insert and find with options in callback', subWrapper(sub, (test, onComplete) ->
_id = testColl.insert { filename: 'testfile', metadata: { x: 1 }, aliases: ["foo"], contentType: 'text/plain' }, (err, retid) ->
test.fail(err) if err
test.isNotNull _id, "No _id returned by insert"
test.isNotNull retid, "No _id returned by insert callback"
test.instanceOf _id, Mongo.ObjectID, "_id is wrong type"
test.instanceOf retid, Mongo.ObjectID, "retid is wrong type"
test.equal _id, retid, "different ids returned in return and callback"
file = testColl.findOne {_id : retid}
test.isNotNull file, "Invalid file returned by findOne"
test.equal typeof file, "object"
test.equal file.length, 0
test.equal file.md5, 'd41d8cd98f00b204e9800998ecf8427e'
test.instanceOf file.uploadDate, Date
test.equal file.chunkSize, 16
test.equal file.filename, 'testfile'
test.equal typeof file.metadata, "object"
test.equal file.metadata.x, 1
test.instanceOf file.aliases, Array
test.equal file.aliases[0], 'foo'
test.equal file.contentType, 'text/plain'
onComplete()
)
if Meteor.isServer
Tinytest.addAsync 'Proper error handling for missing file on import',
(test, onComplete) ->
bogusfile = "/bogus/file.not"
testColl.importFile bogusfile, {}, bind_env (err, doc) ->
test.fail(err) unless err
onComplete()
Tinytest.addAsync 'Server accepts good and rejects bad updates', (test, onComplete) ->
_id = testColl.insert()
testColl.update _id, { $set: { "metadata.test": 1 } }, (err, res) ->
test.fail(err) if err
test.equal res, 1
testColl.update _id, { $inc: { "metadata.test": 1 } }, (err, res) ->
test.fail(err) if err
test.equal res, 1
doc = testColl.findOne _id
test.equal doc.metadata.test, 2
testColl.update _id, { $set: { md5: 1 } }, (err, res) ->
test.isUndefined res
test.instanceOf err, Meteor.Error
testColl.update _id, { $unset: { filename: 1 } }, (err, res) ->
test.isUndefined res
test.instanceOf err, Meteor.Error
testColl.update _id, { foo: "bar" }, (err, res) ->
test.isUndefined res
test.instanceOf err, Meteor.Error
onComplete()
Tinytest.addAsync 'Insert and then Upsert stream to gridfs and read back, write to file system, and re-import',
(test, onComplete) ->
_id = testColl.insert { filename: 'writefile', contentType: 'text/plain' }, (err, _id) ->
test.fail(err) if err
writestream = testColl.upsertStream { _id: _id }
writestream.on 'close', bind_env (file) ->
test.equal typeof file, 'object', "Bad file object after upsert stream"
test.equal file.length, 10, "Improper file length"
test.equal file.md5, 'e807f1fcf82d132f9bb018ca6738a19f', "Improper file md5 hash"
test.equal file.contentType, 'text/plain', "Improper contentType"
test.equal typeof file._id, 'object'
test.equal _id, new Mongo.ObjectID file._id.toHexString()
readstream = testColl.findOneStream {_id: file._id }
readstream.on 'data', bind_env (chunk) ->
test.equal chunk.toString(), '1234567890','Incorrect data read back from stream'
readstream.on 'end', bind_env () ->
testfile = path.join os.tmpdir(), "/FileCollection." + file._id + ".test"
testColl.exportFile file, testfile, bind_env (err, doc) ->
test.fail(err) if err
testColl.importFile testfile, {}, bind_env (err, doc) ->
test.fail(err) if err
test.equal typeof doc, 'object', "No document imported"
if typeof doc == 'object'
readstream = testColl.findOneStream {_id: doc._id }
readstream.on 'data', bind_env (chunk) ->
test.equal chunk.toString(), '1234567890','Incorrect data read back from file stream'
readstream.on 'end', bind_env () ->
onComplete()
else
onComplete()
writestream.write '1234567890'
writestream.end()
Tinytest.addAsync 'Just Upsert stream to gridfs and read back, write to file system, and re-import',
(test, onComplete) ->
writestream = testColl.upsertStream { filename: 'writefile', contentType: 'text/plain' }, bind_env (err, file) ->
test.equal typeof file, 'object', "Bad file object after upsert stream"
test.equal file.length, 10, "Improper file length"
test.equal file.md5, 'e46c309de99c3dfbd6acd9e77751ae98', "Improper file md5 hash"
test.equal file.contentType, 'text/plain', "Improper contentType"
test.equal typeof file._id, 'object'
test.instanceOf file._id, Mongo.ObjectID, "_id is wrong type"
readstream = testColl.findOneStream {_id: file._id }
readstream.on 'data', bind_env (chunk) ->
test.equal chunk.toString(), 'ZYXWVUTSRQ','Incorrect data read back from stream'
readstream.on 'end', bind_env () ->
testfile = path.join os.tmpdir(), "/FileCollection." + file._id + ".test"
testColl.exportFile file, testfile, bind_env (err, doc) ->
test.fail(err) if err
testColl.importFile testfile, {}, bind_env (err, doc) ->
test.fail(err) if err
test.equal typeof doc, 'object', "No document imported"
if typeof doc == 'object'
readstream = testColl.findOneStream {_id: doc._id }
readstream.on 'data', bind_env (chunk) ->
test.equal chunk.toString(), 'ZYXWVUTSRQ','Incorrect data read back from file stream'
readstream.on 'end', bind_env () ->
onComplete()
else
onComplete()
writestream.write 'ZYXWVUTSRQ'
writestream.end()
Tinytest.addAsync 'REST API PUT/GET', (test, onComplete) ->
_id = testColl.insert { filename: 'writefile', contentType: 'text/plain' }, (err, _id) ->
test.fail(err) if err
url = Meteor.absoluteUrl 'test/' + _id
HTTP.put url, { content: '0987654321'}, (err, res) ->
test.fail(err) if err
HTTP.call "OPTIONS", url, (err, res) ->
test.fail(err) if err
test.equal res.headers?['access-control-allow-origin'], 'http://meteor.local'
HTTP.get url, (err, res) ->
test.fail(err) if err
test.equal res.content, '0987654321'
onComplete()
Tinytest.addAsync 'REST API GET null id', (test, onComplete) ->
_id = testColl.insert { filename: 'writefile', contentType: 'text/plain' }, (err, _id) ->
test.fail(err) if err
url = Meteor.absoluteUrl 'test/'
HTTP.get url, (err, res) ->
test.isNotNull err
if err.response? # Not sure why, but under phantomjs the error object is different
test.equal err.response.statusCode, 404
else
console.warn "PhantomJS skipped statusCode check"
onComplete()
Tinytest.addAsync 'maxUploadSize enforced by when HTTP PUT upload is too large', (test, onComplete) ->
_id = testColl.insert { filename: 'writefile', contentType: 'text/plain' }, (err, _id) ->
test.fail(err) if err
url = Meteor.absoluteUrl 'test/' + _id
HTTP.put url, { content: longString }, (err, res) ->
test.isNotNull err
if err.response? # Not sure why, but under phantomjs the error object is different
test.equal err.response.statusCode, 413
else
console.warn "PhantomJS skipped statusCode check"
onComplete()
Tinytest.addAsync 'maxUploadSize enforced by when HTTP POST upload is too large', (test, onComplete) ->
_id = testColl.insert { filename: 'writefile', contentType: 'text/plain' }, (err, _id) ->
test.fail(err) if err
url = Meteor.absoluteUrl 'test/' + _id
content = """
--AaB03x\r
Content-Disposition: form-data; name="blahBlahBlah"\r
Content-Type: text/plain\r
\r
BLAH\r
--AaB03x\r
Content-Disposition: form-data; name="file"; filename="foobar"\r
Content-Type: text/plain\r
\r
#{longString}\r
--AaB03x--\r
"""
HTTP.post url, { headers: { 'Content-Type': 'multipart/form-data; boundary="AaB03x"'}, content: content }, (err, res) ->
test.isNotNull err
if err.response? # Not sure why, but under phantomjs the error object is different
test.equal err.response.statusCode, 413
else
console.warn "PhantomJS skipped statusCode check"
onComplete()
Tinytest.addAsync 'If-Modified-Since header support', (test, onComplete) ->
_id = testColl.insert { filename: 'writefile', contentType: 'text/plain' }, (err, _id) ->
test.fail(err) if err
url = Meteor.absoluteUrl 'test/' + _id
HTTP.get url, (err, res) ->
test.fail(err) if err
test.equal res.statusCode, 200, 'Failed without If-Modified-Since header'
modified = res.headers['last-modified']
test.equal typeof modified, 'string', 'Invalid Last-Modified response'
HTTP.get url, {headers: {'If-Modified-Since': modified}}, (err, res) ->
test.fail(err) if err
test.equal res.statusCode, 304, 'Returned file despite present If-Modified-Since'
HTTP.get url, {headers: {'If-Modified-Since': 'hello'}}, (err, res) ->
test.fail(err) if err
test.equal res.statusCode, 200, 'Skipped file despite unparsable If-Modified-Since'
modified = new Date(Date.parse(modified) - 2000).toUTCString() ## past test
HTTP.get url, {headers: {'If-Modified-Since': modified}}, (err, res) ->
test.fail(err) if err
test.equal res.statusCode, 200, 'Skipped file despite past If-Modified-Since'
modified = new Date(Date.parse(modified) + 4000).toUTCString() ## future test
HTTP.get url, {headers: {'If-Modified-Since': modified}}, (err, res) ->
test.fail(err) if err
test.equal res.statusCode, 304, 'Returned file despite future If-Modified-Since'
onComplete()
Tinytest.addAsync 'REST API POST/GET/DELETE', (test, onComplete) ->
_id = testColl.insert { filename: 'writefile', contentType: 'text/plain' }, (err, _id) ->
test.fail(err) if err
url = Meteor.absoluteUrl 'test/' + _id
content = """
--AaB03x\r
Content-Disposition: form-data; name="blahBlahBlah"\r
Content-Type: text/plain\r
\r
BLAH\r
--AaB03x\r
Content-Disposition: form-data; name="file"; filename="foobar"\r
Content-Type: text/plain\r
\r
ABCDEFGHIJ\r
--AaB03x--\r
"""
HTTP.post url, { headers: { 'Content-Type': 'multipart/form-data; boundary="AaB03x"'}, content: content},
(err, res) ->
test.fail(err) if err
HTTP.get url, (err, res) ->
test.fail(err) if err
test.equal res.content,'ABCDEFGHIJ'
HTTP.del url, (err, res) ->
test.fail(err) if err
onComplete()
createContent = (_id, data, name, chunkNum, chunkSize = 16) ->
totalChunks = Math.floor(data.length / chunkSize)
totalChunks += 1 unless totalChunks*chunkSize is data.length
throw new Error "Bad chunkNum" if chunkNum > totalChunks
begin = (chunkNum - 1) * chunkSize
end = if chunkNum is totalChunks then data.length else chunkNum * chunkSize
"""
--AaB03x\r
Content-Disposition: form-data; name="resumableChunkNumber"\r
Content-Type: text/plain\r
\r
#{chunkNum}\r
--AaB03x\r
Content-Disposition: form-data; name="resumableChunkSize"\r
Content-Type: text/plain\r
\r
#{chunkSize}\r
--AaB03x\r
Content-Disposition: form-data; name="resumableCurrentChunkSize"\r
Content-Type: text/plain\r
\r
#{end - begin}\r
--AaB03x\r
Content-Disposition: form-data; name="resumableTotalSize"\r
Content-Type: text/plain\r
\r
#{data.length}\r
--AaB03x\r
Content-Disposition: form-data; name="resumableType"\r
Content-Type: text/plain\r
\r
text/plain\r
--AaB03x\r
Content-Disposition: form-data; name="resumableIdentifier"\r
Content-Type: text/plain\r
\r
#{_id}\r
--AaB03x\r
Content-Disposition: form-data; name="resumableFilename"\r
Content-Type: text/plain\r
\r
#{name}\r
--AaB03x\r
Content-Disposition: form-data; name="resumableRelativePath"\r
Content-Type: text/plain\r
\r
#{name}\r
--AaB03x\r
Content-Disposition: form-data; name="resumableTotalChunks"\r
Content-Type: text/plain\r
\r
#{totalChunks}\r
--AaB03x\r
Content-Disposition: form-data; name="file"; filename="#{name}"\r
Content-Type: text/plain\r
\r
#{data.substring(begin, end)}\r
--AaB03x--\r
"""
createCheckQuery = (_id, data, name, chunkNum, chunkSize = 16) ->
totalChunks = Math.floor(data.length / chunkSize)
totalChunks += 1 unless totalChunks*chunkSize is data.length
throw new Error "Bad chunkNum" if chunkNum > totalChunks
begin = (chunkNum - 1) * chunkSize
end = if chunkNum is totalChunks then data.length else chunkNum * chunkSize
"?resumableChunkNumber=#{chunkNum}&resumableChunkSize=#{chunkSize}&resumableCurrentChunkSize=#{end-begin}&resumableTotalSize=#{data.length}&resumableType=text/plain&resumableIdentifier=#{_id}&resumableFilename=#{name}&resumableRelativePath=#{name}&resumableTotalChunks=#{totalChunks}"
Tinytest.addAsync 'Basic resumable.js REST interface POST/GET/DELETE', (test, onComplete) ->
testColl.insert { filename: 'writeresumablefile', contentType: 'text/plain' }, (err, _id) ->
test.fail(err) if err
url = Meteor.absoluteUrl "test/_resumable"
data = 'ABCDEFGHIJ'
content = createContent _id, data, "writeresumablefile", 1
HTTP.post url, { headers: { 'Content-Type': 'multipart/form-data; boundary="AaB03x"'}, content: content },
(err, res) ->
test.fail(err) if err
url = Meteor.absoluteUrl 'test/' + _id
HTTP.get url, (err, res) ->
test.fail(err) if err
test.equal res.content, data
HTTP.del url, (err, res) ->
test.fail(err) if err
onComplete()
Tinytest.addAsync 'Basic resumable.js REST interface POST/GET/DELETE, multiple chunks', (test, onComplete) ->
data = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ012345'
testColl.insert { filename: 'writeresumablefile', contentType: 'text/plain' }, (err, _id) ->
test.fail(err) if err
url = Meteor.absoluteUrl "test/_resumable"
content = createContent _id, data, "writeresumablefile", 1
content2 = createContent _id, data, "writeresumablefile", 2
HTTP.post url, { headers: { 'Content-Type': 'multipart/form-data; boundary="AaB03x"'}, content: content },
(err, res) ->
test.fail(err) if err
HTTP.get url + createCheckQuery(_id, data, "writeresumablefile", 2), (err, res) ->
test.fail(err) if err
test.equal res.statusCode, 204
HTTP.call 'head', url + createCheckQuery(_id, data, "writeresumablefile", 1), (err, res) ->
test.fail(err) if err
test.equal res.statusCode, 200
HTTP.post url, { headers: { 'Content-Type': 'multipart/form-data; boundary="AaB03x"'}, content: content2 },
(err, res) ->
test.fail(err) if err
url = Meteor.absoluteUrl 'test/' + _id
HTTP.get url, (err, res) ->
test.fail(err) if err
test.equal res.content, data
HTTP.del url, (err, res) ->
test.fail(err) if err
onComplete()
Tinytest.addAsync 'Basic resumable.js REST interface POST/GET/DELETE, duplicate chunks', (test, onComplete) ->
data = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ012345'
testColl.insert { filename: 'writeresumablefile', contentType: 'text/plain' }, (err, _id) ->
test.fail(err) if err
url = Meteor.absoluteUrl "test/_resumable"
content = createContent _id, data, "writeresumablefile", 1
content2 = createContent _id, data, "writeresumablefile", 2
HTTP.post url, { headers: { 'Content-Type': 'multipart/form-data; boundary="AaB03x"'}, content: content2 },
(err, res) ->
test.fail(err) if err
HTTP.post url, { headers: { 'Content-Type': 'multipart/form-data; boundary="AaB03x"'}, content: content2 },
(err, res) ->
test.fail(err) if err
HTTP.post url, { headers: { 'Content-Type': 'multipart/form-data; boundary="AaB03x"'}, content: content },
(err, res) ->
test.fail(err) if err
url = Meteor.absoluteUrl 'test/' + _id
HTTP.get url, (err, res) ->
test.fail(err) if err
test.equal res.content, data
HTTP.del url, (err, res) ->
test.fail(err) if err
onComplete()
Tinytest.addAsync 'Basic resumable.js REST interface POST/GET/DELETE, duplicate chunks 2', (test, onComplete) ->
data = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ012345'
testColl.insert { filename: 'writeresumablefile', contentType: 'text/plain' }, (err, _id) ->
test.fail(err) if err
url = Meteor.absoluteUrl "test/_resumable"
content = createContent _id, data, "writeresumablefile", 1
content2 = createContent _id, data, "writeresumablefile", 2
HTTP.post url, { headers: { 'Content-Type': 'multipart/form-data; boundary="AaB03x"'}, content: content },
(err, res) ->
test.fail(err) if err
HTTP.post url, { headers: { 'Content-Type': 'multipart/form-data; boundary="AaB03x"'}, content: content2 },
(err, res) ->
test.fail(err) if err
HTTP.post url, { headers: { 'Content-Type': 'multipart/form-data; boundary="AaB03x"'}, content: content },
(err, res) ->
test.fail(err) if err
url = Meteor.absoluteUrl 'test/' + _id
HTTP.get url, (err, res) ->
test.fail(err) if err
test.equal res.content, data
HTTP.del url, (err, res) ->
test.fail(err) if err
onComplete()
Tinytest.addAsync 'Basic resumable.js REST interface POST/GET/DELETE, duplicate chunks 3', (test, onComplete) ->
data = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdef'
testColl.insert { filename: 'writeresumablefile', contentType: 'text/plain' }, (err, _id) ->
test.fail(err) if err
url = Meteor.absoluteUrl "test/_resumable"
content = createContent _id, data, "writeresumablefile", 1
content2 = createContent _id, data, "writeresumablefile", 2
content3 = createContent _id, data, "writeresumablefile", 3
HTTP.post url, { headers: { 'Content-Type': 'multipart/form-data; boundary="AaB03x"'}, content: content2 },
(err, res) ->
test.fail(err) if err
HTTP.post url, { headers: { 'Content-Type': 'multipart/form-data; boundary="AaB03x"'}, content: content },
(err, res) ->
test.fail(err) if err
HTTP.post url, { headers: { 'Content-Type': 'multipart/form-data; boundary="AaB03x"'}, content: content2 },
(err, res) ->
test.fail(err) if err
HTTP.post url, { headers: { 'Content-Type': 'multipart/form-data; boundary="AaB03x"'}, content: content },
(err, res) ->
test.fail(err) if err
HTTP.post url, { headers: { 'Content-Type': 'multipart/form-data; boundary="AaB03x"'}, content: content3 },
(err, res) ->
test.fail(err) if err
url = Meteor.absoluteUrl 'test/' + _id
HTTP.get url, (err, res) ->
test.fail(err) if err
test.equal res.content, data
HTTP.del url, (err, res) ->
test.fail(err) if err
onComplete()
Tinytest.addAsync 'maxUploadSize enforced by when resumable.js upload is too large', (test, onComplete) ->
testColl.insert { filename: 'writeresumablefile', contentType: 'text/plain' }, (err, _id) ->
test.fail(err) if err
url = Meteor.absoluteUrl "test/_resumable"
content = createContent _id, longString, "writeresumablefile", 1
HTTP.post url, { headers: { 'Content-Type': 'multipart/form-data; boundary="AaB03x"'}, content: content },
(err, res) ->
test.isNotNull err
if err.response? # Not sure why, but under phantomjs the error object is different
test.equal err.response.statusCode, 413
else
console.warn "PhantomJS skipped statusCode check"
onComplete()
Tinytest.addAsync 'REST API valid range requests', (test, onComplete) ->
_id = testColl.insert { filename: 'writefile', contentType: 'text/plain' }, (err, _id) ->
test.fail(err) if err
url = Meteor.absoluteUrl 'test/' + _id
HTTP.put url, { content: '0987654321'}, (err, res) ->
test.fail(err) if err
HTTP.get url, { headers: { 'Range': '0-'}},
(err, res) ->
test.fail(err) if err
test.equal res.headers['content-range'], 'bytes 0-9/10'
test.equal res.headers['accept-ranges'], 'bytes'
test.equal res.statusCode, 206
test.equal res.content, '0987654321'
HTTP.get url, { headers: { 'Range': '0-9'}},
(err, res) ->
test.fail(err) if err
test.equal res.headers['content-range'], 'bytes 0-9/10'
test.equal res.headers['accept-ranges'], 'bytes'
test.equal res.statusCode, 206
test.equal res.content, '0987654321'
HTTP.get url, { headers: { 'Range': '5-7'}},
(err, res) ->
test.fail(err) if err
test.equal res.headers['content-range'], 'bytes 5-7/10'
test.equal res.headers['accept-ranges'], 'bytes'
test.equal res.statusCode, 206
test.equal res.content, '543'
onComplete()
Tinytest.addAsync 'REST API invalid range requests', (test, onComplete) ->
_id = testColl.insert { filename: 'writefile', contentType: 'text/plain' }, (err, _id) ->
test.fail(err) if err
url = Meteor.absoluteUrl 'test/' + _id
HTTP.put url, { content: '0987654321'}, (err, res) ->
test.fail(err) if err
HTTP.get url, { headers: { 'Range': '0-10'}}, (err, res) ->
test.isNotNull err
if err.response? # Not sure why, but under phantomjs the error object is different
test.equal err.response.statusCode, 416
else
console.warn "PhantomJS skipped statusCode check"
HTTP.get url, { headers: { 'Range': '5-3'}}, (err, res) ->
test.isNotNull err
if err.response? # Not sure why, but under phantomjs the error object is different
test.equal err.response.statusCode, 416
else
console.warn "PhantomJS skipped statusCode check"
HTTP.get url, { headers: { 'Range': '-1-5'}}, (err, res) ->
test.isNotNull err
if err.response? # Not sure why, but under phantomjs the error object is different
test.equal err.response.statusCode, 416
else
console.warn "PhantomJS skipped statusCode check"
HTTP.get url, { headers: { 'Range': '1-abc'}}, (err, res) ->
test.isNotNull err
if err.response? # Not sure why, but under phantomjs the error object is different
test.equal err.response.statusCode, 416
else
console.warn "PhantomJS skipped statusCode check"
onComplete()
Tinytest.addAsync 'REST API requests header manipilation', (test, onComplete) ->
_id = testColl.insert { filename: 'writefile', contentType: 'text/plain' }, (err, _id) ->
test.fail(err) if err
url = Meteor.absoluteUrl 'test/' + _id
HTTP.put url, { content: '0987654321'}, (err, res) ->
test.fail(err) if err
HTTP.get url+'?download=true', (err, res) ->
test.equal res.headers['content-disposition'], "attachment; filename=\"writefile\"; filename*=UTF-8''writefile"
test.equal res.statusCode, 200
HTTP.get url+'?cache=123456', { headers: { 'Range': '1-5'}},
(err, res) ->
test.equal res.headers['cache-control'], 'max-age=123456, private'
test.equal res.statusCode, 206
HTTP.get url+'?cache=123', (err, res) ->
test.equal res.headers['cache-control'], 'max-age=123, private'
test.equal res.statusCode, 200
onComplete()
Tinytest.addAsync 'REST API requests header manipilation, UTF-8', (test, onComplete) ->
_id = testColl.insert { filename: '中文指南.txt', contentType: 'text/plain' }, (err, _id) ->
test.fail(err) if err
url = Meteor.absoluteUrl 'test/' + _id
HTTP.put url, { content: '0987654321'}, (err, res) ->
test.fail(err) if err
HTTP.get url+'?download=true', (err, res) ->
test.equal res.headers['content-disposition'], "attachment; filename=\"%E4%B8%AD%E6%96%87%E6%8C%87%E5%8D%97.txt\"; filename*=UTF-8''%E4%B8%AD%E6%96%87%E6%8C%87%E5%8D%97.txt"
test.equal res.statusCode, 200
HTTP.get url+'?cache=123456', { headers: { 'Range': '1-5'}},
(err, res) ->
test.equal res.headers['cache-control'], 'max-age=123456, private'
test.equal res.statusCode, 206
HTTP.get url+'?cache=123', (err, res) ->
test.equal res.headers['cache-control'], 'max-age=123, private'
test.equal res.statusCode, 200
onComplete()
if Meteor.isClient
noAllowSub = Meteor.subscribe 'noAllowCollPub'
noReadSub = Meteor.subscribe 'noReadCollPub'
denySub = Meteor.subscribe 'denyCollPub'
Tinytest.addAsync 'Reject insert without true allow rule', subWrapper(noAllowSub, (test, onComplete) ->
_id = noAllowColl.insert {}, (err, retid) ->
if err
test.equal err.error, 403
else
test.fail new Error "Insert without allow succeeded."
onComplete()
)
Tinytest.addAsync 'Reject insert with true deny rule', subWrapper(denySub, (test, onComplete) ->
_id = denyColl.insert {}, (err, retid) ->
if err
test.equal err.error, 403
else
test.fail new Error "Insert with deny succeeded."
onComplete()
)
Tinytest.addAsync 'Reject HTTP GET without true allow rule', subWrapper(noReadSub, (test, onComplete) ->
_id = noReadColl.insert { filename: 'writefile', contentType: 'text/plain' }, (err, _id) ->
test.fail(err) if err
url = Meteor.absoluteUrl 'noread/' + _id
HTTP.put url, { content: '0987654321'}, (err, res) ->
test.fail(err) if err
HTTP.get url, (err, res) ->
test.isNotNull err
if err.response? # Not sure why, but under phantomjs the error object is different
test.equal err.response.statusCode, 403
else
console.warn "PhantomJS skipped statusCode check"
onComplete()
)
Tinytest.addAsync 'Reject HTTP PUT larger than write allow rule allows', subWrapper(noReadSub, (test, onComplete) ->
_id = noReadColl.insert { filename: 'writefile', contentType: 'text/plain' }, (err, _id) ->
test.fail(err) if err
url = Meteor.absoluteUrl 'noread/' + _id
HTTP.put url, { content: '0123456789abcdef'}, (err, res) ->
test.isNotNull err
if err.response? # Not sure why, but under phantomjs the error object is different
test.equal err.response.statusCode, 413
else
console.warn "PhantomJS skipped statusCode check"
onComplete()
)
Tinytest.addAsync 'Client basic localUpdate test', (test, onComplete) ->
id = testColl.insert()
test.equal (typeof testColl.localUpdate), 'function'
testColl.localUpdate { _id: id }, { $set: { 'metadata.test': true } }
doc = testColl.findOne { _id: id }
test.equal doc.metadata.test, true
onComplete()
Tinytest.addAsync 'Client localUpdate server method rejection test', (test, onComplete) ->
id = testColl.insert()
Meteor.call 'updateTest', id, true, (err, res) ->
test.instanceOf err, Meteor.Error
doc = testColl.findOne { _id: id }
test.isUndefined doc.metadata.test2
test.isUndefined res
Meteor.call 'updateTest', id, false, (err, res) ->
test.fail(err) if err
test.equal res, 1
doc = testColl.findOne { _id: id }
test.equal doc.metadata.test2, false
onComplete()
# Resumable.js tests
Tinytest.add 'Client has Resumable', (test) ->
test.instanceOf testColl.resumable, Resumable, "Resumable object not found"
Tinytest.addAsync 'Client resumable.js Upload', (test, onComplete) ->
thisId = null
testColl.resumable.on 'fileAdded', (file) ->
testColl.insert { _id: file.uniqueIdentifier, filename: file.fileName, contentType: file.file.type }, (err, _id) ->
test.fail(err) if err
thisId = "#{_id}"
testColl.resumable.upload()
testColl.resumable.on 'fileSuccess', (file) ->
test.equal thisId, file.uniqueIdentifier
url = Meteor.absoluteUrl 'test/' + file.uniqueIdentifier
HTTP.get url, (err, res) ->
test.fail(err) if err
test.equal res.content,'ABCDEFGHIJ'
HTTP.del url, (err, res) ->
test.fail(err) if err
testColl.resumable.events = []
onComplete()
testColl.resumable.on 'error', (msg, err) ->
test.fail err
myBlob = new Blob [ 'ABCDEFGHIJ' ], { type: 'text/plain' }
myBlob.name = 'resumablefile'
testColl.resumable.addFile myBlob
Tinytest.addAsync 'Client resumable.js Upload, Multichunk', (test, onComplete) ->
thisId = null
testColl.resumable.on 'fileAdded', (file) ->
testColl.insert { _id: file.uniqueIdentifier, filename: file.fileName, contentType: file.file.type }, (err, _id) ->
test.fail(err) if err
thisId = "#{_id}"
testColl.resumable.upload()
testColl.resumable.on 'fileSuccess', (file) ->
test.equal thisId, file.uniqueIdentifier
url = Meteor.absoluteUrl 'test/' + file.uniqueIdentifier
HTTP.get url, (err, res) ->
test.fail(err) if err
test.equal res.content,'ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789'
HTTP.del url, (err, res) ->
test.fail(err) if err
testColl.resumable.events = []
onComplete()
testColl.resumable.on 'error', (msg, err) ->
test.fail err
myBlob = new Blob [ 'ABCDEFGHIJ', 'KLMNOPQRSTUVWXYZ', '0123456789' ], { type: 'text/plain' }
myBlob.name = 'resumablefile'
testColl.resumable.addFile myBlob
gitextract_kvam_bik/
├── .gitignore
├── .gitmodules
├── .travis.yml
├── .versions
├── HISTORY.md
├── LICENSE
├── README.md
├── package.js
├── packages/
│ └── .gitignore
├── src/
│ ├── gridFS.coffee
│ ├── gridFS_client.coffee
│ ├── gridFS_server.coffee
│ ├── http_access_server.coffee
│ ├── resumable_client.coffee
│ ├── resumable_server.coffee
│ └── server_shared.coffee
└── test/
└── file_collection_tests.coffee
Condensed preview — 17 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (166K chars).
[
{
"path": ".gitignore",
"chars": 57,
"preview": ".build*\n.npm\nsampleApp/packages\nsmart.lock\nnpm-debug.log\n"
},
{
"path": ".gitmodules",
"chars": 88,
"preview": "[submodule \"resumable\"]\n\tpath = resumable\n\turl = https://github.com/23/resumable.js.git\n"
},
{
"path": ".travis.yml",
"chars": 346,
"preview": "language: node_js\nnode_js:\n - \"6.9.1\"\n\nbefore_install:\n - \"curl https://install.meteor.com | /bin/sh\"\n - \"npm install"
},
{
"path": ".versions",
"chars": 957,
"preview": "allow-deny@1.0.5\nbabel-compiler@6.14.1\nbabel-runtime@1.0.1\nbase64@1.0.10\nbinary-heap@1.0.10\nblaze@2.1.8\nblaze-tools@1.0."
},
{
"path": "HISTORY.md",
"chars": 12975,
"preview": "## Revision history\n\n### V.NEXT\n\n* Improvements in the Cordova documentation. Thanks @rsmelo92.\n\n### V1.3.8\n\n* Added che"
},
{
"path": "LICENSE",
"chars": 1136,
"preview": "Copyright (C) 2014-2017 by Vaughn Iverson\n\nfile-collection is free software released under the MIT/X11 license:\n\nPermiss"
},
{
"path": "README.md",
"chars": 49946,
"preview": "# file-collection\n\n[](https://travis-ci.org/vsiv"
},
{
"path": "package.js",
"chars": 2286,
"preview": "/***************************************************************************\n### Copyright (C) 2014-2017 by Vaughn I"
},
{
"path": "packages/.gitignore",
"chars": 16,
"preview": "/fileCollection\n"
},
{
"path": "src/gridFS.coffee",
"chars": 3032,
"preview": "############################################################################\n# Copyright (C) 2014-2017 by Vaughn Ive"
},
{
"path": "src/gridFS_client.coffee",
"chars": 4247,
"preview": "############################################################################\n# Copyright (C) 2014-2017 by Vaughn Ive"
},
{
"path": "src/gridFS_server.coffee",
"chars": 14494,
"preview": "############################################################################\n# Copyright (C) 2014-2017 by Vaughn Ive"
},
{
"path": "src/http_access_server.coffee",
"chars": 16930,
"preview": "############################################################################\n# Copyright (C) 2014-2017 by Vaughn Ive"
},
{
"path": "src/resumable_client.coffee",
"chars": 1997,
"preview": "############################################################################\n# Copyright (C) 2014-2017 by Vaughn Ive"
},
{
"path": "src/resumable_server.coffee",
"chars": 11464,
"preview": "############################################################################\n# Copyright (C) 2014-2017 by Vaughn Ive"
},
{
"path": "src/server_shared.coffee",
"chars": 2002,
"preview": "############################################################################\n# Copyright (C) 2014-2017 by Vaughn Ive"
},
{
"path": "test/file_collection_tests.coffee",
"chars": 39413,
"preview": "############################################################################\n# Copyright (C) 2014-2017 by Vaughn Ive"
}
]
About this extraction
This page contains the full source code of the vsivsi/meteor-file-collection GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 17 files (157.6 KB), approximately 38.8k tokens. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.