[ { "path": ".github/FUNDING.yml", "content": "open_collective: dat\n" }, { "path": ".github/issue_template.md", "content": "\n\n\nI am reporting:\n\n\n\n\n\n# Bug Report\n\nPlease give us details about your installation to assist you. Run `dat -v` to see the version of Dat you are using.\n\n* Operating system:\n* Node Version:\n* Dat Version:\n\n### Expected behavior\n\n\n\n### Actual behavior\n\n\n\n### Debug Logs\n\n\n\n```\n```\n" }, { "path": ".gitignore", "content": "node_modules\n.DS_Store\ntmp\n.idea\ndata\nyarn.lock\ntest/fixtures/.dat\ntest/fixtures/dat.json\ntest/**.db\ntest/.datrc-test\ndist\nbuilds" }, { "path": ".travis.yml", "content": "language: node_js\n\nnode_js:\n - 'lts/*'\n - '12'\n - 'node'\nsudo: false\n\nscript:\n - npm test\n\nnotifications:\n irc:\n channels:\n - chat.freenode.net#datbots\n template:\n - '%{repository_slug} - %{commit_subject} - %{result} - %{build_url}'\n skip_join: true\n on_success: change\n on_failure: always\n\nbefore_deploy: npm run package\ndeploy:\n provider: releases\n api_key:\n secure: GF+Ehh9kDu2m+KqSzciZRQmUfubnVGDEfxZKVX+psesKoxxDSq8/wkl7g1yR2H8DO0dg3lW8opbsKbfOOUWztyIfFxFukgwKIawUd7Krtr4XQLyywq49NdYARKP6bSxeEb8N3xVTo5fuq104KT0mMUB9di/iunsO/ITOzbCZyWE=\n skip_cleanup: true\n file_glob: true\n file: dist/*\n on:\n repo: datproject/dat\n node: '12'\n tags: true\n" }, { "path": "CODE_OF_CONDUCT.md", "content": "Our code of conduct is under review in [this repo](https://github.com/datproject/Code-of-Conduct) - please check it out and give us feedback!\n" }, { "path": "CONTRIBUTING.md", "content": "# Welcome to Dat!\n\nPlease take a second to read over this before opening an issue. Providing complete information upfront will help us address any issue (and ship new features!) faster. Our time is limited, please respect it and create complete issues.\n\nWe are available to chat in IRC at #dat. You can join [via Gitter](https://gitter.im/datproject/discussions). You can also [view and search](https://botbot.me/freenode/dat/) chat logs.\n\nWe have a [faq](https://docs.datproject.org/faq) section on our docs that may address non-bug questions.\n\n## Opening an Issue\n\nPlease read this section before opening a new issue. \n\n`dat` is composed of many modules and this repository is just one of them. If you know your issue is with another module, we prefer you open it in that repository. There may be an exiting issue in another repository. If you aren't sure, then this is the perfect place.\n\nAny new issues should be *actionable*. If your issue cannot be solved (and thus closed) please reconsider opening it in our discussion repository or rewording it.\n\n### Bug Reports\n\nA perfect bug report would have the following:\n\n1. Summary of the issue you are experiencing.\n2. Details on what versions of node and dat you have (`node -v` and `dat -v`).\n3. A simple repeatable test case for us to run. Please try to run through it 2-3 times to ensure it is completely repeatable.\n\nWe would like to avoid issues that require a follow up questions to identify the bug. These follow ups are difficult to do unless we have a repeatable test case.\n\nWe can't all be perfect =). Do as much as you can and we'll try to help you with the rest. If we are slow to respond, a more complete bug report will help.\n\n### Feature Requests\n\nFeature requests are more than welcome. Please search exiting issues (open and closed) to make sure it is not a duplicate. A good feature request would have examples of how to use it, some detailed use cases, and any concerns or possible edge cases.\n\nKeep in mind we have specific use cases we are building for (namely scientific data sharing). If, your feature request does not fit within that use case it may not be prioritized or implemented.\n\n### General Discussion Issues\n\nWe prefer to be able to close issues in this repository, which does not lend itself to discussion type questions. Open discussion type issue in the [datproject/discussions](https://github.com/datproject/discussions/issues) repository.\n\n## For Developers\n\nPlease read these guidelines if you are interested in contributing to Dat.\n\n### Submitting pull requests\n\nBefore taking the time to code something, feel free to open an issue first proposing your idea to other contributors, that way you can get feedback on the idea before taking time to write precious code.\n\nFor any new functionality we like to see:\n\n* unit tests so that we can improve long term maintenance and catch regressions in the future\n* updates to the [change log](http://keepachangelog.com/) and relevant documentation\n\n### For Collaborators\n\nMake sure to get a `:thumbsup:`, `+1` or `LGTM` from another collaborator before merging a PR. If you aren't sure if a release should happen, open an issue.\n\nRelease process:\n\n- make sure the tests pass\n- Update changelog\n- `npm version `\n- `git push && git push --tags`\n- `npm publish`\n\n### Development workflow\n\nWe use and write a lot of node modules and it introduces a bit of a learning curve when working on multiple modules simultaneously. There are lots of different and valid solutions to working on lots of modules at once, this is just one way.\n\n#### Developing inside a node_modules folder\n\nFirst make sure you are comfortable with [how require works](https://github.com/maxogden/art-of-node#how-require-works) in node.\n\nWe recommend creating a folder somewhere manually called `node_modules`. For example in `~/code/node_modules`. Clone all of your git copies of modules that you want to work on into here, so for example:\n\n- `~/code/node_modules/dat`\n- `~/code/node_modules/hyperdrive`\n\nWhen you run `npm install` inside of `~/code/node_modules/dat`, dat will get its own copy of `hyperdrive` (one if its dependencies) inside `~/code/node_modules/dat/node_modules`. However, if you encounter a bug in hyperdrive that you need to fix, but you want to test your fix in dat, you want dat to use your git copy of hyperdrive at `~/code/node_modules/hyperdrive` and not the npm copy of hyperdrive at `~/code/node_modules/dat/node_modules/hyperdrive`.\n\nHow do you get dat to use the git copy of hyperdrive? Just delete the npm copy!\n\n```\nrm -rf ~/code/node_modules/dat/node_modules/hyperdrive\n```\n\nNow when you run dat, and it tries to `require('hyperdrive')` it first looks in its own `node_modules` folder at `~/code/node_modules/dat/node_modules` but doesnt find hyperdrive. So it goes up to `~/code/node_modules` and finds `hyperdrive` there and uses that one, your git copy.\n\nIf you want to switch back to an npm copy, just run `npm install` inside `~/code/node_modules/dat/` and npm will download any missing modules into `~/code/node_modules/dat/node_modules` but wont touch anything in `~/code/node_modules`.\n\nThis might seem a bit complicated at first, but is simple once you get the hang of it. Here are some rules to help you get started:\n\n- Never make any meaningful edits to code inside an \"npm-managed\" node_modules folder (such as `~/code/node_modules/dat/node_modules`), because when you run `npm install` inside those folders it could inadvertently delete all of your edits when installing an updated copy of a module. This has happened to me many times, so I just always use my git copy and delete the npm copy (as described above) to make edits to a module.\n- You should never need to run any npm commands in terminal when at your \"manually managed\"\" node_modules folder at `~/code/node_modules`. Never running npm commands at that folder also prevents npm from accidentally erasing your git copies of modules\n- The location of your \"manually managed\" node_modules folder should be somewhere isolated from your normal require path. E.g. if you put it at `~/node_modules`, then when you run `npm install dat` at `~/Desktop` npm might decide to erase your git copy of dat at `~/node_modules/dat` and replace it with a copy from npm, which could make you lose work. Putting your manually managed `node_modules` folder in a sub-folder like `~/code` gets it \"out of the way\" and prevents accidents like that from happening.\n" }, { "path": "LICENSE", "content": "Copyright (c) 2015 Max Ogden. All rights reserved.\nRedistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:\n\n1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.\n\n2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.\n\n3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n" }, { "path": "README.md", "content": "More info on active projects and modules at [dat-ecosystem.org](https://dat-ecosystem.org/) \n\n---\n\n# Dat\n\n> npm install -g dat\n\nUse `dat` command line to share files with version control, back up data to servers, browse remote files on demand, and automate long-term data preservation.\n\n`dat` is the first application based upon the [Hypercore Protocol](https://github.com/hypercore-protocol), and drove the architectural design through iterative development between 2014 and 2017. There exists a large community around it. \n\n\n[][Dat Project]\n\nHave questions? Join our chat via IRC or Gitter:\n\n[![#dat IRC channel on freenode][irc-badge]][irc-channel]\n[![datproject/discussions][gitter-badge]][gitter-chat]\n\n### Thanks to our financial supporters!\n![OpenCollective](https://opencollective.com/dat/tiers/maintainer.svg?avatarHeight=36&width=600\")\n\n### Table of Contents\n\n- [Installation](#installation)\n- [Getting Started](#getting-started)\n- [Using Dat](#usage)\n- [Troubleshooting](#troubleshooting)\n- [Javascript API](#js-api)\n- [For Developers](#for-developers)\n\n## Installation\n\nDat can be used as a command line tool or a javascript library:\n\n* Install the `$ dat` CLI to use in the command line.\n* [require('dat')][dat-node] - dat-node, a library for downloading and sharing dat archives in javascript apps.\n\n### Installing the `$ dat` command line tool\n\nThe recommended way to install dat is through a single file binary distribution version of `dat` with the one line install command below. The binary includes a copy of node and dat packaged inside a single file, so you just have to download one file in order to start sharing data, with no other dependencies needed on your system:\n\n```\nwget -qO- https://raw.githubusercontent.com/datproject/dat/master/download.sh | bash\n```\n\n## Next version\n\nTry the next version of dat! This version (14.0.0) is not compatible with older\nversions (13.x) and below, and works on node v12.\n\n```\nnpm install -g dat@next\n```\n\nMaintainers wanted!\n\n#### NPM Prerequisites\n\n* **Node**: You'll need to [install Node JS][install-node] before installing Dat. Dat needs `node` version 4 or above and `npm` installed. You can run `node -v` to check your version.\n* **npm**: `npm` is installed with node. You can run `npm -v` to make sure it is installed.\n\nOnce you have `npm` ready, install `dat` from npm with the `--global, -g` option, `npm install -g dat`.\n\n## Getting started\n\n#### What is Dat?\n\nShare, backup, and publish your filesystem. You can turn any folder on your computer into a dat. Dat scans your folder, allowing you to:\n\n* Track your files with automatic version history.\n* Share files with others over a secure peer to peer network.\n* Automate live backups to external HDs or remote servers.\n* Publish and share files with built in HTTP server.\n\nDat allows you to focus on the fun work without worrying about moving files around. **Secure**, **distributed**, **fast**.\n\n* Documentation: [docs.datproject.org](https://docs.datproject.org)\n* [Dat white paper](https://github.com/datprotocol/whitepaper/blob/master/dat-paper.pdf)\n\n##### Desktop Applications\n\nRather not use the command line? Check out these options:\n\n* [Beaker Browser] - An experimental p2p browser with built-in support for the Hypercore Protocol.\n* [Dat Desktop](https://github.com/datproject/dat-desktop) - A desktop app to manage multiple dats on your desktop machine. \n\n### JS Library\n\nAdd Dat to your `package.json`, `npm install dat --save`. Dat exports the [dat-node] API via `require('dat')`. Use it in your javascript applications! Dat Desktop and Dat command line both use dat-node to share and download dats.\n\nFull API documentation is available in the [dat-node] repository on Github.\n\nWe have Dat installed, let's use it!\n\nDat's unique design works wherever you store your data. You can create a new dat from any folder on your computer.\n\nA dat is some files from your computer and a `.dat` folder. Each dat has a unique `dat://` link. With your dat link, other users can download your files and live sync any updates.\n\n### Sharing Data\n\nYou can start sharing your files with a single command. Unlike `git`, you do not have to initialize a repository first, `dat share` or simply `dat` will do that for you:\n\n```\ndat \n```\n\nUse `dat` to create a dat and sync your files from your computer to other users. Dat scans your files inside ``, creating metadata in `/.dat`. Dat stores the public link, version history, and file information inside the dat folder.\n\n`dat sync` and `dat share` are aliases for the same command.\n![share-gif]\n\n### Downloading Data\n\n```\ndat dat:// \n```\n\nUse `dat` to download files from a remote computer sharing files with Dat. This will download the files from `dat://` to your ``. The download exits after it completes but you can continue to update the files later after the clone is done. Use `dat pull` to update new files or `dat sync` to live sync changes.\n\n`dat clone` is an alias for the same command.\n\n![clone-gif]\n\n\n### Misc Commands\n\nA few other highlights. Run `dat help` to see the full usage guide.\n\n* `dat create` or `dat init` - Create an empty dat and `dat.json` file.\n* `dat log ~/data/dat-folder/` or `dat log dat://` - view the history and metadata information for a dat.\n\n### Quick Demos\n\nTo get started using Dat, you can try downloading a dat and then sharing a dat of your own.\n\n#### Download Demo\n\nWe made a demo folder just for this exercise. Inside the demo folder is a `dat.json` file and a gif. We shared these files via Dat and now you can download them with our dat key!\n\nSimilar to git, you can download somebody's dat by running `dat clone `. You can also specify the directory:\n\n```\n❯ dat clone dat://778f8d955175c92e4ced5e4f5563f69bfec0c86cc6f670352c457943666fe639 ~/Downloads/dat-demo\ndat v13.5.0\nCreated new dat in /Users/joe/Downloads/dat-demo/.dat\nCloning: 2 files (1.4 MB)\n\n2 connections | Download 614 KB/s Upload 0 B/s\n\ndat sync complete.\nVersion 4\n```\n\nThis will download our demo files to the `~/Downloads/dat-demo` folder. These files are being shared by a server over Dat (to ensure high availability) but you may connect to any number of users also hosting the content.\n\nYou can also also view the files online: [datbase.org/778f8d955175c92e4ced5e4f5563f69bfec0c86cc6f670352c457943666fe639](https://datbase.org/778f8d955175c92e4ced5e4f5563f69bfec0c86cc6f670352c457943666fe639/). datbase.org can download files over Dat and display them on HTTP as long as someone is hosting it. The website temporarily caches data for any visited links (do not view your dat on datbase.org if you do not want us to cache your data).\n\n#### Sharing Demo\n\nDat can share files from your computer to anywhere. If you have a friend going through this demo with you, try sharing to them! If not we'll see what we can do.\n\nFind a folder on your computer to share. Inside the folder can be anything, Dat can handle all sorts of files (Dat works with really big folders too!).\n\nFirst, you can create a new dat inside that folder. Using the `dat create` command also walks us through making a `dat.json` file:\n\n```\n❯ dat create\nWelcome to dat program!\nYou can turn any folder on your computer into a Dat.\nA dat is a folder with some magic.\n```\n\nThis will create a new (empty) dat. Dat will print a link, share this link to give others access to view your files.\n\nOnce we have our dat, run `dat ` to scan your files and sync them to the network. Share the link with your friend to instantly start downloading files.\n\n#### Bonus HTTP Demo\n\nDat makes it really easy to share live files on a HTTP server. This is a cool demo because we can also see how version history works! Serve dat files on HTTP with the `--http` option. For example, `dat --http`, serves your files to a HTTP website with live reloading and version history! This even works for dats you're downloading (add the `--sparse` option to only download files you select via HTTP). The default HTTP port is 8080.\n\n*Hint: Use `localhost:8080/?version=10` to view a specific version.*\n\nGet started using Dat today with the `share` and `clone` commands or read below for more details.\n\n## Usage\n\nThe first time you run a command, a `.dat` folder is created to store the dat metadata.\nOnce a dat is created, you can run all the commands inside that folder, similar to git.\n\nDat keeps secret keys in the `~/.dat/secret_keys` folder. These are required to write to any dats you create.\n\n#### Creating a dat & dat.json\n\n```\ndat create []\n```\n\nThe create command prompts you to make a `dat.json` file and creates a new dat. Import the files with sync or share.\n\nOptionally bypass Title and Description prompt:\n\n```sh\ndat create --title \"MY BITS\" --description \"are ready to synchronize! 😎\"\n```\n\nOptionally bypass `dat.json` creation:\n\n```sh\ndat create --yes\ndat create -y\n```\n\n### Sharing\n\nThe quickest way to get started sharing files is to `share`:\n\n```\n❯ dat \ndat://3e830227b4b2be197679ff1b573cc85e689f202c0884eb8bdb0e1fcecbd93119\nSharing dat: 24 files (383 MB)\n\n0 connections | Download 0 B/s Upload 0 B/s\n\nImporting 528 files to Archive (165 MB/s)\n[=-----------------------------------------] 3%\nADD: data/expn_cd.csv (403 MB / 920 MB)\n```\n\n```\ndat [] [--no-import] [--no-watch]\n```\n\nStart sharing your dat archive over the network. It will import new or updated files since you last ran `create` or `sync`. Dat watches files for changes and imports updated files.\n\n* Use `--no-import` to not import any new or updated files.\n* Use `--no-watch` to not watch directory for changes. `--import` must be true for `--watch` to work.\n\n#### Ignoring Files\n\nBy default, Dat will ignore any files in a `.datignore` file, similar to git. Each file should be separated by a newline. Dat also ignores all hidden folders and files. Supports pattern wildcards (`/*.png`) and directory-wildcards (`/**/cache`).\n\n#### Selecting Files\n\nBy default, Dat will download all files. If you want to only download a subset, you can create a `.datdownload` file which downloads only the files and folders specified. Each should be separated by a newline.\n\n\n### Downloading\n\nStart downloading by running the `clone` command. This creates a folder, downloads the content and metadata, and a `.dat` folder inside. Once you started the download, you can resume at any time.\n\n```\ndat [] [--temp]\n```\n\nClone a remote dat archive to a local folder.\nThis will create a folder with the key name if no folder is specified.\n\n\n#### Downloading via `dat.json` key\n\nYou can use a `dat.json` file to clone also. This is useful when combining Dat and git, for example. To clone a dat you can specify the path to a folder containing a `dat.json`:\n\n```\ngit git@github.com:joehand/dat-clone-sparse-test.git\ndat ./dat-clone-sparse-test\n```\n\nThis will download the dat specified in the `dat.json` file.\n\n#### Updating Downloaded Archives\n\nOnce a dat is clone, you can run either `dat pull` or `dat sync` in the folder to update the archive.\n\n```\ndat pull []\n```\n\nDownload latest files and keep connection open to continue updating as remote source is updated.\n\n### Shortcut commands\n\n* `dat ` will run `dat clone` for new dats or resume the existing dat in ``\n* `dat ` is the same as running `dat sync `\n\n### Key Management & Moving dats\n\n`dat keys` provides a few commands to help you move or backup your dats.\n\nWriting to a dat requires the secret key, stored in the `~/.dat` folder. You can export and import these keys between dats. First, clone your dat to the new location:\n\n* (original) `dat share` \n* (duplicate) `dat clone `\n\nThen transfer the secret key:\n\n* (original) `dat keys export` - copy the secret key printed out.\n* (duplicate) `dat keys import` - this will prompt you for the secret key, paste it in here.\n\n## Troubleshooting\n\nWe've provided some troubleshooting tips based on issues users have seen.\nPlease [open an issue][new-issue] or ask us in our [chat room][gitter-chat] if you need help troubleshooting and it is not covered here.\n\nIf you have trouble sharing/downloading in a directory with a `.dat` folder, try deleting it and running the command again.\n\n#### Check Your Dat Version\n\nKnowing the version is really helpful if you run into any bugs, and will help us troubleshoot your issue.\n\nCheck your Dat version:\n\n```\ndat -v\n```\n\nYou should see the Dat semantic version printed, e.g. `14.0.0`.\n\n### Installation Issues\n\n#### Node & npm\n\nTo use the Dat command line tool you will need to have [node and npm installed][install-node-npm].\nMake sure those are installed correctly before installing Dat.\nYou can check the version of each:\n\n```\nnode -v\nnpm -v\n```\n\n#### Global Install\n\nThe `-g` option installs Dat globally, allowing you to run it as a command.\nMake sure you installed with that option.\n\n* If you receive an `EACCES` error, read [this guide][fixing-npm-permissions] on fixing npm permissions.\n* If you receive an `EACCES` error, you may also install Dat with sudo: `sudo npm install -g dat`.\n* Have other installation issues? Let us know, you can [open an issue][new-issue] or ask us in our [chat room][gitter-chat].\n\n### Debugging Output\n\nIf you are having trouble with a specific command, run with the debug environment variable set to `dat` (and optionally also `dat-node`).\nThis will help us debug any issues:\n\n```\nDEBUG=dat,dat-node dat dat:// dir\n```\n\n### Networking Issues\n\nNetworking capabilities vary widely with each computer, network, and configuration.\nWhenever you run Dat there are several steps to share or download files with peers:\n\n1. Discovering Peers\n2. Connecting to Peers\n3. Sending & Receiving Data\n\nWith successful use, Dat will show `Connected to 1 peer` after connection.\nIf you never see a peer connected, your network may be restricting discovery or connection.\n\n## JS API\n\nYou can use Dat in your javascript application:\n\n```js\nvar Dat = require('dat')\n\nDat('/data', function (err, dat) {\n // use dat\n})\n```\n\n**[Read more][dat-node] about the JS usage provided via `dat-node`.**\n\n## For Developers\n\nPlease see [guidelines on contributing] before submitting an issue or PR.\n\nThis command line library uses [dat-node] to create and manage the archives and networking.\nIf you'd like to build your own Dat application that is compatible with this command line tool, we suggest using dat-node.\n\n### Installing from source\n\nClone this repository and in a terminal inside of the folder you cloned run this command:\n\n```\nnpm link\n```\n\nThis should add a `dat` command line command to your PATH.\nNow you can run the `dat` command to try it out.\n\nThe contribution guide also has more tips on our [development workflow].\n\n* `npm run test` to run tests\n* `npm run auth-server` to run a local auth server for testing\n\n## License\n\nBSD-3-Clause\n\n[Dat Project]: https://datproject.org\n[Code for Science & Society]: https://codeforscience.org\n[Dat white paper]: https://github.com/datproject/docs/blob/master/papers/dat-paper.pdf\n[Dat Desktop]: https://docs.datproject.org/install#desktop-application\n[Beaker Browser]: https://beakerbrowser.com\n[registry server]: https://github.com/datproject/datbase\n[share-gif]: https://raw.githubusercontent.com/datproject/docs/master/docs/assets/cli-share.gif\n[clone-gif]: https://raw.githubusercontent.com/datproject/docs/master/docs/assets/cli-clone.gif\n[Knight Foundation grant]: https://blog.datproject.org/2016/02/01/announcing-publicbits-org/\n[dat-node]: https://github.com/datproject/dat-node\n[dat-ignore]: https://github.com/joehand/dat-ignore\n[new-issue]: https://github.com/datproject/dat/issues/new\n[dat#503]: https://github.com/datproject/dat/issues/503\n[install-node]: https://nodejs.org/en/download/\n[install-node-npm]: https://docs.npmjs.com/getting-started/installing-node\n[fixing-npm-permissions]: https://docs.npmjs.com/getting-started/fixing-npm-permissions\n[guidelines on contributing]: https://github.com/datproject/dat/blob/master/CONTRIBUTING.md\n[development workflow]: https://github.com/datproject/dat/blob/master/CONTRIBUTING.md#development-workflow\n[travis-badge]: https://travis-ci.org/datproject/dat.svg?branch=master\n[travis-build]: https://travis-ci.org/datproject/dat\n[appveyor-badge]: https://ci.appveyor.com/api/projects/status/github/datproject/dat?branch=master&svg=true\n[appveyor-build]: https://ci.appveyor.com/project/joehand/dat/branch/master\n[npm-badge]: https://img.shields.io/npm/v/dat.svg\n[npm-package]: https://npmjs.org/package/dat\n[irc-badge]: https://img.shields.io/badge/irc%20channel-%23dat%20on%20freenode-blue.svg\n[irc-channel]: https://webchat.freenode.net/?channels=dat\n[gitter-badge]: https://badges.gitter.im/Join%20Chat.svg\n[gitter-chat]: https://gitter.im/datproject/discussions\n" }, { "path": "appveyor.yml", "content": "# Test against this version of Node.js\nenvironment:\n matrix:\n - nodejs_version: \"6\"\n - nodejs_version: \"8\"\n - nodejs_version: \"10\"\n\n# Install scripts. (runs after repo cloning)\ninstall:\n - ps: Install-Product node $env:nodejs_version\n - npm install\n\ntest_script:\n # Output useful info for debugging.\n - node --version\n - npm --version\n - npm test\n\nbuild: off\n" }, { "path": "bin/cli.js", "content": "#!/usr/bin/env node\n\nvar subcommand = require('subcommand')\nvar debug = require('debug')('dat')\nvar usage = require('../src/usage')\nvar pkg = require('../package.json')\n\nprocess.title = 'dat'\n\n// Check node version to make sure we support\nvar NODE_VERSION_SUPPORTED = 4\nvar nodeMajorVer = process.version.match(/^v([0-9]+)\\./)[1]\nvar invalidNode = nodeMajorVer < NODE_VERSION_SUPPORTED\nif (invalidNode) exitInvalidNode()\nelse {\n var notifier = require('update-notifier')\n notifier({ pkg: pkg })\n .notify({\n defer: true,\n isGlobal: true,\n boxenOpts: {\n align: 'left',\n borderColor: 'green',\n borderStyle: 'classic',\n padding: 1,\n margin: { top: 1, bottom: 1 }\n }\n })\n}\n\nif (debug.enabled) {\n debug('Dat DEBUG mode engaged, enabling quiet mode')\n}\n\nvar config = {\n defaults: [\n { name: 'dir', abbr: 'd', help: 'set the directory for Dat' },\n { name: 'logspeed', default: 400 },\n { name: 'port', help: 'port to use for connections (default port: 3282 or first available)' },\n { name: 'utp', default: true, boolean: true, help: 'use utp for discovery' },\n { name: 'http', help: 'serve dat over http (default port: 8080)' },\n { name: 'debug', default: !!process.env.DEBUG && !debug.enabled, boolean: true },\n { name: 'quiet', default: debug.enabled, boolean: true }, // use quiet for dat debugging\n { name: 'sparse', default: false, boolean: true, help: 'download only requested data' },\n { name: 'up', help: 'throttle upload bandwidth (1024, 1kb, 2mb, etc.)' },\n { name: 'down', help: 'throttle download bandwidth (1024, 1kb, 2mb, etc.)' }\n ],\n root: {\n options: [\n {\n name: 'version',\n boolean: true,\n default: false,\n abbr: 'v'\n }\n ],\n command: usage\n },\n none: syncShorthand,\n commands: [\n require('../src/commands/clone'),\n require('../src/commands/create'),\n require('../src/commands/log'),\n require('../src/commands/keys'),\n require('../src/commands/publish'),\n require('../src/commands/pull'),\n require('../src/commands/status'),\n require('../src/commands/sync'),\n require('../src/commands/unpublish'),\n require('../src/commands/auth/register'),\n require('../src/commands/auth/whoami'),\n require('../src/commands/auth/logout'),\n require('../src/commands/auth/login')\n ],\n usage: {\n command: showUsageOrRunExtension,\n option: {\n name: 'help',\n abbr: 'h'\n }\n },\n aliases: {\n 'init': 'create',\n 'share': 'sync'\n },\n // whitelist extensions for now\n extensions: [\n 'store'\n ]\n}\n\nif (debug.enabled) {\n debug('dat', pkg.version)\n debug('node', process.version)\n}\n\n// Match Args + Run command\nvar match = subcommand(config)\nmatch(alias(process.argv.slice(2)))\n\nfunction alias (argv) {\n var cmd = argv[0]\n if (!config.aliases[cmd]) return argv\n argv[0] = config.aliases[cmd]\n return argv\n}\n\n// CLI Shortcuts\n// Commands:\n// dat [] - clone/sync a key\n// dat - create dat + share a directory\n// dat \nfunction syncShorthand (opts) {\n if (!opts._.length) return usage(opts)\n debug('Sync shortcut command')\n\n debug('Trying extension', opts._[0])\n // First try extension\n if (config.extensions.indexOf(opts._[0]) > -1) return require('../src/extensions')(opts)\n\n var parsed = require('../src/parse-args')(opts)\n\n // Download Key\n if (parsed.key) {\n // dat [] - clone/resume in [dir]\n debug('Clone sync')\n opts.dir = parsed.dir || parsed.key // put in `process.cwd()/key` if no dir\n opts.exit = opts.exit || false\n return require('../src/commands/clone').command(opts)\n }\n\n // Sync dir\n // dat - sync existing dat in {dir}\n if (parsed.dir) {\n opts.shortcut = true\n debug('Share sync')\n\n // Set default opts. TODO: use default opts in share\n opts.watch = opts.watch || true\n opts.import = opts.import || true\n return require('../src/commands/sync').command(opts)\n }\n\n // All else fails, show usage\n return usage(opts)\n}\n\n// This was needed so that we can show help messages from extensions\nfunction showUsageOrRunExtension (opts, help, usageMessage) {\n if (config.extensions.indexOf(opts._[0]) > -1) return require('../src/extensions')(opts)\n usage(opts, help, usageMessage)\n}\n\nfunction exitInvalidNode () {\n console.error('Node Version:', process.version)\n console.error('Unfortunately, we only support Node >= v4. Please upgrade to use Dat.')\n console.error('You can find the latest version at https://nodejs.org/')\n process.exit(0)\n}\n" }, { "path": "changelog.md", "content": "# Change Log\nAll notable changes to this project will be documented in this file.\nThis project adheres to [Semantic Versioning](https://semver.org/).\n\n## [Unreleased]\n\n## 13.13.0 \n* `dat pull --exit=NN` exits after `NN` number of seconds, when there are no updates to sync.\n\n## 13.9.0 - 2017-10-11\n\n### Changed\n* Use [datbase.org](https://datbase.org) as default registry (instead of datproject.org)\n\n## 13.8.2 - 2017-09-28\n\n### Fixed\n* Error not being handled (https://github.com/datproject/dat/issues/838)\n* Set `opts.debug` properly when using `DEBUG` that isn't `dat`.\n* Move discovery key to option in `dat keys` (#869)\n\n## 13.8.1 - 2017-08-04\n\n### Fixes\n* Error not being handled (https://github.com/datproject/dat/issues/838)\n\n## 13.8.0 - 2017-08-04\n\nWith this release, we are adding an exciting feature that really showcases how powerful Dat is, selective sync. Using the CLI you can now specify which files you want to download either with an option or using the `.datdownload` file. `dat sync` will download and keep updated on the selected files. This means you can put large datasets into Dat but have control over what files you download where.\n\n[Full release notes](https://github.com/datproject/dat/releases/tag/v13.8.0)\n\n## Added\n* Selective Sync (https://github.com/datproject/dat/pull/834)\n* Key management (https://github.com/datproject/dat/pull/828)\n\n## Changed\n* Commands run faster via lazy required modules (https://github.com/datproject/dat/pull/821)\n\n## 13.7.0 - 2017-06-28\n## Added\n* Throttling - sometimes Dat goes too fast, so you can limit the upload + download speeds. (https://github.com/datproject/dat/pull/806)\n* Publish metadata to registry when publishing (https://github.com/datproject/dat/pull/812)\n\n## Changed\n* Use dat-node http support directly (https://github.com/datproject/dat/pull/817)\n\n## Fixed\n* Use npm package for registry testing.\n\n## 13.6.0 - 2017-06-05\nFull support for Dat registries! See our [full release notes](https://github.com/datproject/dat/releases/tag/v13.6.0).\n### Added\n* Improved support for public Dat registries (https://github.com/datproject/dat/pull/794)\n* Add unpublish command\n\n## 13.5.1 - 2017-05-30\n### Changed\n* Big documentation update!\n* Force bump dat-node for a deletion bug that was a bit overzealous in deleting files (https://github.com/mafintosh/hyperdrive/issues/167).\n\n## 13.5.0 - 2017-05-25\n### Added\n* Dat version number is printed in header (https://github.com/datproject/dat/pull/788)\n* Add prompt and introduction to `dat create` command (https://github.com/datproject/dat/pull/782) and create dat.json file (https://github.com/datproject/dat/pull/765).\n* Tell user if new `.dat` was initialized.\n* Add `dat log` command to print archive history and size information (https://github.com/datproject/dat/pull/781).\n* Use `require('dat')` to get `dat-node` JS API (https://github.com/datproject/dat/pull/778).\n\n### Changed\n* Default to upload true for `dat clone` and `dat pull`, enables better hole-punching (https://github.com/datproject/dat/pull/787).\n\n### Fixed\n* Make argument parsing more consistent across commands (https://github.com/datproject/dat/pull/789)\n* Fix usage and help text (various).\n\n## 13.4.1 - 2017-05-16\n### Added\n* Document sparse option in cli help\n* add node/dat version to debug\n\n### Changed\n* Use share for shortcut (create new dat if not created)\n\n### Fixed\n* use exit option on clone shortcut if specified\n* [various ui fixes](https://github.com/datproject/dat/pull/764)\n\n## 13.4.0 - 2017-05-11\n### Added\n* Serve dat over http with `--http` option\n\n## 13.3.0 - 2017-05-10\n### Added\n* Add `--sources` option for debugging network issues\n\n## 13.2.0 - 2017-05-10\n### Added\n* Dat-* extensions ([#740](https://github.com/datproject/dat/pull/740))\n* Ignore directories in import (dat-node v3.3.0)\n\n## 13.1.1 - 2017-05-10\n### Fixed\n* Set directory for publish command\n\n### Changed\n* Improve `--show-key` help output\n* Always show download progress bar and make language more clear.\n\n## 13.1.0 - 2017-05-09\n### Fixed\n* Cleanup dat shortcut + directory creation\n* Check for any swarm.connecting before doing discovery failure.\n\n### Added\n* Check node version, fail for anything older than node v4 (#669)\n* Add show-key option to display key on downloading cmds\n* `dat status` command to show key, file count, dir size, and archive version\n\n## 13.0.0 - 2017-05-08\n### Changed\n* Upgrade to Hyperdrive v8/9 (SLEEP archive format) and Dat-node v2/3. See [dat-node release docs](https://github.com/datproject/dat-node/releases/tag/v2.0.0) for more info.\n* UI updates\n\n## 12.0.3 - 2017-03-29\n### Fixed\n* Content progress for archives with history\n* Change `process.title` to `dat` from `dat-next`\n\n### Changed\n* Use two decimals for content progress\n\n## 12.0.2 - 2017-02-08\n### Fixed\n* Remove `hyperdrive-import-files` from dependencies (it is a dependency of `dat-node`). It was accidentally added.\n* Always verify on read to avoid replication errors.\n\n## 12.0.1 - 2017-02-07\n### Fixed\n* Files getting truncated and edited with bad characters - issue [#626](https://github.com/datproject/dat/issues/626) and [#623](https://github.com/datproject/dat/issues/623)\n* Source files getting overwritten (issue [#628](https://github.com/datproject/dat/issues/628))\n* Duplicate files getting imported\n\n## 12.0.0 - 2017-02-06\nBig new release! See the [release notes](https://github.com/datproject/dat/releases/tag/v12.0.0) on Github.\n\n## 11.6.0 - 2016-11-16\n### Removed\n* webrtc support\n\n### Fixed\n* Fail gracefully if another dat is running in directory\n* Handle `dat.open` errors\n* Progress bar incorrectly showing 100% complete and 0 bytes\n\n### Added\n* Use graceful-fs to avoid EMFILE errors\n\n## 11.5.5 - 2016-11-07\n### Fixed\n* Better download statistics using blocks instead of bytes\n* Fix share stats on resuming without file changes\n* Fix calculating size UI for large files\n\n### Changed\n* Update status logger. Uses [ansi-diff-stream](https://github.com/mafintosh/ansi-diff-stream) for updating CLI output now.\n\n## 11.5.4 - 2016-10-28\n### Changed\n* Turn off `--watchFiles` by default\n* Simplify progress UI\n\n## 11.5.3 - 2016-10-28\n### Fixed\n* Fix `dat` command with no arguments\n\n## 11.5.2 - 2016-10-24\n### Fixed\n* Fix `dat --doctor`\n\n## 11.5.1 - 2016-10-24\n### Fixed\n* Resuming a folder previously shared fixed.\n\n## 11.5.0 - 2016-10-20\n### Added\n* Accept dat.land links\n* Allow `dat ` to resume a downloaded link\n\n### Fixed\n* Improved error output for incorrect params\n\n## 11.4.0 - 2016-10-06\n### Added\n* `--ignore-hidden` option. Ignores hidden files by default.\n* `--signalhub` option to override default signalhub URL.\n\n### Fixed\n* Remove headless option from electron-webrtc. It is detected for us.\n* `utp` is true by default\n\n## 11.3.1 - 2016-09-21\n### Fixed\n* Use `--quiet` mode with `--debug` so output is easier to read.\n\n## 11.3.0 - 2016-09-18\n### Added\n* `--webrtc` option. Uses electron-webrtc to run via webrtc.\n\n## 11.2.0 - 2016-09-14\n### Added\n* `--temp` option. Uses memdb as database instead of `.dat` folder.\n* Print message when download finishes telling user they can exit.\n* Add option for turning off UTP\n* Use dat-js module (includes using hyperdrive-import-files for appending)\n\n### Fixed\n* Download finished message not displayed when dat live updates\n* Download speed removed when download is finished\n\n## 11.1.2 - 2016-07-18\n### Fixed\n* Zero bytes total when downloading Dat with single file\n\n## 11.1.1 - 2016-07-15\n### Fixed\n* Create download directory if doesn't exist\n* Accept dat:// links for dat-desktop\n* Throw error when two different dats are downloaded to same folder\n\n## 11.1.0 - 2016-07-15\n### Fixed\n* Use yolowatch module for recursive live updates\n* Improved stats for edge cases\n* Print link with --quiet argument\n* Better stat & progress output with hyperdrive/hypercore events\n\n### Changed\n* Simplified and clean up CLI output\n* Improve modularity of library\n* Move logger module into own npm package, status-logger\n* Store key in .dat db without encoding as hex string (#498)\n* upgrade to hyperdrive 7\n\n### Removed\n* List download option (will be re-added pending a hyperdrive update)\n\n### Added\n* Accept dat-encoding for 50 character links\n\n## 11.0.2 - 2016-06-23\n### Fixed\n* Live mode with recursive adding files!\n\n### Changed\n* Separate yoloWatch to module\n\n## 11.0.1 - 2016-06-20\n### Fixed\n* Create download directory if it doesn't exist\n\n### Added\n* Updated Docs\n\n## 11.0.0 - 2016-06-17\n### Added\n* Live dat by default\n* Added the `.dat` folder to store metadata and keys\n* Resume dat share and download in existing .dat directory\n* Store metadata using leveldb\n* --list option on download to list files\n* --exit option on download to close process on completion\n\n### Changed\n* New proposed RC2 API\n* --static option change to --snapshot\n* Use Hyperdrive-archive-swarm module for swarm\n\n### Removed\n* --seed option, stays open by default now\n* --no-color option\n* --append option, append by default now\n\n## 10.1.1 - 2016-06-09\n### Fixed\n* Fix file count on live share\n* Fix total percentage on share\n\n## 10.1.0 - 2016-06-08\n### Changed\n* Show progress in CLI output\n\n## 10.0.2 - 2016-06-07\n### Fixed\n* Fix --static sharing\n* Fix --doctor\n\n## 10.0.1 - 2016-06-06\n### Fixed\n* Share argument\n* Argument bugs\n\n## 10.0.0 - 2016-06-06\n### Added\n* Live sharing!\n\n### Changed\n* Update to hyperdrive 6.0\n* Update API to RC2 candidate\n\n## 9.x.x and earlier\n\nThese refer to the pre-1.0 versions of dat and are omitted.\n" }, { "path": "download.sh", "content": "#!/bin/bash\n\n# gets latest dat release zip for platform and extracts runnable binary into ~/.dat\n# usage: wget -qO- https://raw.githubusercontent.com/datproject/dat/master/bin/install.sh | bash\n# based on https://github.com/jpillora/installer/blob/master/scripts/download.sh\n\nDAT_DIR=\"$HOME/.dat/releases\"\n\nfunction cleanup {\n\trm -rf $DAT_DIR/tmp.zip > /dev/null\n}\n\nfunction fail {\n\tcleanup\n\tmsg=$1\n\techo \"============\"\n\techo \"Error: $msg\" 1>&2\n\texit 1\n}\n\nfunction install {\n # bash check\n [ ! \"$BASH_VERSION\" ] && fail \"Please use bash instead\"\n\tGET=\"\"\n\tif which curl > /dev/null; then\n\t\tGET=\"curl\"\n\t\tGET=\"$GET --fail -# -L\"\n\telif which wget > /dev/null; then\n\t\tGET=\"wget\"\n\t\tGET=\"$GET -qO-\"\n\telse\n\t\tfail \"neither wget/curl are installed\"\n\tfi\n case `uname -s` in\n Darwin) OS=\"macos\";;\n Linux) OS=\"linux\";;\n *) fail \"unsupported os: $(uname -s)\";;\n esac\n if uname -m | grep 64 > /dev/null; then\n \tARCH=\"x64\"\n else\n \tfail \"only arch x64 is currently supported for single file install. please use npm instead. your arch is: $(uname -m)\"\n fi\n echo \"Fetching latest Dat release version from GitHub\"\n LATEST=$($GET -qs https://api.github.com/repos/datproject/dat/releases/latest | grep tag_name | head -n 1 | cut -d '\"' -f 4);\n\tmkdir -p $DAT_DIR || fail \"Could not create directory $DAT_DIR, try manually downloading zip and extracting instead.\"\n\tcd $DAT_DIR\n RELEASE=\"dat-${LATEST:1}-${OS}-${ARCH}\"\n URL=\"https://github.com/datproject/dat/releases/download/${LATEST}/${RELEASE}.zip\"\n\twhich unzip > /dev/null || fail \"unzip is not installed\"\n echo \"Downloading $URL\"\n\tbash -c \"$GET $URL\" > $DAT_DIR/tmp.zip || fail \"download failed\"\n\tunzip -o -qq $DAT_DIR/tmp.zip || fail \"unzip failed\"\n BIN=\"$DAT_DIR/$RELEASE/dat\"\n\tchmod +x $BIN || fail \"chmod +x failed\"\n\tcleanup\n printf \"Dat $LATEST has been downloaded successfully. Execute it with this command:\\n\\n${BIN}\\n\\nAdd it to your PATH with this command (add this to .bash_profile/.bashrc):\\n\\nexport PATH=\\\"\\$PATH:$DAT_DIR/$RELEASE\\\"\\n\"\n}\n\ninstall" }, { "path": "index.js", "content": "module.exports = require('dat-node')\n" }, { "path": "package.json", "content": "{\n \"name\": \"dat\",\n \"version\": \"14.0.3\",\n \"description\": \"Dat is the package manager for data. Easily share and version control data.\",\n \"keywords\": [\n \"dat\",\n \"dat protocol\",\n \"hyperdrive\",\n \"decentralized\",\n \"file sharing\"\n ],\n \"main\": \"index.js\",\n \"bin\": {\n \"dat\": \"bin/cli.js\"\n },\n \"scripts\": {\n \"auth-server\": \"DEBUG=* node scripts/auth-server.js\",\n \"install-precommit\": \"echo ./node_modules/.bin/standard > .git/hooks/pre-commit && chmod +x .git/hooks/pre-commit\",\n \"standard\": \"standard\",\n \"deps\": \"dependency-check . && dependency-check . --extra --no-dev\",\n \"test\": \"standard && npm run deps && tape 'test/*.js'\",\n \"test-only\": \"tape 'test/*.js'\",\n \"package\": \"rm -rf builds && npm run pkg && ./package.sh\",\n \"pkg\": \"pkg package.json -o builds/dat\"\n },\n \"repository\": {\n \"type\": \"git\",\n \"url\": \"git://github.com/datproject/dat.git\"\n },\n \"author\": \"Dat Project\",\n \"license\": \"BSD-3-Clause\",\n \"bugs\": {\n \"url\": \"https://github.com/datproject/dat/issues\"\n },\n \"homepage\": \"https://datproject.org\",\n \"directories\": {\n \"test\": \"tests\"\n },\n \"dependencies\": {\n \"bytes\": \"^3.1.0\",\n \"chalk\": \"^2.4.2\",\n \"cli-truncate\": \"^1.0.0\",\n \"dat-encoding\": \"^5.0.1\",\n \"dat-json\": \"^1.0.3\",\n \"dat-link-resolve\": \"^2.3.0\",\n \"dat-log\": \"^2.0.0\",\n \"dat-node\": \"^4.0.0\",\n \"dat-registry\": \"^4.0.1\",\n \"debug\": \"^4.0.0\",\n \"neat-log\": \"^3.1.0\",\n \"prettier-bytes\": \"^1.0.3\",\n \"progress-string\": \"^1.2.1\",\n \"prompt\": \"^1.0.0\",\n \"pump\": \"^3.0.0\",\n \"rimraf\": \"^2.7.1\",\n \"speedometer\": \"^1.1.0\",\n \"subcommand\": \"^2.1.1\",\n \"throttle\": \"^1.0.3\",\n \"update-notifier\": \"^2.3.0\"\n },\n \"devDependencies\": {\n \"cross-zip-cli\": \"^1.0.0\",\n \"dependency-check\": \"^3.4.1\",\n \"hypercore\": \"^6.25.2\",\n \"mkdirp\": \"^0.5.4\",\n \"node-fetch\": \"^2.6.1\",\n \"pkg\": \"^4.4.4\",\n \"random-access-memory\": \"^3.1.1\",\n \"recursive-readdir-sync\": \"^1.0.6\",\n \"standard\": \"^12.0.0\",\n \"tape\": \"^4.13.2\",\n \"tape-spawn\": \"^1.4.2\",\n \"temporary-directory\": \"^1.0.2\"\n },\n \"pkg\": {\n \"assets\": [\n \"./node_modules/utp-native/prebuilds/**\",\n \"./node_modules/blake2b-wasm/blake2b.wasm\",\n \"./node_modules/siphash24/siphash24.wasm\"\n ],\n \"targets\": [\n \"node10-linux-x64\",\n \"node10-macos-x64\",\n \"node10-win-x64\"\n ]\n }\n}\n" }, { "path": "package.sh", "content": "#!/usr/bin/env sh\n# couldnt figure out undocumented 'output template' mode for pkg so wrote this\n# also need to include .node files until pkg supports including them in binary\n\nNODE_ABI=\"node.napi\"\nVERSION=$(node -pe \"require('./package.json').version\")\n\nrm -rf dist\n\nmkdir dist\nmkdir builds/dat-$VERSION-linux-x64\nmkdir builds/dat-$VERSION-macos-x64\nmkdir builds/dat-$VERSION-win-x64\n\nmv builds/dat-linux builds/dat-$VERSION-linux-x64/dat\nmv builds/dat-macos builds/dat-$VERSION-macos-x64/dat\nmv builds/dat-win.exe builds/dat-$VERSION-win-x64/dat.exe\n\ncp node_modules/utp-native/prebuilds/linux-x64/$NODE_ABI.node builds/dat-$VERSION-linux-x64/\ncp node_modules/utp-native/prebuilds/darwin-x64/$NODE_ABI.node builds/dat-$VERSION-macos-x64/\ncp node_modules/utp-native/prebuilds/win32-x64/$NODE_ABI.node builds/dat-$VERSION-win-x64/\n\ncp LICENSE builds/dat-$VERSION-linux-x64/\ncp LICENSE builds/dat-$VERSION-macos-x64/\ncp LICENSE builds/dat-$VERSION-win-x64/\n\ncp README.md builds/dat-$VERSION-linux-x64/README\ncp README.md builds/dat-$VERSION-macos-x64/README\ncp README.md builds/dat-$VERSION-win-x64/README\n\ncd builds\n../node_modules/.bin/cross-zip dat-$VERSION-linux-x64 ../dist/dat-$VERSION-linux-x64.zip\n../node_modules/.bin/cross-zip dat-$VERSION-macos-x64 ../dist/dat-$VERSION-macos-x64.zip\n../node_modules/.bin/cross-zip dat-$VERSION-win-x64 ../dist/dat-$VERSION-win-x64.zip\n\nrm -rf builds\n\n# now travis will upload the 3 zips in dist to the release\n" }, { "path": "scripts/auth-server.js", "content": "var createServer = require('../tests/helpers/auth-server')\n\ncreateServer(process.env.PORT || 8888, function (err, server, closeServer) {\n if (err) throw err\n\n process.on('exit', close)\n process.on('SIGINT', close)\n\n function close (cb) {\n closeServer(function () {\n process.exit()\n })\n }\n})\n" }, { "path": "snap/snapcraft.yaml", "content": "name: dat\nversion: '13.11.4'\nsummary: Share & live sync files anywhere via command line\ndescription: |\n Use Dat command line to share files with version control, \n back up data to servers, browse remote files on demand, \n and automate long-term data preservation.\n\ngrade: 'stable'\nconfinement: 'strict'\n\napps:\n dat: \n command: dat\n plugs:\n - home\n - network\n - network-bind\n - removable-media\n\nparts:\n dat:\n source: https://github.com/datproject/dat.git\n source-tag: 'v13.11.4'\n plugin: nodejs\n node-engine: 10.9.0\n" }, { "path": "src/commands/auth/login.js", "content": "module.exports = {\n name: 'login',\n command: login,\n help: [\n 'Login to a Dat registry server',\n 'Usage: dat login []',\n '',\n 'Publish your dats so other users can discovery them.',\n 'Please register before trying to login.'\n ].join('\\n'),\n options: [\n {\n name: 'server',\n help: 'Your Dat registry server (must be registered to login).'\n }\n ]\n}\n\nfunction login (opts) {\n var prompt = require('prompt')\n var output = require('neat-log/output')\n var chalk = require('chalk')\n var Registry = require('../../registry')\n\n if (opts._[0]) opts.server = opts._[0]\n var welcome = output(`\n Welcome to ${chalk.green(`dat`)} program!\n Login to publish your dats.\n\n `)\n console.log(welcome)\n\n var schema = {\n properties: {\n server: {\n description: chalk.magenta('Dat registry'),\n default: opts.server || 'datbase.org',\n required: true\n },\n email: {\n description: chalk.magenta('Email'),\n message: 'Email required',\n required: true\n },\n password: {\n description: chalk.magenta('Password'),\n message: 'Password required',\n required: true,\n hidden: true,\n replace: '*'\n }\n }\n }\n\n prompt.override = opts\n prompt.message = ''\n prompt.start()\n prompt.get(schema, function (err, results) {\n if (err) return exitErr(err)\n opts.server = results.server\n makeRequest(results)\n })\n\n function makeRequest (user) {\n var client = Registry(opts)\n client.login({\n email: user.email,\n password: user.password\n }, function (err, resp, body) {\n if (err && err.message) return exitErr(err.message)\n else if (err) return exitErr(err.toString())\n\n console.log(output(`\n Logged you in to ${chalk.green(opts.server)}!\n\n Now you can publish dats and share:\n * Run ${chalk.green(`dat publish`)} to publish a dat!\n * View & Share your dats at ${opts.server}\n `))\n process.exit(0)\n })\n }\n}\n\nfunction exitErr (err) {\n console.error(err)\n process.exit(1)\n}\n" }, { "path": "src/commands/auth/logout.js", "content": "module.exports = {\n name: 'logout',\n command: logout,\n help: [\n 'Logout from current Dat registry server',\n 'Usage: dat logout []',\n '',\n 'Specify server if you want to from non-active other server.',\n 'Check active server with `dat whoami`.'\n ].join('\\n'),\n options: [\n {\n name: 'server',\n help: 'Server to log out of. Defaults to active login.'\n }\n ]\n}\n\nfunction logout (opts) {\n var chalk = require('chalk')\n var Registry = require('../../registry')\n\n if (opts._[0]) opts.server = opts._[0]\n\n var client = Registry(opts)\n\n var whoami = client.whoami()\n if (!whoami || !whoami.token) return exitErr('Not currently logged in to that server.')\n client.logout(function (err) {\n if (err) return exitErr(err)\n console.log(`Logged out of ${chalk.green(whoami.server)}`)\n process.exit(0)\n })\n}\n\nfunction exitErr (err) {\n console.error(err)\n process.exit(1)\n}\n" }, { "path": "src/commands/auth/register.js", "content": "module.exports = {\n name: 'register',\n command: register,\n help: [\n 'Register with a public Dat registry',\n 'Usage: dat register []',\n '',\n 'Register with datbase.org or other registries to publish your dats.'\n ].join('\\n'),\n options: [\n {\n name: 'server',\n help: 'Your Dat registry.'\n }\n ]\n}\n\nfunction register (opts) {\n var prompt = require('prompt')\n var output = require('neat-log/output')\n var chalk = require('chalk')\n var Registry = require('../../registry')\n\n // TODO: check if logged in?\n if (opts._[0]) opts.server = opts._[0]\n var welcome = output(`\n Welcome to ${chalk.green(`dat`)} program!\n Create a new account with a Dat registry.\n\n `)\n console.log(welcome)\n\n var schema = {\n properties: {\n server: {\n description: chalk.magenta('Dat registry'),\n default: opts.server || 'datbase.org',\n required: true\n },\n username: {\n description: chalk.magenta('Username'),\n message: 'Username required',\n required: true\n },\n email: {\n description: chalk.magenta('Email'),\n message: 'Email required',\n required: true\n },\n password: {\n description: chalk.magenta('Password'),\n message: 'Password required',\n required: true,\n hidden: true,\n replace: '*'\n }\n }\n }\n\n prompt.override = opts\n prompt.message = ''\n prompt.start()\n prompt.get(schema, function (err, results) {\n if (err) return exitErr(err)\n opts.server = results.server\n makeRequest(results)\n })\n\n function makeRequest (user) {\n var client = Registry(opts)\n\n client.register({\n email: user.email,\n username: user.username,\n password: user.password\n }, function (err) {\n if (err && err.message) return exitErr(err.message)\n else if (err) return exitErr(err.toString())\n console.log(output(`\n Created account on ${chalk.green(opts.server)}!\n\n Login to start publishing: ${chalk.green(`dat login`)}\n `))\n process.exit(0)\n })\n }\n}\n\nfunction exitErr (err) {\n console.error(err)\n process.exit(1)\n}\n" }, { "path": "src/commands/auth/whoami.js", "content": "module.exports = {\n name: 'whoami',\n command: whoami,\n help: [\n 'Get login information',\n 'Usage: dat login []',\n '',\n 'Get information for active registry or specify your registry.'\n ].join('\\n'),\n options: [\n {\n name: 'server',\n help: 'Server to get login information for. Defaults to active login.'\n }\n ]\n}\n\nfunction whoami (opts) {\n var output = require('neat-log/output')\n var chalk = require('chalk')\n var Registry = require('../../registry')\n\n if (opts._[0]) opts.server = opts._[0]\n\n var client = Registry(opts)\n var login = client.whoami()\n if (!login || !login.token) {\n if (!opts.server) return exitErr('No login information found.')\n return exitErr('No login information found for that server.')\n }\n console.log(output(`\n Your active Dat registry information:\n\n ---\n ${chalk.green(login.server)}\n Email: ${login.email}\n Username: ${login.username}\n ---\n\n Change your registry by logging in again:\n ${chalk.dim.green('dat login ')}\n `))\n process.exit(0)\n}\n\nfunction exitErr (err) {\n console.error(err)\n process.exit(1)\n}\n" }, { "path": "src/commands/clone.js", "content": "module.exports = {\n name: 'clone',\n command: clone,\n help: [\n 'Clone a remote Dat archive',\n '',\n 'Usage: dat clone [download-folder]'\n ].join('\\n'),\n options: [\n {\n name: 'empty',\n boolean: false,\n default: false,\n help: 'Do not download files by default. Files must be synced manually.'\n },\n {\n name: 'upload',\n boolean: true,\n default: true,\n help: 'announce your address on link (improves connection capability) and upload data to other downloaders.'\n },\n {\n name: 'show-key',\n boolean: true,\n default: false,\n abbr: 'k',\n help: 'print out the dat key'\n }\n ]\n}\n\nfunction clone (opts) {\n var fs = require('fs')\n var path = require('path')\n var rimraf = require('rimraf')\n var Dat = require('dat-node')\n var linkResolve = require('dat-link-resolve')\n var neatLog = require('neat-log')\n var archiveUI = require('../ui/archive')\n var trackArchive = require('../lib/archive')\n var discoveryExit = require('../lib/discovery-exit')\n var onExit = require('../lib/exit')\n var parseArgs = require('../parse-args')\n var debug = require('debug')('dat')\n\n var parsed = parseArgs(opts)\n opts.key = parsed.key || opts._[0] // pass other links to resolver\n opts.dir = parsed.dir\n opts.showKey = opts['show-key'] // using abbr in option makes printed help confusing\n opts.sparse = opts.empty\n\n debug('clone()')\n\n // cmd: dat /path/to/dat.json (opts.key is path to dat.json)\n if (fs.existsSync(opts.key)) {\n try {\n opts.key = getDatJsonKey()\n } catch (e) {\n debug('error reading dat.json key', e)\n }\n }\n\n debug(Object.assign({}, opts, { key: '', _: null })) // don't show key\n\n var neat = neatLog(archiveUI, { logspeed: opts.logspeed, quiet: opts.quiet, debug: opts.debug })\n neat.use(trackArchive)\n neat.use(discoveryExit)\n neat.use(onExit)\n neat.use(function (state, bus) {\n if (!opts.key) return bus.emit('exit:warn', 'key required to clone')\n\n state.opts = opts\n var createdDirectory = null // so we can delete directory if we get error\n\n // Force these options for clone command\n opts.exit = (opts.exit !== false)\n // opts.errorIfExists = true // TODO: do we want to force this?\n\n linkResolve(opts.key, function (err, key) {\n if (err && err.message.indexOf('Invalid key') === -1) return bus.emit('exit:error', 'Could not resolve link')\n else if (err) return bus.emit('exit:warn', 'Link is not a valid Dat link.')\n\n opts.key = key\n createDir(opts.key, function () {\n bus.emit('key', key)\n runDat()\n })\n })\n\n function createDir (key, cb) {\n debug('Checking directory for clone')\n // Create the directory if it doesn't exist\n // If no dir is specified, we put dat in a dir with name = key\n if (!opts.dir) opts.dir = key\n if (!Buffer.isBuffer(opts.dir) && typeof opts.dir !== 'string') {\n return bus.emit('exit:error', 'Directory path must be a string or Buffer')\n }\n fs.access(opts.dir, fs.F_OK, function (err) {\n if (!err) {\n createdDirectory = false\n return cb()\n }\n debug('No existing directory, creating it.')\n createdDirectory = true\n fs.mkdir(opts.dir, cb)\n })\n }\n\n function runDat () {\n Dat(opts.dir, opts, function (err, dat) {\n if (err && err.name === 'ExistsError') return bus.emit('exit:warn', 'Existing archive in this directory. Use pull or sync to update.')\n if (err) {\n if (createdDirectory) rimraf.sync(dat.path)\n return bus.emit('exit:error', err)\n }\n if (dat.writable) return bus.emit('exit:warn', 'Archive is writable. Cannot clone your own archive =).')\n\n state.dat = dat\n state.title = 'Cloning'\n bus.emit('dat')\n bus.emit('render')\n })\n }\n })\n\n function getDatJsonKey () {\n var datPath = opts.key\n var stat = fs.lstatSync(datPath)\n\n if (stat.isDirectory()) datPath = path.join(datPath, 'dat.json')\n\n if (!fs.existsSync(datPath) || path.basename(datPath) !== 'dat.json') {\n if (stat.isFile()) throw new Error('must specify existing dat.json file to read key')\n throw new Error('directory must contain a dat.json')\n }\n\n debug('reading key from dat.json:', datPath)\n return JSON.parse(fs.readFileSync(datPath, 'utf8')).url\n }\n}\n" }, { "path": "src/commands/create.js", "content": "module.exports = {\n name: 'create',\n command: create,\n help: [\n 'Create an empty dat and dat.json',\n '',\n 'Usage: dat create [directory]'\n ].join('\\n'),\n options: [\n {\n name: 'yes',\n boolean: true,\n default: false,\n abbr: 'y',\n help: 'Skip dat.json creation.'\n },\n {\n name: 'title',\n help: 'the title property for dat.json'\n },\n {\n name: 'description',\n help: 'the description property for dat.json'\n }\n ]\n}\n\nfunction create (opts) {\n var path = require('path')\n var fs = require('fs')\n var Dat = require('dat-node')\n var output = require('neat-log/output')\n var DatJson = require('dat-json')\n var prompt = require('prompt')\n var chalk = require('chalk')\n var parseArgs = require('../parse-args')\n var debug = require('debug')('dat')\n\n debug('dat create')\n if (!opts.dir) {\n opts.dir = parseArgs(opts).dir || process.cwd()\n }\n\n var welcome = `Welcome to ${chalk.green(`dat`)} program!`\n var intro = output(`\n You can turn any folder on your computer into a Dat.\n A Dat is a folder with some magic.\n\n Your dat is ready!\n We will walk you through creating a 'dat.json' file.\n (You can skip dat.json and get started now.)\n\n Learn more about dat.json: ${chalk.blue(`https://github.com/datprotocol/dat.json`)}\n\n ${chalk.dim('Ctrl+C to exit at any time')}\n\n `)\n var outro\n\n // Force certain options\n opts.errorIfExists = true\n\n console.log(welcome)\n Dat(opts.dir, opts, function (err, dat) {\n if (err && err.name === 'ExistsError') return exitErr('\\nArchive already exists.\\nYou can use `dat sync` to update.')\n if (err) return exitErr(err)\n\n outro = output(`\n\n Created empty Dat in ${dat.path}/.dat\n\n Now you can add files and share:\n * Run ${chalk.green(`dat share`)} to create metadata and sync.\n * Copy the unique dat link and securely share it.\n\n ${chalk.blue(`dat://${dat.key.toString('hex')}`)}\n `)\n\n if (opts.yes) return done()\n\n console.log(intro)\n var datjson = DatJson(dat.archive, { file: path.join(opts.dir, 'dat.json') })\n fs.readFile(path.join(opts.dir, 'dat.json'), 'utf-8', function (err, data) {\n if (err || !data) return doPrompt()\n data = JSON.parse(data)\n debug('read existing dat.json data', data)\n doPrompt(data)\n })\n\n function doPrompt (data) {\n if (!data) data = {}\n\n var schema = {\n properties: {\n title: {\n description: chalk.magenta('Title'),\n default: data.title || '',\n // pattern: /^[a-zA-Z\\s\\-]+$/,\n // message: 'Name must be only letters, spaces, or dashes',\n required: false\n },\n description: {\n description: chalk.magenta('Description'),\n default: data.description || ''\n }\n }\n }\n\n prompt.override = { title: opts.title, description: opts.description }\n prompt.message = '' // chalk.green('> ')\n // prompt.delimiter = ''\n prompt.start()\n prompt.get(schema, writeDatJson)\n\n function writeDatJson (err, results) {\n if (err) return exitErr(err) // prompt error\n if (!results.title && !results.description) return done()\n datjson.create(results, done)\n }\n }\n\n function done (err) {\n if (err) return exitErr(err)\n console.log(outro)\n }\n })\n\n function exitErr (err) {\n if (err && err.message === 'canceled') {\n console.log('')\n console.log(outro)\n process.exit(0)\n }\n console.error(err)\n process.exit(1)\n }\n}\n" }, { "path": "src/commands/doctor.js", "content": "module.exports = {\n name: 'doctor',\n help: [\n 'Call the Doctor! Runs two tests:',\n ' 1. Check if you can connect to a peer on a public server.',\n ' 2. Gives you a link to test direct peer connections.',\n '',\n 'Usage: dat doctor []'\n ].join('\\n'),\n options: [],\n command: function (opts) {\n var doctor = require('dat-doctor')\n\n opts.peerId = opts._[0]\n doctor(opts)\n }\n}\n" }, { "path": "src/commands/keys.js", "content": "module.exports = {\n name: 'keys',\n command: keys,\n help: [\n 'View & manage dat keys',\n '',\n 'Usage:',\n '',\n ' dat keys view dat key and discovery key',\n ' dat keys export export dat secret key',\n ' dat keys import import dat secret key to make a dat writable',\n ''\n ].join('\\n'),\n options: [\n {\n name: 'discovery',\n boolean: true,\n default: false,\n help: 'Print Discovery Key'\n }\n ]\n}\n\nfunction keys (opts) {\n var Dat = require('dat-node')\n var parseArgs = require('../parse-args')\n var debug = require('debug')('dat')\n\n debug('dat keys')\n if (!opts.dir) {\n opts.dir = parseArgs(opts).dir || process.cwd()\n }\n opts.createIfMissing = false // keys must always be a resumed archive\n\n Dat(opts.dir, opts, function (err, dat) {\n if (err && err.name === 'MissingError') return exit('Sorry, could not find a dat in this directory.')\n if (err) return exit(err)\n run(dat, opts)\n })\n}\n\nfunction run (dat, opts) {\n var subcommand = require('subcommand')\n var prompt = require('prompt')\n\n var config = {\n root: {\n command: function () {\n console.log(`dat://${dat.key.toString('hex')}`)\n if (opts.discovery) console.log(`Discovery key: ${dat.archive.discoveryKey.toString('hex')}`)\n process.exit()\n }\n },\n commands: [\n {\n name: 'export',\n command: function foo (args) {\n if (!dat.writable) return exit('Dat must be writable to export.')\n console.log(dat.archive.metadata.secretKey.toString('hex'))\n }\n },\n {\n name: 'import',\n command: function bar (args) {\n if (dat.writable) return exit('Dat is already writable.')\n importKey()\n }\n }\n ]\n }\n\n subcommand(config)(process.argv.slice(3))\n\n function importKey () {\n // get secret key & write\n\n var schema = {\n properties: {\n key: {\n pattern: /^[a-z0-9]{128}$/,\n message: 'Use `dat keys export` to get the secret key (128 character hash).',\n hidden: true,\n required: true,\n description: 'dat secret key'\n }\n }\n }\n prompt.message = ''\n prompt.start()\n prompt.get(schema, function (err, data) {\n if (err) return done(err)\n var secretKey = data.key\n if (typeof secretKey === 'string') secretKey = Buffer.from(secretKey, 'hex')\n // Automatically writes the metadata.ogd file\n dat.archive.metadata._storage.secretKey.write(0, secretKey, done)\n })\n\n function done (err) {\n if (err) return exit(err)\n console.log('Successful import. Dat is now writable.')\n exit()\n }\n }\n}\n\nfunction exit (err) {\n if (err) {\n console.error(err)\n process.exit(1)\n }\n process.exit(0)\n}\n" }, { "path": "src/commands/log.js", "content": "\nmodule.exports = {\n name: 'log',\n help: [\n 'View history and information about a dat',\n '',\n 'Usage: dat log [dir|link]'\n ].join('\\n'),\n options: [\n {\n name: 'live',\n boolean: true,\n default: false,\n help: 'View live updates to history.'\n }\n ],\n command: function (opts) {\n var log = require('dat-log')\n log(opts)\n }\n}\n" }, { "path": "src/commands/publish.js", "content": "module.exports = {\n name: 'publish',\n command: publish,\n help: [\n 'Publish your dat to a Dat registry',\n 'Usage: dat publish []',\n '',\n 'By default it will publish to your active registry.',\n 'Specify the server to change where the dat is published.'\n ].join('\\n'),\n options: [\n {\n name: 'server',\n help: 'Publish dat to this registry. Defaults to active login.'\n }\n ]\n}\n\nfunction publish (opts) {\n var path = require('path')\n var Dat = require('dat-node')\n var encoding = require('dat-encoding')\n var output = require('neat-log/output')\n var prompt = require('prompt')\n var chalk = require('chalk')\n var DatJson = require('dat-json')\n var xtend = Object.assign\n var Registry = require('../registry')\n\n if (!opts.dir) opts.dir = process.cwd()\n if (opts._[0]) opts.server = opts._[0]\n if (!opts.server) opts.server = 'datbase.org' // nicer error message if not logged in\n\n var client = Registry(opts)\n var whoami = client.whoami()\n if (!whoami || !whoami.token) {\n var loginErr = output(`\n Welcome to ${chalk.green(`dat`)} program!\n Publish your dats to ${chalk.green(opts.server)}.\n\n ${chalk.bold('Please login before publishing')}\n ${chalk.green('dat login')}\n\n New to ${chalk.green(opts.server)} and need an account?\n ${chalk.green('dat register')}\n\n Explore public dats at ${chalk.blue('datbase.org/explore')}\n `)\n return exitErr(loginErr)\n }\n\n opts.createIfMissing = false // publish must always be a resumed archive\n Dat(opts.dir, opts, function (err, dat) {\n if (err && err.name === 'MissingError') return exitErr('No existing dat in this directory. Create a dat before publishing.')\n else if (err) return exitErr(err)\n\n dat.joinNetwork() // join network to upload metadata\n\n var datjson = DatJson(dat.archive, { file: path.join(dat.path, 'dat.json') })\n datjson.read(publish)\n\n function publish (_, data) {\n // ignore datjson.read() err, we'll prompt for name\n\n // xtend dat.json with opts\n var datInfo = xtend({\n name: opts.name,\n url: 'dat://' + encoding.toStr(dat.key), // force correct url in publish? what about non-dat urls?\n title: opts.title,\n description: opts.description\n }, data)\n var welcome = output(`\n Publishing dat to ${chalk.green(opts.server)}!\n\n `)\n console.log(welcome)\n\n if (datInfo.name) return makeRequest(datInfo)\n\n prompt.message = ''\n prompt.start()\n prompt.get({\n properties: {\n name: {\n description: chalk.magenta('dat name'),\n pattern: /^[a-zA-Z0-9-]+$/,\n message: `A dat name can only have letters, numbers, or dashes.\\n Like ${chalk.bold('cool-cats-12meow')}`,\n required: true\n }\n }\n }, function (err, results) {\n if (err) return exitErr(err)\n datInfo.name = results.name\n makeRequest(datInfo)\n })\n }\n\n function makeRequest (datInfo) {\n console.log(`Please wait, '${chalk.bold(datInfo.name)}' will soon be ready for its great unveiling...`)\n client.dats.create(datInfo, function (err, resp, body) {\n if (err) {\n if (err.message) {\n if (err.message === 'timed out') {\n return exitErr(output(`${chalk.red('\\nERROR: ' + opts.server + ' could not connect to your computer.')}\n Troubleshoot here: ${chalk.green('https://docs.datproject.org/troubleshooting#networking-issues')}\n `))\n }\n var str = err.message.trim()\n if (str === 'jwt expired') return exitErr(`Session expired, please ${chalk.green('dat login')} again`)\n return exitErr('ERROR: ' + err.message) // node error\n }\n\n // server response errors\n return exitErr('ERROR: ' + err.toString())\n }\n if (body.statusCode === 400) return exitErr(new Error(body.message))\n\n datjson.write(datInfo, function (err) {\n if (err) return exitErr(err)\n // TODO: write published url to dat.json (need spec)\n var msg = output(`\n\n We ${body.updated === 1 ? 'updated' : 'published'} your dat!\n ${chalk.blue.underline(`${opts.server}/${whoami.username}/${datInfo.name}`)}\n `)// TODO: get url back? it'd be better to confirm link than guess username/datname structure\n\n console.log(msg)\n if (body.updated === 1) {\n console.log(output(`\n\n ${chalk.dim.green('Cool fact #21')}\n ${opts.server} will live update when you are sharing your dat!\n You only need to publish again if your dat link changes.\n `))\n } else {\n console.log(output(`\n\n Remember to use ${chalk.green('dat share')} before sharing.\n This will make sure your dat is available.\n `))\n }\n process.exit(0)\n })\n })\n }\n })\n}\n\nfunction exitErr (err) {\n console.error(err)\n process.exit(1)\n}\n" }, { "path": "src/commands/pull.js", "content": "module.exports = {\n name: 'pull',\n command: pull,\n help: [\n 'Pull updates from a cloned Dat archive',\n '',\n 'Usage: dat pull'\n ].join('\\n'),\n options: [\n {\n name: 'exit',\n boolean: false,\n help: 'exit after specified number of seconds, to give the dat network time to find updates. (default: 12)'\n },\n\n {\n name: 'upload',\n boolean: true,\n default: true,\n help: 'announce your address on link (improves connection capability) and upload data to other downloaders.'\n },\n {\n name: 'selectFromFile',\n boolean: false,\n default: '.datdownload',\n help: 'Sync only the list of selected files or directories in the given file.',\n abbr: 'select-from-file'\n },\n {\n name: 'select',\n boolean: false,\n default: false,\n help: 'Sync only the list of selected files or directories.'\n },\n {\n name: 'show-key',\n boolean: true,\n default: false,\n abbr: 'k',\n help: 'print out the dat key'\n }\n ]\n}\n\nfunction pull (opts) {\n var Dat = require('dat-node')\n var neatLog = require('neat-log')\n var archiveUI = require('../ui/archive')\n var trackArchive = require('../lib/archive')\n var selectiveSync = require('../lib/selective-sync')\n var discoveryExit = require('../lib/discovery-exit')\n var onExit = require('../lib/exit')\n var parseArgs = require('../parse-args')\n var debug = require('debug')('dat')\n\n debug('dat pull')\n if (!opts.dir) {\n var parsed = parseArgs(opts)\n opts.key = parsed.key\n opts.dir = parsed.dir || process.cwd()\n }\n\n opts.showKey = opts['show-key'] // using abbr in option makes printed help confusing\n\n // Force these options for pull command\n opts.createIfMissing = false\n\n // If --exit is specified without a number of seconds, default to 12\n if (opts.exit) {\n opts.exit = typeof opts.exit === 'number'\n ? opts.exit\n : 12\n }\n\n var neat = neatLog(archiveUI, { logspeed: opts.logspeed, quiet: opts.quiet, debug: opts.debug })\n neat.use(trackArchive)\n neat.use(discoveryExit)\n neat.use(onExit)\n neat.use(function (state, bus) {\n state.opts = opts\n selectiveSync(state, opts)\n\n Dat(opts.dir, opts, function (err, dat) {\n if (err && err.name === 'MissingError') return bus.emit('exit:warn', 'No existing archive in this directory. Use clone to download a new archive.')\n if (err) return bus.emit('exit:error', err)\n if (dat.writable) return bus.emit('exit:warn', 'Archive is writable. Cannot pull your own archive.')\n\n state.dat = dat\n bus.emit('dat')\n bus.emit('render')\n })\n })\n}\n" }, { "path": "src/commands/status.js", "content": "module.exports = {\n name: 'status',\n command: status,\n help: [\n 'Get information on about the Dat in a directory.',\n '',\n 'Usage: dat status'\n ].join('\\n'),\n options: []\n}\n\nfunction status (opts) {\n var Dat = require('dat-node')\n var neatLog = require('neat-log')\n var statusUI = require('../ui/status')\n var onExit = require('../lib/exit')\n var parseArgs = require('../parse-args')\n var debug = require('debug')('dat')\n\n debug('dat status')\n if (!opts.dir) {\n opts.dir = parseArgs(opts).dir || process.cwd()\n }\n opts.createIfMissing = false // sync must always be a resumed archive\n\n var neat = neatLog(statusUI, { logspeed: opts.logspeed, quiet: opts.quiet, debug: opts.debug })\n neat.use(onExit)\n neat.use(function (state, bus) {\n state.opts = opts\n\n Dat(opts.dir, opts, function (err, dat) {\n if (err && err.name === 'MissingError') return bus.emit('exit:warn', 'Sorry, could not find a dat in this directory.')\n if (err) return bus.emit('exit:error', err)\n\n state.dat = dat\n var stats = dat.trackStats()\n if (stats.get().version === dat.version) return exit()\n stats.on('update', function () {\n if (stats.get().version === dat.version) return exit()\n })\n\n function exit () {\n bus.render()\n process.exit(0)\n }\n })\n })\n}\n" }, { "path": "src/commands/sync.js", "content": "module.exports = {\n name: 'sync',\n command: sync,\n help: [\n 'Sync a Dat archive with the network',\n 'Watch and import file changes',\n '',\n 'Usage: dat sync'\n ].join('\\n'),\n options: [\n {\n name: 'import',\n boolean: true,\n default: true,\n help: 'Import files from the directory to the database (when writable).'\n },\n {\n name: 'ignoreHidden',\n boolean: true,\n default: true,\n abbr: 'ignore-hidden'\n },\n {\n name: 'selectFromFile',\n boolean: false,\n default: '.datdownload',\n help: 'Sync only the list of selected files or directories in the given file.',\n abbr: 'select-from-file'\n },\n {\n name: 'select',\n boolean: false,\n default: false,\n help: 'Sync only the list of selected files or directories.'\n },\n {\n name: 'watch',\n boolean: true,\n default: true,\n help: 'Watch for changes and import updated files (Dat Writable).'\n },\n {\n name: 'show-key',\n boolean: true,\n default: true,\n abbr: 'k',\n help: 'Print out the dat key.'\n }\n ]\n}\n\nfunction sync (opts) {\n var Dat = require('dat-node')\n var neatLog = require('neat-log')\n var archiveUI = require('../ui/archive')\n var selectiveSync = require('../lib/selective-sync')\n var trackArchive = require('../lib/archive')\n var onExit = require('../lib/exit')\n var parseArgs = require('../parse-args')\n var debug = require('debug')('dat')\n\n debug('dat sync')\n var parsed = parseArgs(opts)\n opts.key = parsed.key\n opts.dir = parsed.dir || process.cwd()\n opts.showKey = opts['show-key'] // using abbr in option makes printed help confusing\n\n // TODO: if dat-store running, add this dat to the local store and then exit = true\n opts.exit = false\n\n var neat = neatLog(archiveUI, { logspeed: opts.logspeed, quiet: opts.quiet, debug: opts.debug })\n neat.use(trackArchive)\n neat.use(onExit)\n neat.use(function (state, bus) {\n state.opts = opts\n selectiveSync(state, opts)\n Dat(opts.dir, opts, function (err, dat) {\n if (err && err.name === 'IncompatibleError') return bus.emit('exit:warn', 'Directory contains incompatible dat metadata. Please remove the .dat folder in this directory.')\n if (err) return bus.emit('exit:error', err)\n\n state.dat = dat\n bus.emit('dat')\n bus.emit('render')\n })\n })\n}\n" }, { "path": "src/commands/unpublish.js", "content": "module.exports = {\n name: 'unpublish',\n command: unpublish,\n options: [\n {\n name: 'server',\n help: 'Unpublish dat from this Registry.'\n },\n {\n name: 'confirm',\n default: false,\n boolean: true,\n abbr: 'y',\n help: 'Confirm you want to unpublish'\n }\n ]\n}\n\nfunction unpublish (opts) {\n var prompt = require('prompt')\n var path = require('path')\n var Dat = require('dat-node')\n var output = require('neat-log/output')\n var chalk = require('chalk')\n var DatJson = require('dat-json')\n var Registry = require('../registry')\n\n if (opts._[0]) opts.server = opts._[0]\n if (!opts.dir) opts.dir = process.cwd() // run in dir for `dat unpublish`\n\n var client = Registry(opts)\n var whoami = client.whoami()\n if (!whoami || !whoami.token) {\n var loginErr = output(`\n Welcome to ${chalk.green(`dat`)} program!\n\n ${chalk.bold('You must login before unpublishing.')}\n ${chalk.green('dat login')}\n `)\n return exitErr(loginErr)\n }\n\n opts.createIfMissing = false // unpublish dont try to create new one\n Dat(opts.dir, opts, function (err, dat) {\n if (err) return exitErr(err)\n // TODO better error msg for non-existing archive\n if (!dat.writable) return exitErr('Sorry, you can only publish a dat that you created.')\n\n var datjson = DatJson(dat.archive, { file: path.join(dat.path, 'dat.json') })\n datjson.read(function (err, data) {\n if (err) return exitErr(err)\n if (!data.name) return exitErr('Try `dat unpublish ` with this dat, we are having trouble reading it.')\n confirm(data.name)\n })\n })\n\n function confirm (name) {\n console.log(`Unpublishing '${chalk.bold(name)}' from ${chalk.green(whoami.server)}.`)\n prompt.message = ''\n prompt.colors = false\n prompt.start()\n prompt.get([{\n name: 'sure',\n description: 'Are you sure? This cannot be undone. [y/n]',\n pattern: /^[a-zA-Z\\s-]+$/,\n message: '',\n required: true\n }], function (err, results) {\n if (err) return console.log(err.message)\n if (results.sure === 'yes' || results.sure === 'y') makeRequest(name)\n else exitErr('Cancelled.')\n })\n }\n\n function makeRequest (name) {\n client.dats.delete({ name: name }, function (err, resp, body) {\n if (err && err.message) exitErr(err.message)\n else if (err) exitErr(err.toString())\n if (body.statusCode === 400) return exitErr(new Error(body.message))\n console.log(`Removed your dat from ${whoami.server}`)\n process.exit(0)\n })\n }\n}\n\nfunction exitErr (err) {\n console.error(err)\n process.exit(1)\n}\n" }, { "path": "src/extensions.js", "content": "var debug = require('debug')('dat')\nvar os = require('os')\n\nmodule.exports = runExtension\n\nfunction runExtension (opts) {\n debug('Trying Extenion', opts._[0])\n\n var extName = opts._.shift()\n trySpawn(function () {\n console.error('We could not run the extension. Please make sure it is installed:')\n console.error(`npm install -g dat-${extName}`)\n process.exit(1)\n })\n\n function trySpawn (cb) {\n var spawn = require('child_process').spawn\n var name = 'dat-' + extName\n if (os.platform() === 'win32') {\n name += '.cmd'\n }\n var child = spawn(name, process.argv.splice(3))\n child.stdout.pipe(process.stdout)\n child.stderr.pipe(process.stderr)\n child.on('error', function (err) {\n if (err.code === 'ENOENT') return cb()\n throw err\n })\n child.on('close', function (code) {\n process.exit(code)\n })\n }\n}\n" }, { "path": "src/lib/archive.js", "content": "var debug = require('debug')('dat')\nvar path = require('path')\nvar EventEmitter = require('events').EventEmitter\nvar doImport = require('./import-progress')\nvar stats = require('./stats')\nvar network = require('./network')\nvar download = require('./download')\nvar serve = require('./serve-http')\n\nmodule.exports = function (state, bus) {\n state.warnings = state.warnings || []\n bus.once('dat', function () {\n state.writable = state.dat.writable\n state.joinNetwork = !(state.joinNetwork === false)\n\n stats(state, bus)\n if (state.joinNetwork) network(state, bus)\n if (state.opts.http) serve(state, bus)\n\n if (state.writable && state.opts.import) doImport(state, bus)\n else if (state.opts.sparse) selectiveSync(state, bus)\n else download(state, bus)\n\n if (state.dat.archive.content) return bus.emit('archive:content')\n state.dat.archive.once('content', function () {\n bus.emit('archive:content')\n })\n })\n\n bus.once('archive:content', function () {\n state.hasContent = true\n })\n}\n\nfunction selectiveSync (state, bus) {\n var archive = state.dat.archive\n debug('sparse mode. downloading metadata')\n var emitter = new EventEmitter()\n\n function download (entry) {\n debug('selected', entry)\n archive.stat(entry, function (err, stat) {\n if (err) return state.warnings.push(err.message)\n if (stat.isDirectory()) downloadDir(entry, stat)\n if (stat.isFile()) downloadFile(entry, stat)\n })\n }\n\n function downloadDir (dirname, stat) {\n debug('downloading dir', dirname)\n archive.readdir(dirname, function (err, entries) {\n if (err) return bus.emit('exit:error', err)\n entries.forEach(function (entry) {\n emitter.emit('download', path.join(dirname, entry))\n })\n })\n }\n\n function downloadFile (entry, stat) {\n var start = stat.offset\n var end = stat.offset + stat.blocks\n state.selectedByteLength += stat.size\n bus.emit('render')\n if (start === 0 && end === 0) return\n debug('downloading', entry, start, end)\n archive.content.download({ start, end }, function () {\n debug('success', entry)\n })\n }\n\n emitter.on('download', download)\n if (state.opts.selectedFiles) state.opts.selectedFiles.forEach(download)\n\n if (state.opts.empty) {\n archive.metadata.update(function () {\n return bus.emit('exit:warn', `Dat successfully created in empty mode. Download files using pull or sync.`)\n })\n }\n\n archive.on('update', function () {\n debug('archive update')\n bus.emit('render')\n })\n}\n" }, { "path": "src/lib/discovery-exit.js", "content": "var output = require('neat-log/output')\n\nmodule.exports = discoveryExit\n\nfunction discoveryExit (state, bus) {\n bus.once('network:callback', checkExit)\n\n function checkExit () {\n if (state.dat.network.connections || !state.opts.exit) return\n if (state.dat.network.connecting) return setTimeout(checkExit, 500) // wait to see if any connections resolve\n var msg = output(`\n Dat could not find any connections for that link.\n There may not be any sources online.\n\n Ensure that everyone is using the latest version, using dat -v\n `)\n bus.emit('exit:warn', msg)\n }\n}\n" }, { "path": "src/lib/download.js", "content": "var debug = require('debug')('dat')\nvar xtend = Object.assign\n\nmodule.exports = trackDownload\n\nfunction trackDownload (state, bus) {\n if (state.hasContent) return track()\n bus.once('archive:content', track)\n\n function track () {\n var archive = state.dat.archive\n\n state.download = xtend({\n modified: false,\n nsync: false\n }, {})\n\n archive.content.on('clear', function () {\n debug('archive clear')\n state.download.modified = true\n })\n\n archive.content.on('download', function (index, data) {\n state.download.modified = true\n })\n\n archive.on('syncing', function () {\n debug('archive syncing')\n state.download.nsync = false\n })\n\n archive.on('sync', function () {\n debug('archive sync', state.stats.get())\n state.download.nsync = true\n // if we are supposed to exit, do so if we've pulled changes or have given the network the desired wait time\n if (state.opts.exit) {\n if (state.download.modified) {\n return exit()\n } else {\n var delayInMilliseconds = 1000 * state.opts.exit\n setTimeout(exit, delayInMilliseconds)\n }\n }\n if (state.dat.archive.version === 0) {\n // TODO: deal with this.\n // Sync sometimes fires early when it should wait for update.\n }\n bus.emit('render')\n })\n\n archive.on('update', function () {\n debug('archive update')\n bus.emit('render')\n })\n\n function exit () {\n if (state.stats.get().version !== archive.version) {\n return state.stats.on('update', exit)\n }\n state.exiting = true\n bus.render()\n process.exit(0)\n }\n }\n}\n" }, { "path": "src/lib/exit.js", "content": "\nmodule.exports = onExit\n\nfunction onExit (state, bus) {\n bus.on('exit:error', onError)\n bus.on('exit:warn', function (err) {\n onError(err, true)\n })\n bus.on('exit', function () {\n state.exiting = true\n bus.render()\n process.exit()\n })\n\n function onError (err, clear) {\n if (clear) bus.clear()\n console.error(err)\n process.exit(1)\n }\n}\n" }, { "path": "src/lib/import-progress.js", "content": "var xtend = Object.assign\n\nmodule.exports = trackImport\n\nfunction trackImport (state, bus) {\n if (state.dat) return track()\n bus.once('dat', track)\n\n function track () {\n var progress = state.dat.importFiles(state.opts, function (err) {\n if (err) return bus.emit('exit:error', err)\n state.importer.fileImport = null\n state.exiting = true\n bus.emit('render')\n })\n state.importer = xtend({\n importedBytes: 0,\n count: progress.count,\n liveImports: [],\n indexSpeed: progress.indexSpeed\n }, progress)\n bus.emit('dat:importer')\n\n var counting = setInterval(function () {\n // Update file count in progress counting (for big dirs)\n bus.emit('render')\n }, state.opts.logspeed)\n\n progress.on('count', function (count) {\n clearInterval(counting)\n state.count = count\n state.count.done = true\n bus.emit('render')\n })\n\n progress.on('del', function (src, dst) {\n if (src.live) state.importer.liveImports.push({ src: src, dst: dst, type: 'del' })\n })\n\n progress.on('put', function (src, dst) {\n if (src.live) state.importer.liveImports.push({ src: src, dst: dst, type: 'put' })\n if (src.stat.isDirectory()) return\n state.importer.fileImport = {\n src: src,\n dst: dst,\n progress: 0,\n type: 'put'\n }\n bus.emit('render')\n })\n\n progress.on('put-data', function (chunk, src, dst) {\n state.importer.fileImport.progress += chunk.length\n if (!src.live) state.importer.importedBytes += chunk.length // don't include live in total\n state.importer.indexSpeed = progress.indexSpeed\n bus.emit('render')\n })\n }\n}\n" }, { "path": "src/lib/network.js", "content": "var bytes = require('bytes').parse\nvar speed = require('speedometer')\nvar throttle = require('throttle')\nvar pump = require('pump')\nvar debug = require('debug')('dat')\nvar xtend = Object.assign\n\nmodule.exports = trackNetwork\n\nfunction trackNetwork (state, bus) {\n if (state.dat) return track()\n bus.once('dat', track)\n\n function track () {\n var opts = state.opts\n if (state.opts.up || state.opts.down) {\n opts = xtend({}, opts, {\n connect: function (local, remote) {\n var streams = [local, remote, local]\n if (state.opts.up) streams.splice(1, 0, throttle(bytes(state.opts.up)))\n if (state.opts.down) streams.splice(-1, 0, throttle(bytes(state.opts.down)))\n pump(streams)\n }\n })\n }\n var network = state.dat.joinNetwork(opts, function () {\n bus.emit('network:callback')\n })\n\n network.on('error', function (err) {\n if (err.code === 'EADDRINUSE') {\n if (opts.port) {\n bus.emit('exit:warn', `Specified port (${opts.port}) in use. Please use another port.`)\n } else {\n debug(err.message + ' trying random port')\n }\n } else {\n debug('network error:', err.message)\n // TODO return bus.emit('exit:error', err)\n }\n })\n state.network = xtend(network, state.network)\n bus.emit('dat:network')\n\n network.on('connection', function (conn, info) {\n bus.emit('render')\n conn.on('close', function () {\n bus.emit('render')\n })\n })\n\n if (state.opts.sources) trackSources()\n if (state.stats) return trackSpeed()\n bus.once('dat:stats', trackSpeed)\n\n function trackSpeed () {\n setInterval(function () {\n bus.emit('render')\n }, state.opts.logspeed)\n }\n\n function trackSources () {\n state.sources = state.sources || {}\n network.on('connection', function (conn, info) {\n var id = info.id.toString('hex')\n var peerSpeed = speed()\n\n state.sources[id] = info\n state.sources[id].speed = peerSpeed()\n state.sources[id].getProgress = function () {\n\n // TODO: how to get right peer from archive.content?\n // var remote = conn.feeds[1].remoteLength\n // // state.dat.archive.content.sources[0].feed.id.toString('hex')\n // if (!remote) return\n // return remote / dat.archive.content.length\n }\n\n conn.feeds.map(function (feed) {\n feed.stream.on('data', function (data) {\n state.sources[id].speed = peerSpeed(data.length)\n bus.emit('render')\n })\n feed.stream.on('error', function (err) {\n state.sources[id].error = err\n })\n })\n bus.emit('render')\n\n conn.on('close', function () {\n state.sources[id].speed = 0\n state.sources[id].closed = true\n bus.emit('render')\n })\n })\n }\n }\n}\n" }, { "path": "src/lib/selective-sync.js", "content": "var fs = require('fs')\nvar path = require('path')\n\nmodule.exports = function (state, opts) {\n // selective sync stuff\n var parsing = opts.selectFromFile !== '.datdownload' ? opts.selectFromFile : path.join(opts.dir, '.datdownload')\n opts.selectedFiles = parseFiles(parsing)\n if (opts.select && typeof opts.select === 'string') opts.selectedFiles = opts.select.split(',')\n if (opts.selectedFiles) {\n state.title = 'Syncing'\n state.selectedByteLength = 0\n opts.sparse = true\n }\n return state\n}\n\nfunction parseFiles (input) {\n var parsed = null\n\n try {\n if (fs.statSync(input).isFile()) {\n parsed = fs.readFileSync(input).toString().trim().split(/\\r?\\n/)\n }\n } catch (err) {\n if (err && !err.name === 'ENOENT') {\n console.error(err)\n process.exit(1)\n }\n }\n\n return parsed\n}\n" }, { "path": "src/lib/serve-http.js", "content": "\nmodule.exports = runHttp\n\nfunction runHttp (state, bus) {\n if (state.dat) return serve()\n bus.once('dat', serve)\n\n function serve () {\n var port = (typeof state.opts.http === 'boolean') ? 8080 : state.opts.http\n var server = state.dat.serveHttp({ port: port })\n\n server.on('listening', function () {\n state.http = { port: port, listening: true }\n bus.emit('render')\n })\n }\n}\n" }, { "path": "src/lib/stats.js", "content": "var xtend = Object.assign\n\nmodule.exports = trackStats\n\nfunction trackStats (state, bus) {\n if (state.dat) return track()\n bus.once('dat', track)\n\n function track () {\n var stats = state.dat.trackStats(state.opts)\n state.stats = xtend(stats, state.stats)\n stats.on('update', function () {\n bus.emit('stats:update')\n bus.emit('render')\n })\n bus.emit('stats')\n }\n}\n" }, { "path": "src/parse-args.js", "content": "var fs = require('fs')\nvar path = require('path')\nvar encoding = require('dat-encoding')\n\nmodule.exports = function (opts) {\n // dat [] arg1 arg2 [options]\n // parse args without options from opts._\n // return parsed { dir, key }\n var parsed = {\n key: opts.key || null,\n dir: opts.dir || null // process.cwd() ?\n }\n\n // dat []\n if (!opts._.length) return parsed\n\n // dat [] arg1 arg2\n // arg1 = key\n // arg2 = dir\n if (opts._.length === 2) {\n parsed.key = opts._[0]\n parsed.dir = opts._[1]\n return parsed\n }\n\n // dat [] arg\n // arg = dir or key\n\n // First, check if key\n try {\n parsed.key = encoding.toStr(opts._[0])\n return parsed\n } catch (err) {\n if (err && err.message !== 'Invalid key') {\n // catch non-key errors\n console.error(err)\n process.exit(1)\n }\n }\n\n try {\n var stat = fs.statSync(opts._[0])\n if (stat.isFile()) {\n parsed.dir = path.resolve(path.dirname(opts._[0]))\n } else {\n parsed.dir = opts._[0]\n }\n } catch (err) {\n if (err && !err.name === 'ENOENT') {\n console.error(err)\n process.exit(1)\n }\n }\n\n return parsed\n}\n" }, { "path": "src/registry.js", "content": "var xtend = Object.assign\nvar RegistryClient = require('dat-registry')\n\nmodule.exports = function (opts) {\n var townshipOpts = {\n server: opts.server,\n config: {\n filepath: opts.config // defaults to ~/.datrc via dat-registry\n }\n }\n var defaults = {\n // xtend doesn't overwrite when key is present but undefined\n // If we want a default, make sure it's not going to passed as undefined\n }\n var options = xtend(defaults, townshipOpts)\n return RegistryClient(options)\n}\n" }, { "path": "src/ui/archive.js", "content": "var path = require('path')\nvar output = require('neat-log/output')\nvar pretty = require('prettier-bytes')\nvar chalk = require('chalk')\nvar downloadUI = require('./components/download')\nvar importUI = require('./components/import-progress')\nvar warningsUI = require('./components/warnings')\nvar networkUI = require('./components/network')\nvar sourcesUI = require('./components/sources')\nvar keyEl = require('./elements/key')\nvar pluralize = require('./elements/pluralize')\nvar version = require('./elements/version')\nvar pkg = require('../../package.json')\n\nmodule.exports = archiveUI\n\nfunction archiveUI (state) {\n if (!state.dat) return 'Starting Dat program...'\n if (!state.writable && !state.hasContent) return 'Connecting to dat network...'\n if (!state.warnings) state.warnings = []\n\n var dat = state.dat\n var stats = dat.stats.get()\n var title = (state.dat.resumed) ? '' : `Created new dat in ${dat.path}${path.sep}.dat\\n`\n var progressView\n\n if (state.writable || state.opts.showKey) {\n title += `${keyEl(dat.key)}\\n`\n }\n if (state.title) title += state.title\n else if (state.writable) title += 'Sharing dat'\n else title += 'Downloading dat'\n if (state.opts.sparse) title += `: ${state.opts.selectedFiles.length} ${pluralize('file', state.opts.selectedFiles.length)} (${pretty(state.selectedByteLength)})`\n else if (stats.version > 0) title += `: ${stats.files} ${pluralize('file', stats.file)} (${pretty(stats.byteLength)})`\n else if (stats.version === 0) title += ': (empty archive)'\n if (state.http && state.http.listening) title += `\\nServing files over http at http://localhost:${state.http.port}`\n\n if (!state.writable) {\n progressView = downloadUI(state)\n } else {\n if (state.opts.import) {\n progressView = importUI(state)\n } else {\n progressView = 'Not importing files.' // TODO: ?\n }\n }\n\n return output(`\n ${version(pkg.version)}\n ${title}\n ${state.joinNetwork ? '\\n' + networkUI(state) : ''}\n\n ${progressView}\n ${state.opts.sources ? sourcesUI(state) : ''}\n ${state.warnings ? warningsUI(state) : ''}\n ${state.exiting ? 'Exiting the Dat program...' : chalk.dim('Ctrl+C to Exit')}\n `)\n}\n" }, { "path": "src/ui/components/download.js", "content": "var output = require('neat-log/output')\nvar bar = require('progress-string')\n\nmodule.exports = networkUI\n\nfunction networkUI (state) {\n var stats = state.stats.get()\n var download = state.download\n if (!stats || !download) return ''\n\n var title = 'Downloading updates...'\n var downBar = makeBar()\n\n if (download.nsync) {\n if (state.opts.exit && state.dat.archive.version === 0) {\n return 'dat synced. There is no content in this archive.'\n }\n if (state.opts.exit && download.modified) {\n return `dat sync complete.\\nVersion ${stats.version}`\n }\n\n if (!download.modified && state.opts.exit) {\n title = `dat already in sync, waiting for updates.`\n } else {\n title = `dat synced, waiting for updates.`\n }\n }\n\n if (typeof state.opts.exit === 'number') {\n title = `dat synced, exiting in ${state.opts.exit} seconds.`\n }\n\n if (!stats.downloaded || !stats.length) {\n return '' // no metadata yet\n }\n\n return output(`\n ${title}\n ${downBar(stats.downloaded)}\n `)\n\n function makeBar () {\n var total = stats.length\n return bar({\n total: total,\n style: function (a, b) {\n return `[${a}${b}] ${(100 * stats.downloaded / total).toFixed(2)}%`\n }\n })\n }\n}\n" }, { "path": "src/ui/components/import-progress.js", "content": "var output = require('neat-log/output')\nvar pretty = require('prettier-bytes')\nvar bar = require('progress-string')\nvar cliTruncate = require('cli-truncate')\n\nmodule.exports = importUI\n\nfunction importUI (state) {\n var watch = state.opts.watch\n var importState = state.importer\n var indexSpeed = importState.indexSpeed ? `(${pretty(importState.indexSpeed)}/s)` : ''\n\n if (importState.count && !importState.count.done) {\n // dry run in progress\n if (!importState.count.files) return 'Checking for file updates...'\n return output(`\n Metadata created for ${importState.putDone.files} of ${importState.count.files} files ${indexSpeed}\n (Calculating file count...)\n ${fileImport(importState.fileImport)}\n `)\n } else if (importState.putDone.files >= importState.count.files) {\n // Initial import done\n if (!watch) return 'Archive metadata updated for all files.'\n return liveImport()\n }\n\n var total = importState.count.bytes\n var totalBar = bar({\n total: total,\n style: function (a, b) {\n return `[${a}${b}] ${(100 * importState.importedBytes / total).toFixed(0)}%`\n }\n })\n\n return output(`\n Creating metadata for ${importState.count.files} files ${indexSpeed}\n ${totalBar(importState.importedBytes)}\n ${fileImport(importState.fileImport)}\n `)\n\n function liveImport () {\n // Live import\n var imports = importState.liveImports.slice(1).slice(-7)\n return output(`\n Watching for file updates\n ${imports.reverse().map(function (file) { return fileImport(file) }).join('\\n')}\n `)\n }\n\n function fileImport (file) {\n if (!file) return ''\n if (file.type === 'del') return `DEL: ${file.src.name}`\n\n var total = file.src.stat.size\n var name = file.dst.name.substr(1) // remove '/' at start\n var size\n\n // >500 mb show progress\n if (total < 5e8 || !file.progress) size = `(${pretty(total)})`\n else size = `(${pretty(file.progress)} / ${pretty(total)})`\n return output(`\n ADD: ${cliTruncate(name, process.stdout.columns - 7 - size.length, { position: 'start' })} ${size}\n `)\n }\n}\n" }, { "path": "src/ui/components/network.js", "content": "var output = require('neat-log/output')\nvar pretty = require('prettier-bytes')\nvar pluralize = require('../elements/pluralize')\n\nmodule.exports = networkUI\n\nfunction networkUI (state) {\n var network = state.network\n var stats = state.stats\n\n if (!network) return ''\n var peers = stats.peers.total || 0\n // var complete = stats.peers.complete\n return output(`\n ${peers} ${pluralize('connection', peers)} ${speedUI()}\n `)\n\n function speedUI () {\n var output = '| '\n var speed = state.stats.network\n var upSpeed = speed.uploadSpeed || 0\n var downSpeed = speed.downloadSpeed || 0\n output += `Download ${pretty(downSpeed)}/s`\n output += ` Upload ${pretty(upSpeed)}/s `\n return output\n }\n}\n" }, { "path": "src/ui/components/sources.js", "content": "var output = require('neat-log/output')\nvar pretty = require('prettier-bytes')\nvar makeBar = require('progress-string')\n\nmodule.exports = peersUI\n\nfunction peersUI (state) {\n if (!state.network) return ''\n if (Object.keys(state.sources).length === 0) return ''\n\n var peers = state.sources\n // var stats = state.stats\n // var peerCount = stats.peers.total || 0\n // var complete = stats.peers.complete\n var info = Object.keys(peers).map(function (id, i) {\n return peerUI(peers[id], i)\n }).join('\\n')\n\n return `\\n${info}\\n`\n\n function peerUI (peer, i) {\n var progress = peer.getProgress()\n var bar = makeBar({\n total: 100,\n style: function (a, b) {\n return `[${a}${b}] ${(progress).toFixed(2)}%`\n }\n })\n var theBar = progress ? bar(progress) : '' // progress bar todo\n return output(`\n [${i}] ${peer.closed ? 'CLOSED' : peer.type}: ${peer.host}:${peer.port} ${pretty(peer.speed)}/s\n ${peer.error ? peer.error : theBar}\n `)\n }\n}\n" }, { "path": "src/ui/components/warnings.js", "content": "var chalk = require('chalk')\n\nmodule.exports = function (state) {\n var warning = ''\n state.warnings.forEach(function (message) {\n warning += `${chalk.yellow(`Warning: ${message}`)}\\n`\n })\n return warning\n}\n" }, { "path": "src/ui/create.js", "content": "var output = require('neat-log/output')\nvar pretty = require('prettier-bytes')\nvar chalk = require('chalk')\nvar importUI = require('./components/import-progress')\nvar keyEl = require('./elements/key')\nvar pluralize = require('./elements/pluralize')\n\nmodule.exports = createUI\n\nfunction createUI (state) {\n if (!state.dat) {\n return output(`\n Creating a Dat! Add information to your dat.json file:\n `)\n }\n\n var dat = state.dat\n var stats = dat.stats.get()\n var title = '\\n'\n var progressView\n var exitMsg = `\n Your dat is created! Run ${chalk.green('dat sync')} to share:\n ${keyEl(dat.key)}\n `\n if (!state.opts.import) {\n // set exiting right away\n state.exiting = true\n }\n\n if (!state.exiting) {\n // Only show key if not about to exit\n title = `${keyEl(dat.key)}\\n`\n }\n if (state.title) title += state.title\n\n if (stats.version > 0) title += `: ${stats.files} ${pluralize('file', stats.files)} (${pretty(stats.byteLength)})`\n else if (stats.version === 0) title += ': (empty archive)'\n\n if (state.opts.import) {\n progressView = importUI(state) + '\\n'\n } else {\n progressView = 'Not importing files.'\n }\n\n return output(`\n ${title}\n\n ${progressView}\n ${state.exiting ? exitMsg : chalk.dim('Ctrl+C to Exit')}\n `)\n}\n" }, { "path": "src/ui/elements/key.js", "content": "var stringKey = require('dat-encoding').toStr\nvar chalk = require('chalk')\n\nmodule.exports = function (key) {\n return `${chalk.blue(`dat://${stringKey(key)}`)}`\n}\n" }, { "path": "src/ui/elements/pluralize.js", "content": "module.exports = function pluralize (str, val) {\n return `${str}${val === 1 ? '' : 's'}`\n}\n" }, { "path": "src/ui/elements/version.js", "content": "var chalk = require('chalk')\n\nmodule.exports = function (version) {\n return `${chalk.green(`dat v${version}`)}`\n}\n" }, { "path": "src/ui/status.js", "content": "var output = require('neat-log/output')\nvar stringKey = require('dat-encoding').toStr\nvar pretty = require('prettier-bytes')\nvar chalk = require('chalk')\n\nmodule.exports = statusUI\n\nfunction statusUI (state) {\n if (!state.dat) return 'Starting Dat program...'\n\n var dat = state.dat\n var stats = dat.stats.get()\n\n return output(`\n ${chalk.blue('dat://' + stringKey(dat.key))}\n ${stats.files} files (${pretty(stats.byteLength)})\n Version: ${chalk.bold(stats.version)}\n `)\n}\n" }, { "path": "src/usage.js", "content": "module.exports = function (opts, help, usage) {\n if (opts.version) {\n var pkg = require('../package.json')\n console.error(pkg.version)\n process.exit(1)\n }\n var msg = `\n\n dat [] clone or sync link to \n dat create and sync dat in directory\n\nOther commands:\n dat create create empty dat and dat.json\n dat sync live sync files with the network\n dat clone [] download a dat via link to \n dat pull update dat & exit\n dat log log history for a dat\n dat status get key & info about a local dat\n dat keys [import/export] import and export private keys\n\nTroubleshooting & Help:\n dat help print this usage guide\n dat --help, -h print help for a specific command\n dat --version, -v print the dat version\n\n `\n console.error(msg)\n if (usage) {\n console.error('General Options:')\n console.error(usage)\n }\n console.error('Have fun using Dat! Learn more at docs.datproject.org')\n process.exit(0)\n}\n" }, { "path": "test/auth.js", "content": "var test = require('tape')\nvar path = require('path')\nvar fs = require('fs')\nvar rimraf = require('rimraf')\nvar mkdirp = require('mkdirp')\nvar spawn = require('./helpers/spawn')\nvar help = require('./helpers')\nvar authServer = require('./helpers/auth-server')\n\nvar dat = path.resolve(path.join(__dirname, '..', 'bin', 'cli.js'))\nvar baseTestDir = help.testFolder()\nvar fixtures = path.join(__dirname, 'fixtures')\n\nvar port = process.env.PORT || 3000\nvar SERVER = 'http://localhost:' + port\nvar config = path.join(__dirname, '.datrc-test')\nvar opts = ' --server=' + SERVER + ' --config=' + config\n\ndat += opts\nrimraf.sync(config)\n\nauthServer(port, function (err, server, closeServer) {\n if (err) throw err\n if (!server) return\n test('auth - whoami works when not logged in', function (t) {\n var cmd = dat + ' whoami '\n var st = spawn(t, cmd, { cwd: baseTestDir })\n st.stderr.match(function (output) {\n t.same(output.trim(), 'Not logged in.', 'printed correct output')\n return true\n })\n st.stdout.empty()\n st.end()\n })\n\n test('auth - register works', function (t) {\n var cmd = dat + ' register --email=hello@bob.com --password=joe --username=joe'\n var st = spawn(t, cmd, { cwd: baseTestDir })\n st.stdout.match(function (output) {\n t.same(output.trim(), 'Registered successfully.', 'output success message')\n return true\n })\n st.stderr.empty()\n st.end()\n })\n\n test('auth - login works', function (t) {\n var cmd = dat + ' login --email=hello@bob.com --password=joe'\n var st = spawn(t, cmd, { cwd: baseTestDir })\n st.stdout.match(function (output) {\n t.same(output.trim(), 'Logged in successfully.', 'output success message')\n return true\n })\n st.stderr.empty()\n st.end()\n })\n\n test('auth - whoami works', function (t) {\n var cmd = dat + ' whoami'\n var st = spawn(t, cmd, { cwd: baseTestDir })\n st.stdout.match(function (output) {\n t.same('hello@bob.com', output.trim(), 'email printed')\n return true\n })\n st.stderr.empty()\n st.end()\n })\n\n test('auth - publish before create fails', function (t) {\n var cmd = dat + ' publish'\n rimraf.sync(path.join(fixtures, '.dat'))\n var st = spawn(t, cmd, { cwd: fixtures })\n st.stdout.empty()\n st.stderr.match(function (output) {\n t.ok(output.indexOf('existing') > -1, 'Create archive before pub')\n return true\n })\n st.end()\n })\n\n test('auth - create dat to publish', function (t) {\n rimraf.sync(path.join(fixtures, '.dat'))\n rimraf.sync(path.join(fixtures, 'dat.json'))\n var cmd = dat + ' create --no-import'\n var st = spawn(t, cmd, { cwd: fixtures })\n st.stdout.match(function (output) {\n var link = help.matchLink(output)\n if (!link) return false\n t.ok(link, 'prints link')\n return true\n })\n st.stderr.empty()\n st.end()\n })\n\n test('auth - publish our awesome dat', function (t) {\n var cmd = dat + ' publish --name awesome'\n var st = spawn(t, cmd, { cwd: fixtures })\n st.stdout.match(function (output) {\n var published = output.indexOf('Successfully published') > -1\n if (!published) return false\n t.ok(published, 'published')\n return true\n })\n st.stderr.empty()\n st.end()\n })\n\n test('auth - publish our awesome dat with bad dat.json url', function (t) {\n fs.readFile(path.join(fixtures, 'dat.json'), function (err, contents) {\n t.ifError(err)\n var info = JSON.parse(contents)\n var oldUrl = info.url\n info.url = info.url.replace('e', 'a')\n fs.writeFile(path.join(fixtures, 'dat.json'), JSON.stringify(info), function (err) {\n t.ifError(err, 'error after write')\n var cmd = dat + ' publish --name awesome'\n var st = spawn(t, cmd, { cwd: fixtures })\n st.stdout.match(function (output) {\n var published = output.indexOf('Successfully published') > -1\n if (!published) return false\n t.ok(published, 'published')\n t.same(help.datJson(fixtures).url, oldUrl, 'has dat.json with url')\n return true\n })\n st.stderr.empty()\n st.end()\n })\n })\n })\n\n test('auth - clone from registry', function (t) {\n // MAKE SURE THESE MATCH WHAT is published above\n // TODO: be less lazy and make a publish helper\n var shortName = 'localhost:' + port + '/joe/awesome' // they'll never guess who wrote these tests\n var baseDir = path.join(baseTestDir, 'dat_registry_dir')\n mkdirp.sync(baseDir)\n var downloadDir = path.join(baseDir, shortName.split('/').pop())\n var cmd = dat + ' clone ' + shortName\n var st = spawn(t, cmd, { cwd: baseDir })\n st.stdout.match(function (output) {\n var lookingFor = output.indexOf('Looking for') > -1\n if (!lookingFor) return false\n t.ok(lookingFor, 'starts looking for peers')\n t.ok(output.indexOf(downloadDir) > -1, 'prints dir')\n st.kill()\n return true\n })\n st.stderr.empty()\n st.end(function () {\n rimraf.sync(downloadDir)\n })\n })\n\n test('auth - publish our awesome dat without a dat.json file', function (t) {\n rimraf(path.join(fixtures, 'dat.json'), function (err) {\n t.ifError(err)\n var cmd = dat + ' publish --name another-awesome'\n var st = spawn(t, cmd, { cwd: fixtures })\n st.stdout.match(function (output) {\n var published = output.indexOf('Successfully published') > -1\n if (!published) return false\n t.ok(published, 'published')\n t.same(help.datJson(fixtures).name, 'another-awesome', 'has dat.json with name')\n return true\n })\n st.stderr.empty()\n st.end(function () {\n rimraf.sync(path.join(fixtures, '.dat'))\n })\n })\n })\n\n test('auth - bad clone from registry', function (t) {\n var shortName = 'localhost:' + port + '/joe/not-at-all-awesome'\n var baseDir = path.join(baseTestDir, 'dat_registry_dir_too')\n mkdirp.sync(baseDir)\n var downloadDir = path.join(baseDir, shortName.split('/').pop())\n var cmd = dat + ' clone ' + shortName\n var st = spawn(t, cmd, { cwd: baseDir })\n st.stderr.match(function (output) {\n t.same(output.trim(), 'Dat with that name not found.', 'not found')\n st.kill()\n return true\n })\n st.stdout.empty()\n st.end(function () {\n rimraf.sync(downloadDir)\n })\n })\n\n test('auth - logout works', function (t) {\n var cmd = dat + ' logout'\n var st = spawn(t, cmd, { cwd: baseTestDir })\n st.stdout.match(function (output) {\n t.same('Logged out.', output.trim(), 'output correct')\n return true\n })\n st.stderr.empty()\n st.end()\n })\n\n test('auth - logout prints correctly when trying to log out twice', function (t) {\n var cmd = dat + ' logout'\n var st = spawn(t, cmd, { cwd: baseTestDir })\n st.stderr.match(function (output) {\n t.same('Not logged in.', output.trim(), 'output correct')\n return true\n })\n st.stdout.empty()\n st.end()\n })\n\n test('auth - whoami works after logging out', function (t) {\n var cmd = dat + ' whoami'\n var st = spawn(t, cmd, { cwd: baseTestDir })\n st.stderr.match(function (output) {\n t.same('Not logged in.', output.trim())\n return true\n })\n st.stdout.empty()\n st.end()\n })\n\n test.onFinish(function () {\n closeServer(function () {\n fs.unlink(config, function () {\n // done!\n })\n })\n })\n})\n" }, { "path": "test/clone.js", "content": "var fs = require('fs')\nvar path = require('path')\nvar test = require('tape')\nvar tempDir = require('temporary-directory')\nvar spawn = require('./helpers/spawn.js')\nvar help = require('./helpers')\n\nvar dat = path.resolve(path.join(__dirname, '..', 'bin', 'cli.js'))\n\ntest('clone - default opts', function (t) {\n help.shareFixtures(function (_, shareDat) {\n var key = shareDat.key.toString('hex')\n tempDir(function (_, dir, cleanup) {\n var cmd = dat + ' clone ' + key\n var st = spawn(t, cmd, { cwd: dir })\n var datDir = path.join(dir, key)\n\n st.stdout.match(function (output) {\n var downloadFinished = output.indexOf('Exiting') > -1\n if (!downloadFinished) return false\n\n var stats = shareDat.stats.get()\n var fileRe = new RegExp(stats.files + ' files')\n var bytesRe = new RegExp(/1\\.\\d KB/)\n\n t.ok(output.match(fileRe), 'total size: files okay')\n t.ok(output.match(bytesRe), 'total size: bytes okay')\n t.ok(help.isDir(datDir), 'creates download directory')\n\n var fileList = help.fileList(datDir).join(' ')\n var hasCsvFile = fileList.indexOf('all_hour.csv') > -1\n t.ok(hasCsvFile, 'csv file downloaded')\n var hasDatFolder = fileList.indexOf('.dat') > -1\n t.ok(hasDatFolder, '.dat folder created')\n var hasSubDir = fileList.indexOf('folder') > -1\n t.ok(hasSubDir, 'folder created')\n var hasNestedDir = fileList.indexOf('nested') > -1\n t.ok(hasNestedDir, 'nested folder created')\n var hasHelloFile = fileList.indexOf('hello.txt') > -1\n t.ok(hasHelloFile, 'hello.txt file downloaded')\n\n st.kill()\n return true\n })\n st.succeeds('exits after finishing download')\n st.stderr.empty()\n st.end(function () {\n cleanup()\n shareDat.close()\n })\n })\n })\n})\n\n// Right now we aren't forcing this\n// test('clone - errors on existing dir', function (t) {\n// tempDir(function (_, dir, cleanup) {\n// // make empty dat in directory\n// Dat(dir, function (err, shareDat) {\n// t.error(err, 'no error')\n// // Try to clone to same dir\n// shareDat.close(function () {\n// var cmd = dat + ' clone ' + shareDat.key.toString('hex') + ' ' + dir\n// var st = spawn(t, cmd)\n// st.stdout.empty()\n// st.stderr.match(function (output) {\n// t.same(output.trim(), 'Existing archive in this directory. Use pull or sync to update.', 'Existing archive.')\n// st.kill()\n// return true\n// })\n// st.end(cleanup)\n// })\n// })\n// })\n// })\n\ntest('clone - specify dir', function (t) {\n help.shareFixtures(function (_, shareDat) {\n tempDir(function (_, dir, cleanup) {\n var key = shareDat.key.toString('hex')\n var customDir = 'my_dir'\n var cmd = dat + ' clone ' + key + ' ' + customDir\n var st = spawn(t, cmd, { cwd: dir })\n st.stdout.match(function (output) {\n var downloadFinished = output.indexOf('Exiting') > -1\n if (!downloadFinished) return false\n\n t.ok(help.isDir(path.join(dir, customDir)), 'creates download directory')\n st.kill()\n return true\n })\n st.succeeds('exits after finishing download')\n st.stderr.empty()\n st.end(function () {\n cleanup()\n shareDat.close()\n })\n })\n })\n})\n\ntest('clone - dat:// link', function (t) {\n help.shareFixtures(function (_, shareDat) {\n tempDir(function (_, dir, cleanup) {\n var key = 'dat://' + shareDat.key.toString('hex') + '/'\n var cmd = dat + ' clone ' + key + ' '\n var downloadDir = path.join(dir, shareDat.key.toString('hex'))\n var st = spawn(t, cmd, { cwd: dir })\n st.stdout.match(function (output) {\n var downloadFinished = output.indexOf('Exiting') > -1\n if (!downloadFinished) return false\n\n t.ok(help.isDir(path.join(downloadDir)), 'creates download directory')\n st.kill()\n return true\n })\n st.succeeds('exits after finishing download')\n st.stderr.empty()\n st.end(function () {\n cleanup()\n shareDat.close()\n })\n })\n })\n})\n\ntest('clone - datproject.org/key link', function (t) {\n help.shareFixtures(function (_, shareDat) {\n tempDir(function (_, dir, cleanup) {\n var key = 'datproject.org/' + shareDat.key.toString('hex') + '/'\n var cmd = dat + ' clone ' + key + ' '\n var downloadDir = path.join(dir, shareDat.key.toString('hex'))\n var st = spawn(t, cmd, { cwd: dir })\n st.stdout.match(function (output) {\n var downloadFinished = output.indexOf('Exiting') > -1\n if (!downloadFinished) return false\n\n t.ok(help.isDir(path.join(downloadDir)), 'creates download directory')\n st.kill()\n return true\n })\n st.succeeds('exits after finishing download')\n st.stderr.empty()\n st.end(function () {\n cleanup()\n shareDat.close()\n })\n })\n })\n})\n\n// TODO: fix --temp for clones\n// test('clone - with --temp', function (t) {\n// // cmd: dat clone \n// help.shareFixtures(function (_, fixturesDat) {\n// shareDat = fixturesDat\n// var key = shareDat.key.toString('hex')\n// var cmd = dat + ' clone ' + key + ' --temp'\n// var st = spawn(t, cmd, {cwd: baseTestDir})\n// var datDir = path.join(baseTestDir, key)\n// st.stdout.match(function (output) {\n// var downloadFinished = output.indexOf('Download Finished') > -1\n// if (!downloadFinished) return false\n\n// var stats = shareDat.stats.get()\n// var fileRe = new RegExp(stats.filesTotal + ' files')\n// var bytesRe = new RegExp(/1\\.\\d{1,2} kB/)\n\n// t.ok(help.matchLink(output), 'prints link')\n// t.ok(output.indexOf('dat-download-folder/' + key) > -1, 'prints dir')\n// t.ok(output.match(fileRe), 'total size: files okay')\n// t.ok(output.match(bytesRe), 'total size: bytes okay')\n// t.ok(help.isDir(datDir), 'creates download directory')\n\n// var fileList = help.fileList(datDir).join(' ')\n// var hasCsvFile = fileList.indexOf('all_hour.csv') > -1\n// t.ok(hasCsvFile, 'csv file downloaded')\n// var hasDatFolder = fileList.indexOf('.dat') > -1\n// t.ok(!hasDatFolder, '.dat folder not created')\n// var hasSubDir = fileList.indexOf('folder') > -1\n// t.ok(hasSubDir, 'folder created')\n// var hasNestedDir = fileList.indexOf('nested') > -1\n// t.ok(hasNestedDir, 'nested folder created')\n// var hasHelloFile = fileList.indexOf('hello.txt') > -1\n// t.ok(hasHelloFile, 'hello.txt file downloaded')\n\n// st.kill()\n// return true\n// })\n// st.succeeds('exits after finishing download')\n// st.stderr.empty()\n// st.end()\n// })\n// })\n\ntest('clone - invalid link', function (t) {\n var key = 'best-key-ever'\n var cmd = dat + ' clone ' + key\n tempDir(function (_, dir, cleanup) {\n var st = spawn(t, cmd, { cwd: dir })\n var datDir = path.join(dir, key)\n st.stderr.match(function (output) {\n var error = output.indexOf('Could not resolve link') > -1\n if (!error) return false\n t.ok(error, 'has error')\n t.ok(!help.isDir(datDir), 'download dir removed')\n st.kill()\n return true\n })\n st.end(cleanup)\n })\n})\n\ntest('clone - shortcut/stateless clone', function (t) {\n help.shareFixtures(function (_, shareDat) {\n var key = shareDat.key.toString('hex')\n tempDir(function (_, dir, cleanup) {\n var datDir = path.join(dir, key)\n var cmd = dat + ' ' + key + ' ' + datDir + ' --exit'\n var st = spawn(t, cmd)\n\n st.stdout.match(function (output) {\n var downloadFinished = output.indexOf('Exiting') > -1\n if (!downloadFinished) return false\n\n t.ok(help.isDir(datDir), 'creates download directory')\n\n var fileList = help.fileList(datDir).join(' ')\n var hasCsvFile = fileList.indexOf('all_hour.csv') > -1\n t.ok(hasCsvFile, 'csv file downloaded')\n var hasDatFolder = fileList.indexOf('.dat') > -1\n t.ok(hasDatFolder, '.dat folder created')\n var hasSubDir = fileList.indexOf('folder') > -1\n t.ok(hasSubDir, 'folder created')\n var hasNestedDir = fileList.indexOf('nested') > -1\n t.ok(hasNestedDir, 'nested folder created')\n var hasHelloFile = fileList.indexOf('hello.txt') > -1\n t.ok(hasHelloFile, 'hello.txt file downloaded')\n\n st.kill()\n return true\n })\n st.succeeds('exits after finishing download')\n st.stderr.empty()\n st.end(function () {\n cleanup()\n shareDat.close()\n })\n })\n })\n})\n\n// TODO: fix this\n// test('clone - hypercore link', function (t) {\n// help.shareFeed(function (_, key, close) {\n// tempDir(function (_, dir, cleanup) {\n// var cmd = dat + ' clone ' + key\n// var st = spawn(t, cmd, {cwd: dir})\n// var datDir = path.join(dir, key)\n// st.stderr.match(function (output) {\n// var error = output.indexOf('not a Dat Archive') > -1\n// if (!error) return false\n// t.ok(error, 'has error')\n// t.ok(!help.isDir(datDir), 'download dir removed')\n// st.kill()\n// return true\n// })\n// st.end(function () {\n// cleanup()\n// close()\n// })\n// })\n// })\n// })\n\ntest('clone - specify directory containing dat.json', function (t) {\n help.shareFixtures(function (_, shareDat) {\n tempDir(function (_, dir, cleanup) {\n fs.writeFileSync(path.join(dir, 'dat.json'), JSON.stringify({ url: shareDat.key.toString('hex') }), 'utf8')\n\n // dat clone /dir\n var cmd = dat + ' clone ' + dir\n var st = spawn(t, cmd)\n var datDir = dir\n\n st.stdout.match(function (output) {\n var downloadFinished = output.indexOf('Exiting') > -1\n if (!downloadFinished) return false\n\n var fileList = help.fileList(datDir).join(' ')\n var hasCsvFile = fileList.indexOf('all_hour.csv') > -1\n t.ok(hasCsvFile, 'csv file downloaded')\n var hasDatFolder = fileList.indexOf('.dat') > -1\n t.ok(hasDatFolder, '.dat folder created')\n var hasSubDir = fileList.indexOf('folder') > -1\n t.ok(hasSubDir, 'folder created')\n var hasNestedDir = fileList.indexOf('nested') > -1\n t.ok(hasNestedDir, 'nested folder created')\n var hasHelloFile = fileList.indexOf('hello.txt') > -1\n t.ok(hasHelloFile, 'hello.txt file downloaded')\n\n st.kill()\n return true\n })\n st.succeeds('exits after finishing download')\n st.stderr.empty()\n st.end(function () {\n cleanup()\n shareDat.close()\n })\n })\n })\n})\n\ntest('clone - specify directory containing dat.json with cwd', function (t) {\n help.shareFixtures(function (_, shareDat) {\n tempDir(function (_, dir, cleanup) {\n fs.writeFileSync(path.join(dir, 'dat.json'), JSON.stringify({ url: shareDat.key.toString('hex') }), 'utf8')\n\n // cd dir && dat clone /dir/dat.json\n var cmd = dat + ' clone ' + dir\n var st = spawn(t, cmd, { cwd: dir })\n var datDir = dir\n\n st.stdout.match(function (output) {\n var downloadFinished = output.indexOf('Exiting') > -1\n if (!downloadFinished) return false\n\n var fileList = help.fileList(datDir).join(' ')\n var hasCsvFile = fileList.indexOf('all_hour.csv') > -1\n t.ok(hasCsvFile, 'csv file downloaded')\n var hasDatFolder = fileList.indexOf('.dat') > -1\n t.ok(hasDatFolder, '.dat folder created')\n var hasSubDir = fileList.indexOf('folder') > -1\n t.ok(hasSubDir, 'folder created')\n var hasNestedDir = fileList.indexOf('nested') > -1\n t.ok(hasNestedDir, 'nested folder created')\n var hasHelloFile = fileList.indexOf('hello.txt') > -1\n t.ok(hasHelloFile, 'hello.txt file downloaded')\n\n st.kill()\n return true\n })\n st.succeeds('exits after finishing download')\n st.stderr.empty()\n st.end(function () {\n cleanup()\n shareDat.close()\n })\n })\n })\n})\n\ntest('clone - specify dat.json path', function (t) {\n help.shareFixtures(function (_, shareDat) {\n tempDir(function (_, dir, cleanup) {\n var datJsonPath = path.join(dir, 'dat.json')\n fs.writeFileSync(datJsonPath, JSON.stringify({ url: shareDat.key.toString('hex') }), 'utf8')\n\n // dat clone /dir/dat.json\n var cmd = dat + ' clone ' + datJsonPath\n var st = spawn(t, cmd)\n var datDir = dir\n\n st.stdout.match(function (output) {\n var downloadFinished = output.indexOf('Exiting') > -1\n if (!downloadFinished) return false\n\n var fileList = help.fileList(datDir).join(' ')\n var hasCsvFile = fileList.indexOf('all_hour.csv') > -1\n t.ok(hasCsvFile, 'csv file downloaded')\n var hasDatFolder = fileList.indexOf('.dat') > -1\n t.ok(hasDatFolder, '.dat folder created')\n var hasSubDir = fileList.indexOf('folder') > -1\n t.ok(hasSubDir, 'folder created')\n var hasNestedDir = fileList.indexOf('nested') > -1\n t.ok(hasNestedDir, 'nested folder created')\n var hasHelloFile = fileList.indexOf('hello.txt') > -1\n t.ok(hasHelloFile, 'hello.txt file downloaded')\n\n st.kill()\n return true\n })\n st.succeeds('exits after finishing download')\n st.stderr.empty()\n st.end(function () {\n cleanup()\n shareDat.close()\n })\n })\n })\n})\n\ntest('clone - specify dat.json path with cwd', function (t) {\n help.shareFixtures(function (_, shareDat) {\n tempDir(function (_, dir, cleanup) {\n var datJsonPath = path.join(dir, 'dat.json')\n fs.writeFileSync(datJsonPath, JSON.stringify({ url: shareDat.key.toString('hex') }), 'utf8')\n\n // cd /dir && dat clone /dir/dat.json\n var cmd = dat + ' clone ' + datJsonPath\n var st = spawn(t, cmd, { cwd: dir })\n var datDir = dir\n\n st.stdout.match(function (output) {\n var downloadFinished = output.indexOf('Exiting') > -1\n if (!downloadFinished) return false\n\n var fileList = help.fileList(datDir).join(' ')\n var hasCsvFile = fileList.indexOf('all_hour.csv') > -1\n t.ok(hasCsvFile, 'csv file downloaded')\n var hasDatFolder = fileList.indexOf('.dat') > -1\n t.ok(hasDatFolder, '.dat folder created')\n var hasSubDir = fileList.indexOf('folder') > -1\n t.ok(hasSubDir, 'folder created')\n var hasNestedDir = fileList.indexOf('nested') > -1\n t.ok(hasNestedDir, 'nested folder created')\n var hasHelloFile = fileList.indexOf('hello.txt') > -1\n t.ok(hasHelloFile, 'hello.txt file downloaded')\n\n st.kill()\n return true\n })\n st.succeeds('exits after finishing download')\n st.stderr.empty()\n st.end(function () {\n cleanup()\n shareDat.close()\n })\n })\n })\n})\n\ntest('clone - specify dat.json + directory', function (t) {\n help.shareFixtures(function (_, shareDat) {\n tempDir(function (_, dir, cleanup) {\n var datDir = path.join(dir, 'clone-dest')\n var datJsonPath = path.join(dir, 'dat.json') // make dat.json in different dir\n\n fs.mkdirSync(datDir)\n fs.writeFileSync(datJsonPath, JSON.stringify({ url: shareDat.key.toString('hex') }), 'utf8')\n\n // dat clone /dir/dat.json /dir/clone-dest\n var cmd = dat + ' clone ' + datJsonPath + ' ' + datDir\n var st = spawn(t, cmd)\n\n st.stdout.match(function (output) {\n var downloadFinished = output.indexOf('Exiting') > -1\n if (!downloadFinished) return false\n\n var fileList = help.fileList(datDir).join(' ')\n var hasCsvFile = fileList.indexOf('all_hour.csv') > -1\n t.ok(hasCsvFile, 'csv file downloaded')\n var hasDatFolder = fileList.indexOf('.dat') > -1\n t.ok(hasDatFolder, '.dat folder created')\n var hasSubDir = fileList.indexOf('folder') > -1\n t.ok(hasSubDir, 'folder created')\n var hasNestedDir = fileList.indexOf('nested') > -1\n t.ok(hasNestedDir, 'nested folder created')\n var hasHelloFile = fileList.indexOf('hello.txt') > -1\n t.ok(hasHelloFile, 'hello.txt file downloaded')\n\n st.kill()\n return true\n })\n st.succeeds('exits after finishing download')\n st.stderr.empty()\n st.end(function () {\n cleanup()\n shareDat.close()\n })\n })\n })\n})\n" }, { "path": "test/create.js", "content": "var fs = require('fs')\nvar path = require('path')\nvar test = require('tape')\nvar tempDir = require('temporary-directory')\nvar Dat = require('dat-node')\nvar spawn = require('./helpers/spawn.js')\nvar help = require('./helpers')\n\nvar dat = path.resolve(path.join(__dirname, '..', 'bin', 'cli.js'))\nvar fixtures = path.join(__dirname, 'fixtures')\n\n// os x adds this if you view the fixtures in finder and breaks the file count assertions\ntry { fs.unlinkSync(path.join(fixtures, '.DS_Store')) } catch (e) { /* ignore error */ }\n\n// start without dat.json\ntry { fs.unlinkSync(path.join(fixtures, 'dat.json')) } catch (e) { /* ignore error */ }\n\ntest('create - default opts no import', function (t) {\n tempDir(function (_, dir, cleanup) {\n var cmd = dat + ' create --title data --description thing'\n var st = spawn(t, cmd, { cwd: dir })\n\n st.stdout.match(function (output) {\n var datCreated = output.indexOf('Created empty Dat') > -1\n if (!datCreated) return false\n\n t.ok(help.isDir(path.join(dir, '.dat')), 'creates dat directory')\n\n st.kill()\n return true\n })\n st.succeeds('exits after create finishes')\n st.stderr.empty()\n st.end(cleanup)\n })\n})\n\ntest('create - errors on existing archive', function (t) {\n tempDir(function (_, dir, cleanup) {\n Dat(dir, function (err, dat) {\n t.error(err, 'no error')\n dat.close(function () {\n var cmd = dat + ' create --title data --description thing'\n var st = spawn(t, cmd, { cwd: dir })\n st.stderr.match(function (output) {\n t.ok(output, 'errors')\n st.kill()\n return true\n })\n st.end()\n })\n })\n })\n})\n\ntest('create - sync after create ok', function (t) {\n tempDir(function (_, dir, cleanup) {\n var cmd = dat + ' create --title data --description thing'\n var st = spawn(t, cmd, { cwd: dir, end: false })\n st.stdout.match(function (output) {\n var connected = output.indexOf('Created empty Dat') > -1\n if (!connected) return false\n doSync()\n return true\n })\n\n function doSync () {\n var cmd = dat + ' sync '\n var st = spawn(t, cmd, { cwd: dir })\n\n st.stdout.match(function (output) {\n var connected = output.indexOf('Sharing') > -1\n if (!connected) return false\n st.kill()\n return true\n })\n st.stderr.empty()\n st.end(cleanup)\n }\n })\n})\n\ntest('create - init alias', function (t) {\n tempDir(function (_, dir, cleanup) {\n var cmd = dat + ' init --title data --description thing'\n var st = spawn(t, cmd, { cwd: dir })\n\n st.stdout.match(function (output) {\n var datCreated = output.indexOf('Created empty Dat') > -1\n if (!datCreated) return false\n\n t.ok(help.isDir(path.join(dir, '.dat')), 'creates dat directory')\n\n st.kill()\n return true\n })\n st.succeeds('exits after create finishes')\n st.stderr.empty()\n st.end(cleanup)\n })\n})\n\ntest('create - with path', function (t) {\n tempDir(function (_, dir, cleanup) {\n var cmd = dat + ' init ' + dir + ' --title data --description thing'\n var st = spawn(t, cmd)\n st.stdout.match(function (output) {\n var datCreated = output.indexOf('Created empty Dat') > -1\n if (!datCreated) return false\n\n t.ok(help.isDir(path.join(dir, '.dat')), 'creates dat directory')\n\n st.kill()\n return true\n })\n st.succeeds('exits after create finishes')\n st.stderr.empty()\n st.end(cleanup)\n })\n})\n" }, { "path": "test/dat-node.js", "content": "var test = require('tape')\nvar ram = require('random-access-memory')\nvar Dat = require('..')\n\ntest('dat-node: require dat-node + make a dat', function (t) {\n Dat(ram, function (err, dat) {\n t.error(err, 'no error')\n t.ok(dat, 'makes dat')\n t.pass('yay')\n t.end()\n })\n})\n" }, { "path": "test/doctor.js", "content": "// var path = require('path')\n// var test = require('tape')\n// var spawn = require('./helpers/spawn.js')\n// var help = require('./helpers')\n//\n\n// TODO\n// dat-doctor requires interactive testing right now...\n\n// var dat = path.resolve(path.join(__dirname, '..', 'bin', 'cli.js'))\n\n// test('misc - doctor option works ', function (t) {\n// var st = spawn(t, dat + ' doctor', {end: false})\n// st.stderr.match(function (output) {\n// var readyPeer = output.indexOf('Waiting for incoming connections') > -1\n// if (!readyPeer) return false\n\n// if (!process.env.TRAVIS) {\n// // Not working on v4/v7 travis but can't reproduce locally\n// t.ok(output.indexOf('UTP') > -1, 'doctor connects to public peer via UTP')\n// }\n// t.ok(output.indexOf('TCP') > -1, 'doctor connects to public peer via TCP')\n\n// var key = help.matchLink(output)\n// startPhysiciansAssistant(key)\n// return true\n// }, 'doctor started')\n\n// function startPhysiciansAssistant (link) {\n// var assist = spawn(t, dat + ' doctor ' + link, {end: false})\n// assist.stderr.match(function (output) {\n// var readyPeer = output.indexOf('Waiting for incoming connections') > -1\n// if (!readyPeer) return false\n\n// t.same(help.matchLink(output), link, 'key of peer matches')\n// t.ok(readyPeer, 'starts looking for peers')\n// t.skip(output.indexOf('Remote peer echoed expected data back') > -1, 'echo data back')\n// st.kill()\n// return true\n// })\n// assist.end(function () {\n// t.end()\n// })\n// }\n// })\n" }, { "path": "test/fixtures/all_hour.csv", "content": "time,latitude,longitude,depth,mag,magType,nst,gap,dmin,rms,net,id,updated,place,type\n2014-04-30T03:34:57.000Z,60.0366,-141.2214,14.6,1.4,ml,,,,1.51,ak,ak11246293,2014-04-30T03:39:27.956Z,\"67km E of Cape Yakataga, Alaska\",earthquake\n2014-04-30T03:16:54.860Z,33.9233322,-117.9376678,0.81,2.4,ml,71,51,0.02487,0.35,ci,ci37218696,2014-04-30T03:35:24.239Z,\"1km SE of La Habra, California\",earthquake\n2014-04-30T03:03:09.000Z,61.126,-149.7035,28.2,1,ml,,,,0.75,ak,ak11246291,2014-04-30T03:09:51.716Z,\"14km SE of Anchorage, Alaska\",earthquake\n2014-04-30T02:57:51.800Z,37.3798,-122.1912,4.5,1.3,Md,8,104.4,0.01796631,0.02,nc,nc72212216,2014-04-30T03:28:05.271Z,\"2km SSE of Ladera, California\",earthquake\n2014-04-30T02:51:26.000Z,60.2062,-147.7298,0,1.8,ml,,,,0.1,ak,ak11246289,2014-04-30T02:54:55.916Z,\"82km SE of Whittier, Alaska\",earthquake\n2014-04-30T02:48:54.000Z,60.4899,-149.4879,28.5,1.4,ml,,,,0.54,ak,ak11246287,2014-04-30T02:54:51.149Z,\"36km N of Bear Creek, Alaska\",earthquake\n2014-04-30T02:47:44.100Z,38.819,-122.7952,3.3,0.8,Md,12,75.6,0.00898315,0.02,nc,nc72212211,2014-04-30T03:17:09.173Z,\"5km NW of The Geysers, California\",earthquake\n2014-04-30T02:46:11.100Z,37.9812,-122.054,14.7,2.1,Md,35,126,0.09881468,0.09,nc,nc72212206,2014-04-30T03:38:06.391Z,\"1km E of Pacheco, California\",earthquake\n2014-04-30T02:44:49.000Z,61.3482,-151.3611,48.5,1.7,ml,,,,0.92,ak,ak11246285,2014-04-30T02:54:49.809Z,\"73km N of Nikiski, Alaska\",earthquake\n" }, { "path": "test/fixtures/folder/nested/hello.txt", "content": "code for science and society" }, { "path": "test/helpers/auth-server.js", "content": "var path = require('path')\nvar rimraf = require('rimraf')\nvar Server, initDb\ntry {\n Server = require('dat-registry-api/server')\n initDb = require('dat-registry-api/server/database/init')\n} catch (e) {\n console.log('Disabling auth tests, run `npm install dat-registry-api` to enable them.')\n}\n\nmodule.exports = createServer\n\nfunction createServer (port, cb) {\n if (!Server || !initDb) return cb(null)\n var config = {\n mixpanel: 'nothing',\n email: {\n fromEmail: 'hi@example.com'\n },\n township: {\n secret: 'very secret code',\n db: path.join(__dirname, '..', 'test-township.db')\n },\n db: {\n dialect: 'sqlite3',\n connection: { filename: path.join(__dirname, '..', 'test-sqlite.db') },\n useNullAsDefault: true\n },\n archiver: path.join(__dirname, '..', 'test-archiver'),\n whitelist: false,\n port: port || 8888\n }\n rimraf.sync(config.archiver)\n rimraf.sync(config.db.connection.filename)\n rimraf.sync(config.township.db)\n\n initDb(config.db, function (err, db) {\n if (err) return cb(err)\n\n const server = Server(config, db)\n server.listen(config.port, function () {\n console.log('listening', config.port)\n })\n\n cb(null, server, close)\n\n function close (cb) {\n server.close(function () {\n rimraf.sync(config.township.db)\n rimraf.sync(config.db.connection.filename)\n process.exit()\n })\n }\n })\n}\n" }, { "path": "test/helpers/index.js", "content": "var fs = require('fs')\nvar os = require('os')\nvar path = require('path')\nvar mkdirp = require('mkdirp')\nvar rimraf = require('rimraf')\nvar encoding = require('dat-encoding')\nvar recursiveReadSync = require('recursive-readdir-sync')\nvar Dat = require('dat-node')\nvar fetch = require('node-fetch')\n\nmodule.exports.matchLink = matchDatLink\nmodule.exports.isDir = isDir\nmodule.exports.testFolder = newTestFolder\nmodule.exports.datJson = datJson\nmodule.exports.shareFixtures = shareFixtures\nmodule.exports.fileList = fileList\nmodule.exports.fetchText = fetchText\n\nfunction shareFixtures (opts, cb) {\n if (typeof opts === 'function') return shareFixtures(null, opts)\n opts = opts || {}\n var fixtures = path.join(__dirname, '..', 'fixtures')\n // os x adds this if you view the fixtures in finder and breaks the file count assertions\n try { fs.unlinkSync(path.join(fixtures, '.DS_Store')) } catch (e) { /* ignore error */ }\n if (opts.resume !== true) rimraf.sync(path.join(fixtures, '.dat'))\n Dat(fixtures, {}, function (err, dat) {\n if (err) throw err\n dat.trackStats()\n dat.joinNetwork()\n if (opts.import === false) return cb(null, dat)\n dat.importFiles({ watch: false }, function (err) {\n if (err) throw err\n cb(null, dat)\n })\n })\n}\n\nfunction fileList (dir) {\n try {\n return recursiveReadSync(dir)\n } catch (e) {\n return []\n }\n}\n\nfunction newTestFolder () {\n var tmpdir = path.join(os.tmpdir(), 'dat-download-folder')\n rimraf.sync(tmpdir)\n mkdirp.sync(tmpdir)\n return tmpdir\n}\n\nfunction matchDatLink (str) {\n var match = str.match(/[A-Za-z0-9]{64}/)\n if (!match) return false\n var key\n try {\n key = encoding.toStr(match[0].trim())\n } catch (e) {\n return false\n }\n return key\n}\n\nfunction datJson (filepath) {\n try {\n return JSON.parse(fs.readFileSync(path.join(filepath, 'dat.json')))\n } catch (e) {\n return {}\n }\n}\n\nfunction isDir (dir) {\n try {\n return fs.statSync(dir).isDirectory()\n } catch (e) {\n return false\n }\n}\n\nfunction fetchText (url) {\n let resp = {}\n return fetch(url).then(function (res) {\n resp = res\n return resp.text()\n }).then(function (body) {\n return { resp, body, error: null }\n }).catch(function (error) {\n return { error, resp: {}, body: null }\n })\n}\n" }, { "path": "test/helpers/spawn.js", "content": "var spawn = require('tape-spawn')\nvar fs = require('fs')\n\n// happens once at require time\n// https://github.com/AndreasMadsen/execspawn/issues/2\nvar hasBash = fs.existsSync('/bin/bash')\n\nmodule.exports = function (t, cmd, opts) {\n opts = opts || {}\n if (hasBash) opts.shell = '/bin/bash' // override default of /bin/sh\n return spawn(t, cmd, opts)\n}\n" }, { "path": "test/http.js", "content": "var path = require('path')\nvar test = require('tape')\nvar rimraf = require('rimraf')\nvar spawn = require('./helpers/spawn.js')\nvar { fetchText } = require('./helpers')\n\nvar dat = path.resolve(path.join(__dirname, '..', 'bin', 'cli.js'))\nvar fixtures = path.join(__dirname, 'fixtures')\n\ntest('http - share with http', function (t) {\n rimraf.sync(path.join(fixtures, '.dat'))\n var cmd = dat + ' share --http'\n var st = spawn(t, cmd, { cwd: fixtures })\n\n st.stdout.match(function (output) {\n var sharingHttp = output.indexOf('Serving files over http') > -1\n if (!sharingHttp) return false\n\n fetchText('http://localhost:8080').then(function ({ resp, body, error }) {\n t.error(error, 'no error')\n t.ok(resp.status === 200, 'okay status')\n t.ok(body)\n\n fetchText('http://localhost:8080/folder/nested/hello.txt').then(function ({ resp, body, error }) {\n t.error(error, 'no error')\n t.ok(resp.status === 200, 'okay status')\n t.same(body, 'code for science and society', 'body of file okay')\n\n st.kill()\n })\n })\n return true\n })\n st.stderr.empty()\n st.end()\n})\n\ntest('http - share with http other port', function (t) {\n rimraf.sync(path.join(fixtures, '.dat'))\n var cmd = dat + ' share --http 3333'\n var st = spawn(t, cmd, { cwd: fixtures })\n\n st.stdout.match(function (output) {\n var sharingHttp = output.indexOf('Serving files over http') > -1\n if (!sharingHttp) return false\n\n fetchText('http://localhost:3333').then(function ({ resp, body, error }) {\n t.error(error, 'no error')\n t.ok(resp.status === 200, 'okay status')\n t.ok(body)\n\n fetchText('http://localhost:3333/folder/nested/hello.txt').then(function ({ resp, body, error }) {\n t.error(error, 'no error')\n t.ok(resp.status === 200, 'okay status')\n t.same(body, 'code for science and society', 'body of file okay')\n\n st.kill()\n })\n })\n return true\n })\n st.stderr.empty()\n st.end()\n})\n" }, { "path": "test/keys.js", "content": "var fs = require('fs')\nvar path = require('path')\nvar test = require('tape')\nvar rimraf = require('rimraf')\nvar tempDir = require('temporary-directory')\nvar spawn = require('./helpers/spawn.js')\nvar help = require('./helpers')\n\nvar dat = path.resolve(path.join(__dirname, '..', 'bin', 'cli.js'))\nvar fixtures = path.join(__dirname, 'fixtures')\n\ntest('keys - print keys', function (t) {\n help.shareFixtures(function (_, shareDat) {\n shareDat.close(function () {\n var cmd = dat + ' keys '\n var st = spawn(t, cmd, { cwd: fixtures })\n\n st.stdout.match(function (output) {\n if (output.indexOf('dat://') === -1) return false\n t.ok(output.indexOf(shareDat.key.toString('hex') > -1), 'prints key')\n st.kill()\n return true\n })\n st.stderr.empty()\n st.end()\n })\n })\n})\n\ntest('keys - print discovery key', function (t) {\n help.shareFixtures(function (_, shareDat) {\n shareDat.close(function () {\n var cmd = dat + ' keys --discovery'\n var st = spawn(t, cmd, { cwd: fixtures })\n\n st.stdout.match(function (output) {\n if (output.indexOf('Discovery') === -1) return false\n t.ok(output.indexOf(shareDat.key.toString('hex') > -1), 'prints key')\n t.ok(output.indexOf(shareDat.archive.discoveryKey.toString('hex') > -1), 'prints discovery key')\n st.kill()\n return true\n })\n st.stderr.empty()\n st.end()\n })\n })\n})\n\nif (!process.env.TRAVIS) {\n test('keys - export & import secret key', function (t) {\n help.shareFixtures(function (_, shareDat) {\n var key = shareDat.key.toString('hex')\n tempDir(function (_, dir, cleanup) {\n var cmd = dat + ' clone ' + key\n var st = spawn(t, cmd, { cwd: dir, end: false })\n var datDir = path.join(dir, key)\n\n st.stdout.match(function (output) {\n var downloadFinished = output.indexOf('Exiting') > -1\n if (!downloadFinished) return false\n st.kill()\n shareDat.close(exchangeKeys)\n return true\n })\n st.stderr.empty()\n\n function exchangeKeys () {\n var secretKey = null\n\n var exportKey = dat + ' keys export'\n var st = spawn(t, exportKey, { cwd: fixtures, end: false })\n st.stdout.match(function (output) {\n if (!output) return false\n secretKey = output.trim()\n st.kill()\n importKey()\n return true\n })\n st.stderr.empty()\n\n function importKey () {\n var exportKey = dat + ' keys import'\n var st = spawn(t, exportKey, { cwd: datDir })\n st.stdout.match(function (output) {\n if (!output.indexOf('secret key') === -1) return false\n st.stdin.write(secretKey + '\\r')\n if (output.indexOf('Successful import') === -1) return false\n t.ok(fs.statSync(path.join(datDir, '.dat', 'metadata.ogd')), 'original dat file exists')\n st.kill()\n return true\n })\n st.stderr.empty()\n st.end(function () {\n rimraf.sync(path.join(fixtures, '.dat'))\n cleanup()\n })\n }\n }\n })\n })\n })\n}\n" }, { "path": "test/pull.js", "content": "var path = require('path')\nvar test = require('tape')\nvar tempDir = require('temporary-directory')\nvar spawn = require('./helpers/spawn.js')\nvar help = require('./helpers')\n\nvar dat = path.resolve(path.join(__dirname, '..', 'bin', 'cli.js'))\n\ntest('pull - errors without clone first', function (t) {\n tempDir(function (_, dir, cleanup) {\n var cmd = dat + ' pull'\n var st = spawn(t, cmd, { cwd: dir })\n st.stderr.match(function (output) {\n t.ok('No existing archive', 'Error: no existing archive')\n st.kill()\n return true\n })\n st.end(cleanup)\n })\n})\n\ntest('pull - default opts', function (t) {\n // import false so we can pull files later\n help.shareFixtures({ import: false }, function (_, fixturesDat) {\n tempDir(function (_, dir, cleanup) {\n // clone initial dat\n var cmd = dat + ' clone ' + fixturesDat.key.toString('hex') + ' ' + dir\n var st = spawn(t, cmd, { end: false })\n st.stdout.match(function (output) {\n var synced = output.indexOf('dat synced') > -1\n if (!synced) return false\n st.kill()\n fixturesDat.close(doPull)\n return true\n })\n\n function doPull () {\n // TODO: Finish this one. Need some bug fixes on empty pulls =(\n help.shareFixtures({ resume: true, import: true }, function (_, fixturesDat) {\n var cmd = dat + ' pull'\n var st = spawn(t, cmd, { cwd: dir })\n st.stdout.match(function (output) {\n var downloadFinished = output.indexOf('dat sync') > -1\n if (!downloadFinished) return false\n st.kill()\n return true\n })\n st.succeeds('exits after finishing download')\n st.stderr.empty()\n st.end(function () {\n fixturesDat.close()\n })\n })\n }\n })\n })\n})\n\n// test('pull - default opts', function (t) {\n// // cmd: dat pull\n// // import the files to the sharer so we can pull new data\n// shareDat.importFiles(function (err) {\n// if (err) throw err\n\n// var datDir = path.join(baseTestDir, shareKey)\n// var cmd = dat + ' pull'\n// var st = spawn(t, cmd, {cwd: datDir})\n// st.stdout.match(function (output) {\n// var downloadFinished = output.indexOf('Download Finished') > -1\n// if (!downloadFinished) return false\n\n// var stats = shareDat.stats.get()\n// var fileRe = new RegExp(stats.filesTotal + ' files')\n// var bytesRe = new RegExp(/1\\.\\d{1,2} kB/)\n\n// t.ok(help.matchLink(output), 'prints link')\n// t.ok(output.indexOf('dat-download-folder/' + shareKey) > -1, 'prints dir')\n// t.ok(output.match(fileRe), 'total size: files okay')\n// t.ok(output.match(bytesRe), 'total size: bytes okay')\n// t.ok(help.isDir(datDir), 'creates download directory')\n\n// var fileList = help.fileList(datDir).join(' ')\n// var hasCsvFile = fileList.indexOf('all_hour.csv') > -1\n// t.ok(hasCsvFile, 'csv file downloaded')\n// var hasDatFolder = fileList.indexOf('.dat') > -1\n// t.ok(hasDatFolder, '.dat folder created')\n// var hasSubDir = fileList.indexOf('folder') > -1\n// t.ok(hasSubDir, 'folder created')\n// var hasNestedDir = fileList.indexOf('nested') > -1\n// t.ok(hasNestedDir, 'nested folder created')\n// var hasHelloFile = fileList.indexOf('hello.txt') > -1\n// t.ok(hasHelloFile, 'hello.txt file downloaded')\n\n// st.kill()\n// return true\n// })\n// st.succeeds('exits after finishing download')\n// st.stderr.empty()\n// st.end()\n// })\n// })\n\n// test('pull - with dir arg', function (t) {\n// var dirName = shareKey\n// var datDir = path.join(baseTestDir, shareKey)\n// var cmd = dat + ' pull ' + dirName\n// var st = spawn(t, cmd, {cwd: baseTestDir})\n// st.stdout.match(function (output) {\n// var downloadFinished = output.indexOf('Download Finished') > -1\n// if (!downloadFinished) return false\n\n// t.ok(output.indexOf('dat-download-folder/' + dirName) > -1, 'prints dir')\n// t.ok(help.isDir(datDir), 'creates download directory')\n\n// st.kill()\n// return true\n// })\n// st.succeeds('exits after finishing download')\n// st.stderr.empty()\n// st.end()\n// })\n" }, { "path": "test/share.js", "content": "// var fs = require('fs')\n// var path = require('path')\n// var test = require('tape')\n// var rimraf = require('rimraf')\n// var spawn = require('./helpers/spawn.js')\n// var help = require('./helpers')\n\n// var dat = path.resolve(path.join(__dirname, '..', 'bin', 'cli.js'))\n// if (process.env.TRAVIS) dat += ' --no-watch '\n// var fixtures = path.join(__dirname, 'fixtures')\n\n// // os x adds this if you view the fixtures in finder and breaks the file count assertions\n// try { fs.unlinkSync(path.join(fixtures, '.DS_Store')) } catch (e) { /* ignore error */ }\n\n// // start without dat.json\n// try { fs.unlinkSync(path.join(fixtures, 'dat.json')) } catch (e) { /* ignore error */ }\n\n// test('share - default opts', function (t) {\n// rimraf.sync(path.join(fixtures, '.dat'))\n// var cmd = dat + ' share'\n// var st = spawn(t, cmd, {cwd: fixtures})\n\n// st.stdout.match(function (output) {\n// var importFinished = output.indexOf('Total Size') > -1\n// if (!importFinished) return false\n\n// t.ok(help.isDir(path.join(fixtures, '.dat')), 'creates dat directory')\n// t.ok(output.indexOf('Looking for connections') > -1, 'network')\n\n// st.kill()\n// return true\n// })\n// st.stderr.empty()\n// st.end()\n// })\n\n// test('share - with dir arg', function (t) {\n// rimraf.sync(path.join(fixtures, '.dat'))\n// var cmd = dat + ' share ' + fixtures\n// var st = spawn(t, cmd)\n\n// st.stdout.match(function (output) {\n// var importFinished = output.indexOf('Total Size') > -1\n// if (!importFinished) return false\n\n// t.ok(help.isDir(path.join(fixtures, '.dat')), 'creates dat directory')\n// t.ok(output.indexOf('Looking for connections') > -1, 'network')\n\n// st.kill()\n// return true\n// })\n// st.stderr.empty()\n// st.end()\n// })\n\n// test.onFinish(function () {\n// rimraf.sync(path.join(fixtures, '.dat'))\n// })\n" }, { "path": "test/sync-owner.js", "content": "// var fs = require('fs')\n// var net = require('net')\n// var path = require('path')\n// var test = require('tape')\n// var mkdirp = require('mkdirp')\n// var rimraf = require('rimraf')\n// var Dat = require('dat-node')\n// var spawn = require('./helpers/spawn.js')\n// var help = require('./helpers')\n\n// var dat = path.resolve(path.join(__dirname, '..', 'bin', 'cli.js'))\n// if (process.env.TRAVIS) dat += ' --no-watch '\n// var fixtures = path.join(__dirname, 'fixtures')\n// var downDat\n\n// // os x adds this if you view the fixtures in finder and breaks the file count assertions\n// try { fs.unlinkSync(path.join(fixtures, '.DS_Store')) } catch (e) { /* ignore error */ }\n\n// test('sync-owner - errors without create first', function (t) {\n// rimraf.sync(path.join(fixtures, '.dat'))\n// // cmd: dat sync\n// var cmd = dat + ' sync'\n// var st = spawn(t, cmd, {cwd: fixtures})\n\n// st.stderr.match(function (output) {\n// var hasError = output.indexOf('No existing archive') > -1\n// t.ok(hasError, 'emits error')\n// st.kill()\n// return true\n// })\n// st.end()\n// })\n\n// test('sync-owner - create a dat for syncing', function (t) {\n// rimraf.sync(path.join(fixtures, '.dat'))\n// // cmd: dat create\n// var cmd = dat + ' create --import'\n// var st = spawn(t, cmd, {cwd: fixtures})\n// st.stdout.match(function (output) {\n// var importFinished = output.indexOf('import finished') > -1\n// if (!importFinished) return false\n// st.kill()\n// return true\n// })\n// st.stderr.empty()\n// st.end()\n// })\n\n// test('sync-owner - default opts', function (t) {\n// // cmd: dat sync\n// var cmd = dat + ' sync'\n// var st = spawn(t, cmd, {cwd: fixtures, end: false})\n\n// var key\n\n// st.stdout.match(function (output) {\n// var sharing = output.indexOf('Dat Network') > -1\n// if (!sharing) return false\n\n// key = help.matchLink(output)\n\n// t.ok(key, 'prints link')\n// t.ok(output.indexOf('tests/fixtures') > -1, 'prints dir')\n\n// downloadDat()\n// return true\n// })\n// st.stderr.empty()\n// st.end()\n\n// function downloadDat () {\n// var downloadDir = path.join(help.testFolder(), '' + Date.now())\n// mkdirp.sync(downloadDir)\n\n// Dat(downloadDir, { key: key }, function (err, tmpDat) {\n// if (err) throw err\n\n// downDat = tmpDat\n// downDat.joinNetwork()\n\n// downDat.network.swarm.once('connection', function () {\n// t.pass('downloader connects')\n// downDat.close(function () {\n// rimraf.sync(downDat.path)\n// t.end()\n// })\n// })\n// })\n// }\n// })\n\n// test('sync-owner - create without import for syncing', function (t) {\n// rimraf.sync(path.join(fixtures, '.dat'))\n// // cmd: dat create\n// var cmd = dat + ' create'\n// var st = spawn(t, cmd, {cwd: fixtures})\n// st.stdout.match(function (output) {\n// if (output.indexOf('created') > -1) return true\n// return false\n// })\n// st.succeeds()\n// st.end()\n// })\n\n// test('sync-owner - imports after no-import create', function (t) {\n// // cmd: dat sync\n// var cmd = dat + ' sync'\n// var st = spawn(t, cmd, {cwd: fixtures})\n\n// st.stdout.match(function (output) {\n// // have to check both for local test (watching) and travis (sharing)\n// var sharing = output.indexOf('Watching') > -1 || output.indexOf('Sharing latest') > -1\n// if (!sharing) return false\n\n// var fileRe = new RegExp('2 files')\n// var bytesRe = new RegExp(/1\\.\\d{1,2} kB/)\n\n// t.ok(help.matchLink(output), 'prints link')\n// t.ok(output.indexOf('tests/fixtures') > -1, 'prints dir')\n// t.ok(output.match(fileRe), 'total size: files okay')\n// t.ok(output.match(bytesRe), 'total size: bytes okay')\n\n// st.kill()\n// return true\n// })\n// st.stderr.empty()\n// st.end()\n// })\n\n// // TODO: this test is causing serious memory issues.\n// // HELP. Maybe related to https://github.com/datproject/dat-node/issues/71\n// // test('sync-owner - turn off ignore hidden', function (t) {\n// // // cmd: dat sync\n// // var hiddenFile = path.join(fixtures, '.hidden-file')\n// // var cmd = dat + ' sync --no-ignore-hidden'\n// // fs.writeFile(hiddenFile, 'You cannot see me', function (err) {\n// // t.error(err)\n\n// // var st = spawn(t, cmd, {cwd: fixtures, end: false})\n// // var key\n\n// // st.stdout.match(function (output) {\n// // var sharing = output.indexOf('Dat Network') > -1\n// // if (!sharing) return false\n\n// // key = help.matchLink(output)\n\n// // downloadDat()\n// // return true\n// // })\n// // st.stderr.empty()\n// // st.end()\n\n// // function downloadDat () {\n// // var downloadDir = path.join(help.testFolder(), '' + Date.now())\n// // mkdirp.sync(downloadDir)\n\n// // Dat(downloadDir, { key: key }, function (err, downDat) {\n// // if (err) throw err\n\n// // downDat.joinNetwork()\n\n// // downDat.network.swarm.once('connection', function () {\n// // downDat.archive.list({live: false}, function (err, data) {\n// // t.error(err)\n// // var hasHiddenFile = data.filter(function (entry) {\n// // return entry.name === '.hidden-file'\n// // })\n// // t.ok(hasHiddenFile.length, 'hidden file in archive')\n// // downDat.network.swarm.close(function () {\n// // process.nextTick(function () {\n// // downDat.close(function () {\n// // rimraf(downDat.path, function () {\n// // fs.unlink(hiddenFile, function () {\n// // t.end()\n// // })\n// // })\n// // })\n// // })\n// // })\n// // })\n// // })\n// // })\n// // }\n// // })\n// // })\n\n// test('sync-owner - port and utp options', function (t) {\n// var port = 3281\n// var cmd = dat + ' sync --port ' + port + ' --no-utp'\n// var st = spawn(t, cmd, {cwd: fixtures, end: false})\n// st.stderr.empty()\n\n// var server = net.createServer()\n// server.once('error', function (err) {\n// if (err.code !== 'EADDRINUSE') return t.error(err)\n// t.skip('TODO: correct port in use')\n// done()\n// })\n// server.once('listening', function () {\n// t.skip(`TODO: port ${server.address().port} should be in use`)\n// done()\n// })\n// server.listen(port)\n\n// t.skip('TODO: check utp option') // TODO: how to check utp?\n\n// function done () {\n// server.close(function () {\n// st.kill()\n// t.end()\n// })\n// }\n// })\n\n// test('sync-owner - shorthand', function (t) {\n// var cmd = dat + ' .'\n// var st = spawn(t, cmd, {cwd: fixtures})\n\n// st.stdout.match(function (output) {\n// var sharing = output.indexOf('Looking for connections') > -1\n// if (!sharing) return false\n\n// t.ok(help.matchLink(output), 'prints link')\n\n// st.kill()\n// return true\n// })\n// st.stderr.empty()\n// st.end()\n// })\n\n// test('sync-owner - dir argument', function (t) {\n// var cmd = dat + ' sync ' + fixtures\n// var st = spawn(t, cmd)\n\n// st.stdout.match(function (output) {\n// var sharing = output.indexOf('Looking for connections') > -1\n// if (!sharing) return false\n\n// t.ok(help.matchLink(output), 'prints link')\n\n// st.kill()\n// return true\n// })\n// st.stderr.empty()\n// st.end()\n// })\n\n// if (!process.env.TRAVIS) {\n// test('sync-owner - live', function (t) {\n// var liveFile = path.join(fixtures, 'live.txt')\n// var wroteFile = false\n\n// var cmd = dat + ' sync --watch'\n// var st = spawn(t, cmd, {cwd: fixtures})\n\n// st.stdout.match(function (output) {\n// var watching = output.indexOf('Watching for file changes') > -1\n// if (!watching) return false\n// else if (!wroteFile) {\n// fs.writeFileSync(liveFile, 'hello')\n// wroteFile = true\n// }\n// var fileImported = output.indexOf('live.txt') > -1\n// if (!fileImported) return false\n\n// t.ok(fileImported, 'prints live file output')\n// t.ok(output.indexOf('3 files') > -1, 'total size: files okay')\n\n// fs.unlinkSync(liveFile)\n// st.kill()\n// return true\n// })\n// st.stderr.empty()\n// st.end()\n// })\n// }\n\n// test.onFinish(function () {\n// rimraf.sync(path.join(fixtures, '.dat'))\n// })\n" }, { "path": "test/sync-remote.js", "content": "// var path = require('path')\n// var test = require('tape')\n// var rimraf = require('rimraf')\n// var spawn = require('./helpers/spawn.js')\n// var help = require('./helpers')\n\n// var dat = path.resolve(path.join(__dirname, '..', 'bin', 'cli.js'))\n// var baseTestDir = help.testFolder()\n// var shareDat\n// var syncDir\n\n// test('sync-remote - default opts', function (t) {\n// // cmd: dat sync\n// var key\n\n// help.shareFixtures({import: false}, function (_, fixturesDat) {\n// shareDat = fixturesDat\n// key = shareDat.key.toString('hex')\n// syncDir = path.join(baseTestDir, key)\n\n// makeClone(function () {\n// shareDat.importFiles(function () {\n// var cmd = dat + ' sync'\n// var st = spawn(t, cmd, {cwd: syncDir})\n// st.stdout.match(function (output) {\n// var updated = output.indexOf('Files updated') > -1\n// if (!updated) return false\n\n// var fileRe = new RegExp('3 files')\n// var bytesRe = new RegExp(/1\\.\\d{1,2} kB/)\n\n// key = help.matchLink(output)\n\n// t.ok(key, 'prints link')\n// t.ok(output.indexOf('dat-download-folder/' + key) > -1, 'prints dir')\n// t.ok(output.match(fileRe), 'total size: files okay')\n// t.ok(output.match(bytesRe), 'total size: bytes okay')\n\n// st.kill()\n// return true\n// })\n// st.stderr.empty()\n// st.end()\n// })\n// })\n// })\n\n// function makeClone (cb) {\n// var cmd = dat + ' clone ' + key\n// var st = spawn(t, cmd, {cwd: baseTestDir, end: false})\n// st.stdout.match(function (output) {\n// var downloadFinished = output.indexOf('Download Finished') > -1\n// if (!downloadFinished) return false\n\n// st.kill()\n// cb()\n// return true\n// })\n// st.stderr.empty()\n// }\n// })\n\n// test('sync-remote - shorthand sync', function (t) {\n// // cmd: dat sync\n// var cmd = dat + ' .'\n// var st = spawn(t, cmd, {cwd: syncDir})\n// st.stdout.match(function (output) {\n// var syncing = output.indexOf('Syncing Dat Archive') > -1\n// if (!syncing) return false\n// t.ok(help.matchLink(output), 'prints link')\n// st.kill()\n// return true\n// })\n// st.stderr.empty()\n// st.end()\n// })\n\n// test('sync-remote - dir arg', function (t) {\n// var cmd = dat + ' ' + syncDir\n// var st = spawn(t, cmd)\n// st.stdout.match(function (output) {\n// var syncing = output.indexOf('Syncing Dat Archive') > -1\n// if (!syncing) return false\n// t.ok(help.matchLink(output), 'prints link')\n// st.kill()\n// return true\n// })\n// st.stderr.empty()\n// st.end()\n// })\n\n// test('close sharer', function (t) {\n// shareDat.close(function () {\n// rimraf.sync(path.join(shareDat.path, '.dat'))\n// t.end()\n// })\n// })\n\n// test.onFinish(function () {\n// rimraf.sync(baseTestDir)\n// })\n" }, { "path": "test/usage.js", "content": "var path = require('path')\nvar test = require('tape')\nvar spawn = require('./helpers/spawn.js')\n\nvar dat = path.resolve(path.join(__dirname, '..', 'bin', 'cli.js'))\nvar version = require('../package.json').version\n\ntest('usage - prints usage', function (t) {\n var d = spawn(t, dat)\n d.stderr.match(function (output) {\n var usage = output.indexOf('dat ') > -1\n if (!usage) return false\n return true\n })\n d.end()\n})\n\ntest('usage - prints version', function (t) {\n var d = spawn(t, dat + ' -v')\n d.stderr.match(function (output) {\n var ver = output.indexOf(version) > -1\n if (!ver) return false\n return true\n })\n d.end()\n})\n\ntest('usage - also prints version', function (t) {\n var d = spawn(t, dat + ' -v')\n d.stderr.match(function (output) {\n var ver = output.indexOf(version) > -1\n if (!ver) return false\n return true\n })\n d.end()\n})\n\ntest('usage - help prints usage', function (t) {\n var d = spawn(t, dat + ' help')\n d.stderr.match(function (output) {\n var usage = output.indexOf('dat ') > -1\n if (!usage) return false\n return true\n })\n d.end()\n})\n" } ]