Showing preview only (404K chars total). Download the full file or copy to clipboard to get everything.
Repository: chainpoint/chainpoint-node
Branch: master
Commit: 87a35270802f
Files: 56
Total size: 383.0 KB
Directory structure:
gitextract_zhinso_3/
├── .eslintrc.json
├── .gitignore
├── .prettierrc
├── Dockerfile
├── LICENSE
├── Makefile
├── README.md
├── chainpoint-gateway-openapi-3.yaml
├── cloudbuild.yaml
├── docker-compose.yaml
├── init/
│ └── index.js
├── ip-blacklist.txt
├── lib/
│ ├── aggregator.js
│ ├── analytics.js
│ ├── api-server.js
│ ├── cached-proofs.js
│ ├── cores.js
│ ├── endpoints/
│ │ ├── calendar.js
│ │ ├── config.js
│ │ ├── hashes.js
│ │ ├── proofs.js
│ │ └── verify.js
│ ├── lightning.js
│ ├── logger.js
│ ├── models/
│ │ └── RocksDB.js
│ ├── parse-env.js
│ └── utils.js
├── package.json
├── scripts/
│ ├── install_deps.sh
│ ├── prod_secrets_expand.sh
│ └── run_prod.sh
├── server.js
├── swarm-compose.yaml
└── tests/
├── RocksDB.js
├── aggregator.js
├── api-server.js
├── cached-proofs.js
├── calendar.js
├── config.js
├── cores.js
├── hashes.js
├── parse-env.js
├── proofs.js
├── sample-data/
│ ├── btc-proof.chp.json
│ ├── cal-proof-l.chp.json
│ ├── cal-proof.chp.json
│ ├── core-btc-proof.chp.json
│ ├── core-cal-proof.chp.json
│ ├── core-tbtc-proof.chp.json
│ ├── core-tcal-proof.chp.json
│ ├── lsat-data.json
│ ├── tbtc-proof-l.chp.json
│ ├── tbtc-proof.chp.json
│ └── tcal-proof.chp.json
├── utils.js
└── verify.js
================================================
FILE CONTENTS
================================================
================================================
FILE: .eslintrc.json
================================================
{
"env": {
"es6": true,
"node": true
},
"extends": ["eslint:recommended", "plugin:prettier/recommended"],
"parserOptions": {
"ecmaVersion": 2018
},
"plugins": ["prettier"],
"rules": {
"prettier/prettier": "error",
"linebreak-style": ["error", "unix"],
"no-console": "off",
"camelcase": [
"error",
{
"properties": "never",
"ignoreDestructuring": true
}
]
}
}
================================================
FILE: .gitignore
================================================
.env
.DS_Store
.data/
.vscode
.vscode/
.nyc_output
node_modules/
npm-debug.log*
yarn-debug.log*
yarn-error.log*
test.js
init/init.json
================================================
FILE: .prettierrc
================================================
{
"singleQuote": true,
"semi": false,
"printWidth": 120
}
================================================
FILE: Dockerfile
================================================
# Node.js 8.x LTS on Debian Stretch Linux
# see: https://github.com/nodejs/LTS
# see: https://hub.docker.com/_/node/
FROM node:12.14.1-stretch
LABEL MAINTAINER="Jacob Henderson <jacob@tierion.com>"
# The `node` user and its home dir is provided by
# the base image. Create a subdir where app code lives.
RUN mkdir /home/node/app
WORKDIR /home/node/app
COPY package.json yarn.lock server.js /home/node/app/
RUN yarn policies set-version 1.22.10
RUN yarn
RUN mkdir -p /home/node/app/scripts
COPY ./scripts/*.sh /home/node/app/scripts/
RUN mkdir -p /home/node/app/lib
COPY ./lib/*.js /home/node/app/lib/
RUN mkdir -p /home/node/app/lib/endpoints
COPY ./lib/endpoints/*.js /home/node/app/lib/endpoints/
RUN mkdir -p /home/node/app/lib/models
COPY ./lib/models/*.js /home/node/app/lib/models/
RUN mkdir -p /root/.lnd
RUN mkdir -p /root/.chainpoint/gateway/data/rocksdb
RUN chmod -R 777 /root
EXPOSE 80
CMD ["yarn", "start"]
================================================
FILE: LICENSE
================================================
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "{}"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright {yyyy} {name of copyright owner}
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
================================================
FILE: Makefile
================================================
# First target in the Makefile is the default.
all: help
# without this 'source' won't work.
SHELL := /bin/bash
# Get the location of this makefile.
ROOT_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
# Get home directory of current users
GATEWAY_DATADIR := $(shell eval printf "~$$USER")/.chainpoint/gateway
# Get home directory of current users
HOMEDIR := $(shell eval printf "~$$USER")
CORE_DATADIR := ${HOMEDIR}/.chainpoint/core
UID := $(shell id -u $$USER)
GID := $(shell id -g $$USER)
.PHONY : help
help : Makefile
@sed -n 's/^##//p' $<
## logs : Tail Gateway logs
.PHONY : logs
logs:
docker service logs -f chainpoint-gateway_chainpoint-gateway --raw
## up : Start Gateway in dev mode
.PHONY : up
up: build-config build build-rocksdb
docker-compose up -d
## down : Shutdown Gateway
.PHONY : down
down:
docker-compose down
## clean : Shutdown and **destroy** all local Gateway data
.PHONY : clean
clean: stop
@rm -rf ${GATEWAY_DATADIR}/data/rocksdb/*
@chmod 777 ${GATEWAY_DATADIR}/data/rocksdb
## burn : Shutdown and **destroy** all local Gateway data
.PHONY : burn
burn: clean
@rm -rf ${HOMEDIR}/.chainpoint/gateway/.lnd
@rm -rf init/init.json
@docker swarm leave --force || echo "already left swarm"
## restart : Restart Gateway in dev mode
.PHONY : restart
restart: down up
## build : Build Gateway image
.PHONY : build
build:
docker build -t chainpoint-gateway .
docker tag chainpoint-gateway gcr.io/chainpoint-registry/github-chainpoint-chainpoint-gateway:latest
docker container prune -f
## build-config : Copy the .env config from .env.sample
.PHONY : build-config
build-config:
@[ ! -f ./.env ] && \
cp .env.sample .env && \
echo 'Copied config .env.sample to .env' || true
## build-rocksdb : Ensure the RocksDB data dir exists
.PHONY : build-rocksdb
build-rocksdb:
@echo Setting up directories...
@mkdir -p ${GATEWAY_DATADIR}/data/rocksdb && chmod 777 ${GATEWAY_DATADIR}/data/rocksdb
## pull : Pull Docker images
.PHONY : pull
pull:
docker-compose pull
## git-pull : Git pull latest
.PHONY : git-pull
git-pull:
@git pull --all
@git submodule update --init --remote --recursive
## upgrade : Same as `make down && git pull && make up`
.PHONY : upgrade
upgrade: down git-pull up
## install-deps : Install system dependencies
install-deps:
scripts/install_deps.sh
@echo Please login and logout to enable docker
## init : Bring up yarn, swarm, and generate secrets
init: build-rocksdb init-yarn init-swarm
## init-yarn : Initialize dependencies
init-yarn:
@echo Installing packages...
@yarn >/dev/null
## init-swarm : Initialize a docker swarm
.PHONY : init-swarm
init-swarm:
@node ./init/index.js
## init-swarm-restart : Initialize a docker swarm, abandon current configuration
.PHONY : init-swarm-restart
init-swarm-restart: stop
@rm -rf ~/.chainpoint/gateway/.lnd
@rm -rf ./init/init.json
@node ./init/index.js
@docker swarm leave --force || echo "already left swarm"
## init-restart : Bring up yarn, swarm, and generate secrets, abondon current configuration
init-restart: build-rocksdb init-yarn init-swarm-restart
## deploy : deploys a swarm stack
deploy:
set -a && source .env && set +a && export USERID=${UID} && export GROUPID=${GID} && docker stack deploy -c swarm-compose.yaml chainpoint-gateway
## optimize-network: increases number of sockets host can use
optimize-network:
@sudo sysctl net.core.somaxconn=1024
@sudo sysctl net.ipv4.tcp_fin_timeout=30
@sudo sysctl net.ipv4.tcp_tw_reuse=1
@sudo sysctl net.core.netdev_max_backlog=2000
@sudo sysctl net.ipv4.tcp_max_syn_backlog=2048
## stop : removes a swarm stack
stop:
docker stack rm chainpoint-gateway
rm -rf ${HOMEDIR}/.chainpoint/gateway/.lnd/tls.*
================================================
FILE: README.md
================================================
# Chainpoint Gateway
[](https://github.com/prettier/prettier)
[](https://opensource.org/licenses/Apache-2.0)
See [Chainpoint Start](https://github.com/chainpoint/chainpoint-start) for an overview of the Chainpoint Network.
A Chainpoint Gateway is a dedicated server for generating many Chainpoint proofs with a single request to the Chainpoint Network.
Each Gateway has an integrated Lightning Node running [LND](https://github.com/lightningnetwork/lnd). Gateways use [Lightning Service Authentication Tokens](https://www.npmjs.com/package/lsat-js) (LSATs) to pay Cores an `anchor fee` when submitting a Merkle root. The default anchor fee is 2 [satoshis](<https://en.bitcoin.it/wiki/Satoshi_(unit)>). Core operators can set their `anchor fee` to adapt to changing market conditions, and compete to receive transactions from Gateways
Gateway setup takes 45 - 90 mins, due to activities that require the automated setup tools to interact with the Bitcoin Blockchain.
- Lightning Node sync (10 - 15 minutes)
- Funding the Lightning wallet and waiting for 3 confirmations (avg 30 mins)
## Installation
### Requirements
The following software is required:
- `*Nix-based OS (Ubuntu Linux and MacOS have been tested)`
- `BASH`
- `Git`
- `Docker`
A BASH script to install all other dependencies (make, openssl, nodejs, yarn) on Ubuntu and Mac can be run from `make install-deps`.
Chainpoint Gateway has been tested with different hardware configurations.
Minimum:
- `4GB RAM`
- `1 CPU Cores`
- `128+ GB SSD`
- `Public IPv4 address`
Mid-Range:
- `8GB RAM`
- `2 CPU Cores`
- `256+ GB SSD`
- `Public IPv4 address`
### Deployment
Run the following commands to initiate your Gateway:
#### Install Dependencies
```bash
$ sudo apt-get install make git
$ git clone https://github.com/chainpoint/chainpoint-gateway.git
$ cd chainpoint-gateway
$ make install-deps
Logout and login to allow your user to use Docker
$ exit
```
#### Configure Gateway
```
$ ssh user@<your_ip>
$ cd chainpoint-gateway
$ make init
██████╗██╗ ██╗ █████╗ ██╗███╗ ██╗██████╗ ██████╗ ██╗███╗ ██╗████████╗ ██████╗ █████╗ ████████╗███████╗██╗ ██╗ █████╗ ██╗ ██╗
██╔════╝██║ ██║██╔══██╗██║████╗ ██║██╔══██╗██╔═══██╗██║████╗ ██║╚══██╔══╝ ██╔════╝ ██╔══██╗╚══██╔══╝██╔════╝██║ ██║██╔══██╗╚██╗ ██╔╝
██║ ███████║███████║██║██╔██╗ ██║██████╔╝██║ ██║██║██╔██╗ ██║ ██║ ██║ ███╗███████║ ██║ █████╗ ██║ █╗ ██║███████║ ╚████╔╝
██║ ██╔══██║██╔══██║██║██║╚██╗██║██╔═══╝ ██║ ██║██║██║╚██╗██║ ██║ ██║ ██║██╔══██║ ██║ ██╔══╝ ██║███╗██║██╔══██║ ╚██╔╝
╚██████╗██║ ██║██║ ██║██║██║ ╚████║██║ ╚██████╔╝██║██║ ╚████║ ██║ ╚██████╔╝██║ ██║ ██║ ███████╗╚███╔███╔╝██║ ██║ ██║
╚═════╝╚═╝ ╚═╝╚═╝ ╚═╝╚═╝╚═╝ ╚═══╝╚═╝ ╚═════╝ ╚═╝╚═╝ ╚═══╝ ╚═╝ ╚═════╝ ╚═╝ ╚═╝ ╚═╝ ╚══════╝ ╚══╝╚══╝ ╚═╝ ╚═╝ ╚═╝
? Will this Gateway use Bitcoin mainnet or testnet? testnet
? Enter your Gateways's Public IP Address: 104.154.83.163
```
#### Initialize Lightning
```
Initializing Lightning wallet...
Create new address for wallet...
Creating Docker secrets...
****************************************************
Lightning initialization has completed successfully.
****************************************************
Lightning Wallet Password: kPlIshurrduurSQoXa
LND Wallet Seed: absorb behind drop safe like herp derp celery galaxy wait orient sign suit castle awake gadget pass pipe sudden ethics hill choose six orphan
Lightning Wallet Address:tb1qglvlrlg0velrserjuuy7s4uhrsrhuzwgl8hvgm
******************************************************
You should back up this information in a secure place.
******************************************************
TODO: REMOVE Please fund the Lightning Wallet Address above with Bitcoin and wait for 6 confirmations before running 'make deploy'
How many Cores would you like to connect to? (max 4) 2
Would you like to specify any Core IPs manually? No
You have chosen to connect to 2 Core(s).
You will now need to fund you wallet with a minimum amount of BTC to cover costs of the initial channel creation and future Core submissions.
? How many Satoshi to commit to each channel/Core? (min 120000) 500000
? 500000 per channel will require 1000000 Satoshi total funding. Is this OK? (Y/n) y
**************************************************************************************************************
Please send 1000000 Satoshi (0.01 BTC) to your wallet with address tb1qglvlrlg0velrserjuuy7s4uhrsrhuzwgl8hvgm
**************************************************************************************************************
This initialization process will now wait until your Lightning node is fully synced and your wallet is funded with at least 400000 Satoshi. The init process should resume automatically.
2020-02-24T17:12:12.244Z> Syncing in progress... currently at block height 1576000
2020-02-24T17:12:42.259Z> Syncing in progress... currently at block height 1596000
2020-02-24T17:13:12.269Z> Syncing in progress... currently at block height 1608000
2020-02-24T17:13:42.279Z> Syncing in progress... currently at block height 1626000
2020-02-24T17:14:12.286Z> Syncing in progress... currently at block height 1650000
2020-02-24T17:14:42.297Z> Syncing in progress... currently at block height 1662000
*****************************************
Your Lightning node is fully synced.
*****************************************
***********************************************
Your Lightning wallet is adequately funded.
***********************************************
*********************************************************************************
Chainpoint Gateway and integrated Lightning node have been successfully initialized.
*********************************************************************************
$ make deploy
```
After running `make deploy`, the Gateway will automatically peer and open Lightning channels with Cores. This may take up to 6 confirmations (~60 minutes). This will allow the Gateway to authenticate with Cores and pay for Anchor Fees. This process may take several minutes upon first run.
## Troubleshooting
If your issue isn't addressed here, please [submit an issue](https://github.com/chainpoint/chainpoint-core/issues) to the Chainpoint Core repo.
### Init Problems
If `make init` fails and the Lightning wallet hasn't yet been generated and funded, run `make init-restart`, then run `make init` again. If the Lightning wallet has already been generated and funded, you can usually just run `make init` again to continue the initialization process.
### Docker Secrets
Gateway uses docker secrets to store sensitive credentials. If you receive a `secret not found` error for either `HOT_WALLET_PASS` or `HOT_WALLET_ADDRESS` while deploying, this can be remedied by using your saved lnd credentials to recreate the secrets:
`printf <hot wallet password without quotes> | docker secret create HOT_WALLET_PASS -` or `printf <hot wallet address without quotes> | docker secret create HOT_WALLET_ADDRESS -`.
## Gateway Public API
Every Gateway provides a public HTTP API. This is documented in greater detail on the [Gateway HTTP API wiki](https://github.com/chainpoint/chainpoint-gateway/wiki/Gateway-HTTP-API)
Additionally, lightning node information for your Gateway can be found at `http://<gateway_ip>/config`.
## License
[Apache License, Version 2.0](https://opensource.org/licenses/Apache-2.0)
```text
Copyright (C) 2017-2020 Tierion
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
```
================================================
FILE: chainpoint-gateway-openapi-3.yaml
================================================
openapi: 3.0.0
info:
title: 'Chainpoint Node'
description: 'Documentation for the Chainpoint Node API'
version: '2.0.0'
license:
name: 'Apache 2.0'
url: 'https://tldrlegal.com/license/apache-license-2.0-(apache-2.0)'
servers:
- url: 'http://35.231.41.69'
description: 'Development server (produces testnet proofs)'
paths:
'/hashes':
post:
summary: 'Submit one or more hashes for anchoring'
operationId: 'submitHashes'
requestBody:
description: 'An array of hex string hashes to be anchored'
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/PostHashesRequest'
responses:
'200':
description: 'An array of hash object and supporting meta information for that array'
content:
'application/json':
schema:
$ref: '#/components/schemas/PostHashesResponse'
'409':
description: 'There was an invalid argument in the request'
content:
'application/json':
schema:
$ref: '#/components/schemas/ErrorResponse'
tags:
- 'Hashes'
'/proofs/{proof_id}':
get:
summary: 'Retrieves a proof by proof_id'
operationId: 'getProof'
parameters:
- name: 'proof_id'
in: 'path'
required: true
description: 'The proof_id of the proof to retrieve'
schema:
type: 'string'
format: 'uuid'
responses:
'200':
description: 'The requested proof object'
content:
'application/json':
schema:
$ref: '#/components/schemas/GetProofsBase64Response'
'application/vnd.chainpoint.ld+json':
schema:
$ref: '#/components/schemas/GetProofsJSONResponse'
'application/vnd.chainpoint.json+base64':
schema:
$ref: '#/components/schemas/GetProofsBase64Response'
'409':
description: 'There was an invalid argument in the request'
content:
'application/json':
schema:
$ref: '#/components/schemas/ErrorResponse'
'application/vnd.chainpoint.ld+json':
schema:
$ref: '#/components/schemas/ErrorResponse'
'application/vnd.chainpoint.json+base64':
schema:
$ref: '#/components/schemas/ErrorResponse'
tags:
- 'Proofs'
'/proofs':
get:
summary: 'Retrieves one or more proofs by proofids supplied in header'
operationId: 'getProofs'
parameters:
- name: 'proofids'
in: 'header'
required: true
description: 'Comma separated proof_id list of the proofs to retrieve'
schema:
type: 'string'
responses:
'200':
description: 'An array of the requested proof objects'
content:
'application/json':
schema:
type: 'array'
items:
$ref: '#/components/schemas/GetProofsBase64Response'
'application/vnd.chainpoint.ld+json':
schema:
type: 'array'
items:
$ref: '#/components/schemas/GetProofsJSONResponse'
'application/vnd.chainpoint.json+base64':
schema:
type: 'array'
items:
$ref: '#/components/schemas/GetProofsBase64Response'
'409':
description: 'There was an invalid argument in the request'
content:
'application/json':
schema:
$ref: '#/components/schemas/ErrorResponse'
'application/vnd.chainpoint.ld+json':
schema:
$ref: '#/components/schemas/ErrorResponse'
'application/vnd.chainpoint.json+base64':
schema:
$ref: '#/components/schemas/ErrorResponse'
tags:
- 'Proofs'
'/verify':
post:
summary: 'Submit one or more proofs for verification'
operationId: 'verifyProofs'
requestBody:
description: 'Array of one or more proofs to be verified'
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/PostVerifyRequest'
responses:
'200':
description: 'An array of the verification results'
content:
'application/json':
schema:
type: array
items:
$ref: '#/components/schemas/PostVerifyResponse'
'409':
description: 'There was an invalid argument in the request'
content:
'application/json':
schema:
$ref: '#/components/schemas/ErrorResponse'
tags:
- 'Verify'
'/calendar/{tx_id}/data':
get:
summary: 'Retrieves the the data embedded in a calendar transaction'
operationId: 'getCalendarTxData'
parameters:
- name: 'tx_id'
in: 'path'
required: true
description: 'The calendar transaction id from which to retrieve the embedded data'
schema:
type: 'string'
responses:
'200':
description: 'The data value embedded within the calendar transaction'
content:
'application/json':
schema:
type: 'string'
example: 'f18bf0968b224f73528d99cc83ca9e79d467f34875e85f36e2c1f074ff2dc657'
'404':
description: 'The requested transaction was not found'
content:
'application/json':
schema:
$ref: '#/components/schemas/ErrorResponse'
'409':
description: 'There was an invalid argument in the request'
content:
'application/json':
schema:
$ref: '#/components/schemas/ErrorResponse'
tags:
- 'Calendar'
'/config':
get:
summary: 'Retrieves some basic information for the Node'
operationId: 'getNodeConfig'
responses:
'200':
description: 'Basic information about the Node and it''s environment'
content:
'application/json':
schema:
$ref: '#/components/schemas/GetConfigResponse'
tags:
- 'Config'
components:
schemas:
PostHashesRequest:
type: 'object'
properties:
hashes:
type: 'array'
items:
type: 'string'
example: '1957db7fe23e4be1740ddeb941ddda7ae0a6b782e536a9e00b5aa82db1e84547'
pattern: '^([a-fA-F0-9]{2}){20,64}$'
minLength: 40
maxLength: 128
PostHashesResponse:
type: 'object'
properties:
meta:
type: 'object'
properties:
submitted_at:
type: 'string'
format: 'date-time'
example: '2017-05-02T15:16:44Z'
processing_hints:
type: 'object'
properties:
cal:
type: 'string'
format: 'date-time'
example: '2017-05-02T15:17:44Z'
btc:
type: 'string'
format: 'date-time'
example: '2017-05-02T16:17:44Z'
hashes:
type: 'array'
items:
type: 'object'
properties:
proof_id:
type: 'string'
example: '5a001650-2f4a-11e7-ad22-37b426116bc4'
hash:
type: 'string'
example: '11cd8a380e8d5fd3ac47c1f880390341d40b11485e8ae946d8fa3d466f23fe89'
GetProofsJSONResponse:
type: 'object'
properties:
proof_id:
type: 'string'
example: '577c6c90-78d5-11e9-9c57-010a193d9f8c'
proof:
type: 'object'
example:
'@context': 'https://w3id.org/chainpoint/v3'
type: 'Chainpoint'
hash: '11cd8a380e8d5fd3ac47c1f880390341d40b11485e8ae946d8fa3d466f23fe89'
proof_id: '577c6c90-78d5-11e9-9c57-010a193d9f8c'
hash_received: '2019-05-17T18:55:30Z'
branches:
- label: 'cal_anchor_branch'
ops:
- l: 'node_id:577c6c90-78d5-11e9-9c57-010a193d9f8c'
- op: 'sha-256'
- l: 'core_id:5a22fb80-78d5-11e9-8186-01d1f712eccc'
- op: 'sha-256'
- l: 'nistv2:1558119240000:eb591780782f746fda5e7ac8011064fda657ae451bd1ae6b71e2f5d7e24e9d49bdc25db6d901ccf8736bbf135c451d1edc9c6065b577d69f3fd9be6a1a8d0763'
- op: 'sha-256'
- l: '1766c5a6c10cf8ae5cce76c6d89cb9bc8696a2acf8e7ed4dbe05a71802cae38a'
- op: 'sha-256'
- anchors:
- type: 'cal'
anchor_id: 'b220c0443b5f8b1394a38a102892590b4c21d8ad1382cd9e4d59b9834f6a769f'
uris:
- 'http://35.245.9.90/calendar/b220c0443b5f8b1394a38a102892590b4c21d8ad1382cd9e4d59b9834f6a769f/data'
anchors_complete:
type: 'array'
items:
type: 'string'
example: 'cal'
GetProofsBase64Response:
type: 'object'
properties:
proof_id:
type: 'string'
example: '577c6c90-78d5-11e9-9c57-010a193d9f8c'
proof:
type: 'string'
example: 'eJyNk79u1EAQh1+Gksvt7P91dRKvQJXmNDszy1k67JPtBCgDDW0KOpqQoAREg4QoeY97G+y7S4AAUrbzrr5vfyP/9u3NgtpmkJfDj9UwbPpqPn9haj5qu2dzWmHdbNq6Gean5mp4tZHPT+62rlbYr7YLAOKIJiqJ7AobJBsISozKJGUssFUZwEYnESVZz7GgYet90aZITF8mzbLmZdOybB+5EMhTUrMw6mYAkmaJXJgpUAjJcCqRvu+Q/iQ/r4dB9uQSh29aQZqpkQpPIVbOVUYd3+mp7SY9al1y/F0fIfpRz1ACaCH6Sz+R/9bb45vcYUMr6c9ff1xjlvVXwvVy2mq75f7sst30788u1tvHu5w1Vw+Z8exDu7nuVzjTzu/gXYoJfsAE9+F3Td0Pp7oC5yJA0laNq5LsEoSoQtQlWF8YnQSkqACUt+OndwHFOsgMKD4HEF0cB9FWEtuUmbTj7DkpICoxGJ9zAeNoZBiEKZFX3uVxXvapmMIpi0fAyCp4cz/lAoL35NATqFGH4ogkeBorkyinTNEnjxrHMwnClrMohwGi0oRiIt4XJqO10irezm1MqKC6bXk++lXvqe3V+OeqA3F20W0XQRBJMhoxbABRUzBG5zFkZBNztihZNEDBWGL23hvtQhFO1hSXAvwR53rfif78ze4dXY6XfTrUpObrw7VXJ13dn2+P/hdxPlLSMHbzAzCfqvoTAAQ82A=='
anchors_complete:
type: 'array'
items:
type: 'string'
example: 'cal'
PostVerifyRequest:
type: 'object'
properties:
proofs:
type: 'array'
items:
type: 'string'
example: 'eJyNk79u1EAQh1+Gksvt7P91dRKvQJXmNDszy1k67JPtBCgDDW0KOpqQoAREg4QoeY97G+y7S4AAUrbzrr5vfyP/9u3NgtpmkJfDj9UwbPpqPn9haj5qu2dzWmHdbNq6Gean5mp4tZHPT+62rlbYr7YLAOKIJiqJ7AobJBsISozKJGUssFUZwEYnESVZz7GgYet90aZITF8mzbLmZdOybB+5EMhTUrMw6mYAkmaJXJgpUAjJcCqRvu+Q/iQ/r4dB9uQSh29aQZqpkQpPIVbOVUYd3+mp7SY9al1y/F0fIfpRz1ACaCH6Sz+R/9bb45vcYUMr6c9ff1xjlvVXwvVy2mq75f7sst30788u1tvHu5w1Vw+Z8exDu7nuVzjTzu/gXYoJfsAE9+F3Td0Pp7oC5yJA0laNq5LsEoSoQtQlWF8YnQSkqACUt+OndwHFOsgMKD4HEF0cB9FWEtuUmbTj7DkpICoxGJ9zAeNoZBiEKZFX3uVxXvapmMIpi0fAyCp4cz/lAoL35NATqFGH4ogkeBorkyinTNEnjxrHMwnClrMohwGi0oRiIt4XJqO10irezm1MqKC6bXk++lXvqe3V+OeqA3F20W0XQRBJMhoxbABRUzBG5zFkZBNztihZNEDBWGL23hvtQhFO1hSXAvwR53rfif78ze4dXY6XfTrUpObrw7VXJ13dn2+P/hdxPlLSMHbzAzCfqvoTAAQ82A=='
minItems: 1
maxItems: 1000
PostVerifyResponse:
type: 'object'
properties:
proof_index:
type: 'integer'
example: 0
hash:
type: 'string'
example: '11cd8a380e8d5fd3ac47c1f880390341d40b11485e8ae946d8fa3d466f23fe89'
proof_id:
type: 'string'
example: '577c6c90-78d5-11e9-9c57-010a193d9f8c'
hash_received:
type: 'string'
format: 'date-time'
example: '2019-05-17T18:55:30Z'
anchors:
type: 'array'
items:
type: 'object'
properties:
branch:
type: 'string'
example: 'cal_anchor_branch'
type:
type: 'string'
example: 'cal'
valid:
type: 'boolean'
example: true
status:
type: 'string'
example: 'verified'
GetConfigResponse:
type: 'object'
properties:
version:
type: 'string'
example: '2.0.0'
time:
type: 'string'
format: 'date-time'
example: '2019-05-17T19:53:49.140Z'
ErrorResponse:
type: 'object'
properties:
code:
type: 'string'
message:
type: 'string'
tags:
- name: 'Hashes'
description: 'Your hashes to be anchored'
- name: 'Proofs'
description: 'Your Chainpoint proofs created for each of your hashes'
- name: 'Verify'
description: 'Verification process for your proofs'
- name: 'Calendar'
description: 'Chainpoint calendar transaction data'
- name: 'Config'
description: 'Configuration information about the Node'
externalDocs:
description: 'Find out more about Chainpoint'
url: 'https://chainpoint.org'
================================================
FILE: cloudbuild.yaml
================================================
steps:
- name: 'gcr.io/cloud-builders/git'
args: ['submodule', 'update', '--init', '--recursive']
- name: 'gcr.io/cloud-builders/docker'
args: [ 'build', '-f', 'Dockerfile', '-t', 'gcr.io/chainpoint-registry/$REPO_NAME:$COMMIT_SHA', '-t', 'gcr.io/chainpoint-registry/$REPO_NAME:latest', '.' ]
id: 'chainpoint-gateway'
timeout: 1000s
images:
- 'gcr.io/chainpoint-registry/$REPO_NAME:latest'
- 'gcr.io/chainpoint-registry/$REPO_NAME:$COMMIT_SHA'
options:
machineType: 'N1_HIGHCPU_8'
================================================
FILE: docker-compose.yaml
================================================
version: '3.4'
networks:
chainpoint-gateway:
driver: bridge
services:
chainpoint-gateway:
restart: on-failure
volumes:
- ./ip-blacklist.txt:/home/node/app/ip-blacklist.txt:ro
- ~/.chainpoint/gateway/data/rocksdb:/root/.chainpoint/gateway/data/rocksdb
- ~/.chainpoint/gateway/.lnd:/root/.lnd:ro
build: .
container_name: chainpoint-gateway
ports:
# - '${PORT}:${CHAINPOINT_NODE_PORT}'
- '80:8080'
networks:
- chainpoint-gateway
environment:
HOME: /root
HOT_WALLET_PASS: ${HOT_WALLET_PASS}
HOT_WALLET_ADDRESS: ${HOT_WALLET_ADDRESS}
LND_SOCKET: ${LND_SOCKET}
LND_MACAROON: ${LND_MACAROON}
LND_TLS_CERT: ${LND_TLS_CERT}
CHAINPOINT_CORE_CONNECT_IP_LIST: '${CHAINPOINT_CORE_CONNECT_IP_LIST}'
PORT: '${PORT:-80}'
AGGREGATION_INTERVAL_SECONDS: '${AGGREGATION_INTERVAL_SECONDS}'
PROOF_EXPIRE_MINUTES: '${PROOF_EXPIRE_MINUTES}'
CHAINPOINT_NODE_PORT: '${CHAINPOINT_NODE_PORT:-9090}'
POST_HASHES_MAX: '${POST_HASHES_MAX}'
POST_VERIFY_PROOFS_MAX: '${POST_VERIFY_PROOFS_MAX}'
GET_PROOFS_MAX: '${GET_PROOFS_MAX}'
MAX_SATOSHI_PER_HASH: '${MAX_SATOSHI_PER_HASH}'
NETWORK: ${NETWORK}
NODE_ENV: ${NODE_ENV}
CHANNEL_AMOUNT: ${CHANNEL_AMOUNT}
FUND_AMOUNT: ${FUND_AMOUNT}
NO_LSAT_CORE_WHITELIST: ${NO_LSAT_CORE_WHITELIST}
GOOGLE_UA_ID: ''
PUBLIC_IP: ${LND_PUBLIC_IP}
tty: true
# Lightning node
lnd:
image: tierion/lnd:${NETWORK:-testnet}-0.9.2
user: ${USERID}:${GROUPID}
entrypoint: './start-lnd.sh'
container_name: lnd-node
ports:
- target: 8080
published: 8080
protocol: tcp
mode: host
- target: 9735
published: 9735
protocol: tcp
mode: host
- target: 10009
published: 10009
protocol: tcp
mode: host
restart: always
environment:
- PUBLICIP=${LND_PUBLIC_IP}
- RPCUSER
- RPCPASS
- NETWORK=${NETWORK:-testnet}
- CHAIN
- DEBUG=info
- BACKEND=neutrino
- NEUTRINO=faucet.lightning.community:18333
- LND_REST_PORT
- LND_RPC_PORT
- TLSPATH
- TLSEXTRADOMAIN=lnd
volumes:
- ~/.chainpoint/gateway/.lnd:/root/.lnd:z
networks:
- chainpoint-gateway
# ln-accounting
# Returns accounting reports in harmony format for lnd node
ln-accounting:
image: tierion/ln-accounting
ports:
- '9000'
environment:
NETWORK: ${NETWORK}
LND_DIR: /root/.lnd
LND_SOCKET: ${LND_SOCKET}
ACCOUNTING_PORT: ${ACCOUNTING_PORT:-9000}
volumes:
- ~/.chainpoint/gateway/.lnd:/root/.lnd:ro
networks:
- chainpoint-gateway
================================================
FILE: init/index.js
================================================
/* Copyright (C) 2019 Tierion
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
const inquirer = require('inquirer')
const validator = require('validator')
const chalk = require('chalk')
const exec = require('executive')
const generator = require('generate-password')
const lightning = require('../lib/lightning')
const home = require('os').homedir()
const fs = require('fs')
const path = require('path')
const utils = require('../lib/utils.js')
const _ = require('lodash')
const rp = require('request-promise-native')
const retry = require('async-retry')
const QUIET_OUTPUT = true
const LND_SOCKET = '127.0.0.1:10009'
const MIN_CHANNEL_SATOSHI = 100000
const CHANNEL_OPEN_OVERHEAD_SAFE = 20000
const CORE_SEED_IPS_MAINNET = ['18.220.31.138']
const CORE_SEED_IPS_TESTNET = ['3.133.119.65', '52.14.49.31', '3.135.54.225']
const initQuestionConfig = [
{
type: 'list',
name: 'NETWORK',
message: 'Will this Gateway use Bitcoin mainnet or testnet?',
choices: [
{
name: 'Mainnet',
value: 'mainnet'
},
{
name: 'Testnet',
value: 'testnet'
}
],
default: 'mainnet'
},
{
type: 'input',
name: 'LND_PUBLIC_IP',
message: "Enter your Node's Public IP Address:",
validate: input => {
if (input) {
return validator.isIP(input, 4)
} else {
return true
}
}
}
]
function displayTitleScreen() {
const txt = `
██████╗██╗ ██╗ █████╗ ██╗███╗ ██╗██████╗ ██████╗ ██╗███╗ ██╗████████╗
██╔════╝██║ ██║██╔══██╗██║████╗ ██║██╔══██╗██╔═══██╗██║████╗ ██║╚══██╔══╝
██║ ███████║███████║██║██╔██╗ ██║██████╔╝██║ ██║██║██╔██╗ ██║ ██║
██║ ██╔══██║██╔══██║██║██║╚██╗██║██╔═══╝ ██║ ██║██║██║╚██╗██║ ██║
╚██████╗██║ ██║██║ ██║██║██║ ╚████║██║ ╚██████╔╝██║██║ ╚████║ ██║
╚═════╝╚═╝ ╚═╝╚═╝ ╚═╝╚═╝╚═╝ ╚═══╝╚═╝ ╚═════╝ ╚═╝╚═╝ ╚═══╝ ╚═╝
`
const gateway = `
██████╗ █████╗ ████████╗███████╗██╗ ██╗ █████╗ ██╗ ██╗
██╔════╝ ██╔══██╗╚══██╔══╝██╔════╝██║ ██║██╔══██╗╚██╗ ██╔╝
██║ ███╗███████║ ██║ █████╗ ██║ █╗ ██║███████║ ╚████╔╝
██║ ██║██╔══██║ ██║ ██╔══╝ ██║███╗██║██╔══██║ ╚██╔╝
╚██████╔╝██║ ██║ ██║ ███████╗╚███╔███╔╝██║ ██║ ██║
╚═════╝ ╚═╝ ╚═╝ ╚═╝ ╚══════╝ ╚══╝╚══╝ ╚═╝ ╚═╝ ╚═╝
`
console.log('\n')
console.log(chalk.dim.magenta(txt))
console.log(chalk.dim.magenta(gateway))
console.log('\n')
}
async function startLndNodeAsync(initAnswers) {
try {
let uid = (await exec.quiet('id -u $USER')).stdout.trim()
let gid = (await exec.quiet('id -g $USER')).stdout.trim()
console.log(chalk.yellow(`Starting Lightning node...`))
await exec.quiet([
`docker-compose pull lnd &&
mkdir -p ${home}/.chainpoint/gateway/.lnd &&
export USERID=${uid} &&
export GROUPID=${gid} &&
export NETWORK=${initAnswers.NETWORK} &&
export PUBLICIP=${initAnswers.LND_PUBLIC_IP} &&
docker-compose run -d --service-ports lnd`
])
} catch (error) {
throw new Error(`Could not start Lightning node : ${error.message}`)
}
await utils.sleepAsync(20000)
}
async function initializeLndNodeAsync(initAnswers) {
await startLndNodeAsync(initAnswers)
let walletSecret = generator.generate({ length: 20, numbers: false })
try {
console.log(chalk.yellow(`Initializing Lightning wallet...`))
let lnd = new lightning(LND_SOCKET, initAnswers.NETWORK, true, true)
let seed = await lnd.callMethodRawAsync('unlocker', 'genSeedAsync', {}, true)
await lnd.callMethodRawAsync('unlocker', 'initWalletAsync', {
wallet_password: walletSecret,
cipher_seed_mnemonic: seed.cipher_seed_mnemonic
})
await utils.sleepAsync(10000)
console.log(chalk.yellow(`Create new address for wallet...`))
lnd = new lightning(LND_SOCKET, initAnswers.NETWORK, false, true)
let newAddress = await lnd.callMethodAsync('lightning', 'newAddressAsync', { type: 0 }, walletSecret)
return { cipherSeedMnemonic: seed.cipher_seed_mnemonic, newAddress: newAddress.address, walletSecret: walletSecret }
} catch (error) {
throw new Error(`Could not initialize Lightning wallet : ${error.message}`)
}
}
async function createDockerSecretsAsync(initAnswers, walletInfo) {
try {
console.log(chalk.yellow('Creating Docker secrets...'))
await exec.quiet([`docker swarm init --advertise-addr=${initAnswers.LND_PUBLIC_IP}`])
await utils.sleepAsync(2000) // wait for swarm to initialize
await exec.quiet([
`printf ${walletInfo.walletSecret} | docker secret create HOT_WALLET_PASS -`,
`printf '${walletInfo.cipherSeedMnemonic.join(' ')}' | docker secret create HOT_WALLET_SEED -`,
`printf ${walletInfo.newAddress} | docker secret create HOT_WALLET_ADDRESS -`
])
} catch (error) {
throw new Error(`Could not create Docker secrets : ${error.message}`)
}
}
function displayInitResults(walletInfo) {
console.log(chalk.green(`\n****************************************************`))
console.log(chalk.green(`Lightning initialization has completed successfully.`))
console.log(chalk.green(`****************************************************\n`))
console.log(chalk.yellow(`Lightning Wallet Password: `) + walletInfo.walletSecret)
console.log(chalk.yellow(`Lightning Wallet Seed: `) + walletInfo.cipherSeedMnemonic.join(' '))
console.log(chalk.yellow(`Lightning Wallet Address:`) + walletInfo.newAddress)
console.log(chalk.magenta(`\n******************************************************`))
console.log(chalk.magenta(`You should back up this information in a secure place.`))
console.log(chalk.magenta(`******************************************************\n\n`))
console.log(
chalk.green(
`\nPlease fund the Lightning Wallet Address above with Bitcoin and wait for 6 confirmation before running 'make deploy'\n`
)
)
}
async function setENVValuesAsync(newENVData) {
// check for existence of .env file
let envFileExists = fs.existsSync(path.resolve(__dirname, '../', '.env'))
// load .env file if it exists, otherwise load the .env.sample file
let envContents = fs.readFileSync(path.resolve(__dirname, '../', `.env${!envFileExists ? '.sample' : ''}`)).toString()
let updatedEnvContents = Object.keys(newENVData).reduce((contents, key) => {
let regexMatch = new RegExp(`^${key}=.*`, 'gim')
if (!contents.match(regexMatch)) return contents + `\n${key}=${newENVData[key]}`
return contents.replace(regexMatch, `${key}=${newENVData[key]}`)
}, envContents)
fs.writeFileSync(path.resolve(__dirname, '../', '.env'), updatedEnvContents)
}
async function getCorePeerListAsync(seedIPs) {
seedIPs = _.shuffle(seedIPs)
let peersReceived = false
while (!peersReceived && seedIPs.length > 0) {
let targetIP = seedIPs.pop()
let options = {
uri: `http://${targetIP}/peers`,
method: 'GET',
json: true,
gzip: true,
resolveWithFullResponse: true
}
try {
let response = await retry(async () => await rp(options), { retries: 3 })
return response.body.concat([targetIP])
} catch (error) {
console.log(`Core IP ${targetIP} not repsonding to peers requests`)
}
}
throw new Error('Unable to retrieve Core peer list')
}
async function askCoreConnectQuestionsAsync(progress) {
let peerList = _.shuffle(
await getCorePeerListAsync(progress.network === 'maininet' ? CORE_SEED_IPS_MAINNET : CORE_SEED_IPS_TESTNET)
)
let peerCount = peerList.length
const coreConnectQuestion = [
{
type: 'number',
name: 'CORE_COUNT',
message: `Connecting with more Cores improves the reliability but is more expensive (2 is recommended).\nHow many Cores would you like to connect to? (max ${peerCount})`,
validate: input => input > 0 && input <= peerCount
},
{
type: 'confirm',
name: 'MANUAL_IP',
message: 'Would you like to specify any Core IPs manually?',
default: false
}
]
let coreConnectAnswers = await inquirer.prompt(coreConnectQuestion)
let coreConnectIPs = []
if (coreConnectAnswers.MANUAL_IP) {
let manualCount = await inquirer.prompt({
type: 'number',
name: 'TOTAL',
message: `How many Core IPs would you like to specify manually? (max ${coreConnectAnswers.CORE_COUNT})`,
validate: input => input > 0 && input <= coreConnectAnswers.CORE_COUNT
})
for (let x = 0; x < manualCount.TOTAL; x++) {
let manualInput = await inquirer.prompt({
type: 'input',
name: 'IP',
message: `Enter Core IP manual entry #${x + 1}:`,
validate: input => peerList.includes(input) && !coreConnectIPs.includes(input)
})
coreConnectIPs.push(manualInput.IP)
}
}
let randomCoreIPCount = coreConnectAnswers.CORE_COUNT - coreConnectIPs.length
let unusedPeers = peerList.filter(ip => !coreConnectIPs.includes(ip))
for (let x = 0; x < randomCoreIPCount; x++) coreConnectIPs.push(unusedPeers.pop())
// Update ENV file with core IP list
await setENVValuesAsync({ CHAINPOINT_CORE_CONNECT_IP_LIST: coreConnectIPs.join(',') })
let coreLNDUris = []
for (let coreIP of coreConnectIPs) {
let options = {
uri: `http://${coreIP}/status`,
method: 'GET',
json: true,
gzip: true,
resolveWithFullResponse: true
}
try {
let response = await retry(async () => await rp(options), { retries: 3 })
coreLNDUris.push(response.body.uris[0])
} catch (error) {
throw new Error(`Unable to retrive status of Core at ${coreIP}`)
}
}
progress.coreLNDUris = coreLNDUris
writeInitProgress(progress)
return coreLNDUris
}
async function askFundAmountAsync(progress) {
const coreConnectCount = progress.coreLNDUris.length
console.log(chalk.yellow(`\nYou have chosen to connect to ${coreConnectCount} Core(s).`))
console.log(
chalk.yellow(
'You will now need to fund your wallet with a minimum amount of BTC to cover costs of the initial Lightning channel(s) creation and future Core submissions.\nThe init process will wait for your funding to confirm with the Bitcoin Network.'
)
)
const minAmount = MIN_CHANNEL_SATOSHI + CHANNEL_OPEN_OVERHEAD_SAFE
let finalFundAmount = null
let finalChannelAmount = null
while (finalFundAmount === null) {
const fundQuestion1 = [
{
type: 'number',
name: 'AMOUNT',
message: `How many Satoshi to commit to each channel/Core? (min ${minAmount})`,
validate: input => input >= minAmount
}
]
let fundAnswer1 = await inquirer.prompt(fundQuestion1)
const totalFundsNeeded = fundAnswer1.AMOUNT * coreConnectCount
const fundQuestion2 = [
{
type: 'confirm',
name: 'AGREE',
message: `${fundAnswer1.AMOUNT} per channel will require ${totalFundsNeeded} Satoshi total funding. Is this OK?`,
default: true
}
]
let fundAnswer2 = await inquirer.prompt(fundQuestion2)
if (fundAnswer2.AGREE) {
finalChannelAmount = fundAnswer1.AMOUNT
finalFundAmount = totalFundsNeeded
}
}
console.log(
chalk.magenta(
`\n**************************************************************************************************************`
)
)
console.log(
chalk.magenta(
`Please send ${finalFundAmount} Satoshi (${finalFundAmount / 10 ** 8} BTC) to your wallet with address ${
progress.walletAddress
}`
)
)
console.log(
chalk.magenta(
`**************************************************************************************************************\n`
)
)
progress.finalFundAmount = finalFundAmount
progress.finalChannelAmount = finalChannelAmount
writeInitProgress(progress)
return progress
}
async function waitForSyncAndFundingAsync(progress) {
console.log(
chalk.yellow(
`This initialization process will now wait until your Lightning node is fully synced and your wallet is funded with at least ${progress.finalFundAmount} Satoshi. The init process should resume automatically. \n`
)
)
let isSynced = false
let isFunded = false
let lnd = new lightning(LND_SOCKET, progress.network, false, true)
while (!isSynced) {
try {
let info = await lnd.callMethodAsync('lightning', 'getInfoAsync', null, progress.walletSecret)
if (info.synced_to_chain) {
console.log(chalk.green('\n*****************************************'))
console.log(chalk.green('Your lightning node is fully synced.'))
console.log(chalk.green('*****************************************'))
isSynced = true
} else {
console.log(
chalk.magenta(
`${new Date().toISOString()}> Syncing in progress... currently at block height ${info.block_height}`
)
)
}
} catch (error) {
console.log(chalk.red(`An error occurred while checking node state : ${error.message}`))
} finally {
if (!isSynced) await utils.sleepAsync(30000)
}
}
while (!isFunded) {
try {
let balance = await lnd.callMethodAsync('lightning', 'walletBalanceAsync', null, progress.walletSecret)
if (balance.confirmed_balance >= progress.finalFundAmount) {
console.log(chalk.green('\n***********************************************'))
console.log(chalk.green('Your lightning wallet is adequately funded.'))
console.log(chalk.green('***********************************************\n'))
console.log(
chalk.yellow(
'Your wallet may require up to 5 more confirmations (~60 Minutes) before your gateway can open payment channels to submit hashes\n'
)
)
isFunded = true
} else {
console.log(
chalk.magenta(
`${new Date().toISOString()}> Awaiting funds for wallet... wallet has a current balance of ${
balance.confirmed_balance
}`
)
)
}
} catch (error) {
console.log(chalk.red(`An error occurred while checking wallet balance : ${error.message}`))
} finally {
if (!isFunded) await utils.sleepAsync(30000)
}
}
}
function displayFinalConnectionSummary() {
console.log(chalk.green('\n*********************************************************************************'))
console.log(chalk.green('Chainpoint Gateway and supporting Lighning node have been successfully initialized.'))
console.log(chalk.green('*********************************************************************************\n'))
}
function readInitProgress() {
// check for existence of .init file
let initFileExists = fs.existsSync(path.resolve(__dirname, './init.json'))
// load .init file if it exists
if (initFileExists) return JSON.parse(fs.readFileSync(path.resolve(__dirname, './init.json')).toString())
return {}
}
function writeInitProgress(progress) {
fs.writeFileSync(path.resolve(__dirname, './init.json'), JSON.stringify(progress, null, 2))
}
function setInitProgressCompleteAsync() {
fs.writeFileSync(path.resolve(__dirname, './init.json'), JSON.stringify({ complete: true }, null, 2))
}
async function start() {
try {
// Check if init has already recorded some progress
// This allows recovery from last known step in process
// Exit of initialization has already completed successfully
let progress = await readInitProgress()
if (progress.complete) {
console.log(chalk.green('Initialization has already been completed successfully.'))
return
}
// Display the title screen
displayTitleScreen()
if (!progress.walletAddress) {
// Ask initialization questions
let initAnswers = await inquirer.prompt(initQuestionConfig)
// Initialize the LND wallet and create a new address
let walletInfo = await initializeLndNodeAsync(initAnswers)
// Store relevant values as Docker secrets
await createDockerSecretsAsync(initAnswers, walletInfo)
// Update the .env file with generated data
await setENVValuesAsync(initAnswers)
// Display the generated wallet information to the user
displayInitResults(walletInfo)
progress.network = initAnswers.NETWORK
progress.walletAddress = walletInfo.newAddress
progress.walletSecret = walletInfo.walletSecret
writeInitProgress(progress)
} else {
await startLndNodeAsync({ NETWORK: progress.network })
}
if (!progress.coreLNDUris) {
// Determine which Core(s) to connect to
let coreLNDUris = await askCoreConnectQuestionsAsync(progress)
progress.coreLNDUris = coreLNDUris
}
// Ask funding questions
if (!progress.finalChannelAmount > 0) {
progress = await askFundAmountAsync(progress)
}
// Wait for sync and wallet funding
await waitForSyncAndFundingAsync(progress)
await setENVValuesAsync({
CHANNEL_AMOUNT: progress.finalChannelAmount,
FUND_AMOUNT: progress.finalFundAmount
})
await displayFinalConnectionSummary()
// Initialization complete, mark progress file as such
setInitProgressCompleteAsync()
} catch (error) {
console.error(chalk.red(`An unexpected error has occurred : ${error.message}. Please run 'make init' again.`))
} finally {
try {
console.log(chalk.yellow(`Shutting down Lightning node...`))
await exec([`docker-compose down`], { quiet: QUIET_OUTPUT })
console.error(chalk.yellow(`Shutdown complete`))
} catch (error) {
console.error(chalk.red(`Unable to shut down Lightning node : ${error.message}`))
}
}
}
start()
================================================
FILE: ip-blacklist.txt
================================================
# ip-blacklist.txt
#
# Add a single IPv4 address per line that you'd
# like to block from connecting to this Node.
#
# Lines beginning with '#' are ignored.
================================================
FILE: lib/aggregator.js
================================================
/**
* Copyright 2019 Tierion
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
const MerkleTools = require('merkle-tools')
const uuidTime = require('uuid-time')
let cores = require('./cores.js')
const BLAKE2s = require('blake2s-js')
let rocksDB = require('./models/RocksDB.js')
const logger = require('./logger.js')
const env = require('./parse-env.js').env
const utils = require('./utils.js')
const { UlidMonotonic } = require('id128')
// a boolean value indicating whether or not aggregateSubmitAndPersistAsync is currently running
let AGG_IN_PROCESS = false
// The merkle tools object for building trees and generating proof paths
const merkleTools = new MerkleTools()
async function getSubmittedHashData() {
let submittedHashData = [[], []]
try {
submittedHashData = await rocksDB.getIncomingHashesUpToAsync(Date.now())
} catch (error) {
logger.error('Could not read submitted hash data')
return [[], []]
}
return submittedHashData
}
// Build a merkle tree from HashData queued in RocksDB, submit the root to Core, persist resulting state data
async function aggregateSubmitAndPersistAsync() {
// Return if the previous call to this function has not yet completed
if (AGG_IN_PROCESS) {
return
} else {
AGG_IN_PROCESS = true
}
let [hashesForTree, hashDataForTreeDeleteOps] = await getSubmittedHashData()
let aggregationRoot = null
if (hashesForTree.length > 0) {
// clear the merkleTools instance to prepare for a new tree
merkleTools.resetTree()
// concatenate and hash the hash ids and hash values into new array
let leaves = hashesForTree.map(hashObj => {
return Buffer.from(hashObj.hash, 'hex')
})
// Add every hash in hashesForTree to new Merkle tree
merkleTools.addLeaves(leaves)
merkleTools.makeTree()
let nodeProofDataItems = []
aggregationRoot = merkleTools.getMerkleRoot().toString('hex')
let treeSize = merkleTools.getLeafCount()
for (let x = 0; x < treeSize; x++) {
// push the hash_id and corresponding proof onto the array
let nodeProofDataItem = {}
nodeProofDataItem.proofId = hashesForTree[x].proof_id
nodeProofDataItem.hash = hashesForTree[x].hash
nodeProofDataItem.proofState = merkleTools.getProof(x, true)
nodeProofDataItems.push(nodeProofDataItem)
}
// submit merkle root to Core
try {
let submitResults = await submitHashToCoresAsync(aggregationRoot)
if (submitResults.length === 0) throw new Error(`Unable to submit hash to Core : No Cores responded`)
submitResults = submitResults.filter(submitResult => {
// Log all proofId values returned by Cores
logger.info(
`Aggregator : Core IP : ${submitResult.coreIP} : proofId : ${submitResult.proofId} : ${JSON.stringify(
submitResult
)} : ${hashesForTree.length}`
)
if (utils.isULID(submitResult.proofId)) {
try {
UlidMonotonic.fromCanonical(submitResult.proofId)
return true
} catch (error) {
logger.error(`unable to validate ProofID ulid ${error.message}`)
}
} else if (utils.isUUID(submitResult.proofId)) {
// validate BLAKE2s
let hashTimestampMS = parseInt(uuidTime.v1(submitResult.proofId))
let h = new BLAKE2s(32, {
personalization: Buffer.from('CHAINPNT')
})
let hashStr = [
hashTimestampMS.toString(),
hashTimestampMS.toString().length,
submitResult.hash,
submitResult.hash.length
].join(':')
h.update(Buffer.from(hashStr))
let expectedData = Buffer.concat([Buffer.from([0x01]), h.digest().slice(27)]).toString('hex')
let embeddedData = submitResult.proofId.slice(24)
if (embeddedData == expectedData) {
return true
} else {
logger.error(
`Aggregator : Submit : ProofID UUID from Core refused : Cannot validate embedded BLAKE2s data : ${embeddedData} != ${expectedData}`
)
logger.error(`hashStr was ${hashStr}`)
}
}
return false
})
if (submitResults.length === 0)
throw new Error(`Unable to submit hash to Core : No Cores responded with valid HashID`)
// add the submission info including core IP and proofId values from Cores for each item in proofDataItems
let submitId = UlidMonotonic.generate().toCanonical() // the identifier for all hashes submitted in this batch
let coreInfo = submitResults.map(submitResult => {
return {
ip: submitResult.coreIP,
proofId: submitResult.proofId
}
})
nodeProofDataItems = nodeProofDataItems.map(nodeProofDataItem => {
nodeProofDataItem.submission = {
submitId: submitId,
cores: coreInfo
}
return nodeProofDataItem
})
// persist these proofDataItems to storage
try {
await rocksDB.saveProofStatesBatchAsync(nodeProofDataItems)
} catch (error) {
throw new Error(`Unable to persist proof state data to disk : ${error.message}`)
}
try {
// Submission to Core was successful, purge hashes that were delivered from RocksDB
await rocksDB.deleteBatchAsync(hashDataForTreeDeleteOps)
} catch (error) {
logger.warn(`Aggregator : Submit to Core : Could not purge submitted hashes : ${error.message}`)
}
let submittedIPs = submitResults.map(item => item.coreIP).toString()
logger.info(`Aggregator : ${hashesForTree.length} hash(es) : Core IPs : ${submittedIPs} `)
} catch (error) {
logger.error(`Aggregator : Submit : ${error.message}`)
if (error.stack) {
logger.error(`Stacktrace: ${error.stack}`)
}
}
}
AGG_IN_PROCESS = false
return aggregationRoot
}
async function submitHashToCoresAsync(hash) {
let response = await cores.submitHashAsync(hash)
if (response.length == 0) {
throw new Error('No Response from any Core')
}
return response.map(item => {
return {
coreIP: item.ip,
proofId: item.response.proof_id,
hash: item.response.hash
}
})
}
function startAggInterval() {
return setInterval(aggregateSubmitAndPersistAsync, env.AGGREGATION_INTERVAL_SECONDS * 1000)
}
module.exports = {
startAggInterval: startAggInterval,
// additional functions for testing purposes
aggregateSubmitAndPersistAsync: aggregateSubmitAndPersistAsync,
setRocksDB: db => {
rocksDB = db
},
setCores: c => {
cores = c
}
}
================================================
FILE: lib/analytics.js
================================================
/**
* Copyright 2019 Tierion
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
// load environment variables
let env = require('./parse-env.js').env
let ua = require('universal-analytics')
const logger = require('./logger.js')
let visitor
if (env.GOOGLE_UA_ID) {
visitor = ua(env.GOOGLE_UA_ID, env.PUBLIC_IP, { strictCidFormat: false })
logger.info(`Setup analytics for analytics id ${env.GOOGLE_UA_ID}`)
}
function setClientID(clientID) {
if (env.GOOGLE_UA_ID) {
visitor = ua(env.GOOGLE_UA_ID, clientID, { strictCidFormat: false })
}
}
function sendEvent(params) {
if (params && visitor) {
logger.info(`Sending event ${params.ea}`)
visitor.event(params).send()
}
}
module.exports = {
setClientID: setClientID,
sendEvent: sendEvent
}
================================================
FILE: lib/api-server.js
================================================
/**
* Copyright 2019 Tierion
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
const restify = require('restify')
const errors = require('restify-errors')
const corsMiddleware = require('restify-cors-middleware')
let rp = require('request-promise-native')
let fs = require('fs')
const _ = require('lodash')
const validator = require('validator')
let rocksDB = require('./models/RocksDB.js')
const logger = require('./logger.js')
const { version } = require('../package.json')
const apiHashes = require('./endpoints/hashes.js')
const apiCalendar = require('./endpoints/calendar.js')
const apiProofs = require('./endpoints/proofs.js')
const apiVerify = require('./endpoints/verify.js')
const apiConfig = require('./endpoints/config.js')
// RESTIFY SETUP
// 'version' : all routes will default to this version
const httpOptions = {
name: 'chainpoint-node',
version: '2.0.0',
formatters: {
'application/json': restify.formatters['application/json; q=0.4'],
'application/javascript': restify.formatters['application/json; q=0.4'],
'text/html': restify.formatters['application/json; q=0.4'],
'application/octet-stream': restify.formatters['application/json; q=0.4']
}
}
const TOR_IPS_KEY = 'blacklist:tor:ips'
const CHAINPOINT_NODE_HTTP_PORT = process.env.CHAINPOINT_NODE_PORT || 8080
// state indicating if the Node is ready to accept new hashes for processing
let acceptingHashes = true
let registrationPassed = true
// the list of IP to refuse connections from
let IPBlacklist = []
// middleware to ensure the Node is accepting hashes
function ensureAcceptingHashes(req, res, next) {
if (!acceptingHashes || !registrationPassed) {
return next(new errors.ServiceUnavailableError('Service is not currently accepting hashes'))
}
return next()
}
async function refreshIPBlacklistAsync() {
let torExitIPs = await getTorExitIPAsync()
let localIPBlacklist = await getLocalIPBlacklistAsync()
let mergedIPBlacklist = torExitIPs.concat(localIPBlacklist)
return _.uniq(mergedIPBlacklist)
}
async function getTorExitIPAsync() {
let options = {
headers: {
'User-Agent': `chainpoint-node/${version}`
},
method: 'GET',
uri: 'https://check.torproject.org/exit-addresses',
gzip: true,
timeout: 10000
}
// Retrieve latest exit IP list
let extractedTorExitIPs = null
try {
let response = await rp(options)
extractedTorExitIPs = parseTorExitIPs(response)
} catch (error) {
logger.error('Firewall : Unable to refresh Tor exit IP list from check.torproject.org')
}
// Save IPs to cache and return if retrieval succeeded
if (extractedTorExitIPs !== null) {
let compactTorExitIPs = _.compact(extractedTorExitIPs)
let uniqueTorExitIPs = _.uniq(compactTorExitIPs)
try {
await rocksDB.setAsync(TOR_IPS_KEY, uniqueTorExitIPs)
} catch (error) {
logger.error('Firewall : Unable to save Tor exit IP list to cache')
}
return uniqueTorExitIPs
} else {
// otherwise, read existing from cache and return
try {
let cachedTorExitIPs = await rocksDB.getAsync(TOR_IPS_KEY)
return cachedTorExitIPs.split(',')
} catch (error) {
logger.error('Firewall : Unable to load Tor exit IP list from cache')
return []
}
}
}
function parseTorExitIPs(response) {
let exitIPs = []
if (!_.isString(response)) {
return exitIPs
}
let respArr = response.split('\n')
if (!_.isArray(respArr)) {
return exitIPs
}
_.forEach(respArr, value => {
if (/^ExitAddress/.test(value)) {
// The second segment of the ExitAddress line is the IP
let ip = value.split(' ')[1]
// Confirm its an IPv4 address
if (validator.isIP(ip.toString(), 4)) {
exitIPs.push(ip)
}
}
})
return exitIPs
}
async function getLocalIPBlacklistAsync() {
try {
if (fs.existsSync('./ip-blacklist.txt')) {
let blacklist = fs.readFileSync('./ip-blacklist.txt', 'utf-8')
let splitBlacklist = blacklist.split('\n')
let compactBlacklist = _.compact(splitBlacklist)
let uniqBlacklist = _.uniq(compactBlacklist)
let ipList = []
_.forEach(uniqBlacklist, ip => {
// any line that doesn't start with '#' comment
if (/^[^#]/.test(ip)) {
// Confirm its an IPv4 or IPv6 address
// IPv6 allowed is to handle the macOS/Docker
// situation where the IP is like: ::ffff:172.18.0.1
// See : https://stackoverflow.com/a/33790357/3902629
if (validator.isIP(ip.toString())) {
ipList.push(ip)
}
}
})
return ipList
} else {
return []
}
} catch (error) {
logger.warn('Firewall : Unable to parse local IP blacklist (ip-blacklist.txt) ')
return []
}
}
function ipFilter(req, res, next) {
var reqIPs = []
if (req.headers['x-forwarded-for']) {
let fwdIPs = req.headers['x-forwarded-for'].split(',')
reqIPs.push(fwdIPs[0])
}
reqIPs.push(req.connection.remoteAddress || '')
reqIPs = reqIPs
.filter(ip => validator.isIP(ip))
.reduce((ips, ip) => {
ips.push(ip, ip.replace(/^.*:/, ''))
return ips
}, [])
for (let ip of reqIPs) {
if (IPBlacklist.includes(ip)) return next(new errors.ForbiddenError())
}
return next()
}
// Put any routing, response, etc. logic here.
function setupCommonRestifyConfigAndRoutes(server) {
// limit responses to only requests for acceptable types
server.pre(
restify.plugins.acceptParser([
'application/json',
'application/javascript',
'text/html',
'application/octet-stream',
'application/vnd.chainpoint.ld+json',
'application/vnd.chainpoint.json+base64'
])
)
// Clean up sloppy paths like //todo//////1//
server.pre(restify.pre.sanitizePath())
// Checks whether the user agent is curl. If it is, it sets the
// Connection header to "close" and removes the "Content-Length" header
// See : http://restify.com/#server-api
server.pre(restify.pre.userAgentConnection())
// CORS
// See : https://github.com/TabDigital/restify-cors-middleware
// See : https://github.com/restify/node-restify/issues/1151#issuecomment-271402858
//
// Test w/
//
// curl \
// --verbose \
// --request OPTIONS \
// http://127.0.0.1:9090/hashes \
// --header 'Origin: http://localhost:9292' \
// --header 'Access-Control-Request-Headers: Origin, Accept, Content-Type' \
// --header 'Access-Control-Request-Method: POST' \
// --header 'proofids: da5b6c70-d628-11e7-a676-0102636501e0'
//
let cors = corsMiddleware({
preflightMaxAge: 600,
origins: ['*'],
allowHeaders: ['proofids,auth'],
exposeHeaders: ['proofids']
})
server.pre(cors.preflight)
server.use(cors.actual)
server.use(restify.plugins.gzipResponse())
server.use(
restify.plugins.queryParser({
mapParams: true
})
)
server.use(
restify.plugins.bodyParser({
mapParams: true
})
)
// DROP all requests from blacklisted IP addresses
server.use(ipFilter)
const applyMiddleware = (middlewares = []) => {
if (process.env.NODE_ENV === 'development' || process.env.NETWORK === 'testnet') {
return []
} else {
return middlewares
}
}
let throttle = (burst, rate, opts = { ip: true }) => {
return restify.plugins.throttle(Object.assign({}, { burst, rate }, opts))
}
// API RESOURCES
// IMPORTANT : These routes MUST come after the firewall initialization!
// submit hash(es)
server.post(
{ path: '/hashes', version: '2.0.0' },
...applyMiddleware([throttle(50, 25)]),
ensureAcceptingHashes,
apiHashes.postHashesAsync
)
// get a data value from a calendar transaction
server.get(
{ path: '/calendar/:tx_id/data', version: '2.0.0' },
...applyMiddleware([throttle(15, 5)]),
apiCalendar.getDataValueByIDAsync
)
// get a single proof with a single hash_id
server.get(
{ path: '/proofs/:proof_id', version: '2.0.0' },
...applyMiddleware([throttle(15, 5)]),
apiProofs.getProofsByIDAsync
)
// get multiple proofs with 'proofids' header param
server.get({ path: '/proofs', version: '2.0.0' }, ...applyMiddleware([throttle(15, 5)]), apiProofs.getProofsByIDAsync)
// verify one or more proofs
server.post(
{ path: '/verify', version: '2.0.0' },
...applyMiddleware([throttle(15, 5)]),
apiVerify.postProofsForVerificationAsync
)
// get configuration information for this Node
server.get({ path: '/config', version: '2.0.0' }, ...applyMiddleware([throttle(1, 1)]), apiConfig.getConfigInfoAsync)
server.get({ path: '/login', version: '2.0.0' }, function(req, res, next) {
res.redirect('/', next)
})
server.get({ path: '/about', version: '2.0.0' }, function(req, res, next) {
res.redirect('/', next)
})
}
// HTTP Server
async function startInsecureRestifyServerAsync() {
let restifyServer = restify.createServer(httpOptions)
setupCommonRestifyConfigAndRoutes(restifyServer)
// Begin listening for requests
return new Promise((resolve, reject) => {
restifyServer.listen(CHAINPOINT_NODE_HTTP_PORT, err => {
if (err) return reject(err)
logger.info(`App : Chainpoint Node listening on port ${CHAINPOINT_NODE_HTTP_PORT}`)
return resolve(restifyServer)
})
})
}
function startIPBlacklistRefreshInterval() {
return setInterval(async () => {
IPBlacklist = await refreshIPBlacklistAsync()
}, 24 * 60 * 60 * 1000) // refresh IPBlacklist every 24 hours
}
async function startAsync(lnd) {
try {
apiConfig.setLnd(lnd)
IPBlacklist = await refreshIPBlacklistAsync()
await startInsecureRestifyServerAsync()
} catch (error) {
logger.error(`Startup : ${error.message}`)
}
}
module.exports = {
startAsync: startAsync,
startInsecureRestifyServerAsync: startInsecureRestifyServerAsync,
startIPBlacklistRefreshInterval: startIPBlacklistRefreshInterval,
// additional functions for testing purposes
refreshIPBlacklistAsync: refreshIPBlacklistAsync,
setAcceptingHashes: isAccepting => {
acceptingHashes = isAccepting
},
setRocksDB: db => {
rocksDB = db
},
setRP: RP => {
rp = RP
},
setFS: FS => {
fs = FS
}
}
================================================
FILE: lib/cached-proofs.js
================================================
/**
* Copyright 2019 Tierion
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
let cores = require('./cores.js')
const utils = require('./utils.js')
const _ = require('lodash')
const logger = require('./logger.js')
let env = require('./parse-env.js').env
// The object containing the cached Core proof objects
let CORE_PROOF_CACHE = {}
const PRUNE_EXPIRED_INTERVAL_SECONDS = 10
async function getCachedCoreProofsAsync(coreSubmissions) {
// determine max core submission count for these submissions... the largest cores array of all submissions
// this will be constant under normal usage, only varying if the master submission count is updated
let maxCoreSubmissionCount = coreSubmissions.reduce((result, item) => {
if (item.cores.length > result) result = item.cores.length
return result
}, 0)
// create `proofId` to `submitId` lookup object, keyed by `proofId`
// and simultaneously create `coreSubmissionsLookup` object keyed by the coreSubmissions `submitId`
let [submitIdForHashIdCoreLookup, coreSubmissionsLookup] = coreSubmissions.reduce(
(result, item) => {
for (let core of item.cores) {
result[0][core.proofId] = item.submitId
}
result[1][item.submitId] = { cores: item.cores, proof: undefined }
return [result[0], result[1]]
},
[{}, {}]
)
// Attempt to read proofs from the cache.
// For all proofs that are not found (not cached), keep `proof` value as undefined
for (let submitId in coreSubmissionsLookup) {
coreSubmissionsLookup[submitId].proof = (function() {
if (CORE_PROOF_CACHE[submitId] && _.isNil(CORE_PROOF_CACHE[submitId].coreProof)) return null
return CORE_PROOF_CACHE[submitId] ? _.get(CORE_PROOF_CACHE, `${submitId}.coreProof`, undefined) : undefined
})()
}
// loop through the 1st, 2nd, ... cores array object for each submission until all proofs have been returned
// under normal operation, with all Cores operating as expected, only one iteration will be performed
// if a Core is offline, the second iterations will attempt to request proofs from the second IP/HashIdCore pair
// iterations will continue until all values have been retrieved, or all IP/HashIdCore pairs have been attempted
//
// use `newProofSubmitIds` to keep track of the submitIds that have new proof data returned from Core
// this information is used later to determine what new data needs to be cached
let newProofSubmitIds = []
for (let index = 0; index < maxCoreSubmissionCount; index++) {
// find core submissions that have `undefined` proofs, they will be requested from Core at cores index `index`
let undefinedProofSubmissions = Object.keys(coreSubmissionsLookup).reduce((result, submitId) => {
let coreInfo = coreSubmissionsLookup[submitId].cores[index]
// if the proof is undefined, and Core info exists in this submission for this `index`, add to results
if (coreSubmissionsLookup[submitId].proof === undefined && coreInfo) {
result.push({ submitId: submitId, ip: coreInfo.ip, proofId: coreInfo.proofId })
} else if (
coreSubmissionsLookup[submitId].proof !== undefined &&
coreSubmissionsLookup[submitId].proof != null &&
coreSubmissionsLookup[submitId].proof.hash_received !== undefined &&
coreInfo
) {
let timer = Date.parse(coreSubmissionsLookup[submitId].proof.hash_received)
let btcDue = Date.now() - timer > 7200000 // 120 min have passed
// if 90 minutes have passed and we have a cal anchor and not a btc anchor
// then we add the proof to undefinedProofSubmissions to ensure re-retrieval
let containsRelevantAnchors =
coreSubmissionsLookup[submitId].anchorsComplete !== undefined &&
(!coreSubmissionsLookup[submitId].anchorsComplete.includes('btc') ||
!coreSubmissionsLookup[submitId].anchorsComplete.includes('tbtc'))
if (btcDue && containsRelevantAnchors) {
logger.info(`time to check for btc proof ${coreInfo.proofId} from ${coreInfo.ip}`)
result.push({ submitId: submitId, ip: coreInfo.ip, proofId: coreInfo.proofId })
}
}
return result
}, [])
// if none were found, then we have received proof data for all requested submissions, exit loop
if (undefinedProofSubmissions.length === 0) break
// split `undefinedProofSubmissions` into distinct array by unique Core IP
// this is needed so that we may request proofs from Core in batches grouped by Core IP
// start by determining all the unique IPs in play in undefinedProofSubmissions
let uniqueIPs = undefinedProofSubmissions.reduce((result, item) => {
// using unshift to build list in reverse for efficiency because we must iterate in reverse later
if (!result.includes(item.ip)) result.unshift(item.ip)
return result
}, [])
// build an array of submissions for each unique Core IP found
let submissionsGroupByIPs = []
for (let ip of uniqueIPs) {
let result = []
for (let x = undefinedProofSubmissions.length - 1; x >= 0; x--) {
if (undefinedProofSubmissions[x].ip === ip) result.push(...undefinedProofSubmissions.splice(x, 1))
}
submissionsGroupByIPs.push(result)
}
// for each resulting submission group array, request the proofs from Core
for (let submissionsGroup of submissionsGroupByIPs) {
// flatten the submission data for use in the getProofsAsync call
let flattenedSubmission = submissionsGroup.reduce(
(result, item) => {
result.ip = item.ip // need to make the setting only once, but will be the same for every item
result.proofIds.push(item.proofId)
return result
},
{ ip: '', proofIds: [] }
)
// attempt to retrieve proofs from Core
let getProofsFromCoreResults = []
try {
getProofsFromCoreResults = await cores.getProofsAsync(flattenedSubmission.ip, flattenedSubmission.proofIds)
} catch (err) {
logger.error(
`getCachedCoreProofsAsync : Core ${flattenedSubmission.ip} : ProofID Count ${
flattenedSubmission.proofIds.length
} (${JSON.stringify(flattenedSubmission.proofIds)}) : ${err.message}`
)
// Cache as `null` Proof to prevent subsequent retries for 1min
CORE_PROOF_CACHE[submissionsGroup.submitId] = {
coreProof: null,
expiresAt: Date.now() + 1 * 60 * 1000 // 1min
}
}
// assign the returned proof values back to the `coreSubmissions` object,
// for each item in the results, using the `submitIdForHashIdCoreLookup` object
for (let result of getProofsFromCoreResults) {
let submitIdForResult = submitIdForHashIdCoreLookup[result.proof_id]
// be sure that the `submitIdForResult` is known in `coreSubmissions`
// assuming it is, as it always should be unless error, assign the proof to that key
if (coreSubmissionsLookup.hasOwnProperty(submitIdForResult)) {
coreSubmissionsLookup[submitIdForResult].proof = result.proof
coreSubmissionsLookup[submitIdForResult].anchorsComplete = _.isNil(
coreSubmissionsLookup[submitIdForResult].proof
)
? []
: utils.parseAnchorsComplete(coreSubmissionsLookup[submitIdForResult].proof, env.NETWORK)
// track this submitId for caching later
newProofSubmitIds.push(submitIdForResult)
}
}
}
}
// cache any new results returned from Core
if (newProofSubmitIds.length > 0) {
// for all new proofs received from Core, create an proofType lookup object for use in determining
// proper cache TTL for the proof
let proofTypeLookup = newProofSubmitIds.reduce((result, submitId) => {
if (!_.isNil(coreSubmissionsLookup[submitId].proof)) {
if (env.NETWORK === 'mainnet') {
result[submitId] = coreSubmissionsLookup[submitId].anchorsComplete.includes('btc') ? 'btc' : 'cal'
} else {
result[submitId] = coreSubmissionsLookup[submitId].anchorsComplete.includes('tbtc') ? 'tbtc' : 'tcal'
}
}
return result
}, {})
// Store the non-null AND null proofs from Core in the local cache for subsequent requests
// First, create the array of objects to be written to the cache
let uncachedCoreProofObjects = newProofSubmitIds.reduce((result, submitId) => {
// `null` cached proofs expire after 1 minute
// `(t)cal` cached proofs expire after 15 minutes
// `(t)btc` cached proofs expire after 25 hours
let expMinutes = 1
if (coreSubmissionsLookup[submitId].proof !== null) {
expMinutes = ['btc', 'tbtc'].includes(proofTypeLookup[submitId]) ? 25 * 60 : 15
}
result.push({
submitId: submitId,
coreProof: coreSubmissionsLookup[submitId].proof,
expiresAt: Date.now() + expMinutes * 60 * 1000
})
return result
}, [])
// Next,write proofs to cache
for (let coreProofObject of uncachedCoreProofObjects) {
CORE_PROOF_CACHE[coreProofObject.submitId] = {
coreProof: coreProofObject.coreProof,
expiresAt: coreProofObject.expiresAt
}
}
}
// format `coreSubmissions` into the proper result object to return from this function
let finalProofResults = Object.keys(coreSubmissionsLookup).map(submitId => {
if (_.isNil(coreSubmissionsLookup[submitId].proof)) {
return { submitId: submitId, proof: null }
}
// A proof exists and has been found for this submitId
// Identify the anchors completed in this proof and append that information
return {
submitId: submitId,
proof: coreSubmissionsLookup[submitId].proof,
anchorsComplete: coreSubmissionsLookup[submitId].anchorsComplete
}
})
// Finally, return the proof items array
return finalProofResults
}
function pruneExpiredItems() {
let now = Date.now()
for (let key in CORE_PROOF_CACHE) {
if (CORE_PROOF_CACHE[key].expiresAt <= now) {
delete CORE_PROOF_CACHE[key]
}
}
}
function startPruneExpiredItemsInterval() {
return setInterval(pruneExpiredItems, PRUNE_EXPIRED_INTERVAL_SECONDS * 1000)
}
module.exports = {
getCachedCoreProofsAsync: getCachedCoreProofsAsync,
startPruneExpiredItemsInterval: startPruneExpiredItemsInterval,
// additional functions for testing purposes
pruneExpiredItems: pruneExpiredItems,
getPruneExpiredIntervalSeconds: () => PRUNE_EXPIRED_INTERVAL_SECONDS,
getCoreProofCache: () => CORE_PROOF_CACHE,
setCoreProofCache: obj => {
CORE_PROOF_CACHE = obj
},
setCores: c => {
cores = c
},
setENV: obj => {
env = obj
}
}
================================================
FILE: lib/cores.js
================================================
/**
* Copyright 2019 Tierion
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
// load environment variables
let env = require('./parse-env.js').env
let rp = require('request-promise-native')
const { version } = require('../package.json')
const retry = require('async-retry')
const { Lsat } = require('lsat-js')
const _ = require('lodash')
const logger = require('./logger.js')
const utils = require('./utils.js')
const chalk = require('chalk')
const lightning = require('./lightning')
const PRUNE_EXPIRED_INTERVAL_SECONDS = 10
let CONNECTED_CORE_IPS = []
let CONNECTED_CORE_LN_URIS = []
let coreConnectionCount = 1
// In some cases we may want a list of all Core Nodes (whether or not the Chainpoint Node is connected to it or not)
let ALL_CORE_IPS = []
// This is the local in-memory cache of calendar transactions
// CORE_TX_CACHE is an object keyed by txId, storing the transaction object
let CORE_TX_CACHE = {}
// Initialize the Lightning grpc object
let lnd = new lightning(env.LND_SOCKET, env.NETWORK)
async function connectAsync() {
// Retrieve the list of Core IPs we can work with
let coreIPList = []
if (!_.isEmpty(env.CHAINPOINT_CORE_CONNECT_IP_LIST)) {
// Core IPs have been supplied in CHAINPOINT_CORE_CONNECT_IP_LIST, use those IPs only
coreIPList = env.CHAINPOINT_CORE_CONNECT_IP_LIST
}
//coreConnectionCount = coreIPList.length
logger.info(`Connecting to Cores...`)
// Select and establish connection to Core(s)
let connectedCoreIPResult = await getConnectedCoreIPsAsync(coreIPList, coreConnectionCount)
// warn users about env mismatch
if (connectedCoreIPResult.networkMismatch)
logger.warn(`Unable to connect to Cores with a different network setting. This Node is set to '${env.NETWORK}'`)
// ensure we have successfully communicated with `coreConnectionCount` Cores
// if we have not, the Node cannot continue, log error and exit
if (!connectedCoreIPResult.connected) {
throw new Error(`Unable to connect to ${coreConnectionCount} Core(s) as required`)
}
CONNECTED_CORE_IPS = connectedCoreIPResult.ips
CONNECTED_CORE_LN_URIS = connectedCoreIPResult.lnUris
// if we're not paying cores, then skip lightning
// TODO: ability to pay some cores but not others
if (env.NO_LSAT_CORE_WHITELIST.length > 0) {
return
}
let done = false
while (!done) {
try {
try {
logger.info(`unlocking lightning...`)
await lnd.handleUnlock(null, env.HOT_WALLET_PASS)
} catch (error) {
utils.sleepAsync(2000)
}
logger.info(`syncing lightning...`)
await waitForSync()
logger.info(`peering with lightning nodes...`)
await createCoreLNDPeerConnectionsAsync(CONNECTED_CORE_LN_URIS)
utils.sleepAsync(10000) // takes a few seconds for peers to connect
logger.info(`creating lightning channels...`)
await createCoreLNDChannelsAsync(CONNECTED_CORE_LN_URIS)
done = true
logger.info(`opening lightning channels to Core complete`)
} catch (error) {
console.log(`open peer or channel failed: ${error.message}`)
done = false
utils.sleepAsync(2000)
}
}
logger.info(`App : Core IPs : ${CONNECTED_CORE_IPS}`)
}
async function waitForSync() {
let isSynced = false
while (!isSynced) {
try {
let info = await lnd.callMethodAsync('lightning', 'getInfoAsync', null, env.HOT_WALLET_PASS)
if (info.synced_to_chain) {
console.log(chalk.green('\n*****************************************'))
console.log(chalk.green('Your lightning node is fully synced.'))
console.log(chalk.green('*****************************************'))
isSynced = true
} else {
console.log(
chalk.magenta(
`${new Date().toISOString()}> Syncing in progress... currently at block height ${info.block_height}`
)
)
}
} catch (error) {
if (
!(
error.message.includes('failed to connect to all addresses') ||
error.message.includes('unknown service lnrpc.Lightning')
)
) {
console.log(chalk.red(`An error occurred while checking lnd state : ${error.message}`))
}
} finally {
if (!isSynced) await utils.sleepAsync(5000)
}
}
}
async function createCoreLNDPeerConnectionsAsync(lnUris) {
let peerPubKeys = []
try {
let peerList = await lnd.callMethodAsync('lightning', 'listPeersAsync', null, env.HOT_WALLET_PASS)
for (let peer of peerList.peers) {
peerPubKeys.push(peer.pub_key)
}
} catch (error) {
throw new Error('Could not retrieve LND peer list')
}
for (let lndUri of lnUris) {
let [pubkey, host] = lndUri.split('@')
if (peerPubKeys.includes(pubkey)) continue // already peered to this node, skip
try {
await lnd.callMethodAsync(
'lightning',
'connectPeerAsync',
{ addr: { pubkey, host }, perm: true },
env.HOT_WALLET_PASS
)
console.log(chalk.yellow(`Peer connection established with ${lndUri}`))
} catch (error) {
throw new Error(`Unable to establish a peer connection with ${lndUri} : ${error.message}`)
}
}
}
async function getConnectedCoreIPsAsync(coreIPList, coreConnectionCount) {
let getStatusOptions = buildRequestOptions(null, 'GET', '/status')
let connectedCoreIPs = []
let lnUris = []
let networkMismatch = false
coreIPList = _.shuffle(coreIPList)
for (let coreIP of coreIPList) {
logger.info(`Connecting to core ${coreIP}`)
try {
let coreResponse = await coreRequestAsync(getStatusOptions, coreIP, 0)
let networkMatch = coreResponse.network === env.NETWORK
let isCoreSynced = coreResponse.sync_info.catching_up === false
if (!networkMatch) networkMismatch = true
if (networkMatch && isCoreSynced) connectedCoreIPs.push(coreIP)
if (coreResponse.uris.length > 0) lnUris.push(coreResponse.uris[0])
} catch (error) {
console.log(`unable to contact core ${coreIP}: ${error.message}`)
}
// if we've made enough connections, break out of loop and return IPs
if (connectedCoreIPs.length >= coreConnectionCount) break
}
return {
connected: connectedCoreIPs.length >= coreConnectionCount,
ips: connectedCoreIPs,
lnUris: lnUris,
networkMismatch
}
}
async function createCoreLNDChannelsAsync(lnUris) {
let channelPubKeys = {}
try {
let channelList = await lnd.callMethodAsync('lightning', 'listChannelsAsync', {}, env.HOT_WALLET_PASS)
for (let channel of channelList.channels) {
channelPubKeys[channel.remote_pubkey] = channel
}
} catch (error) {
let msg = `Could not retrieve LND channel list: ${error.message}`
console.log(chalk.red(msg))
throw new Error(msg)
}
try {
let channelList = await lnd.callMethodAsync('lightning', 'pendingChannelsAsync', {}, env.HOT_WALLET_PASS)
for (let pendingChannel of channelList.pending_open_channels) {
channelPubKeys[pendingChannel.channel.remote_node_pub] = pendingChannel
console.log(
chalk.green(
`Channel to ${pendingChannel.channel.remote_node_pub} is pending and may require up to 6 confirmations (~60 minutes) to fully open`
)
)
}
} catch (error) {
let msg = `Could not retrieve pending LND channel list: ${error.message}`
console.log(chalk.red(msg))
throw new Error(msg)
}
for (let lndUri of lnUris) {
let pubkey = lndUri.split('@')[0]
if (channelPubKeys.hasOwnProperty(pubkey)) {
let chan = channelPubKeys[pubkey]
// close channel if all local funds are used up
if (
chan.hasOwnProperty('total_satoshis_sent') &&
chan.total_satoshis_sent > 0 &&
chan.local_balance <= env.MAX_SATOSHI_PER_HASH
) {
try {
let fee = await lnd.callMethodAsync(
'lightning',
'estimateFeeAsync',
{ target_confirmations: 6 },
env.HOT_WALLET_PASS
)
let closeChanReq = {
channel_point: {
funding_txid_bytes: Buffer.from(chan.transaction_id, 'hex').reverse(),
output_index: chan.transaction_vout
},
delivery_address: env.HOT_WALLET_ADDRESS,
force: false,
sat_per_byte: fee.tokens_per_vbyte,
target_conf: 6
}
let close = await lnd.callMethodAsync('lightning', 'closeChannelAsync', closeChanReq, env.HOT_WALLET_PASS)
console.log(`channel close txid: ${close.close_pending.txid.reverse().toString('hex')}`)
} catch (error) {
console.log(`unable to close channel: ${error.message}`)
}
} else {
continue
}
}
try {
let channelTxInfo = await lnd.callMethodAsync(
'lightning',
'openChannelSyncAsync',
{
node_pubkey_string: pubkey,
local_funding_amount: env.CHANNEL_AMOUNT,
push_sat: 0
},
env.HOT_WALLET_PASS
)
console.log(
`Channel created with ${lndUri} with the following transaction Id: ${Buffer.from(
channelTxInfo.funding_txid_bytes.data
).toString('hex')}`
)
} catch (error) {
let msg = `Unable to create a channel with ${lndUri} : ${error.message}`
console.log(chalk.red(msg))
throw new Error(msg)
}
}
}
async function coreRequestAsync(options, coreIP, retryCount = 3, timeout = 500) {
options.headers['X-Node-Version'] = version
options.uri = `http://${coreIP}${options.uriPath}`
let response
if (retryCount <= 0) {
try {
response = await rp(options)
} catch (error) {
if (error.statusCode === 402 || error.status === 402) return error.response
throw error
}
} else {
await retry(
async bail => {
try {
response = await rp(options)
} catch (error) {
// If no response was received or there is a status code >= 500 or payment is still pending/held,
// then we should retry the call, throw an error
if (!error.statusCode || error.statusCode >= 500 || error.statusCode === 402) throw error
// errors like 409 Conflict or 400 Bad Request are not retried because the request is bad and will never succeed
bail(error)
}
},
{
retries: retryCount, // The maximum amount of times to retry the operation. Default is 3
factor: 1, // The exponential factor to use. Default is 2
minTimeout: timeout, // The number of milliseconds before starting the first retry. Default is 200
randomize: true,
onRetry: error => {
if (error.statusCode === 402)
logger.warn(`Core request : 402: Payment Required. Core ${coreIP} : Request ${options.uri}. Retrying`)
else
logger.warn(
`Core request : ${error.statusCode || 'no response'} : Core ${coreIP} : Request ${
options.uri
} - ${JSON.stringify(options, null, 2)} : ${error.message} : retrying`
)
}
}
)
}
return response.body
}
async function submitHashAsync(hash) {
let responses = []
for (let coreIP of env.CHAINPOINT_CORE_CONNECT_IP_LIST) {
try {
// send initial request without LSAT. Expecting an LSAT w/ invoice in response
let postHashOptions = buildRequestOptions(null, 'POST', '/hash', { hash })
let submitResponse = await coreRequestAsync(postHashOptions, coreIP, 0)
if (!env.NO_LSAT_CORE_WHITELIST.includes(coreIP)) {
logger.info(`Aggregator : Response received from Core ${coreIP} : response : ${JSON.stringify(submitResponse)}`)
// get invoice for hash submission from LSAT challenge
let lsat = parse402Response(submitResponse)
let submitHashInvoiceId = lsat.paymentHash
let invoiceAmount = lsat.invoiceAmount
let decodedPaymentRequest = lsat.invoice
logger.info(
`Aggregator : Invoice received from Core ${coreIP} : invoiceId : ${submitHashInvoiceId} : Invoice Amount : ${invoiceAmount} Satoshis`
)
// ensure that the invoice amount does not exceed max payment amount
if (invoiceAmount > env.MAX_SATOSHI_PER_HASH)
throw new Error(
`Aggregator : Invoice amount exceeds max setting of ${env.MAX_SATOSHI_PER_HASH} : invoiceId : ${submitHashInvoiceId} : ${invoiceAmount}`
)
// pay the invoice
// since this is a hodl invoice, the request will stall until it is settled
// so we don't want to await the response but rather continue trying submission until complete
payInvoiceAsync(decodedPaymentRequest, submitHashInvoiceId)
.then(() => {
logger.info(
`Aggregator : Invoice paid to Core ${coreIP} : invoiceId : ${submitHashInvoiceId} : ${invoiceAmount}`
)
})
.catch(e => {
logger.error(e.message)
})
// submit hash with paid invoice id
let headers = { Authorization: lsat.toToken() }
postHashOptions = buildRequestOptions(headers, 'POST', '/hash', { hash })
// setting retries to 5 since we can't await invoice payment
// and don't know exactly when the invoice is paid and held which is when the hash can be submitted
submitResponse = await coreRequestAsync(postHashOptions, coreIP, 5, 1000)
logger.info(
`Aggregator : Hash submitted to Core ${coreIP} : invoiceId : ${submitHashInvoiceId} : ${invoiceAmount}`
)
} else {
logger.info(`Aggregator : Hash submitted to Core ${coreIP}`)
}
responses.push({ ip: coreIP, response: submitResponse })
} catch (error) {
// Ignore and try next coreIP
logger.warn(`submitHashAsync : Unable to submit to Core ${coreIP} : Hash = ${hash} : ${error.message}`)
}
}
return responses
}
async function payInvoiceAsync(invoice, submitHashInvoiceId) {
return new Promise(async (resolve, reject) => {
var call = await lnd.callMethodAsync('lightning', 'sendPayment', {})
call.on('data', function(response) {
// A response was received from the server.
call.end()
if (response.payment_error)
return reject(
new Error(`Error paying invoice : SubmitHashInvoiceId = ${submitHashInvoiceId} : ${response.payment_error}`)
)
return resolve(response)
})
call.on('error', err => {
console.log(`Error paying invoice : SubmitHashInvoiceId = ${submitHashInvoiceId} : ${err.message}`)
;(async () => await lnd.handleUnlock(err))()
return reject(new Error(`Error paying invoice : SubmitHashInvoiceId = ${submitHashInvoiceId} : ${err.message}`))
})
call.write({ payment_request: invoice })
})
}
async function getProofsAsync(coreIP, proofIds) {
let getProofsOptions = buildRequestOptions(
{
proofids: proofIds.join(',')
},
'GET',
'/proofs',
null,
20000
)
try {
let coreResponse = await coreRequestAsync(getProofsOptions, coreIP, 0)
return coreResponse
} catch (error) {
if (error.statusCode) throw new Error(`Invalid response on GET proof : ${error.statusCode} : ${error.message}`)
throw new Error(`Invalid response received on GET proof : ${error.message || error}`)
}
}
async function getLatestCalBlockInfoAsync() {
let getStatusOptions = buildRequestOptions(null, 'GET', '/status')
let lastError = null
for (let coreIP of env.CHAINPOINT_CORE_CONNECT_IP_LIST) {
try {
let coreResponse = await coreRequestAsync(getStatusOptions, coreIP, 0)
// if the Core is catching up, we cannot use its status to retrieve the latest cal block hash
if (coreResponse.sync_info.catching_up) throw 'Core not fully synced'
return coreResponse.sync_info
} catch (error) {
// Record most recent error, ignore, and try next coreIP
lastError = error
}
}
if (lastError.statusCode) throw new Error(`Invalid response on GET status : ${lastError.statusCode}`)
throw new Error('Invalid response received on GET status')
}
async function getCachedTransactionAsync(txID) {
// if the transaction already exists in the cache, return it
if (CORE_TX_CACHE[txID]) return CORE_TX_CACHE[txID].transaction
// otherwise, get the tranasction from Core
let getTxOptions = buildRequestOptions(null, 'GET', `/calendar/${txID}/data`)
for (let coreIP of env.CHAINPOINT_CORE_CONNECT_IP_LIST) {
try {
let transaction = await coreRequestAsync(getTxOptions, coreIP)
if (transaction) {
// cache the result and return the transaction
CORE_TX_CACHE[txID] = {
transaction: transaction,
expiresAt: Date.now() + 120 * 60 * 1000 // in 2 hours
}
return transaction
}
} catch (error) {
console.log(chalk.red(error.message))
}
}
return null
}
function buildRequestOptions(headerValues, method, uriPath, body, timeout = 3000) {
return {
headers: headerValues || {},
method: method,
uriPath: uriPath,
body: body || undefined,
json: true,
gzip: true,
timeout: timeout,
resolveWithFullResponse: true,
agent: false,
forever: true
}
}
function pruneExpiredItems() {
let now = Date.now()
for (let key in CORE_TX_CACHE) {
if (CORE_TX_CACHE[key].expiresAt <= now) {
delete CORE_TX_CACHE[key]
}
}
}
function startPruneExpiredItemsInterval() {
return setInterval(pruneExpiredItems, PRUNE_EXPIRED_INTERVAL_SECONDS * 1000)
}
function startConnectionMonitoringInterval() {
return setInterval(async () => {
await connectAsync()
}, 3600000)
}
function parse402Response(response) {
if (response.statusCode !== 402) throw new Error('Expected a 402 response')
if (!response.headers['www-authenticate'])
throw new Error('Missing www-authenticate header. Cannot parse LSAT challenge')
try {
const lsat = Lsat.fromChallenge(response.headers['www-authenticate'])
return lsat
} catch (e) {
logger.error(`Could not generate LSAT from challenge: ${e.message}`)
throw new Error('Problem processing www-authenticate header challenge for LSAT')
}
}
module.exports = {
connectAsync: connectAsync,
coreRequestAsync: coreRequestAsync,
submitHashAsync: submitHashAsync,
getProofsAsync: getProofsAsync,
getLatestCalBlockInfoAsync: getLatestCalBlockInfoAsync,
getCachedTransactionAsync: getCachedTransactionAsync,
startPruneExpiredItemsInterval: startPruneExpiredItemsInterval,
parse402Response: parse402Response,
getAllCoreIPs: () => ALL_CORE_IPS,
// additional functions for testing purposes
setCoreConnectionCount: c => (coreConnectionCount = c),
getCoreConnectedIPs: () => CONNECTED_CORE_IPS,
startConnectionMonitoringInterval: startConnectionMonitoringInterval,
pruneExpiredItems: pruneExpiredItems,
getPruneExpiredIntervalSeconds: () => PRUNE_EXPIRED_INTERVAL_SECONDS,
getCoreTxCache: () => CORE_TX_CACHE,
setCoreTxCache: obj => {
CORE_TX_CACHE = obj
},
setENV: obj => {
env = obj
},
setRP: RP => {
rp = RP
},
setLN: LN => {
lnd = LN
},
getLn: () => {
return lnd
}
}
================================================
FILE: lib/endpoints/calendar.js
================================================
/**
* Copyright 2019 Tierion
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
const errors = require('restify-errors')
let cores = require('../cores.js')
async function getDataValueByIDAsync(req, res, next) {
res.contentType = 'application/json'
let txId = req.params.tx_id
// validate TM txId is well formed
let containsValidTxId = /^([a-fA-F0-9]{2}){32}$/.test(txId)
if (!containsValidTxId) {
return next(new errors.InvalidArgumentError('invalid JSON body, invalid txId present'))
}
// check Core for the transaction
let txInfo = await cores.getCachedTransactionAsync(txId)
if (txInfo === null) {
return next(new errors.NotFoundError())
}
res.contentType = 'text/plain'
res.send(txInfo.toLowerCase())
return next()
}
module.exports = {
getDataValueByIDAsync: getDataValueByIDAsync,
// additional functions for testing purposes
setCores: c => {
cores = c
}
}
================================================
FILE: lib/endpoints/config.js
================================================
/**
* Copyright 2019 Tierion
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
const { version } = require('../../package.json')
let env = require('../parse-env.js').env
/**
* GET /config handler
*
* Returns a configuration information object
*/
let lnd
async function getConfigInfoAsync(req, res, next) {
res.contentType = 'application/json'
let info
try {
info = await lnd.callMethodAsync('lightning', 'getInfoAsync', null, env.HOT_WALLET_PASS)
} catch (error) {
info = { error: error.message }
}
let balance
try {
balance = await lnd.callMethodAsync('lightning', 'walletBalanceAsync', null, env.HOT_WALLET_PASS)
} catch (error) {
balance = { error: error.message }
}
res.send({
lndInfo: info,
lightning_balance: balance,
version: version,
time: new Date().toISOString()
})
return next()
}
module.exports = {
getConfigInfoAsync: getConfigInfoAsync,
setLnd: lndObj => {
lnd = lndObj
}
}
================================================
FILE: lib/endpoints/hashes.js
================================================
/**
* Copyright 2019 Tierion
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
// load environment variables
let env = require('../parse-env.js').env
const errors = require('restify-errors')
const _ = require('lodash')
const utils = require('../utils.js')
let rocksDB = require('../models/RocksDB.js')
const logger = require('../logger.js')
const analytics = require('../analytics.js')
const { UlidMonotonic } = require('id128')
/**
* Generate the values for the 'meta' property in a POST /hashes response.
*
* Returns an Object with metadata about a POST /hashes request
* including a 'timestamp', and hints for estimated time to completion
* for various operations.
*
* @returns {Object}
*/
function generatePostHashesResponseMetadata() {
let metaDataObj = {}
let timestamp = new Date()
metaDataObj.hash_received = utils.formatDateISO8601NoMs(timestamp)
metaDataObj.processing_hints = generateProcessingHints(timestamp)
return metaDataObj
}
/**
* Generate the expected proof ready times for each proof stage
*
* @param {Date} timestampDate - The hash submission timestamp
* @returns {Object} An Object with 'cal' and 'btc' properties
*
*/
function generateProcessingHints(timestampDate) {
// cal proof aggregation occurs at :30 seconds past each minute
// allow and extra 30 seconds for processing
let maxLocalAggregationFromTimestamp = utils.addSeconds(timestampDate, env.AGGREGATION_INTERVAL_SECONDS)
let maxSeconds = maxLocalAggregationFromTimestamp.getSeconds()
let secondsUntil30Past = maxSeconds < 30 ? 30 - maxSeconds : 90 - maxSeconds
let calHint = utils.formatDateISO8601NoMs(utils.addSeconds(maxLocalAggregationFromTimestamp, secondsUntil30Past + 30))
let twoHoursFromTimestamp = utils.addMinutes(timestampDate, 120)
let btcHint = utils.formatDateISO8601NoMs(twoHoursFromTimestamp)
return {
cal: calHint,
btc: btcHint
}
}
/**
* Converts an array of hash strings to a object suitable to
* return to HTTP clients.
*
* @param {string[]} hashes - An array of string hashes to process
* @returns {Object} An Object with 'meta' and 'hashes' properties
*/
function generatePostHashesResponse(ip, hashes) {
let lcHashes = utils.lowerCaseHashes(hashes)
let hashObjects = lcHashes.map(hash => {
let proofId
try {
proofId = UlidMonotonic.generate().toCanonical()
} catch (error) {
UlidMonotonic.reset()
proofId = UlidMonotonic.generate().toCanonical()
}
let hashObj = {}
hashObj.proof_id = proofId
hashObj.hash = hash
logger.info(`Created proof_id ${proofId}`)
//send event to google UA
var hashEvent = {
ec: env.GATEWAY_NAME,
ea: 'CreateProof',
el: proofId,
cd1: hash,
cd2: utils.formatDateISO8601NoMs(new Date()),
cd3: env.PUBLIC_IP,
cd4: ip,
dp: '/hash'
}
analytics.setClientID(proofId)
analytics.sendEvent(hashEvent)
return hashObj
})
return {
meta: generatePostHashesResponseMetadata(hashObjects),
hashes: hashObjects
}
}
/**
* POST /hashes handler
*
* Expects a JSON body with the form:
* {"hashes": ["hash1", "hash2", "hashN"]}
*
* The `hashes` key must reference a JSON Array
* of strings representing each hash to anchor.
*
* Each hash must be:
* - in Hexadecimal form [a-fA-F0-9]
* - minimum 40 chars long (e.g. 20 byte SHA1)
* - maximum 128 chars long (e.g. 64 byte SHA512)
* - an even length string
*/
async function postHashesAsync(req, res, next) {
res.contentType = 'application/json'
// validate content-type sent was 'application/json'
if (req.contentType() !== 'application/json') {
return next(new errors.InvalidArgumentError('invalid content type'))
}
// validate params has parse a 'hashes' key
if (!req.params.hasOwnProperty('hashes')) {
return next(new errors.InvalidArgumentError('invalid JSON body, missing hashes'))
}
// validate hashes param is an Array
if (!_.isArray(req.params.hashes)) {
return next(new errors.InvalidArgumentError('invalid JSON body, hashes is not an Array'))
}
// validate hashes param Array has at least one hash
if (_.size(req.params.hashes) < 1) {
return next(new errors.InvalidArgumentError('invalid JSON body, hashes Array is empty'))
}
// validate hashes param Array is not larger than allowed max length
if (_.size(req.params.hashes) > env.POST_HASHES_MAX) {
return next(
new errors.InvalidArgumentError(`invalid JSON body, hashes Array max size of ${env.POST_HASHES_MAX} exceeded`)
)
}
// validate hashes are individually well formed
let containsValidHashes = _.every(req.params.hashes, hash => {
return /^([a-fA-F0-9]{2}){20,64}$/.test(hash)
})
if (!containsValidHashes) {
return next(new errors.InvalidArgumentError('invalid JSON body, invalid hashes present'))
}
let ip = utils.getClientIP(req)
logger.info(`Incoming hash from ${ip}`)
let responseObj = generatePostHashesResponse(ip, req.params.hashes)
// store hash data for later aggregation
try {
await rocksDB.queueIncomingHashObjectsAsync(responseObj.hashes)
} catch (error) {
return next(new errors.InternalServerError('Could not save hash data'))
}
res.send(responseObj)
return next()
}
module.exports = {
postHashesAsync: postHashesAsync,
generatePostHashesResponse: generatePostHashesResponse,
// additional functions for testing purposes
setRocksDB: db => {
rocksDB = db
},
setENV: obj => {
env = obj
}
}
================================================
FILE: lib/endpoints/proofs.js
================================================
/**
* Copyright 2019 Tierion
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
let env = require('../parse-env.js').env
const errors = require('restify-errors')
const utils = require('../utils.js')
const uuidValidate = require('uuid-validate')
const uuidTime = require('uuid-time')
const chpBinary = require('chainpoint-binary')
const _ = require('lodash')
let cachedProofs = require('../cached-proofs.js')
let rocksDB = require('../models/RocksDB.js')
const logger = require('../logger.js')
const analytics = require('../analytics.js')
const { UlidMonotonic } = require('id128')
// The custom MIME type for JSON proof array results containing Base64 encoded proof data
const BASE64_MIME_TYPE = 'application/vnd.chainpoint.json+base64'
// The custom MIME type for JSON proof array results containing Base64 encoded proof data
const JSONLD_MIME_TYPE = 'application/vnd.chainpoint.ld+json'
/**
* Converts proof path array output from the merkle-tools package
* to a Chainpoint v3 ops array
*
* @param {proof object array} proof - The proof array generated by merkle-tools
* @param {string} op - The hash type performed throughout merkle tree construction (sha-256, sha-512, sha-256-x2, etc.)
* @returns {ops object array}
*/
function formatAsChainpointV3Ops(proof, op) {
let ChainpointV3Ops = proof.reduce((result, item) => {
if (item.left) {
item = { l: item.left }
} else {
item = { r: item.right }
}
result.push(item, { op: op })
return result
}, [])
return ChainpointV3Ops
}
/**
* GET /proofs/:proof_id handler
*
* Expects a path parameter 'proof_id' in the form of a Version 1 UUID
*
* Returns a chainpoint proof for the requested Proof ID
*/
async function getProofsByIDAsync(req, res, next) {
res.contentType = 'application/json'
let ip = utils.getClientIP(req)
let proofIds = []
// check if proof_id parameter was included
if (req.params && req.params.proof_id) {
// a proof_id was specified in the url, so use that proof_id only
proofIds.push(req.params.proof_id)
} else if (req.headers && req.headers.proofids) {
// no proof_id was specified in url, read from headers.proofids
proofIds = req.headers.proofids.split(',').map(_.trim)
}
// ensure at least one proof_id was submitted
if (proofIds.length === 0) {
return next(new errors.InvalidArgumentError('invalid request, at least one proof id required'))
}
// ensure that the request count does not exceed the maximum setting
if (proofIds.length > env.GET_PROOFS_MAX) {
return next(new errors.InvalidArgumentError('invalid request, too many proof ids (' + env.GET_PROOFS_MAX + ' max)'))
}
// ensure all proof_ids are valid, send event
for (let proofId of proofIds) {
if (!(uuidValidate(proofId, 1) || utils.isULID(proofId))) {
return next(new errors.InvalidArgumentError('invalid request, bad proof_id'))
}
// Proof endpoint (always logged regardless of proof retrieval success)
var startProofEvent = {
ec: env.GATEWAY_NAME,
ea: 'GetProof',
el: proofId,
cd1: 'Start',
cd2: utils.formatDateISO8601NoMs(new Date()),
cd3: env.PUBLIC_IP,
cd4: ip,
dp: '/proof'
}
analytics.setClientID(proofId)
analytics.sendEvent(startProofEvent)
}
let requestedType =
req.accepts(JSONLD_MIME_TYPE) && !req.accepts(BASE64_MIME_TYPE) ? JSONLD_MIME_TYPE : BASE64_MIME_TYPE
// retrieve all the nodeProofDataItems for the requested proofIds
let nodeProofDataItems = []
try {
nodeProofDataItems = await rocksDB.getProofStatesBatchByProofIdsAsync(proofIds)
} catch (error) {
return next(new errors.InternalError('error retrieving node proof data items'))
}
// convert all proof state value to chpv3
nodeProofDataItems = nodeProofDataItems.map(nodeProofDataItem => {
if (nodeProofDataItem.proofState === null) return nodeProofDataItem
nodeProofDataItem.proofState = formatAsChainpointV3Ops(nodeProofDataItem.proofState, 'sha-256')
return nodeProofDataItem
})
// get an array of all unique core submission objects from the proof data items by submitId
let knownSubmitIds = []
let uniqueCoreSubmissions = nodeProofDataItems.reduce((result, val) => {
// if the node data item was not found for that proofId, skip this item
if (_.isNil(val.submission)) return result
// add this submission object to the results if it is unique
let submitId = val.submission.submitId
if (!knownSubmitIds.includes(submitId)) {
knownSubmitIds.push(submitId)
result.push(val.submission)
}
return result
}, [])
// get proofs for each unique submitId
let coreProofDataItems
try {
coreProofDataItems = await cachedProofs.getCachedCoreProofsAsync(uniqueCoreSubmissions)
} catch (error) {
logger.error(`Could not get proofs from Core : ${error.message}`)
return next(new errors.InternalError('error retrieving proofs from Core'))
}
// assemble all Core proofs received into an object keyed by submitId
coreProofDataItems = coreProofDataItems.reduce((results, coreProofDataItem) => {
results[coreProofDataItem.submitId] = coreProofDataItem
return results
}, {})
// build the resulting proofs from the collected data for each proof_id
let results = []
for (let nodeProofDataItem of nodeProofDataItems) {
let submitId = nodeProofDataItem.submission ? nodeProofDataItem.submission.submitId : null
let coreProof = coreProofDataItems[submitId] ? coreProofDataItems[submitId].proof : null
let fullProof = null
if (coreProof) {
fullProof = buildFullProof(coreProof, nodeProofDataItem)
}
let proofResult = fullProof
if (requestedType === BASE64_MIME_TYPE && fullProof) {
try {
proofResult = chpBinary.objectToBase64Sync(fullProof)
} catch (error) {
if (nodeProofDataItem) {
logger.error(`Could not convert binary proof for proof id : ${nodeProofDataItem.proofId}`)
logger.error(`Binary Conversion Error : ${error.message}`)
if (fullProof) {
logger.error(JSON.stringify(fullProof))
}
}
return next(error)
}
}
let result = {
proof_id: nodeProofDataItem.proofId,
proof: proofResult,
anchors_complete: coreProofDataItems[submitId] ? coreProofDataItems[submitId].anchorsComplete : []
}
results.push(result)
//send events
var event = {
ec: env.GATEWAY_NAME,
el: result.proof_id,
cd1: result.proof ? result.proof.hash : 'null',
cd2: utils.formatDateISO8601NoMs(new Date()),
cd3: env.PUBLIC_IP,
cd4: ip,
dp: '/proof'
}
analytics.setClientID(result.proof_id)
if (fullProof) {
try {
let proofString = JSON.stringify(fullProof)
if (proofString.includes('btc') || proofString.includes('tbtc')) { // eslint-disable-line
event.ea = 'GetProofSuccessBtc'
analytics.sendEvent(event)
} else if (proofString.includes('cal') || proofString.includes('tcal')) { // eslint-disable-line
event.ea = 'GetProofSuccessCal'
analytics.sendEvent(event)
}
} catch (error) {
logger.error(`could not get proof size of proof ${result.proof_id}`)
}
} else {
event.ea = 'GetProofFail'
analytics.sendEvent(event)
}
}
res.send(results)
return next()
}
function buildFullProof(coreProof, nodeProofDataItem) {
if (!coreProof || !nodeProofDataItem) return null
let fullProof = _.cloneDeep(coreProof)
let coreBranches = _.cloneDeep(fullProof.branches)
fullProof.proof_id = nodeProofDataItem.proofId
fullProof.hash = nodeProofDataItem.hash
if (utils.isULID(nodeProofDataItem.proofId)) {
fullProof.hash_received = utils.formatDateISO8601NoMs(UlidMonotonic.fromCanonical(nodeProofDataItem.proofId).time)
} else {
fullProof.hash_received = utils.formatDateISO8601NoMs(new Date(parseInt(uuidTime.v1(nodeProofDataItem.proofId))))
}
fullProof.branches[0].label = 'aggregator'
fullProof.branches[0].ops = nodeProofDataItem.proofState
fullProof.branches[0].branches = coreBranches
fullProof = utils.jsonTransform(
fullProof,
// eslint-disable-next-line no-unused-vars
(key, value) => key.includes('uris'),
(key, value) => {
value = value.replace('https://', 'http://')
if (!value.includes('http://')) {
value = 'http://' + value
}
return value
}
)
return fullProof
}
module.exports = {
getProofsByIDAsync: getProofsByIDAsync,
// additional functions for testing purposes
setRocksDB: db => {
rocksDB = db
},
setCachedProofs: cp => {
cachedProofs = cp
},
setENV: obj => {
env = obj
}
}
================================================
FILE: lib/endpoints/verify.js
================================================
/**
* Copyright 2019 Tierion
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
let env = require('../parse-env.js').env
const _ = require('lodash')
const chpParse = require('chainpoint-parse')
const errors = require('restify-errors')
let cores = require('../cores.js')
const parallel = require('async-await-parallel')
const logger = require('../logger.js')
async function ProcessVerifyTasksAsync(verifyTasks) {
let processedTasks = []
for (let x = 0; x < verifyTasks.length; x++) {
let verifyTask = verifyTasks[x]
// check for malformatted proofs
let status = verifyTask.status
if (status === 'malformed') {
processedTasks.push({
proof_index: verifyTask.proof_index,
status: status
})
continue
}
// check for anchors on unsupported network
let supportedAnchorTypes = []
// if this Node is running in mainnet mode, only accept mainnet anchors
if (env.NETWORK === 'mainnet') supportedAnchorTypes = ['cal', 'btc']
// if this Node is running in testnet mode, only accept testnet anchors
if (env.NETWORK === 'testnet') supportedAnchorTypes = ['tcal', 'tbtc']
// if there is a network mismatch, do not attempt to verify proof
let mismatchFound = false
for (let x = 0; x < verifyTask.anchors.length; x++) {
if (!supportedAnchorTypes.includes(verifyTask.anchors[x].anchor.type)) {
processedTasks.push({
proof_index: verifyTask.proof_index,
status: `This is a '${env.NETWORK}' Node supporting '${supportedAnchorTypes.join(
"' and '"
)}' anchor types. Cannot verify '${verifyTask.anchors[x].anchor.type}' anchors.`
})
mismatchFound = true
break
}
}
if (mismatchFound) continue
let totalCount = 0
let validCount = 0
let anchorResults = []
let confirmTasks = []
for (let x = 0; x < verifyTask.anchors.length; x++) {
confirmTasks.push(async () => {
return confirmExpectedValueAsync(verifyTask.anchors[x].anchor)
})
}
let confirmResults = []
if (confirmTasks.length > 0) {
try {
confirmResults = await parallel(confirmTasks, 20)
} catch (error) {
logger.error(`Could not confirm proof data ${error.message}`)
throw new Error('error confirming proof data')
}
}
for (let x = 0; x < verifyTask.anchors.length; x++) {
try {
let anchor = verifyTask.anchors[x]
let confirmResult = confirmResults[x]
let anchorResult = {
branch: anchor.branch || null,
type: anchor.anchor.type,
valid: confirmResult
}
totalCount++
validCount = validCount + (anchorResult.valid === true ? 1 : 0)
anchorResults.push(anchorResult)
} catch (error) {
logger.error('Verification error')
}
}
if (validCount === 0) {
status = 'invalid'
} else if (validCount === totalCount) {
status = 'verified'
} else {
status = 'mixed'
}
let result = {
proof_index: verifyTask.proof_index,
hash: verifyTask.hash,
proof_id: verifyTask.proof_id,
hash_received: verifyTask.hash_received,
anchors: anchorResults,
status: status
}
processedTasks.push(result)
}
return processedTasks
}
function BuildVerifyTaskList(proofs) {
let results = []
let proofIndex = 0
// extract id, time, anchors, and calculate expected values
_.forEach(proofs, proof => {
try {
let parseObj = chpParse.parse(proof)
results.push(buildResultObject(parseObj, proofIndex++))
} catch (error) {
// continue regardless of error
results.push(buildResultObject(null, proofIndex++))
}
})
return results
}
function buildResultObject(parseObj, proofIndex) {
let hash = parseObj !== null ? parseObj.hash : undefined
let proofId = parseObj !== null ? parseObj.proof_id : undefined
let hashReceived = parseObj !== null ? parseObj.hash_received : undefined
let expectedValues = parseObj !== null ? flattenExpectedValues(parseObj.branches) : undefined
return {
proof_index: proofIndex,
hash: hash,
proof_id: proofId,
hash_received: hashReceived,
anchors: expectedValues,
status: parseObj === null ? 'malformed' : ''
}
}
async function confirmExpectedValueAsync(anchorInfo) {
let anchorUri = anchorInfo.uris[0]
let anchorTxId = _.takeRight(anchorUri.split('/'), 2)[0]
let expectedValue = anchorInfo.expected_value
// check Core for the transaction
let txInfo = await cores.getCachedTransactionAsync(anchorTxId)
if (txInfo === null) {
throw new Error('Unable to retrieve transaction from Core to the confirm the anchor value')
}
return txInfo === expectedValue
}
function flattenExpectedValues(branchArray) {
let results = []
for (let b = 0; b < branchArray.length; b++) {
let anchors = branchArray[b].anchors
if (anchors.length > 0) {
for (let a = 0; a < anchors.length; a++) {
results.push({
branch: branchArray[b].label || undefined,
anchor: anchors[a]
})
}
}
if (branchArray[b].branches) {
results = results.concat(flattenExpectedValues(branchArray[b].branches))
}
return results
}
}
/**
* POST /verify handler
*
* Expects a JSON body with the form:
* {"proofs": [ {proofJSON1}, {proofJSON2}, {proofJSON3} ]}
* or
* {"proofs": [ "proof binary 1", "proof binary 2", "proof binary 3" ]}
*
* The `proofs` key must reference a JSON Array of chainpoint proofs.
* Proofs may be in either JSON form or base64 encoded binary form.
*
*/
async function postProofsForVerificationAsync(req, res, next) {
res.contentType = 'application/json'
// validate content-type sent was 'application/json'
if (req.contentType() !== 'application/json') {
return next(new errors.InvalidArgumentError('Invalid content type'))
}
// validate params has parse a 'proofs' key
if (!req.params.hasOwnProperty('proofs')) {
return next(new errors.InvalidArgumentError('Invalid JSON body, missing proofs'))
}
// validate proofs param is an Array
if (!_.isArray(req.params.proofs)) {
return next(new errors.InvalidArgumentError('Invalid JSON body, proofs is not an Array'))
}
// validate proofs param Array has at least one hash
if (_.size(req.params.proofs) < 1) {
return next(new errors.InvalidArgumentError('Invalid JSON body, proofs Array is empty'))
}
// validate proofs param Array is not larger than allowed max length
if (_.size(req.params.proofs) > env.POST_VERIFY_PROOFS_MAX) {
return next(
new errors.InvalidArgumentError(
`Invalid JSON body, proofs Array max size of ${env.POST_VERIFY_PROOFS_MAX} exceeded`
)
)
}
let verifyTasks
try {
verifyTasks = BuildVerifyTaskList(req.params.proofs)
} catch (error) {
logger.error(`Could not build verify list: ${error.message}`)
return next(new errors.InternalError('Node internal error verifying proof(s)'))
}
let verifyResults
try {
verifyResults = await ProcessVerifyTasksAsync(verifyTasks)
} catch (error) {
return next(new errors.InternalError('Node internal error verifying proof(s)'))
}
res.send(verifyResults)
return next()
}
module.exports = {
postProofsForVerificationAsync: postProofsForVerificationAsync,
// additional functions for testing purposes
setCores: c => {
cores = c
},
setENV: obj => {
env = obj
}
}
================================================
FILE: lib/lightning.js
================================================
/**
* Copyright 2019 Tierion
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
// load environment variables
let env = require('./parse-env.js').env
const lnRPCNodeClient = require('lnrpc-node-client')
const retry = require('async-retry')
const dotenv = require('dotenv')
// track if unlock is in process to prevent multiple simultaneous unlock errors
let IS_UNLOCKING = false
let LND_DIR
let LND_SOCKET
let LND_TLS_CERT
let LND_MACAROON
dotenv.config()
let lnd = function(socket, network, unlockOnly = false, inHostContext = false) {
if (!['mainnet', 'testnet'].includes(network)) throw new Error('Invalid network value')
LND_DIR = process.env.LND_DIR || `${process.env.HOME}/${inHostContext ? '.chainpoint/gateway/' : ''}.lnd`
LND_SOCKET = socket
LND_TLS_CERT = process.env.LND_TLS_CERT || `${LND_DIR}/tls.cert`
LND_MACAROON = process.env.LND_MACAROON || `${LND_DIR}/data/chain/bitcoin/${network}/admin.macaroon`
if (unlockOnly) {
lnRPCNodeClient.setTls(LND_SOCKET, LND_TLS_CERT)
} else {
lnRPCNodeClient.setCredentials(LND_SOCKET, LND_MACAROON, LND_TLS_CERT)
}
// Call the service method with the given parameter
// If a locked wallet is detected, wallet unlock is attempted and call is retried
this.callMethodAsync = async (service, method, params, hotWalletPass = null) => {
let lndService = lnRPCNodeClient[service]()
return await retry(async () => await lndService[method](params), {
retries: 30,
factor: 1,
minTimeout: 1000,
onRetry: async error => {
if (!error.message.includes('failed to connect to all addresses')) {
if (!error.message.includes('unknown service lnrpc.Lightning')) {
console.log(error)
}
await this.handleUnlock(error, hotWalletPass)
}
}
})
}
this.handleUnlock = async (err, hotWalletPass) => {
if (err == null || err.code === 2 || err.code === 12) {
// error code 12 indicates wallet may be locked
if (!IS_UNLOCKING) {
IS_UNLOCKING = true
try {
await lnRPCNodeClient
.unlocker()
.unlockWalletAsync({ wallet_password: hotWalletPass || env.HOT_WALLET_PASS, recovery_window: 10000 })
} catch (error) {
throw new Error(`Unable to unlock LND wallet : ${error.message}`)
} finally {
IS_UNLOCKING = false
}
}
}
}
// Call the service method with the given parameter
// No unlocking or retrying is performed with this method
this.callMethodRawAsync = async (service, method, params) => await lnRPCNodeClient[service]()[method](params)
}
module.exports = lnd
================================================
FILE: lib/logger.js
================================================
/**
* Copyright 2019 Tierion
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
// load environment variables
let env = require('./parse-env.js').env
const winston = require('winston')
const myFormat = winston.format.printf(({ level, message, timestamp }) => `${timestamp} [${level}] ${message}`)
let consoleOpts = {
level: 'info',
stderrLevels: ['error'],
format: winston.format.combine(winston.format.colorize({ all: true }), winston.format.timestamp(), myFormat)
}
if (env.NODE_ENV === 'test') consoleOpts.silent = true
const logger = winston.createLogger({
transports: [new winston.transports.Console(consoleOpts)]
})
module.exports = logger
================================================
FILE: lib/models/RocksDB.js
================================================
/**
* Copyright 2019 Tierion
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
let env = require('../parse-env.js').env
const level = require('level-rocksdb')
const crypto = require('crypto')
const path = require('path')
const JSBinaryType = require('js-binary').Type
const logger = require('../logger.js')
const utils = require('../utils.js')
const { Ulid } = require('id128')
// See Options: https://github.com/level/leveldown#options
// Setup with options, all default except:
// cacheSize : which was increased from 8MB to 32MB
let options = {
createIfMissing: true,
errorIfExists: false,
compression: true,
cacheSize: 32 * 1024 * 1024,
writeBufferSize: 4 * 1024 * 1024,
blockSize: 4096,
maxOpenFiles: 1000,
blockRestartInterval: 16,
maxFileSize: 2 * 1024 * 1024,
keyEncoding: 'binary',
valueEncoding: 'binary'
}
const prefixBuffers = {
PROOF_STATE_INDEX: Buffer.from('b1a1', 'hex'),
PROOF_STATE_VALUE: Buffer.from('b1a2', 'hex'),
INCOMING_HASH_OBJECTS: Buffer.from('b1b1', 'hex'),
REP_ITEM_VALUE: Buffer.from('b1c1', 'hex'),
REP_ITEM_ID_INDEX: Buffer.from('b1c2', 'hex'),
REP_ITEM_PROOF_VALUE: Buffer.from('b1c3', 'hex')
}
const PRUNE_BATCH_SIZE = 1000
const PRUNE_INTERVAL_SECONDS = 10
let PRUNE_IN_PROGRESS = false
let db = null
async function openConnectionAsync(dir = `${process.env.HOME}/.chainpoint/gateway/data/rocksdb`) {
return new Promise(resolve => {
level(path.resolve(dir), options, (err, conn) => {
if (err) {
logger.error(`Unable to open database : ${err.message}`)
process.exit(0)
} else {
db = conn
resolve(db)
}
})
})
}
/****************************************************************************************************
* DEFINE SCHEMAS
****************************************************************************************************/
// #region SCHEMAS
const nodeProofDataItemSchema = new JSBinaryType({
proofId: 'Buffer',
hash: 'Buffer',
proofState: ['Buffer'],
submission: {
submitId: 'Buffer',
cores: [{ ip: 'string', proofId: 'Buffer' }]
}
})
const incomingHashObjectsSchema = new JSBinaryType([
{
proof_id: 'Buffer',
hash: 'Buffer'
}
])
// #endregion SCHEMAS
/****************************************************************************************************
* PROOF STATE FUNCTIONS
****************************************************************************************************/
// #region PROOF STATE FUNCTIONS
function encodeBinaryProofStateId(proofIdNode) {
let idBuffer
let idType
if (utils.isUUID(proofIdNode)) {
idBuffer = Buffer.from(proofIdNode.replace(/-/g, ''), 'hex')
idType = 'uuid'
} else if (utils.isULID(proofIdNode)) {
idBuffer = Buffer.from(Ulid.fromCanonicalTrusted(proofIdNode).bytes)
idType = 'ulid'
}
return { idType: idType, idBuffer: idBuffer }
}
function decodeBinaryProofStateId(idType, proofId) {
return idType == 'uuid'
? utils.hexToUUIDv1(proofId.toString('hex'))
: Ulid.construct(Uint8Array.from(proofId)).toCanonical()
}
function encodeBinaryProofStateValueKey(proofIdNode) {
let idBuffer = encodeBinaryProofStateId(proofIdNode).idBuffer
return Buffer.concat([prefixBuffers.PROOF_STATE_VALUE, idBuffer])
}
function createBinaryProofStateTimeIndexKey() {
// generate a new key for the current time
let timestampBuffer = Buffer.alloc(8)
timestampBuffer.writeDoubleBE(Date.now())
let rndBuffer = crypto.randomBytes(16)
return Buffer.concat([prefixBuffers.PROOF_STATE_INDEX, timestampBuffer, rndBuffer])
}
function createBinaryProofStateTimeIndexMin() {
// generate the minimum key value for range query
let minBoundsBuffer = Buffer.alloc(24, 0)
return Buffer.concat([prefixBuffers.PROOF_STATE_INDEX, minBoundsBuffer])
}
function createBinaryProofStateTimeIndexMax(timestamp) {
// generate the maximum key value for range query up to given timestamp
let timestampBuffer = Buffer.alloc(8, 0)
timestampBuffer.writeDoubleBE(timestamp)
let rndBuffer = Buffer.alloc(16, 'ff', 'hex')
return Buffer.concat([prefixBuffers.PROOF_STATE_INDEX, timestampBuffer, rndBuffer])
}
function encodeProofStateValue(nodeProofDataItem) {
let stateObj = {
proofId: encodeBinaryProofStateId(nodeProofDataItem.proofId).idBuffer,
hash: Buffer.from(nodeProofDataItem.hash, 'hex'),
proofState: nodeProofDataItem.proofState,
submission: {
submitId: encodeBinaryProofStateId(nodeProofDataItem.submission.submitId).idBuffer,
cores: nodeProofDataItem.submission.cores.map(core => {
return { ip: core.ip, proofId: encodeBinaryProofStateId(core.proofId).idBuffer }
})
}
}
return nodeProofDataItemSchema.encode(stateObj)
}
function decodeProofStateValue(proofStateValue, idType) {
let leftCode = Buffer.from('\x00')
let rightCode = Buffer.from('\x01')
let stateObj = nodeProofDataItemSchema.decode(proofStateValue)
let nodeProofDataItem = {
proofId: decodeBinaryProofStateId(idType, stateObj.proofId),
hash: stateObj.hash.toString('hex'),
proofState: stateObj.proofState.reduce((result, op, index, proofState) => {
if (op.equals(leftCode)) result.push({ left: proofState[index + 1].toString('hex') })
if (op.equals(rightCode)) result.push({ right: proofState[index + 1].toString('hex') })
return result
}, []),
submission: {
submitId: decodeBinaryProofStateId(idType, stateObj.submission.submitId),
cores: stateObj.submission.cores.map(core => {
return {
ip: core.ip,
proofId: decodeBinaryProofStateId(idType, core.proofId)
}
})
}
}
return nodeProofDataItem
}
async function getProofStatesBatchByProofIdsAsync(proofIds) {
let results = []
for (let proofId of proofIds) {
try {
let proofIdType = encodeBinaryProofStateId(proofId).idType
let proofStateValueKey = encodeBinaryProofStateValueKey(proofId)
let proofStateValue = await db.get(proofStateValueKey)
let nodeProofDataItem = decodeProofStateValue(proofStateValue, proofIdType)
results.push(nodeProofDataItem)
} catch (error) {
if (error.notFound) {
results.push({
proofId: proofId,
hash: null,
proofState: null,
submission: null
})
} else {
let err = `Unable to read proof state for hash with proofId = ${proofId} : ${error.message}`
throw err
}
}
}
return results
}
async function saveProofStatesBatchAsync(nodeProofDataItems) {
let ops = []
for (let nodeProofDataItem of nodeProofDataItems) {
let proofStateValueKey = encodeBinaryProofStateValueKey(nodeProofDataItem.proofId)
let proofStateTimeIndexKey = createBinaryProofStateTimeIndexKey()
let proofStateValue = encodeProofStateValue(nodeProofDataItem)
ops.push({ type: 'put', key: proofStateValueKey, value: proofStateValue })
ops.push({ type: 'put', key: proofStateTimeIndexKey, value: proofStateValueKey })
}
try {
await db.batch(ops)
} catch (error) {
let err = `Unable to write proof state : ${error.message}`
throw err
}
}
async function pruneProofStateDataSince(timestampMS) {
return new Promise((resolve, reject) => {
let delOps = []
let minKey = createBinaryProofStateTimeIndexMin()
let maxKey = createBinaryProofStateTimeIndexMax(timestampMS)
db.createReadStream({ gte: minKey, lte: maxKey })
.on('data', async data => {
delOps.push({ type: 'del', key: data.key })
delOps.push({ type: 'del', key: data.value })
// Execute in batches of PRUNE_BATCH_SIZE
if (delOps.length >= PRUNE_BATCH_SIZE) {
try {
let delOpsBatch = delOps.splice(0)
await db.batch(delOpsBatch)
} catch (error) {
let err = `Error during proof state batch delete : ${error.message}`
return reject(err)
}
}
})
.on('error', error => {
let err = `Error reading proof state keys for pruning : ${error.message}`
return reject(err)
})
.on('end', async () => {
try {
await db.batch(delOps)
} catch (error) {
return reject(error.message)
}
return resolve()
})
})
}
async function pruneOldProofStateDataAsync() {
let pruneTime = Date.now() - env.PROOF_EXPIRE_MINUTES * 60 * 1000
try {
await pruneProofStateDataSince(pruneTime)
} catch (error) {
logger.warn(`An error occurred during proof state pruning : ${error.message}`)
}
}
// #endregion PROOF STATE FUNCTIONS
/****************************************************************************************************
* INCOMING HASH QUEUE FUNCTIONS
****************************************************************************************************/
// #region INCOMING HASH QUEUE FUNCTIONS
function createBinaryIncomingHashObjectsTimeIndexKey() {
// generate a new key for the current time
let timestampBuffer = Buffer.alloc(8)
timestampBuffer.writeDoubleBE(Date.now())
let rndBuffer = crypto.randomBytes(16)
return Buffer.concat([prefixBuffers.INCOMING_HASH_OBJECTS, timestampBuffer, rndBuffer])
}
function createBinaryIncomingHashObjectsTimeIndexMin() {
// generate the minimum key value for range query
let minBoundsBuffer = Buffer.alloc(24, 0)
return Buffer.concat([prefixBuffers.INCOMING_HASH_OBJECTS, minBoundsBuffer])
}
function createBinaryIncomingHashObjectsTimeIndexMax(timestamp) {
// generate the maximum key value for range query up to given timestamp
let timestampBuffer = Buffer.alloc(8, 0)
timestampBuffer.writeDoubleBE(timestamp)
let rndBuffer = Buffer.alloc(16, 'ff', 'hex')
return Buffer.concat([prefixBuffers.INCOMING_HASH_OBJECTS, timestampBuffer, rndBuffer])
}
function encodeIncomingHashObjectsValue(hashObjects) {
hashObjects = hashObjects.map(hashObject => {
return {
proof_id: encodeBinaryProofStateId(hashObject.proof_id).idBuffer,
hash: Buffer.from(hashObject.hash, 'hex')
}
})
return incomingHashObjectsSchema.encode(hashObjects)
}
function decodeIncomingHashObjectsValue(hashObjectsBinary) {
if (hashObjectsBinary.length === 0) return []
let hashObjects = incomingHashObjectsSchema.decode(hashObjectsBinary)
hashObjects = hashObjects.map(hashObject => {
return {
proof_id: Ulid.construct(Uint8Array.from(hashObject.proof_id)).toCanonical(),
hash: hashObject.hash.toString('hex')
}
})
return hashObjects
}
async function queueIncomingHashObjectsAsync(hashObjects) {
try {
let incomingHashObjectsBinaryKey = createBinaryIncomingHashObjectsTimeIndexKey()
let incomingHashObjectsValue = encodeIncomingHashObjectsValue(hashObjects)
await db.put(incomingHashObjectsBinaryKey, incomingHashObjectsValue)
} catch (error) {
let err = `Unable to write incoming hash data : ${error.message}`
throw err
}
}
async function getIncomingHashesUpToAsync(maxTimestamp) {
return new Promise((resolve, reject) => {
let hashesObjects = []
let delOps = []
let minIncomingHashObjectsBinaryKey = createBinaryIncomingHashObjectsTimeIndexMin()
let maxIncomingHashObjectsBinaryKey = createBinaryIncomingHashObjectsTimeIndexMax(maxTimestamp)
db.createReadStream({ gte: minIncomingHashObjectsBinaryKey, lte: maxIncomingHashObjectsBinaryKey })
.on('data', async data => {
hashesObjects.push(...decodeIncomingHashObjectsValue(data.value))
delOps.push({
type: 'del',
key: data.key
})
})
.on('error', error => {
let err = `Error reading incoming hashes : ${error.message}`
return reject(err)
})
.on('end', async () => {
return resolve([hashesObjects, delOps])
})
})
}
// #endregion INCOMING HASH QUEUE FUNCTIONS
/****************************************************************************************************
* GENERAL KEY - VALUE FUNCTIONS
****************************************************************************************************/
// #region GENERAL KEY - VALUE FUNCTIONS
async function setAsync(key, value) {
try {
await db.put(Buffer.from(`custom_key:${key}`), Buffer.from(value.toString()))
} catch (error) {
let err = `Unable to write key : ${error.message}`
throw new Error(err)
}
}
async function getAsync(key) {
try {
let result = await db.get(Buffer.from(`custom_key:${key}`))
return result.toString()
} catch (error) {
if (error.notFound) {
return null
} else {
let err = `Unable to read key : ${error.message}`
throw new Error(err)
}
}
}
async function deleteBatchAsync(delOps) {
return db.batch(delOps)
}
// #endregion GENERAL KEY - VALUE FUNCTIONS
/****************************************************************************************************
* SUPPORT FUNCTIONS
****************************************************************************************************/
// #region SUPPORT FUNCTIONS
// #endregion SUPPORT FUNCTIONS
/****************************************************************************************************
* SET AUTOMATIC PRUNING INTERVALS
****************************************************************************************************/
// #region SET AUTOMATIC PRUNING INTERVALS
function startPruningInterval() {
return setInterval(async () => {
if (!PRUNE_IN_PROGRESS) {
PRUNE_IN_PROGRESS = true
await pruneOldProofStateDataAsync()
PRUNE_IN_PROGRESS = false
}
}, PRUNE_INTERVAL_SECONDS * 1000)
}
// #endregion SET AUTOMATIC PRUNING INTERVALS
module.exports = {
openConnectionAsync: openConnectionAsync,
getProofStatesBatchByProofIdsAsync: getProofStatesBatchByProofIdsAsync,
saveProofStatesBatchAsync: saveProofStatesBatchAsync,
queueIncomingHashObjectsAsync: queueIncomingHashObjectsAsync,
getIncomingHashesUpToAsync: getIncomingHashesUpToAsync,
setAsync: setAsync,
getAsync: getAsync,
deleteBatchAsync: deleteBatchAsync,
startPruningInterval: startPruningInterval,
pruneOldProofStateDataAsync: pruneOldProofStateDataAsync,
// additional functions for testing purposes
setENV: obj => {
env = obj
}
}
================================================
FILE: lib/parse-env.js
================================================
/**
* Copyright 2019 Tierion
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
const envalid = require('envalid')
const ip = require('ip')
function valCoreIPList(list) {
// If IP list supplied, ensure it is valid, or continue with empty string
if (list === '') return ''
let IPs = list.split(',')
for (let val of IPs) {
if ((!ip.isV4Format(val) && !ip.isV6Format(val)) || val === '')
throw new Error('The Core IP list contains an invalid entry')
}
// ensure each IP is unique
let ipSet = new Set(IPs)
if (ipSet.size !== IPs.length) throw new Error('The Core IP list cannot contain duplicates')
return IPs
}
function valNetwork(name) {
if (name === '' || name === 'mainnet') return 'mainnet'
if (name === 'testnet') return 'testnet'
throw new Error('The NETWORK value is invalid')
}
const validateCoreIPList = envalid.makeValidator(valCoreIPList)
const validateNetwork = envalid.makeValidator(valNetwork)
let envDefinitions = {
// Chainpoint Node environment related variables
NODE_ENV: envalid.str({ default: 'production', desc: 'The type of environment in which the service is running' }),
NETWORK: validateNetwork({ default: 'mainnet', desc: `The network to use, 'mainnet' or 'testnet'` }),
LND_SOCKET: envalid.str({ default: 'lnd:10009', desc: 'Lightning GRPC host and port' }),
PUBLIC_IP: envalid.str({ default: '127.0.0.1', desc: 'IP host and port' }),
GATEWAY_NAME: envalid.str({ default: 'UNNAMED', desc: 'A, B, or C' }),
GOOGLE_UA_ID: envalid.str({ default: '', desc: 'Google Universal Analytics ID' }),
// Chainpoint Core
CHAINPOINT_CORE_CONNECT_IP_LIST: validateCoreIPList({
default: '',
desc: 'A comma separated list of specific Core IPs to connect to (instead of using Core discovery)'
}),
AGGREGATION_INTERVAL_SECONDS: envalid.num({
default: 60,
desc: `The aggregation and Core submission frequency, in seconds`
}),
MAX_SATOSHI_PER_HASH: envalid.num({
default: 10,
desc: `The maximum amount you are willing to spend for each hash submission to Core, in Satoshi`
}),
PROOF_EXPIRE_MINUTES: envalid.num({
default: 1440,
desc: `The length of time proofs as stored on the node for retrieval, in minutes`
}),
POST_HASHES_MAX: envalid.num({
default: 1000,
desc: `The maximum number of hashes accepted in a single submit request`
}),
POST_VERIFY_PROOFS_MAX: envalid.num({
default: 1000,
desc: `The maximum number of proofs accepted in a single verification request`
}),
GET_PROOFS_MAX: envalid.num({
default: 250,
desc: `The maximum number of proofs to be returned in a single request`
}),
CHANNEL_AMOUNT: envalid.num({
default: 120000,
desc: `The amount to fund a channel with`
}),
FUND_AMOUNT: envalid.num({
default: 360000,
desc: `The total wallet funding required`
}),
NO_LSAT_CORE_WHITELIST: validateCoreIPList({
default: '',
desc: 'A comma separated list of specific Core IPs to skip LSAT auth'
})
}
module.exports = {
env: envalid.cleanEnv(process.env, envDefinitions, { strict: false }),
// additional functions for testing purposes
valCoreIPList: valCoreIPList,
valNetwork: valNetwork
}
================================================
FILE: lib/utils.js
================================================
/**
* Copyright 2019 Tierion
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
const jmespath = require('jmespath')
const _ = require('lodash')
const flattenKeys = (obj, path = []) =>
!_.isObject(obj)
? { [path.join('.')]: obj }
: _.reduce(obj, (cum, next, key) => _.merge(cum, flattenKeys(next, [...path, key])), {})
// wait for a specified number of milliseconds to elapse
function sleepAsync(ms) {
return new Promise(resolve => setTimeout(resolve, ms))
}
/**
* Add specified seconds to a Date object
*
* @param {Date} date - The starting date
* @param {number} seconds - The seconds of seconds to add to the date
* @returns {Date}
*/
function addSeconds(date, seconds) {
return new Date(date.getTime() + seconds * 1000)
}
/**
* Add specified minutes to a Date object
*
* @param {Date} date - The starting date
* @param {number} minutes - The number of minutes to add to the date
* @returns {Date}
*/
function addMinutes(date, minutes) {
return new Date(date.getTime() + minutes * 60000)
}
/**
* Convert Date to ISO8601 string, stripping milliseconds
* '2017-03-19T23:24:32Z'
*
* @param {Date} date - The date to convert
* @returns {string} An ISO8601 formatted time string
*/
function formatDateISO8601NoMs(date) {
return date.toISOString().slice(0, 19) + 'Z'
}
/**
* Convert strings in an Array of hashes to lower case
*
* @param {string[]} hashes - An array of string hashes to convert to lower case
* @returns {string[]} An array of lowercase hash strings
*/
function lowerCaseHashes(hashes) {
return hashes.map(hash => {
return hash.toLowerCase()
})
}
function parseAnchorsComplete(proofObject, network) {
// Because the minimum proof will contain a cal anchor, always start with cal
let anchorsComplete = [network === 'mainnet' ? 'cal' : 'tcal'].concat(
jmespath.search(proofObject, '[branches[].branches[].ops[].anchors[].type] | [0]')
)
return anchorsComplete
}
/**
* Checks if value is a hexadecimal string
*
* @param {string} value - The value to check
* @returns {bool} true if value is a hexadecimal string, otherwise false
*/
function isHex(value) {
var hexRegex = /^[0-9a-f]{2,}$/i
var isHex = hexRegex.test(value) && !(value.length % 2)
return isHex
}
/**
* Checks if value is a uuid string
*
* @param {string} value - The value to check
* @returns {bool} true if value is a hexadecimal string, otherwise false
*/
function isUUID(value) {
var uuidRegex = /^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}$/i
return uuidRegex.test(value)
}
/**
* Checks if value is a ulid string
*
* @param {string} value - The value to check
* @returns {bool} true if value is a hexadecimal string, otherwise false
*/
function isULID(value) {
var ulidRegex = /^[0123456789ABCDEFGHJKMNPQRSTVWXYZ]{26}$/i
return ulidRegex.test(value)
}
/**
* converts a hex value to a uuid
*
* @param {string} value - The value to convert
* @returns {string} the segmented uuid string
*/
function hexToUUIDv1(hexString) {
if (hexString.length < 32) return null
let segment1 = hexString.substring(0, 8)
let segment2 = hexString.substring(8, 12)
let segment3 = hexString.substring(12, 16)
let segment4 = hexString.substring(16, 20)
let segment5 = hexString.substring(20, 32)
return `${segment1}-${segment2}-${segment3}-${segment4}-${segment5}`
}
/**
* Returns a random Integer between min and max
*
* @param {Integer} min - The min value to be returned
* @param {Integer} max - The max value to be returned
* @returns {Integer} The selected random Integer between min and max
*/
function randomIntFromInterval(min, max) {
return Math.floor(Math.random() * (max - min + 1) + min)
}
function nodeUIPasswordBooleanCheck(pw = '') {
if (_.isBoolean(pw) && pw === false) {
return false
} else {
let password = pw.toLowerCase()
if (password === 'false') return false
}
return pw
}
/**
* Extracts the IP address from a Restify request object
*
* @param {req} value - The Restify request object
* @returns {string} - The IP address, or null if it cannot be determined
*/
function getClientIP(req) {
let xff, rcr, rsa
try {
xff = req.headers['x-forwarded-for']
} catch (error) {
xff = null
}
try {
rcr = req.connection.remoteAddress
} catch (error) {
rcr = null
}
try {
rsa = req.socket.remoteAddress
} catch (error) {
rsa = null
}
let result = xff || rcr || rsa
if (result) result = result.replace('::ffff:', '')
return result || null
}
function jsonTransform(json, conditionFn, modifyFn) {
// transform { responses: { category: 'first' } } to { 'responses.category': 'first' }
const flattenedKeys = Object.keys(flattenKeys(json))
// Easily iterate over the flat json
for (let i = 0; i < flattenedKeys.length; i++) {
const key = flattenedKeys[i]
const value = _.get(json, key)
// Did the condition match the one we passed?
if (conditionFn(key, value)) {
// Replace the value to the new one
_.set(json, key, modifyFn(key, value))
}
}
return json
}
module.exports = {
sleepAsync: sleepAsync,
addMinutes: addMinutes,
addSeconds: addSeconds,
formatDateISO8601NoMs: formatDateISO8601NoMs,
lowerCaseHashes: lowerCaseHashes,
parseAnchorsComplete: parseAnchorsComplete,
isHex: isHex,
isUUID: isUUID,
isULID: isULID,
hexToUUIDv1: hexToUUIDv1,
randomIntFromInterval: randomIntFromInterval,
nodeUIPasswordBooleanCheck: nodeUIPasswordBooleanCheck,
getClientIP: getClientIP,
jsonTransform: jsonTransform
}
================================================
FILE: package.json
================================================
{
"name": "chainpoint-gateway",
"description": "A Chainpoint Network Gateway is a key part of a scalable solution for anchoring data to public blockchains.",
"version": "1.2.0",
"main": "server.js",
"scripts": {
"start": "node server.js",
"eslint-check": "eslint --print-config . | eslint-config-prettier-check",
"test": "mocha tests/*.js"
},
"husky": {
"hooks": {
"pre-commit": "lint-staged"
}
},
"lint-staged": {
"linters": {
"*.js": [
"eslint --fix",
"git add"
],
"*.{json,css,md}": [
"prettier --write",
"git add"
]
}
},
"keywords": [
"Chainpoint",
"bitcoin",
"lightning",
"Tierion",
"node",
"hash",
"blockchain",
"crypto",
"cryptography",
"sha256"
],
"author": "Jason Bukowski <jason@tierion.com> (https://tierion.com)",
"license": "Apache-2.0",
"devDependencies": {
"chai": "^4.2.0",
"eslint": "^5.4.0",
"eslint-config-prettier": "^3.0.1",
"eslint-plugin-prettier": "^2.6.2",
"eslint-plugin-react": "^7.11.1",
"husky": "^1.3.1",
"lint-staged": "^8.1.0",
"mocha": "^5.2.0",
"prettier": "^1.14.2",
"rimraf": "^2.6.3",
"supertest": "^3.4.2"
},
"dependencies": {
"async-await-parallel": "^1.0.0",
"async-retry": "^1.2.3",
"blake2s-js": "^1.3.0",
"bluebird": "^3.5.5",
"chainpoint-binary": "^5.1.1",
"chainpoint-parse": "^5.0.1",
"chalk": "^2.4.2",
"dotenv": "^8.2.0",
"envalid": "^4.2.0",
"executive": "^1.6.3",
"generate-password": "^1.4.2",
"id128": "^1.6.6",
"ip": "^1.1.5",
"jmespath": "^0.15.0",
"js-binary": "^1.2.0",
"level-rocksdb": "^4.0.0",
"lnrpc-node-client": "^1.1.2",
"lodash": "^4.17.11",
"lsat-js": "^2.0.0",
"merkle-tools": "^1.4.0",
"request": "^2.88.0",
"request-promise-native": "^1.0.7",
"restify": "^8.3.3",
"restify-cors-middleware": "^1.1.1",
"restify-errors": "^6.1.1",
"universal-analytics": "^0.4.23",
"uuid": "^3.3.2",
"uuid-time": "^1.0.0",
"uuid-validate": "^0.0.3",
"validator": "^10.11.0",
"winston": "^3.2.1",
"winston-papertrail": "^1.0.5"
}
}
================================================
FILE: scripts/install_deps.sh
================================================
#!/bin/bash
if [ -x "$(command -v docker)" ]; then
echo "Docker already installed"
else
echo "Install docker"
curl -fsSL https://get.docker.com -o get-docker.sh
bash get-docker.sh
sudo usermod -aG docker $USER
fi
if [[ "$OSTYPE" == "linux-gnu" ]]; then
sudo apt-get -qq update -y
sudo apt-get -qq install -y apt-utils
sudo apt-get -qq install -y git make jq openssl
curl -sL https://deb.nodesource.com/setup_12.x | sudo -E bash -
sudo apt-get install -y nodejs
curl -sL https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
sudo apt-get update && sudo apt-get install yarn
sudo curl -L "https://github.com/docker/compose/releases/download/1.25.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
sudo ln -sf /usr/local/bin/docker-compose /usr/bin/docker-compose || echo Binary is not at usual location or is already linked
elif [[ "$OSTYPE" == "darwin"* ]]; then
ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
brew install caskroom/cask/brew-cask
brew cask install docker-toolbox
brew install jq
brew install homebrew/core/make
brew install git
brew install node
brew install yarn
brew install openssl
fi
yarn
================================================
FILE: scripts/prod_secrets_expand.sh
================================================
#!/bin/sh
: ${ENV_SECRETS_DIR:=/run/secrets}
function env_secret_debug() {
if [ ! -z "$ENV_SECRETS_DEBUG" ]; then
echo -e "\033[1m$@\033[0m"
fi
}
# usage: env_secret_expand VAR
# ie: env_secret_expand 'XYZ_DB_PASSWORD'
# (will check for "$XYZ_DB_PASSWORD" variable value for a placeholder that defines the
# name of the docker secret to use instead of the original value. For example:
# XYZ_DB_PASSWORD=DOCKER-SECRET->my-db.secret
env_secret_expand() {
var="$1"
eval val=\$$var
if secret_name=$(expr match "$val" "DOCKER-SECRET->\([^}]\+\)$"); then
secret="${ENV_SECRETS_DIR}/${secret_name}"
env_secret_debug "Secret file for $var: $secret"
if [ -f "$secret" ]; then
val=$(cat "${secret}")
export "$var"="$val"
env_secret_debug "Expanded variable: $var=$val"
else
env_secret_debug "Secret file does not exist! $secret"
fi
fi
}
env_secrets_expand() {
for env_var in $(printenv | cut -f1 -d"=")
do
env_secret_expand $env_var
done
if [ ! -z "$ENV_SECRETS_DEBUG" ]; then
echo -e "\n\033[1mExpanded environment variables\033[0m"
printenv
fi
}
env_secrets_expand
================================================
FILE: scripts/run_prod.sh
================================================
#!/bin/bash
cd $(dirname $0)
source ./prod_secrets_expand.sh
yarn start
================================================
FILE: server.js
================================================
/**
* Copyright 2019 Tierion
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
// load environment variables
const env = require('./lib/parse-env.js').env
const apiServer = require('./lib/api-server.js')
const aggregator = require('./lib/aggregator.js')
const { version } = require('./package.json')
const rocksDB = require('./lib/models/RocksDB.js')
const cachedProofs = require('./lib/cached-proofs.js')
const cores = require('./lib/cores.js')
const logger = require('./lib/logger.js')
// establish a connection with the database
async function openStorageConnectionAsync() {
await rocksDB.openConnectionAsync()
}
// process all steps need to start the application
async function startAsync() {
try {
logger.info(`App : Startup : Version ${version}`)
// display NETWORK value
logger.info(`App : Startup : Network : ${env.NETWORK}`)
// establish a connection with the database
await openStorageConnectionAsync()
logger.info(`App : Startup : Storage Connection Opened`)
// connect to the Cores listed in .env and check/open lightning connections
await cores.connectAsync()
logger.info(`App : Startup : Cores Connected`)
// start API server
await apiServer.startAsync(cores.getLn())
logger.info(`App : Startup : API Started`)
// start the interval processes for refreshing the IP blocklist
apiServer.startIPBlacklistRefreshInterval()
// start the interval processes for aggregating and submitting hashes to Core
aggregator.startAggInterval()
// start the interval processes for pruning expired proof state data from RocksDB
rocksDB.startPruningInterval()
// start the interval processes for pruning cached proof data from memory
cachedProofs.startPruneExpiredItemsInterval()
// start the interval processes for pruning cached transaction data from memory
cores.startPruneExpiredItemsInterval()
// start monitoring health of peer/channel connections
cores.startConnectionMonitoringInterval()
logger.info(`App : Startup : Complete`)
} catch (err) {
logger.error(`App : Startup : ${err.message}`)
// Unrecoverable Error : Exit cleanly (!), so Docker Compose `on-failure` policy
// won't force a restart since this situation will not resolve itself.
process.exit(0)
}
}
// get the whole show started
startAsync()
================================================
FILE: swarm-compose.yaml
================================================
version: "3.7"
networks:
chainpoint-gateway:
secrets:
HOT_WALLET_PASS:
external: true
HOT_WALLET_ADDRESS:
external: true
services:
chainpoint-gateway:
restart: on-failure
entrypoint: /home/node/app/scripts/run_prod.sh
volumes:
- ./ip-blacklist.txt:/home/node/app/ip-blacklist.txt:ro
- ~/.chainpoint/gateway/data/rocksdb:/root/.chainpoint/gateway/data/rocksdb
- ./.env:/home/node/app/.env
- ~/.chainpoint/gateway/.lnd:/root/.lnd:ro
image: gcr.io/chainpoint-registry/github_chainpoint_chainpoint-gateway:${DOCKER_TAG:-latest}
user: ${USERID}:${GROUPID}
build: .
deploy:
mode: global
placement:
constraints: [node.role==manager]
restart_policy:
condition: any
delay: 5s
max_attempts: 15
window: 90s
depends_on:
- lnd
ports:
- target: 8080
published: 80
protocol: tcp
mode: host
networks:
- chainpoint-gateway
secrets:
- HOT_WALLET_PASS
- HOT_WALLET_ADDRESS
environment:
HOME: /root
HOT_WALLET_PASS: DOCKER-SECRET->HOT_WALLET_PASS
HOT_WALLET_ADDRESS: DOCKER-SECRET->HOT_WALLET_ADDRESS
LND_SOCKET: ${LND_SOCKET}
CHAINPOINT_CORE_CONNECT_IP_LIST: "${CHAINPOINT_CORE_CONNECT_IP_LIST}"
AGGREGATION_INTERVAL_SECONDS: "${AGGREGATION_INTERVAL_SECONDS}"
PROOF_EXPIRE_MINUTES: "${PROOF_EXPIRE_MINUTES}"
POST_HASHES_MAX: "${POST_HASHES_MAX}"
POST_VERIFY_PROOFS_MAX: "${POST_VERIFY_PROOFS_MAX}"
GET_PROOFS_MAX: "${GET_PROOFS_MAX}"
MAX_SATOSHI_PER_HASH: "${MAX_SATOSHI_PER_HASH}"
NETWORK: ${NETWORK}
NODE_ENV: ${NODE_ENV}
CHANNEL_AMOUNT: ${CHANNEL_AMOUNT}
FUND_AMOUNT: ${FUND_AMOUNT}
NO_LSAT_CORE_WHITELIST: ${NO_LSAT_CORE_WHITELIST}
GOOGLE_UA_ID: ${GOOGLE_UA_ID}
PUBLIC_IP: ${LND_PUBLIC_IP}
tty: true
logging:
driver: 'json-file'
options:
max-size: '1g'
max-file: '5'
# Lightning node
lnd:
image: tierion/lnd:${NETWORK:-testnet}-v0.14.1
user: ${USERID}:${GROUPID}
entrypoint: "./start-lnd.sh"
ports:
- target: 8080
published: 8080
protocol: tcp
mode: host
- target: 9735
published: 9735
protocol: tcp
mode: host
- target: 10009
published: 10009
protocol: tcp
mode: host
deploy:
restart_policy:
condition: any
delay: 5s
max_attempts: 15
window: 90s
endpoint_mode: dnsrr
environment:
- PUBLICIP=${LND_PUBLIC_IP}
- RPCUSER
- RPCPASS
- NETWORK=${NETWORK:-testnet}
- CHAIN
- DEBUG=info
- BACKEND=neutrino
- NEUTRINO=faucet.lightning.community:18333
- LND_REST_PORT
- LND_RPC_PORT
- TLSPATH
- TLSEXTRADOMAIN=lnd
volumes:
- ~/.chainpoint/gateway/.lnd:/root/.lnd:z
networks:
- chainpoint-gateway
logging:
driver: 'json-file'
options:
max-size: '1g'
max-file: '5'
================================================
FILE: tests/RocksDB.js
================================================
/* global describe, it, before, after */
process.env.NODE_ENV = 'test'
// test related packages
const expect = require('chai').expect
const rocksDB = require('../lib/models/RocksDB.js')
const rmrf = require('rimraf')
const uuidv1 = require('uuid/v1')
const crypto = require('crypto')
const TEST_ROCKS_DIR = './test_db'
let insertedProofStateHashIdNodes = null
describe('RocksDB Methods', () => {
let db = null
before(async () => {
db = await rocksDB.openConnectionAsync(TEST_ROCKS_DIR)
expect(db).to.be.a('object')
})
after(() => {
db.close(() => {
rmrf.sync(TEST_ROCKS_DIR)
})
})
describe('Proof State Functions', () => {
it('should return the same data that was inserted', async () => {
let sampleData = generateSampleProofStateData(100)
await rocksDB.saveProofStatesBatchAsync(sampleData.state)
let queriedState = await rocksDB.getProofStatesBatchByProofIdsAsync(sampleData.proofIdNodes)
insertedProofStateHashIdNodes = sampleData.proofIdNodes
queriedState = convertStateBackToBinaryForm(queriedState)
expect(queriedState).to.deep.equal(sampleData.state)
})
})
describe('Incoming Hash Functions', () => {
let delOps = []
it('should return the same data that was inserted', async () => {
let sampleData = generateSampleHashObjects(100)
await rocksDB.queueIncomingHashObjectsAsync(sampleData)
let getResults = await rocksDB.getIncomingHashesUpToAsync(Date.now)
let queriedHashes = getResults[0]
delOps = getResults[1]
expect(queriedHashes).to.deep.equal(sampleData)
})
after(async () => {
await db.batch(delOps)
})
})
describe('Generic key/value Functions', () => {
it('should return the same value that was inserted', async () => {
let keys = []
let values = []
for (let x = 0; x < 100; x++) {
keys.push(`testKey${x}${crypto.randomBytes(8).toString('hex')}`)
values.push(crypto.randomBytes(8).toString('hex'))
}
for (let x = 0; x < 100; x++) {
await rocksDB.setAsync(keys[x], values[x])
}
let getValues = []
for (let x = 0; x < 100; x++) {
getValues.push(await rocksDB.getAsync(keys[x]))
}
expect(getValues).to.deep.equal(values)
let delOps = []
for (let key of keys) {
delOps.push({ type: 'del', key: key })
}
rocksDB.deleteBatchAsync(delOps).then(async () => {
let getValues = []
for (let x = 0; x < 100; x++) {
getValues.push(await rocksDB.getAsync(keys[x]))
}
expect(getValues).to.deep.equal(values)
})
})
})
describe('Delete/Prune Functions', () => {
before(() => {
rocksDB.setENV({
PROOF_EXPIRE_MINUTES: 0
})
})
it('should batch delete as expected', async () => {
let keys = []
let values = []
for (let x = 0; x < 100; x++) {
keys.push(`testKey${x}${crypto.randomBytes(8).toString('hex')}`)
values.push(crypto.randomBytes(8).toString('hex'))
}
for (let x = 0; x < 100; x++) {
await rocksDB.setAsync(keys[x], values[x])
}
let getValues = []
for (let x = 0; x < 100; x++) {
getValues.push(await rocksDB.getAsync(keys[x]))
}
expect(getValues).to.deep.equal(values)
let delOps = []
for (let key of keys) {
delOps.push({ type: 'del', key: `custom_key:${key}` })
}
await rocksDB.deleteBatchAsync(delOps)
getValues = []
for (let x = 0; x < 100; x++) {
let getResult = await rocksDB.getAsync(keys[x])
expect(getResult).to.equal(null)
}
})
it('should initiate prune interval as expected', async () => {
let interval = rocksDB.startPruningInterval()
expect(interval).to.be.a('object')
clearInterval(interval)
})
it('should prune proof state data as expected', async () => {
// retrieve inserted proof state, confirm it still exists
let queriedState = await rocksDB.getProofStatesBatchByProofIdsAsync(insertedProofStateHashIdNodes)
expect(queriedState).to.be.a('array')
expect(queriedState.length).to.be.greaterThan(0)
for (let x = 0; x < queriedState.length; x++) {
expect(queriedState[x]).to.have.property('hash')
expect(queriedState[x].hash).to.be.a('string')
}
// prune all proof state data (0 minute expiration)
await rocksDB.pruneOldProofStateDataAsync()
// retrieve inserted proof state, confirm it has all beed pruned
queriedState = await rocksDB.getProofStatesBatchByProofIdsAsync(insertedProofStateHashIdNodes)
expect(queriedState).to.be.a('array')
expect(queriedState.length).to.be.greaterThan(0)
for (let x = 0; x < queriedState.length; x++) {
expect(queriedState[x]).to.have.property('hash')
expect(queriedState[x].hash).to.equal(null)
}
})
})
describe('Other Functions', () => {
it('hexToUUIDv1 should return null with invalid hex value', done => {
let result = rocksDB.hexToUUIDv1('deadbeefcafe')
expect(result).to.equal(null)
done()
})
it('hexToUUIDv1 should return the expected result with proper hex value', done => {
let result = rocksDB.hexToUUIDv1('ed60c311ede60102689f66a9e98feab6')
expect(result)
.to.be.a('string')
.and.to.equal('ed60c311-ede6-0102-689f-66a9e98feab6')
done()
})
})
})
// support functions
function generateSampleProofStateData(batchSize) {
let results = {}
results.state = []
results.proofIdNodes = []
for (let x = 0; x < batchSize; x++) {
let newHashIdNode = uuidv1()
let submitId = uuidv1()
results.state.push({
proofId: newHashIdNode,
hash: crypto.randomBytes(32).toString('hex'),
proofState: [Buffer.from(Math.round(Math.random()) ? '00' : '01', 'hex'), crypto.randomBytes(32)],
submission: {
submitId: submitId,
cores: [
{ ip: '65.1.12.122', proofId: uuidv1() },
{ ip: '65.1.12.123', proofId: uuidv1() },
{ ip: '65.1.12.124', proofId: uuidv1() }
]
}
})
results.proofIdNodes.push(newHashIdNode)
}
return results
}
function convertStateBackToBinaryForm(queriedState) {
for (let stateItem of queriedState) {
let binState
for (let psItem of stateItem.proofState) {
binState = []
if (psItem.left) {
binState.push(Buffer.from('00', 'hex'))
binState.push(Buffer.from(psItem.left, 'hex'))
} else {
binState.push(Buffer.from('01', 'hex'))
binState.push(Buffer.from(psItem.right, 'hex'))
}
}
stateItem.proofState = binState
}
return queriedState
}
function generateSampleHashObjects(batchSize) {
let results = []
for (let x = 0; x < batchSize; x++) {
results.push({
proof_id: uuidv1(),
hash: crypto.randomBytes(32).toString('hex')
})
}
return results
}
================================================
FILE: tests/aggregator.js
================================================
/* global describe, it, before, after */
process.env.NODE_ENV = 'test'
// test related packages
const expect = require('chai').expect
const aggregator = require('../lib/aggregator.js')
const uuidv1 = require('uuid/v1')
const crypto = require('crypto')
const BLAKE2s = require('blake2s-js')
const MerkleTools = require('merkle-tools')
describe('Aggregator Methods', () => {
describe('startAggInterval', () => {
it('should initiate interval as expected', async () => {
let interval = aggregator.startAggInterval()
expect(interval).to.be.a('object')
clearInterval(interval)
})
})
describe('aggregateSubmitAndPersistAsync with 0 hashes', () => {
let hashCount = 0
let IncomingHashes = generateIncomingHashData(hashCount)
let ProofStateData = []
before(() => {
aggregator.setRocksDB({
getIncomingHashesUpToAsync: async () => {
let delOps = IncomingHashes.map(item => {
return { type: 'del', key: item.proof_id }
})
return [IncomingHashes, delOps]
}
})
})
after(() => {})
it('should complete successfully', async () => {
expect(IncomingHashes.length).to.equal(hashCount)
await aggregator.aggregateSubmitAndPersistAsync()
expect(IncomingHashes.length).to.equal(0)
expect(ProofStateData.length).to.equal(hashCount)
})
})
describe('aggregateSubmitAndPersistAsync with 100 hashes', () => {
let hashCount = 100
let IncomingHashes = generateIncomingHashData(hashCount)
let newHashIdCore1 = null
let newHashIdCore2 = null
let ProofStateData = null
let ip1 = '65.21.21.122'
let ip2 = '65.21.21.123'
before(() => {
aggregator.setRocksDB({
getIncomingHashesUpToAsync: async () => {
let delOps = IncomingHashes.map(item => {
return { type: 'del', key: item.proof_id }
})
return [IncomingHashes, delOps]
},
deleteBatchAsync: async delOps => {
let delHashIds = delOps.map(item => item.key)
IncomingHashes = IncomingHashes.filter(item => !delHashIds.includes(item.proof_id))
},
saveProofStatesBatchAsync: async items => {
ProofStateData = items
}
})
aggregator.setCores({
submitHashAsync: async () => {
let hash = crypto.randomBytes(32).toString('hex')
newHashIdCore1 = generateBlakeEmbeddedUUID(hash)
newHashIdCore2 = generateBlakeEmbeddedUUID(hash)
return [
{ ip: ip1, response: { hash_id: newHashIdCore1, hash: hash, processing_hints: 'hints' } },
{ ip: ip2, response: { hash_id: newHashIdCore2, hash: hash, processing_hints: 'hints' } }
]
}
})
})
after(() => {})
it('should complete successfully', async () => {
var merkleTools = new MerkleTools()
expect(IncomingHashes.length).to.equal(hashCount)
let aggRoot = await aggregator.aggregateSubmitAndPersistAsync()
expect(IncomingHashes.length).to.equal(0)
expect(ProofStateData.length).to.equal(hashCount)
for (let x = 0; x < hashCount; x++) {
expect(ProofStateData[x])
.to.have.property('proofId')
.and.and.be.a('string')
expect(ProofStateData[x])
.to.have.property('hash')
.and.and.be.a('string')
expect(ProofStateData[x])
.to.have.property('proofState')
.and.and.be.a('array')
// add the additional nodeId operation to get final leaf values
let proofIdBuffer = Buffer.from(`node_id:${ProofStateData[x].proofId}`, 'utf8')
let hashBuffer = Buffer.from(ProofStateData[x].hash, 'hex')
ProofStateData[x].hash = crypto
.createHash('sha256')
.update(Buffer.concat([proofIdBuffer, hashBuffer]))
.digest()
// convert from binary
let proofState = []
for (let y = 0; y < ProofStateData[x].proofState.length; y += 2) {
let operand = ProofStateData[x].proofState[y + 1].toString('hex')
let isLeft = ProofStateData[x].proofState[y].toString('hex') === '00'
let fullOp
if (isLeft) {
fullOp = { left: operand }
} else {
fullOp = { right: operand }
}
proofState.push(fullOp)
}
expect(merkleTools.validateProof(proofState, ProofStateData[x].hash, aggRoot)).to.equal(true)
expect(ProofStateData[x]).to.have.property('submission')
expect(ProofStateData[x].submission).to.be.a('object')
expect(ProofStateData[x].submission)
.to.have.property('submitId')
.and.and.be.a('string')
expect(ProofStateData[x].submission).to.have.property('cores')
expect(ProofStateData[x].submission.cores).to.to.a('array')
expect(ProofStateData[x].submission.cores.length).to.equal(2)
expect(ProofStateData[x].submission.cores[0]).to.be.a('object')
expect(ProofStateData[x].submission.cores[0])
.to.have.property('ip')
.and.and.be.a('string')
.and.to.equal(ip1)
expect(ProofStateData[x].submission.cores[0])
.to.have.property('proofId')
.and.and.be.a('string')
.and.to.equal(newHashIdCore1)
expect(ProofStateData[x].submission.cores[1]).to.be.a('object')
expect(ProofStateData[x].submission.cores[1])
.to.have.property('ip')
.and.and.be.a('string')
.and.to.equal(ip2)
expect(ProofStateData[x].submission.cores[1])
.to.have.property('proofId')
.and.and.be.a('string')
.and.to.equal(newHashIdCore2)
}
})
})
})
// support functions
function generateIncomingHashData(batchSize) {
let hashes = []
for (let x = 0; x < batchSize; x++) {
let newHashIdNode = uuidv1()
hashes.push({
proof_id: newHashIdNode,
hash: crypto.randomBytes(32).toString('hex')
})
}
return hashes
}
function generateBlakeEmbeddedUUID(hash) {
let timestampDate = new Date()
let timestampMS = timestampDate.getTime()
// 5 byte length BLAKE2s hash w/ personalization
let h = new BLAKE2s(5, { personalization: Buffer.from('CHAINPNT') })
let hashStr = [timestampMS.toString(), timestampMS.toString().length, hash, hash.length].join(':')
h.update(Buffer.from(hashStr))
return uuidv1({
msecs: timestampMS,
node: Buffer.concat([Buffer.from([0x01]), h.digest()])
})
}
================================================
FILE: tests/api-server.js
================================================
/* global describe, it, beforeEach, before */
process.env.NODE_ENV = 'test'
// test related packages
const expect = require('chai').expect
const apiServer = require('../lib/api-server.js')
let rocksData = {}
const TOR_IPS_KEY = 'blacklist:tor:ips'
describe('API Server Methods', () => {
beforeEach(() => {
apiServer.setRocksDB({
getAsync: async key => rocksData[key],
setAsync: async (key, value) => {
rocksData[key] = value.toString()
}
})
})
describe('startIPBlacklistRefreshInterval', () => {
it('should initiate interval as expected', async () => {
let interval = apiServer.startIPBlacklistRefreshInterval()
expect(interval).to.be.a('object')
clearInterval(interval)
})
})
describe('refreshIPBlacklistAsync with tor request failure, cache read failure', () => {
before(() => {
apiServer.setRP(async () => {
throw 'bad'
})
apiServer.setRocksDB(null)
})
it('should return expected value', async () => {
let ips = await apiServer.refreshIPBlacklistAsync()
expect(ips).to.be.a('array')
expect(ips.length).to.equal(0)
})
})
describe('refreshIPBlacklistAsync with tor request failure, empty cache', () => {
before(() => {
apiServer.setRP(async () => {
throw 'bad'
})
})
it('should return expected value', async () => {
let ips = await apiServer.refreshIPBlacklistAsync()
expect(ips).to.be.a('array')
expect(ips.length).to.equal(0)
})
})
describe('refreshIPBlacklistAsync with tor request failure, cache present', () => {
before(() => {
apiServer.setRP(async () => {
throw 'bad'
})
rocksData[TOR_IPS_KEY] = '65.1.1.1,202.10.0.12'
})
it('should return expected value', async () => {
let ips = await apiServer.refreshIPBlacklistAsync()
expect(ips).to.be.a('array')
expect(ips.length).to.equal(2)
expect(ips[0])
.to.be.a('string')
.and.to.equal('65.1.1.1')
expect(ips[1])
.to.be.a('string')
.and.to.equal('202.10.0.12')
})
})
describe('refreshIPBlacklistAsync with tor request success, cache write failure', () => {
before(() => {
apiServer.setRP(async () => {
return 'ExitNode 0011BD2485AD45D984EC4159C88FC066E5E3300E\nPublished 2019-02-17 23:51:28\nLastStatus 2019-02-18 01:18:34\nExitAddress 162.247.74.201 2019-02-18 01:22:36\nExitNode 003D78825E0B9609EECFF5E4E0529717772E53C7\nPublished 2019-02-18 14:56:19\nLastStatus 2019-02-18 16:02:35\nExitAddress 104.218.63.73 2019-02-18 16:07:38'
})
apiServer.setRocksDB(null)
})
it('should return expected value', async () => {
let ips = await apiServer.refreshIPBlacklistAsync()
expect(ips).to.be.a('array')
expect(ips.length).to.equal(2)
expect(ips[0])
.to.be.a('string')
.and.to.equal('162.247.74.201')
expect(ips[1])
.to.be.a('string')
.and.to.equal('104.218.63.73')
})
})
describe('refreshIPBlacklistAsync with tor request success, cache write success', () => {
before(() => {
apiServer.setRP(async () => {
return 'ExitNode 0011BD2485AD45D984EC4159C88FC066E5E3300E\nPublished 2019-02-17 23:51:28\nLastStatus 2019-02-18 01:18:34\nExitAddress 162.247.74.201 2019-02-18 01:22:36\nExitNode 003D78825E0B9609EECFF5E4E0529717772E53C7\nPublished 2019-02-18 14:56:19\nLastStatus 2019-02-18 16:02:35\nExitAddress 104.218.63.73 2019-02-18 16:07:38'
})
})
it('should return expected value', async () => {
let ips = await apiServer.refreshIPBlacklistAsync()
expect(ips).to.be.a('array')
expect(ips.length).to.equal(2)
expect(ips[0])
.to.be.a('string')
.and.to.equal('162.247.74.201')
expect(ips[1])
.to.be.a('string')
.and.to.equal('104.218.63.73')
})
})
describe('refreshIPBlacklistAsync with tor request success, no local list', () => {
before(() => {
apiServer.setRP(async () => {
return 'ExitNode 0011BD2485AD45D984EC4159C88FC066E5E3300E\nPublished 2019-02-17 23:51:28\nLastStatus 2019-02-18 01:18:34\nExitAddress 162.247.74.201 2019-02-18 01:22:36\nExitNode 003D78825E0B9609EECFF5E4E0529717772E53C7\nPublished 2019-02-18 14:56:19\nLastStatus 2019-02-18 16:02:35\nExitAddress 104.218.63.73 2019-02-18 16:07:38'
})
apiServer.setFS({
existsSync: () => false
})
})
it('should return expected value', async () => {
let ips = await apiServer.refreshIPBlacklistAsync()
expect(ips).to.be.a('array')
expect(ips.length).to.equal(2)
expect(ips[0])
.to.be.a('string')
.and.to.equal('162.247.74.201')
expect(ips[1])
.to.be.a('string')
.and.to.equal('104.218.63.73')
})
})
describe('refreshIPBlacklistAsync with tor request success, empty local list', () => {
before(() => {
apiServer.setRP(async () => {
return 'ExitNode 0011BD2485AD45D984EC4159C88FC066E5E3300E\nPublished 2019-02-17 23:51:28\nLastStatus 2019-02-18 01:18:34\nExitAddress 162.247.74.201 2019-02-18 01:22:36\nExitNode 003D78825E0B9609EECFF5E4E0529717772E53C7\nPublished 2019-02-18 14:56:19\nLastStatus 2019-02-18 16:02:35\nExitAddress 104.218.63.73 2019-02-18 16:07:38'
})
apiServer.setFS({
existsSync: () => true,
readFileSync: () => ''
})
})
it('should return expected value', async () => {
let ips = await apiServer.refreshIPBlacklistAsync()
expect(ips).to.be.a('array')
expect(ips.length).to.equal(2)
expect(ips[0])
.to.be.a('string')
.and.to.equal('162.247.74.201')
expect(ips[1])
.to.be.a('string')
.and.to.equal('104.218.63.73')
})
})
describe('refreshIPBlacklistAsync with tor request success, malformatted local list', () => {
before(() => {
apiServer.setRP(async () => {
return 'ExitNode 0011BD2485AD45D984EC4159C88FC066E5E3300E\nPublished 2019-02-17 23:51:28\nLastStatus 2019-02-18 01:18:34\nExitAddress 162.247.74.201 2019-02-18 01:22:36\nExitNode 003D78825E0B9609EECFF5E4E0529717772E53C7\nPublished 2019-02-18 14:56:19\nLastStatus 2019-02-18 16:02:35\nExitAddress 104.218.63.73 2019-02-18 16:07:38'
})
apiServer.setFS({
ex
gitextract_zhinso_3/
├── .eslintrc.json
├── .gitignore
├── .prettierrc
├── Dockerfile
├── LICENSE
├── Makefile
├── README.md
├── chainpoint-gateway-openapi-3.yaml
├── cloudbuild.yaml
├── docker-compose.yaml
├── init/
│ └── index.js
├── ip-blacklist.txt
├── lib/
│ ├── aggregator.js
│ ├── analytics.js
│ ├── api-server.js
│ ├── cached-proofs.js
│ ├── cores.js
│ ├── endpoints/
│ │ ├── calendar.js
│ │ ├── config.js
│ │ ├── hashes.js
│ │ ├── proofs.js
│ │ └── verify.js
│ ├── lightning.js
│ ├── logger.js
│ ├── models/
│ │ └── RocksDB.js
│ ├── parse-env.js
│ └── utils.js
├── package.json
├── scripts/
│ ├── install_deps.sh
│ ├── prod_secrets_expand.sh
│ └── run_prod.sh
├── server.js
├── swarm-compose.yaml
└── tests/
├── RocksDB.js
├── aggregator.js
├── api-server.js
├── cached-proofs.js
├── calendar.js
├── config.js
├── cores.js
├── hashes.js
├── parse-env.js
├── proofs.js
├── sample-data/
│ ├── btc-proof.chp.json
│ ├── cal-proof-l.chp.json
│ ├── cal-proof.chp.json
│ ├── core-btc-proof.chp.json
│ ├── core-cal-proof.chp.json
│ ├── core-tbtc-proof.chp.json
│ ├── core-tcal-proof.chp.json
│ ├── lsat-data.json
│ ├── tbtc-proof-l.chp.json
│ ├── tbtc-proof.chp.json
│ └── tcal-proof.chp.json
├── utils.js
└── verify.js
SYMBOL INDEX (140 symbols across 19 files)
FILE: init/index.js
constant QUIET_OUTPUT (line 31) | const QUIET_OUTPUT = true
constant LND_SOCKET (line 33) | const LND_SOCKET = '127.0.0.1:10009'
constant MIN_CHANNEL_SATOSHI (line 34) | const MIN_CHANNEL_SATOSHI = 100000
constant CHANNEL_OPEN_OVERHEAD_SAFE (line 35) | const CHANNEL_OPEN_OVERHEAD_SAFE = 20000
constant CORE_SEED_IPS_MAINNET (line 37) | const CORE_SEED_IPS_MAINNET = ['18.220.31.138']
constant CORE_SEED_IPS_TESTNET (line 38) | const CORE_SEED_IPS_TESTNET = ['3.133.119.65', '52.14.49.31', '3.135.54....
function displayTitleScreen (line 71) | function displayTitleScreen() {
function startLndNodeAsync (line 94) | async function startLndNodeAsync(initAnswers) {
function initializeLndNodeAsync (line 115) | async function initializeLndNodeAsync(initAnswers) {
function createDockerSecretsAsync (line 139) | async function createDockerSecretsAsync(initAnswers, walletInfo) {
function displayInitResults (line 154) | function displayInitResults(walletInfo) {
function setENVValuesAsync (line 171) | async function setENVValuesAsync(newENVData) {
function getCorePeerListAsync (line 186) | async function getCorePeerListAsync(seedIPs) {
function askCoreConnectQuestionsAsync (line 209) | async function askCoreConnectQuestionsAsync(progress) {
function askFundAmountAsync (line 282) | async function askFundAmountAsync(progress) {
function waitForSyncAndFundingAsync (line 347) | async function waitForSyncAndFundingAsync(progress) {
function displayFinalConnectionSummary (line 408) | function displayFinalConnectionSummary() {
function readInitProgress (line 414) | function readInitProgress() {
function writeInitProgress (line 422) | function writeInitProgress(progress) {
function setInitProgressCompleteAsync (line 426) | function setInitProgressCompleteAsync() {
function start (line 430) | async function start() {
FILE: lib/aggregator.js
constant AGG_IN_PROCESS (line 25) | let AGG_IN_PROCESS = false
function getSubmittedHashData (line 30) | async function getSubmittedHashData() {
function aggregateSubmitAndPersistAsync (line 42) | async function aggregateSubmitAndPersistAsync() {
function submitHashToCoresAsync (line 171) | async function submitHashToCoresAsync(hash) {
function startAggInterval (line 186) | function startAggInterval() {
FILE: lib/analytics.js
function setClientID (line 25) | function setClientID(clientID) {
function sendEvent (line 31) | function sendEvent(params) {
FILE: lib/api-server.js
constant TOR_IPS_KEY (line 44) | const TOR_IPS_KEY = 'blacklist:tor:ips'
constant CHAINPOINT_NODE_HTTP_PORT (line 45) | const CHAINPOINT_NODE_HTTP_PORT = process.env.CHAINPOINT_NODE_PORT || 8080
function ensureAcceptingHashes (line 56) | function ensureAcceptingHashes(req, res, next) {
function refreshIPBlacklistAsync (line 63) | async function refreshIPBlacklistAsync() {
function getTorExitIPAsync (line 70) | async function getTorExitIPAsync() {
function parseTorExitIPs (line 112) | function parseTorExitIPs(response) {
function getLocalIPBlacklistAsync (line 139) | async function getLocalIPBlacklistAsync() {
function ipFilter (line 171) | function ipFilter(req, res, next) {
function setupCommonRestifyConfigAndRoutes (line 194) | function setupCommonRestifyConfigAndRoutes(server) {
function startInsecureRestifyServerAsync (line 309) | async function startInsecureRestifyServerAsync() {
function startIPBlacklistRefreshInterval (line 323) | function startIPBlacklistRefreshInterval() {
function startAsync (line 329) | async function startAsync(lnd) {
FILE: lib/cached-proofs.js
constant CORE_PROOF_CACHE (line 21) | let CORE_PROOF_CACHE = {}
constant PRUNE_EXPIRED_INTERVAL_SECONDS (line 23) | const PRUNE_EXPIRED_INTERVAL_SECONDS = 10
function getCachedCoreProofsAsync (line 25) | async function getCachedCoreProofsAsync(coreSubmissions) {
function pruneExpiredItems (line 224) | function pruneExpiredItems() {
function startPruneExpiredItemsInterval (line 233) | function startPruneExpiredItemsInterval() {
FILE: lib/cores.js
constant PRUNE_EXPIRED_INTERVAL_SECONDS (line 27) | const PRUNE_EXPIRED_INTERVAL_SECONDS = 10
constant CONNECTED_CORE_IPS (line 29) | let CONNECTED_CORE_IPS = []
constant CONNECTED_CORE_LN_URIS (line 30) | let CONNECTED_CORE_LN_URIS = []
constant ALL_CORE_IPS (line 35) | let ALL_CORE_IPS = []
constant CORE_TX_CACHE (line 39) | let CORE_TX_CACHE = {}
function connectAsync (line 44) | async function connectAsync() {
function waitForSync (line 99) | async function waitForSync() {
function createCoreLNDPeerConnectionsAsync (line 131) | async function createCoreLNDPeerConnectionsAsync(lnUris) {
function getConnectedCoreIPsAsync (line 159) | async function getConnectedCoreIPsAsync(coreIPList, coreConnectionCount) {
function createCoreLNDChannelsAsync (line 189) | async function createCoreLNDChannelsAsync(lnUris) {
function coreRequestAsync (line 276) | async function coreRequestAsync(options, coreIP, retryCount = 3, timeout...
function submitHashAsync (line 323) | async function submitHashAsync(hash) {
function payInvoiceAsync (line 386) | async function payInvoiceAsync(invoice, submitHashInvoiceId) {
function getProofsAsync (line 409) | async function getProofsAsync(coreIP, proofIds) {
function getLatestCalBlockInfoAsync (line 429) | async function getLatestCalBlockInfoAsync() {
function getCachedTransactionAsync (line 448) | async function getCachedTransactionAsync(txID) {
function buildRequestOptions (line 474) | function buildRequestOptions(headerValues, method, uriPath, body, timeou...
function pruneExpiredItems (line 489) | function pruneExpiredItems() {
function startPruneExpiredItemsInterval (line 498) | function startPruneExpiredItemsInterval() {
function startConnectionMonitoringInterval (line 502) | function startConnectionMonitoringInterval() {
function parse402Response (line 508) | function parse402Response(response) {
FILE: lib/endpoints/calendar.js
function getDataValueByIDAsync (line 17) | async function getDataValueByIDAsync(req, res, next) {
FILE: lib/endpoints/config.js
function getConfigInfoAsync (line 25) | async function getConfigInfoAsync(req, res, next) {
FILE: lib/endpoints/hashes.js
function generatePostHashesResponseMetadata (line 34) | function generatePostHashesResponseMetadata() {
function generateProcessingHints (line 51) | function generateProcessingHints(timestampDate) {
function generatePostHashesResponse (line 74) | function generatePostHashesResponse(ip, hashes) {
function postHashesAsync (line 128) | async function postHashesAsync(req, res, next) {
FILE: lib/endpoints/proofs.js
constant BASE64_MIME_TYPE (line 28) | const BASE64_MIME_TYPE = 'application/vnd.chainpoint.json+base64'
constant JSONLD_MIME_TYPE (line 31) | const JSONLD_MIME_TYPE = 'application/vnd.chainpoint.ld+json'
function formatAsChainpointV3Ops (line 41) | function formatAsChainpointV3Ops(proof, op) {
function getProofsByIDAsync (line 62) | async function getProofsByIDAsync(req, res, next) {
function buildFullProof (line 221) | function buildFullProof(coreProof, nodeProofDataItem) {
FILE: lib/endpoints/verify.js
function ProcessVerifyTasksAsync (line 22) | async function ProcessVerifyTasksAsync(verifyTasks) {
function BuildVerifyTaskList (line 119) | function BuildVerifyTaskList(proofs) {
function buildResultObject (line 137) | function buildResultObject(parseObj, proofIndex) {
function confirmExpectedValueAsync (line 153) | async function confirmExpectedValueAsync(anchorInfo) {
function flattenExpectedValues (line 167) | function flattenExpectedValues(branchArray) {
function postProofsForVerificationAsync (line 199) | async function postProofsForVerificationAsync(req, res, next) {
FILE: lib/lightning.js
constant IS_UNLOCKING (line 22) | let IS_UNLOCKING = false
constant LND_DIR (line 23) | let LND_DIR
constant LND_SOCKET (line 24) | let LND_SOCKET
constant LND_TLS_CERT (line 25) | let LND_TLS_CERT
constant LND_MACAROON (line 26) | let LND_MACAROON
FILE: lib/models/RocksDB.js
constant PRUNE_BATCH_SIZE (line 48) | const PRUNE_BATCH_SIZE = 1000
constant PRUNE_INTERVAL_SECONDS (line 49) | const PRUNE_INTERVAL_SECONDS = 10
constant PRUNE_IN_PROGRESS (line 50) | let PRUNE_IN_PROGRESS = false
function openConnectionAsync (line 54) | async function openConnectionAsync(dir = `${process.env.HOME}/.chainpoin...
function encodeBinaryProofStateId (line 97) | function encodeBinaryProofStateId(proofIdNode) {
function decodeBinaryProofStateId (line 110) | function decodeBinaryProofStateId(idType, proofId) {
function encodeBinaryProofStateValueKey (line 116) | function encodeBinaryProofStateValueKey(proofIdNode) {
function createBinaryProofStateTimeIndexKey (line 121) | function createBinaryProofStateTimeIndexKey() {
function createBinaryProofStateTimeIndexMin (line 129) | function createBinaryProofStateTimeIndexMin() {
function createBinaryProofStateTimeIndexMax (line 135) | function createBinaryProofStateTimeIndexMax(timestamp) {
function encodeProofStateValue (line 143) | function encodeProofStateValue(nodeProofDataItem) {
function decodeProofStateValue (line 158) | function decodeProofStateValue(proofStateValue, idType) {
function getProofStatesBatchByProofIdsAsync (line 183) | async function getProofStatesBatchByProofIdsAsync(proofIds) {
function saveProofStatesBatchAsync (line 209) | async function saveProofStatesBatchAsync(nodeProofDataItems) {
function pruneProofStateDataSince (line 228) | async function pruneProofStateDataSince(timestampMS) {
function pruneOldProofStateDataAsync (line 263) | async function pruneOldProofStateDataAsync() {
function createBinaryIncomingHashObjectsTimeIndexKey (line 279) | function createBinaryIncomingHashObjectsTimeIndexKey() {
function createBinaryIncomingHashObjectsTimeIndexMin (line 287) | function createBinaryIncomingHashObjectsTimeIndexMin() {
function createBinaryIncomingHashObjectsTimeIndexMax (line 293) | function createBinaryIncomingHashObjectsTimeIndexMax(timestamp) {
function encodeIncomingHashObjectsValue (line 301) | function encodeIncomingHashObjectsValue(hashObjects) {
function decodeIncomingHashObjectsValue (line 311) | function decodeIncomingHashObjectsValue(hashObjectsBinary) {
function queueIncomingHashObjectsAsync (line 323) | async function queueIncomingHashObjectsAsync(hashObjects) {
function getIncomingHashesUpToAsync (line 334) | async function getIncomingHashesUpToAsync(maxTimestamp) {
function setAsync (line 365) | async function setAsync(key, value) {
function getAsync (line 374) | async function getAsync(key) {
function deleteBatchAsync (line 388) | async function deleteBatchAsync(delOps) {
function startPruningInterval (line 406) | function startPruningInterval() {
FILE: lib/parse-env.js
function valCoreIPList (line 17) | function valCoreIPList(list) {
function valNetwork (line 31) | function valNetwork(name) {
FILE: lib/utils.js
function sleepAsync (line 23) | function sleepAsync(ms) {
function addSeconds (line 34) | function addSeconds(date, seconds) {
function addMinutes (line 45) | function addMinutes(date, minutes) {
function formatDateISO8601NoMs (line 56) | function formatDateISO8601NoMs(date) {
function lowerCaseHashes (line 66) | function lowerCaseHashes(hashes) {
function parseAnchorsComplete (line 72) | function parseAnchorsComplete(proofObject, network) {
function isHex (line 86) | function isHex(value) {
function isUUID (line 98) | function isUUID(value) {
function isULID (line 109) | function isULID(value) {
function hexToUUIDv1 (line 120) | function hexToUUIDv1(hexString) {
function randomIntFromInterval (line 137) | function randomIntFromInterval(min, max) {
function nodeUIPasswordBooleanCheck (line 141) | function nodeUIPasswordBooleanCheck(pw = '') {
function getClientIP (line 159) | function getClientIP(req) {
function jsonTransform (line 183) | function jsonTransform(json, conditionFn, modifyFn) {
FILE: server.js
function openStorageConnectionAsync (line 25) | async function openStorageConnectionAsync() {
function startAsync (line 30) | async function startAsync() {
FILE: tests/RocksDB.js
constant TEST_ROCKS_DIR (line 13) | const TEST_ROCKS_DIR = './test_db'
function generateSampleProofStateData (line 169) | function generateSampleProofStateData(batchSize) {
function convertStateBackToBinaryForm (line 196) | function convertStateBackToBinaryForm(queriedState) {
function generateSampleHashObjects (line 214) | function generateSampleHashObjects(batchSize) {
FILE: tests/aggregator.js
function generateIncomingHashData (line 152) | function generateIncomingHashData(batchSize) {
function generateBlakeEmbeddedUUID (line 166) | function generateBlakeEmbeddedUUID(hash) {
FILE: tests/api-server.js
constant TOR_IPS_KEY (line 11) | const TOR_IPS_KEY = 'blacklist:tor:ips'
Condensed preview — 56 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (412K chars).
[
{
"path": ".eslintrc.json",
"chars": 440,
"preview": "{\n \"env\": {\n \"es6\": true,\n \"node\": true\n },\n \"extends\": [\"eslint:recommended\", \"plugin:prettier/recommended\"],\n"
},
{
"path": ".gitignore",
"chars": 135,
"preview": ".env\n.DS_Store\n.data/\n.vscode\n.vscode/\n.nyc_output\nnode_modules/\nnpm-debug.log*\nyarn-debug.log*\nyarn-error.log*\ntest.js\n"
},
{
"path": ".prettierrc",
"chars": 64,
"preview": "{\n \"singleQuote\": true,\n \"semi\": false,\n \"printWidth\": 120\n}\n"
},
{
"path": "Dockerfile",
"chars": 930,
"preview": "# Node.js 8.x LTS on Debian Stretch Linux\n# see: https://github.com/nodejs/LTS\n# see: https://hub.docker.com/_/node/\nFRO"
},
{
"path": "LICENSE",
"chars": 11357,
"preview": " Apache License\n Version 2.0, January 2004\n "
},
{
"path": "Makefile",
"chars": 3864,
"preview": "# First target in the Makefile is the default.\nall: help\n\n# without this 'source' won't work.\nSHELL := /bin/bash\n\n# Get "
},
{
"path": "README.md",
"chars": 8194,
"preview": "# Chainpoint Gateway\n\n[![code style: prettier](https://img.shields.io/badge/code_style-prettier-ff69b4.svg?style=flat-sq"
},
{
"path": "chainpoint-gateway-openapi-3.yaml",
"chars": 13240,
"preview": "openapi: 3.0.0\ninfo:\n title: 'Chainpoint Node'\n description: 'Documentation for the Chainpoint Node API'\n version: '2"
},
{
"path": "cloudbuild.yaml",
"chars": 488,
"preview": "steps:\n- name: 'gcr.io/cloud-builders/git'\n args: ['submodule', 'update', '--init', '--recursive']\n- name: 'gcr.io/clou"
},
{
"path": "docker-compose.yaml",
"chars": 2748,
"preview": "version: '3.4'\n\nnetworks:\n chainpoint-gateway:\n driver: bridge\n\nservices:\n chainpoint-gateway:\n restart: on-fail"
},
{
"path": "init/index.js",
"chars": 18238,
"preview": "/* Copyright (C) 2019 Tierion\n *\n * This program is free software: you can redistribute it and/or modify\n * it under the"
},
{
"path": "ip-blacklist.txt",
"chars": 157,
"preview": "# ip-blacklist.txt\n#\n# Add a single IPv4 address per line that you'd\n# like to block from connecting to this Node.\n#\n# L"
},
{
"path": "lib/aggregator.js",
"chars": 7125,
"preview": "/**\n * Copyright 2019 Tierion\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this"
},
{
"path": "lib/analytics.js",
"chars": 1271,
"preview": "/**\n * Copyright 2019 Tierion\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this"
},
{
"path": "lib/api-server.js",
"chars": 10715,
"preview": "/**\n * Copyright 2019 Tierion\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this"
},
{
"path": "lib/cached-proofs.js",
"chars": 11220,
"preview": "/**\n * Copyright 2019 Tierion\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this"
},
{
"path": "lib/cores.js",
"chars": 19713,
"preview": "/**\n * Copyright 2019 Tierion\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this"
},
{
"path": "lib/endpoints/calendar.js",
"chars": 1418,
"preview": "/**\n * Copyright 2019 Tierion\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this"
},
{
"path": "lib/endpoints/config.js",
"chars": 1467,
"preview": "/**\n * Copyright 2019 Tierion\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this"
},
{
"path": "lib/endpoints/hashes.js",
"chars": 5996,
"preview": "/**\n * Copyright 2019 Tierion\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this"
},
{
"path": "lib/endpoints/proofs.js",
"chars": 9246,
"preview": "/**\n * Copyright 2019 Tierion\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this"
},
{
"path": "lib/endpoints/verify.js",
"chars": 8001,
"preview": "/**\n * Copyright 2019 Tierion\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this"
},
{
"path": "lib/lightning.js",
"chars": 3152,
"preview": "/**\n * Copyright 2019 Tierion\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this"
},
{
"path": "lib/logger.js",
"chars": 1163,
"preview": "/**\n * Copyright 2019 Tierion\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this"
},
{
"path": "lib/models/RocksDB.js",
"chars": 14743,
"preview": "/**\n * Copyright 2019 Tierion\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this"
},
{
"path": "lib/parse-env.js",
"chars": 3703,
"preview": "/**\n * Copyright 2019 Tierion\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this"
},
{
"path": "lib/utils.js",
"chars": 6093,
"preview": "/**\n * Copyright 2019 Tierion\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this"
},
{
"path": "package.json",
"chars": 2218,
"preview": "{\n \"name\": \"chainpoint-gateway\",\n \"description\": \"A Chainpoint Network Gateway is a key part of a scalable solution fo"
},
{
"path": "scripts/install_deps.sh",
"chars": 1429,
"preview": "#!/bin/bash\n\nif [ -x \"$(command -v docker)\" ]; then\n echo \"Docker already installed\"\nelse\n echo \"Install docker\"\n "
},
{
"path": "scripts/prod_secrets_expand.sh",
"chars": 1230,
"preview": "#!/bin/sh\n\n: ${ENV_SECRETS_DIR:=/run/secrets}\n\nfunction env_secret_debug() {\n if [ ! -z \"$ENV_SECRETS_DEBUG\" ]; then\n"
},
{
"path": "scripts/run_prod.sh",
"chars": 71,
"preview": "#!/bin/bash\ncd $(dirname $0)\nsource ./prod_secrets_expand.sh\nyarn start"
},
{
"path": "server.js",
"chars": 2854,
"preview": "/**\n * Copyright 2019 Tierion\n * Licensed under the Apache License, Version 2.0 (the \"License\");\n * you may not use this"
},
{
"path": "swarm-compose.yaml",
"chars": 3050,
"preview": "version: \"3.7\"\n\nnetworks:\n chainpoint-gateway:\n\nsecrets:\n HOT_WALLET_PASS:\n external: true\n HOT_WALLET_ADDRESS:\n "
},
{
"path": "tests/RocksDB.js",
"chars": 6984,
"preview": "/* global describe, it, before, after */\n\nprocess.env.NODE_ENV = 'test'\n\n// test related packages\nconst expect = require"
},
{
"path": "tests/aggregator.js",
"chars": 6488,
"preview": "/* global describe, it, before, after */\n\nprocess.env.NODE_ENV = 'test'\n\n// test related packages\nconst expect = require"
},
{
"path": "tests/api-server.js",
"chars": 13037,
"preview": "/* global describe, it, beforeEach, before */\n\nprocess.env.NODE_ENV = 'test'\n\n// test related packages\nconst expect = re"
},
{
"path": "tests/cached-proofs.js",
"chars": 38111,
"preview": "/* global describe, it, before */\n\nprocess.env.NODE_ENV = 'test'\n\n// test related packages\nconst expect = require('chai'"
},
{
"path": "tests/calendar.js",
"chars": 3810,
"preview": "/* global describe, it beforeEach, afterEach */\n\nprocess.env.NODE_ENV = 'test'\n\n// test related packages\nconst expect = "
},
{
"path": "tests/config.js",
"chars": 1102,
"preview": "/* global describe, it, beforeEach, afterEach */\n\nprocess.env.NODE_ENV = 'test'\n\n// test related packages\nconst expect ="
},
{
"path": "tests/cores.js",
"chars": 29513,
"preview": "/* global describe, it, before, beforeEach, afterEach */\n\nprocess.env.NODE_ENV = 'test'\n\n// test related packages\nconst "
},
{
"path": "tests/hashes.js",
"chars": 7078,
"preview": "/* global describe, it beforeEach, afterEach, before, after */\n\nprocess.env.NODE_ENV = 'test'\n\n// test related packages\n"
},
{
"path": "tests/parse-env.js",
"chars": 6351,
"preview": "/* global describe, it */\n\nprocess.env.NODE_ENV = 'test'\n\n// test related packages\nconst expect = require('chai').expect"
},
{
"path": "tests/proofs.js",
"chars": 9595,
"preview": "/* global describe, it beforeEach, afterEach */\n\nprocess.env.NODE_ENV = 'test'\n\n// test related packages\nconst expect = "
},
{
"path": "tests/sample-data/btc-proof.chp.json",
"chars": 6280,
"preview": "{\n \"@context\": \"https://w3id.org/chainpoint/v3\",\n \"type\": \"Chainpoint\",\n \"hash\": \"ffff27222fe366d0b8988b7312c6ba60ee4"
},
{
"path": "tests/sample-data/cal-proof-l.chp.json",
"chars": 1866,
"preview": "{\n \"@context\": \"https://w3id.org/chainpoint/v3\",\n \"type\": \"Chainpoint\",\n \"hash\": \"ffff27222fe366d0b8988b7312c6ba60ee4"
},
{
"path": "tests/sample-data/cal-proof.chp.json",
"chars": 2014,
"preview": "{\n \"@context\": \"https://w3id.org/chainpoint/v3\",\n \"type\": \"Chainpoint\",\n \"hash\": \"ffff27222fe366d0b8988b7312c6ba60ee4"
},
{
"path": "tests/sample-data/core-btc-proof.chp.json",
"chars": 6533,
"preview": "{\n \"@context\": \"https://w3id.org/chainpoint/v3\",\n \"type\": \"Chainpoint\",\n \"hash\": \"18af1184ae64160f8a4019f43ddc825db95"
},
{
"path": "tests/sample-data/core-cal-proof.chp.json",
"chars": 1895,
"preview": "{\n \"@context\": \"https://w3id.org/chainpoint/v3\",\n \"type\": \"Chainpoint\",\n \"hash\": \"18af1184ae64160f8a4019f43ddc825db95"
},
{
"path": "tests/sample-data/core-tbtc-proof.chp.json",
"chars": 6535,
"preview": "{\n \"@context\": \"https://w3id.org/chainpoint/v3\",\n \"type\": \"Chainpoint\",\n \"hash\": \"18af1184ae64160f8a4019f43ddc825db95"
},
{
"path": "tests/sample-data/core-tcal-proof.chp.json",
"chars": 1896,
"preview": "{\n \"@context\": \"https://w3id.org/chainpoint/v3\",\n \"type\": \"Chainpoint\",\n \"hash\": \"18af1184ae64160f8a4019f43ddc825db95"
},
{
"path": "tests/sample-data/lsat-data.json",
"chars": 1768,
"preview": "{\n \"challenge1000\": \"LSAT macaroon=\\\"MDAxY2xvY2F0aW9uIDEyNy4wLjAuMTo4MDgwCjAwOTRpZGVudGlmaWVyIDAwMDA3Y2VmOTNmMmM1MWFhNj"
},
{
"path": "tests/sample-data/tbtc-proof-l.chp.json",
"chars": 5978,
"preview": "{\n \"@context\": \"https://w3id.org/chainpoint/v3\",\n \"type\": \"Chainpoint\",\n \"hash\": \"ffff27222fe366d0b8988b7312c6ba60ee4"
},
{
"path": "tests/sample-data/tbtc-proof.chp.json",
"chars": 6282,
"preview": "{\n \"@context\": \"https://w3id.org/chainpoint/v3\",\n \"type\": \"Chainpoint\",\n \"hash\": \"ffff27222fe366d0b8988b7312c6ba60ee4"
},
{
"path": "tests/sample-data/tcal-proof.chp.json",
"chars": 2015,
"preview": "{\n \"@context\": \"https://w3id.org/chainpoint/v3\",\n \"type\": \"Chainpoint\",\n \"hash\": \"ffff27222fe366d0b8988b7312c6ba60ee4"
},
{
"path": "tests/utils.js",
"chars": 6202,
"preview": "/* global describe, it */\n\nprocess.env.NODE_ENV = 'test'\n\n// test related packages\nconst expect = require('chai').expect"
},
{
"path": "tests/verify.js",
"chars": 45161,
"preview": "/* global describe, it beforeEach, afterEach */\n\nprocess.env.NODE_ENV = 'test'\n\n// test related packages\nconst expect = "
}
]
About this extraction
This page contains the full source code of the chainpoint/chainpoint-node GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 56 files (383.0 KB), approximately 109.6k tokens, and a symbol index with 140 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.