![](\"https://raw.githubusercontent.com/anthonybudd/s3-from-scratch/master/_img/s3.png\")
\n\n \n
\n ![](\"https://raw.githubusercontent.com/anthonybudd/s3-from-scratch/master/_img/live-demo.gif\")
\n
\n ![](\"https://raw.githubusercontent.com/anthonybudd/s3-from-scratch/master/_img/infrastructure.png?v=4\")
\n
![](\"https://raw.githubusercontent.com/anthonybudd/s3-from-scratch/master/_img/node.png\")
\n\nSince this project needs to be \"Enterprise-grade\" we need a distinct and replicable compute unit that we can buy and build in bulk. I call this a \"Node\" which is a Raspberry Pi with a 1TB SSD and POE hat. I have also 3D printed a rack-mount solution (Source: [Merocle From UpTimeLabs](https://www.thingiverse.com/thing:4756812)) for easy install into a rack. \n\n### [Console](./sections/console.md)\n![](\"https://raw.githubusercontent.com/anthonybudd/s3-from-scratch/master/_img/console-close-up.png\")
\n\nWe will need a \"console\" so we can locally interact with the infrastructure. In the past I have tried using a Raspberry Pi with a monitor and keyboard attached but I have found that using an old MacBook Pro works best for this. In this section I explain how to set-up the console so you can use it to store secrets, manage the network, provision K3s clusters and deploy pods.\n\n### [Frontend](./frontend/ReadMe.md)\n![](\"https://raw.githubusercontent.com/anthonybudd/s3-from-scratch/master/_img/frontend.gif\")
\n\nThis represents the AWS management console found at [aws.amazon.com/console](https://aws.amazon.com/console/). This is a Vue.js SPA that makes HTTP requests to the [S3 REST API](./api/ReadMe.md) for users to create, manage and delete their S3 buckets.\n\n\n### [API](./api/ReadMe.md)\n```sh\ncurl -X POST \\\n -H 'Authorization: Bearer $JWT' \\\n -H 'Content-Type: application/json' \\\n -d '{ \"name\":\"s3-test-bucket\"}' \\\n https://s3-api.anthonybudd.io/buckets\n```\n\nThis API simulates the back-end of the AWS Console. A user can sign-up, login, create a bucket then delete the bucket.\n\n### [Source Control](./sections/gitlab.md)\n![](\"https://raw.githubusercontent.com/anthonybudd/s3-from-scratch/master/_img/gitlab-logo.svg\"){kind=logo}
\n\nWe will need a way to store the code for the landing website, the front-end ui and the back-end REST API. We will also need CI/CD to compile the code and deploy it into our infrastructure. GitLab will work perfectly for these two tasks. This will allow us to commit code to the local GitLab instance, compile it into a Docker image, store the image in a repository and deploy from it directy into our local kubernetes cluster. \n\n### [Networking](./sections/networking.md)\n![](\"https://raw.githubusercontent.com/anthonybudd/s3-from-scratch/master/_img/openwrt_.png\")
\n\nWe will need a network for the nodes to communicate. For the a router I have chosen OpenWRT. This allows me to use a Raspberry Pi with a USB 3.0 Ethernet adapter so it can work as a router between the internet and the datacenter.\n\n### [Automation](./sections/automated-bucket-deployment.md)\nWhen you create an S3 bucket on AWS, everything is automated, there isn’t a human in a datacenter somewhere typing out CLI commands to get your bucket scheduled. I want my project to work the same way, when a user wants to create a bucket it must not require any human input to provision and deploy it.\n\n### [Resource Utilization](./sections/storage-cluster.md)\nAWS doesn't give each user their own dedicated server with a hard drive attached, instead the hardware is virtualized, allowing multiple tenants to share a single physical CPU. Similarly it would not be practical to assign a whole node and SSD to each bucket, to maximize resource utilization my platform must be able to allow multiple tenants to share the pool of compute and SSD storage space available. In addition, AWS S3 buckets can store an unlimited amount of data, so my platform will also need to allow a user to have a dynamically increasing volume that will auto-scale based on the storage space required.\n\n\n### Notes\nYou will need to SSH into multiple devices simultaneously I have added an annotation (example: `[Console] nano /boot/config.txt`) to all commands in this repo, to show where you should be executing each command. Generally you will see `[Console]` and `[Node X]`.\n\nBecause this is still very much a work-in-progress you will see my notes in italics \"_AB:_\" throughout, please ignore."
},
{
"path": "ansible/.gitignore",
"content": "Notes.md"
},
{
"path": "ansible/.yamllint",
"content": "---\nextends: default\n\nrules:\n line-length:\n max: 120\n level: warning\n truthy:\n allowed-values: ['true', 'false', 'yes', 'no']\n"
},
{
"path": "ansible/README.md",
"content": "# Ansible\n\nSource: [https://github.com/k3s-io/k3s-ansible](https://github.com/k3s-io/k3s-ansible)\n"
},
{
"path": "ansible/ansible.cfg",
"content": "[defaults]\nnocows = True\nroles_path = ./roles\ninventory = ./hosts.ini\n\nremote_tmp = $HOME/.ansible/tmp\nlocal_tmp = $HOME/.ansible/tmp\npipelining = True\nbecome = True\nhost_key_checking = False\ndeprecation_warnings = False\ncallback_whitelist = profile_tasks\n"
},
{
"path": "ansible/inventory/.gitignore",
"content": "*-cluster/\n!exmaple/\n!.gitignore\n!sample/"
},
{
"path": "ansible/inventory/example/group_vars/all.yml",
"content": "---\nk3s_version: v1.26.9+k3s1\nansible_user: node\nsystemd_dir: /etc/systemd/system\nmaster_ip: \"{{ hostvars[groups['master'][0]]['ansible_host'] | default(groups['master'][0]) }}\"\nextra_server_args: \"\"\nextra_agent_args: \"\"\n"
},
{
"path": "ansible/inventory/example/hosts.ini",
"content": "[master]\n10.0.0.5\n\n[node]\n10.0.0.5\n10.0.0.6\n10.0.0.7\n\n[k3s_cluster:children]\nmaster\nnode\n"
},
{
"path": "ansible/reset.yml",
"content": "---\n\n- hosts: k3s_cluster\n gather_facts: yes\n become: yes\n roles:\n - role: reset\n"
},
{
"path": "ansible/roles/download/tasks/main.yml",
"content": "---\n\n- name: Download k3s binary x64\n get_url:\n url: https://github.com/k3s-io/k3s/releases/download/{{ k3s_version }}/k3s\n checksum: sha256:https://github.com/k3s-io/k3s/releases/download/{{ k3s_version }}/sha256sum-amd64.txt\n dest: /usr/local/bin/k3s\n owner: root\n group: root\n mode: 0755\n when: ansible_facts.architecture == \"x86_64\"\n\n- name: Download k3s binary arm64\n get_url:\n url: https://github.com/k3s-io/k3s/releases/download/{{ k3s_version }}/k3s-arm64\n checksum: sha256:https://github.com/k3s-io/k3s/releases/download/{{ k3s_version }}/sha256sum-arm64.txt\n dest: /usr/local/bin/k3s\n owner: root\n group: root\n mode: 0755\n when:\n - ( ansible_facts.architecture is search(\"arm\") and\n ansible_facts.userspace_bits == \"64\" ) or\n ansible_facts.architecture is search(\"aarch64\")\n\n- name: Download k3s binary armhf\n get_url:\n url: https://github.com/k3s-io/k3s/releases/download/{{ k3s_version }}/k3s-armhf\n checksum: sha256:https://github.com/k3s-io/k3s/releases/download/{{ k3s_version }}/sha256sum-arm.txt\n dest: /usr/local/bin/k3s\n owner: root\n group: root\n mode: 0755\n when:\n - ansible_facts.architecture is search(\"arm\")\n - ansible_facts.userspace_bits == \"32\"\n"
},
{
"path": "ansible/roles/k3s/master/tasks/main.yml",
"content": "---\n\n- name: Copy K3s service file\n register: k3s_service\n template:\n src: \"k3s.service.j2\"\n dest: \"{{ systemd_dir }}/k3s.service\"\n owner: root\n group: root\n mode: 0644\n\n- name: Enable and check K3s service\n systemd:\n name: k3s\n daemon_reload: yes\n state: restarted\n enabled: yes\n\n- name: Wait for node-token\n wait_for:\n path: /var/lib/rancher/k3s/server/node-token\n\n- name: Register node-token file access mode\n stat:\n path: /var/lib/rancher/k3s/server\n register: p\n\n- name: Change file access node-token\n file:\n path: /var/lib/rancher/k3s/server\n mode: \"g+rx,o+rx\"\n\n- name: Read node-token from master\n slurp:\n src: /var/lib/rancher/k3s/server/node-token\n register: node_token\n\n- name: Store Master node-token\n set_fact:\n token: \"{{ node_token.content | b64decode | regex_replace('\\n', '') }}\"\n\n- name: Restore node-token file access\n file:\n path: /var/lib/rancher/k3s/server\n mode: \"{{ p.stat.mode }}\"\n\n- name: Create directory .kube\n file:\n path: ~{{ ansible_user }}/.kube\n state: directory\n owner: \"{{ ansible_user }}\"\n mode: \"u=rwx,g=rx,o=\"\n\n- name: Copy config file to user home directory\n copy:\n src: /etc/rancher/k3s/k3s.yaml\n dest: ~{{ ansible_user }}/.kube/config\n remote_src: yes\n owner: \"{{ ansible_user }}\"\n mode: \"u=rw,g=,o=\"\n\n- name: Replace https://localhost:6443 by https://master-ip:6443\n command: >-\n k3s kubectl config set-cluster default\n --server=https://{{ master_ip }}:6443\n --kubeconfig ~{{ ansible_user }}/.kube/config\n changed_when: true\n\n- name: Create kubectl symlink\n file:\n src: /usr/local/bin/k3s\n dest: /usr/local/bin/kubectl\n state: link\n\n- name: Create crictl symlink\n file:\n src: /usr/local/bin/k3s\n dest: /usr/local/bin/crictl\n state: link\n"
},
{
"path": "ansible/roles/k3s/master/templates/k3s.service.j2",
"content": "[Unit]\nDescription=Lightweight Kubernetes\nDocumentation=https://k3s.io\nAfter=network-online.target\n\n[Service]\nType=notify\nExecStartPre=-/sbin/modprobe br_netfilter\nExecStartPre=-/sbin/modprobe overlay\nExecStart=/usr/local/bin/k3s server {{ extra_server_args | default(\"\") }}\nKillMode=process\nDelegate=yes\n# Having non-zero Limit*s causes performance problems due to accounting overhead\n# in the kernel. We recommend using cgroups to do container-local accounting.\nLimitNOFILE=1048576\nLimitNPROC=infinity\nLimitCORE=infinity\nTasksMax=infinity\nTimeoutStartSec=0\nRestart=always\nRestartSec=5s\n\n[Install]\nWantedBy=multi-user.target\n"
},
{
"path": "ansible/roles/k3s/node/tasks/main.yml",
"content": "---\n\n- name: Copy K3s service file\n template:\n src: \"k3s.service.j2\"\n dest: \"{{ systemd_dir }}/k3s-node.service\"\n owner: root\n group: root\n mode: 0755\n\n- name: Enable and check K3s service\n systemd:\n name: k3s-node\n daemon_reload: yes\n state: restarted\n enabled: yes\n"
},
{
"path": "ansible/roles/k3s/node/templates/k3s.service.j2",
"content": "[Unit]\nDescription=Lightweight Kubernetes\nDocumentation=https://k3s.io\nAfter=network-online.target\n\n[Service]\nType=notify\nExecStartPre=-/sbin/modprobe br_netfilter\nExecStartPre=-/sbin/modprobe overlay\nExecStart=/usr/local/bin/k3s agent --server https://{{ master_ip }}:6443 --token {{ hostvars[groups['master'][0]]['token'] }} {{ extra_agent_args | default(\"\") }}\nKillMode=process\nDelegate=yes\n# Having non-zero Limit*s causes performance problems due to accounting overhead\n# in the kernel. We recommend using cgroups to do container-local accounting.\nLimitNOFILE=1048576\nLimitNPROC=infinity\nLimitCORE=infinity\nTasksMax=infinity\nTimeoutStartSec=0\nRestart=always\nRestartSec=5s\n\n[Install]\nWantedBy=multi-user.target\n"
},
{
"path": "ansible/roles/prereq/tasks/main.yml",
"content": "---\n- name: Set SELinux to disabled state\n selinux:\n state: disabled\n when: ansible_distribution in ['CentOS', 'Red Hat Enterprise Linux']\n\n- name: Enable IPv4 forwarding\n sysctl:\n name: net.ipv4.ip_forward\n value: \"1\"\n state: present\n reload: yes\n\n- name: Enable IPv6 forwarding\n sysctl:\n name: net.ipv6.conf.all.forwarding\n value: \"1\"\n state: present\n reload: yes\n\n- name: Add br_netfilter to /etc/modules-load.d/\n copy:\n content: \"br_netfilter\"\n dest: /etc/modules-load.d/br_netfilter.conf\n mode: \"u=rw,g=,o=\"\n when: ansible_distribution in ['CentOS', 'Red Hat Enterprise Linux']\n\n- name: Load br_netfilter\n modprobe:\n name: br_netfilter\n state: present\n when: ansible_distribution in ['CentOS', 'Red Hat Enterprise Linux']\n\n- name: Set bridge-nf-call-iptables (just to be sure)\n sysctl:\n name: \"{{ item }}\"\n value: \"1\"\n state: present\n reload: yes\n when: ansible_distribution in ['CentOS', 'Red Hat Enterprise Linux']\n loop:\n - net.bridge.bridge-nf-call-iptables\n - net.bridge.bridge-nf-call-ip6tables\n\n- name: Add /usr/local/bin to sudo secure_path\n lineinfile:\n line: 'Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin'\n regexp: \"Defaults(\\\\s)*secure_path(\\\\s)*=\"\n state: present\n insertafter: EOF\n path: /etc/sudoers\n validate: 'visudo -cf %s'\n when: ansible_distribution in ['CentOS', 'Red Hat Enterprise Linux']\n"
},
{
"path": "ansible/roles/raspberrypi/handlers/main.yml",
"content": "---\n- name: reboot\n reboot:\n"
},
{
"path": "ansible/roles/raspberrypi/tasks/main.yml",
"content": "---\n- name: Test for raspberry pi /proc/cpuinfo\n command: grep -E \"Raspberry Pi|BCM2708|BCM2709|BCM2835|BCM2836\" /proc/cpuinfo\n register: grep_cpuinfo_raspberrypi\n failed_when: false\n changed_when: false\n\n- name: Test for raspberry pi /proc/device-tree/model\n command: grep -E \"Raspberry Pi\" /proc/device-tree/model\n register: grep_device_tree_model_raspberrypi\n failed_when: false\n changed_when: false\n\n- name: Set raspberry_pi fact to true\n set_fact:\n raspberry_pi: true\n when:\n grep_cpuinfo_raspberrypi.rc == 0 or grep_device_tree_model_raspberrypi.rc == 0\n\n- name: Set detected_distribution to Raspbian\n set_fact:\n detected_distribution: Raspbian\n when: >\n raspberry_pi|default(false) and\n ( ansible_facts.lsb.id|default(\"\") == \"Raspbian\" or\n ansible_facts.lsb.description|default(\"\") is match(\"[Rr]aspbian.*\") )\n\n- name: Set detected_distribution to Raspbian (ARM64 on Debian Buster)\n set_fact:\n detected_distribution: Raspbian\n when:\n - ansible_facts.architecture is search(\"aarch64\")\n - raspberry_pi|default(false)\n - ansible_facts.lsb.description|default(\"\") is match(\"Debian.*buster\")\n\n- name: Set detected_distribution_major_version\n set_fact:\n detected_distribution_major_version: \"{{ ansible_facts.lsb.major_release }}\"\n when:\n - detected_distribution | default(\"\") == \"Raspbian\"\n\n- name: execute OS related tasks on the Raspberry Pi\n include_tasks: \"{{ item }}\"\n with_first_found:\n - \"prereq/{{ detected_distribution }}-{{ detected_distribution_major_version }}.yml\"\n - \"prereq/{{ detected_distribution }}.yml\"\n - \"prereq/{{ ansible_distribution }}-{{ ansible_distribution_major_version }}.yml\"\n - \"prereq/{{ ansible_distribution }}.yml\"\n - \"prereq/default.yml\"\n when:\n - raspberry_pi|default(false)\n"
},
{
"path": "ansible/roles/raspberrypi/tasks/prereq/CentOS.yml",
"content": "---\n- name: Enable cgroup via boot commandline if not already enabled for Centos\n lineinfile:\n path: /boot/cmdline.txt\n backrefs: yes\n regexp: '^((?!.*\\bcgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory\\b).*)$'\n line: '\\1 cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory'\n notify: reboot\n"
},
{
"path": "ansible/roles/raspberrypi/tasks/prereq/Raspbian.yml",
"content": "---\n- name: Activating cgroup support\n lineinfile:\n path: /boot/cmdline.txt\n regexp: '^((?!.*\\bcgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory\\b).*)$'\n line: '\\1 cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory'\n backrefs: true\n notify: reboot\n\n- name: Flush iptables before changing to iptables-legacy\n iptables:\n flush: true\n changed_when: false # iptables flush always returns changed\n\n- name: Changing to iptables-legacy\n alternatives:\n path: /usr/sbin/iptables-legacy\n name: iptables\n register: ip4_legacy\n\n- name: Changing to ip6tables-legacy\n alternatives:\n path: /usr/sbin/ip6tables-legacy\n name: ip6tables\n register: ip6_legacy\n"
},
{
"path": "ansible/roles/raspberrypi/tasks/prereq/Ubuntu.yml",
"content": "---\n- name: Enable cgroup via boot commandline if not already enabled for Ubuntu on a Raspberry Pi\n lineinfile:\n path: /boot/firmware/cmdline.txt\n backrefs: yes\n regexp: '^((?!.*\\bcgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory\\b).*)$'\n line: '\\1 cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory'\n notify: reboot\n"
},
{
"path": "ansible/roles/raspberrypi/tasks/prereq/default.yml",
"content": "---\n"
},
{
"path": "ansible/roles/reset/tasks/main.yml",
"content": "---\n- name: Disable services\n systemd:\n name: \"{{ item }}\"\n state: stopped\n enabled: no\n failed_when: false\n with_items:\n - k3s\n - k3s-node\n\n- name: pkill -9 -f \"k3s/data/[^/]+/bin/containerd-shim-runc\"\n register: pkill_containerd_shim_runc\n command: pkill -9 -f \"k3s/data/[^/]+/bin/containerd-shim-runc\"\n changed_when: \"pkill_containerd_shim_runc.rc == 0\"\n failed_when: false\n\n- name: Umount k3s filesystems\n include_tasks: umount_with_children.yml\n with_items:\n - /run/k3s\n - /var/lib/kubelet\n - /run/netns\n - /var/lib/rancher/k3s\n loop_control:\n loop_var: mounted_fs\n\n- name: Remove service files, binaries and data\n file:\n name: \"{{ item }}\"\n state: absent\n with_items:\n - /usr/local/bin/k3s\n - \"{{ systemd_dir }}/k3s.service\"\n - \"{{ systemd_dir }}/k3s-node.service\"\n - /etc/rancher/k3s\n - /var/lib/kubelet\n - /var/lib/rancher/k3s\n\n- name: daemon_reload\n systemd:\n daemon_reload: yes\n"
},
{
"path": "ansible/roles/reset/tasks/umount_with_children.yml",
"content": "---\n- name: Get the list of mounted filesystems\n shell: set -o pipefail && cat /proc/mounts | awk '{ print $2}' | grep -E \"^{{ mounted_fs }}\"\n register: get_mounted_filesystems\n args:\n executable: /bin/bash\n failed_when: false\n changed_when: get_mounted_filesystems.stdout | length > 0\n check_mode: false\n\n- name: Umount filesystem\n mount:\n path: \"{{ item }}\"\n state: unmounted\n with_items:\n \"{{ get_mounted_filesystems.stdout_lines | reverse | list }}\"\n"
},
{
"path": "ansible/site.yml",
"content": "---\n\n- hosts: k3s_cluster\n gather_facts: yes\n become: yes\n roles:\n - role: prereq\n - role: download\n - role: raspberrypi\n\n- hosts: master\n become: yes\n roles:\n - role: k3s/master\n\n- hosts: node\n become: yes\n roles:\n - role: k3s/node\n"
},
{
"path": "api/.dockerignore",
"content": "node_modules\npackage-lock.json"
},
{
"path": "api/.eslintignore",
"content": "src/database/\ntests"
},
{
"path": "api/.gitignore",
"content": "node_modules/\n.vol/\n.env\ndev.js\nprivate.pem\npublic.pem\n.DS_Store\n*/.DS_Store\nk8s/secrets.yml\nkubeconfig.yml"
},
{
"path": "api/.gitlab-ci.yml",
"content": "stages:\n - build\n\nbuild-job:\n image: docker:dind\n stage: build\n services:\n - docker:dind\n variables:\n IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG\n script:\n - docker login $CI_REGISTRY -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD\n - docker build -t $IMAGE_TAG .\n - docker push $IMAGE_TAG"
},
{
"path": "api/.sequelizerc",
"content": "const path = require('path');\n\nmodule.exports = {\n 'config': path.resolve('src/providers', 'connections.js'),\n 'models-path': path.resolve('src/', 'models'),\n 'seeders-path': path.resolve('src/database', 'seeders'),\n 'migrations-path': path.resolve('src/database', 'migrations')\n}"
},
{
"path": "api/Dockerfile",
"content": "FROM node:20\n\nRUN apt-get update && apt-get install -y curl nano\nRUN curl -LO \"https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/arm64/kubectl\"\nRUN install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl\n\nRUN npm install -g nodemon mocha sequelize sequelize-cli mysql2 eslint\n\nWORKDIR /app\nCOPY . /app\nRUN npm install\n\nENTRYPOINT [ \"node\", \"/app/src/index.js\" ]\n"
},
{
"path": "api/LICENSE",
"content": "The MIT License\n\nCopyright Anthony C. Budd\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in\nall copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN\nTHE SOFTWARE."
},
{
"path": "api/ReadMe.md",
"content": "# S3 API\n\nThis API simulates the back-end of the AWS Console. A user can sign-up, login, create a bucket then delete the bucket.\n\nThis API was built using my project [anthonybudd/express-api-boilerplate.](https://github.com/anthonybudd/express-api-boilerplate)\n\n### Main Files\n- Auth Controller: [./src/routes/Auth.js](./src/routes/auth.js)\n- Bucket Controller: [./src/routes/Buckets.js](./src/routes/Buckets.js)\n- Model: [./src/models/Bucket.js](./src/models/Bucket.js)\n\n\n### Set-up\n```\ncp .env.example .env\nnpm install\n\n# Private RSA key for JWT signing\nopenssl genrsa -out private.pem 2048\nopenssl rsa -in private.pem -outform PEM -pubout -out public.pem\n\n# Start the app\ndocker compose up\nnpm run _db:refresh\nnpm run _test\n```\n\n\n### Routes\n| Method | Route | Description | Payload | Response | \n| ----------- | -------------------------------- | ------------------------------------- | ------------------------------------- | ----------------- | \n| **Buckets** | | | | | \n| GET | `/api/v1/buckets` | Get all buckets for the current user | -- | [Bucket, Bucket] | \n| POST | `/api/v1/buckets` | Create new bucket | { name: \"test-bucket\" } | {Bucket} | \n| GET | `/api/v1/buckets/:bucketID` | Get a single bucket | -- | {Bucket} | \n| DELETE | `/api/v1/buckets/:bucketID` | Returns HTTP 202 {id} | -- | {bucketID} | \n| **Auth** | | | | | \n| POST | `/api/v1/auth/login` | Login | {email, password} | {accessToken} | \n| POST | `/api/v1/auth/sign-up` | Sign-up | {email, password, firstName, tos} | {accessToken} | \n| GET | `/api/v1/_authcheck` | Returns {auth: true} if has auth | -- | {auth: true} | \n| **User** | | | | | \n| GET | `/api/v1/user` | Get the current user | | {User} | \n| POST | `/api/v1/user` | Update the current user | {firstName, lastName} | {User} | \n\n\n"
},
{
"path": "api/docker-compose.yml",
"content": "version: \"3\"\n\nservices:\n s3-api:\n build: .\n entrypoint: \"nodemon /app/src/index.js --watch /app --legacy-watch\"\n container_name: s3-api\n volumes:\n - ./:/app\n - ./.vol/tmp:/tmp\n links:\n - s3-api-db\n - s3-api-db-test\n ports:\n - \"8888:80\"\n environment:\n PORT: 80\n\n s3-api-db:\n image: mysql:oracle\n container_name: s3-api-db\n ports:\n - \"3306:3306\"\n volumes:\n - ./.vol/s3-api:/var/lib/mysql\n environment:\n MYSQL_ROOT_PASSWORD: supersecret\n MYSQL_DATABASE: $DB_DATABASE\n MYSQL_USER: $DB_USERNAME\n MYSQL_PASSWORD: $DB_PASSWORD\n\n s3-api-db-test:\n image: mysql:oracle\n container_name: s3-api-db-test\n ports:\n - \"3307:3306\"\n volumes:\n - ./.vol/s3-api-test:/var/lib/mysql\n environment:\n MYSQL_ROOT_PASSWORD: supersecret\n MYSQL_DATABASE: $DB_DATABASE\n MYSQL_USER: $DB_USERNAME\n MYSQL_PASSWORD: $DB_PASSWORD\n"
},
{
"path": "api/k8s/Deploy.md",
"content": "# Deploy\n\n\n\n\nkubectl --kubeconfig=.kube/config create namespace s3-api\nnamespace/s3-api created\n[Console]:~> kubectl --kubeconfig=.kube/config apply -f db.yml \nservice/s3-db created\ndeployment.apps/s3-db created\n\n\n\n\n----\n\nFind & Replace (case-sensaive, whole repo): \"s3-api\" => \"your-api-name\" \n\nSave kubeconfig.yml to root of repo\n\n\n### Namespace\nCreate a namespace\n`kubectl --kubeconfig=./kubeconfig.yml create namespace s3-api`\n\n\n### JWT\n```\nopenssl genrsa -out private.pem 2048\nopenssl rsa -in private.pem -outform PEM -pubout -out public.pem\nkubectl --kubeconfig=.kube/config --namespace=s3-api create secret generic s3-api-jwt-secret \\\n --from-file=./private.pem \\\n --from-file=./public.pem\nrm ./private.pem ./public.pem \n```\n\n\n### Secrets\nMake a new secrets config file\n`cp secrets.example.yml secrets.yml`\n\n__Add Secrets in Base64__\nHint: `echo -n 'my-secret-string' | base64`\n\nCreate the secrets\n`kubectl --kubeconfig=./kubeconfig.yml apply -f ./k8s/secrets.yml`\n\n\n### Build & Push Container Image\n```\ndocker buildx build --platform linux/amd64 --push -t registry.digitalocean.com/s3-api/app:latest\n```\n\n### Create Deployment\n```\nkubectl --kubeconfig=./kubeconfig.yml apply -f ./k8s/api.deployment.yml\nkubectl --kubeconfig=./kubeconfig.yml apply -f ./k8s/api.service.yml\n```\n\n### Deploy\n```\ndocker buildx build --platform linux/amd64 --push -t registry.digitalocean.com/s3-api/app:latest . && \nkubectl --kubeconfig=./kubeconfig.yml rollout restart deployment s3-api && \\\nkubectl --kubeconfig=./kubeconfig.yml get pods -w\n```\n\n\n### Migrate\nMigrate the DB\n```\nexport POD=\"$(kubectl --kubeconfig=kubeconfig.yml --namespace=s3-api get pods --field-selector=status.phase==Running --no-headers -o custom-columns=\":metadata.name\")\"\nkubectl --kubeconfig=./kubeconfig.yml --namespace=s3-api exec -ti $POD -- /bin/bash -c 'sequelize db:migrate:undo:all && sequelize db:migrate && sequelize db:seed:all'\n```\n\n### SSL\nReadMore: https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nginx-ingress-with-cert-manager-on-digitalocean-kubernetes\n\n```\nkubectl --kubeconfig=./kubeconfig.yml apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/do/deploy.yaml\n\nkubectl --kubeconfig=./kubeconfig.yml get pods -n ingress-nginx -l app.kubernetes.io/name=ingress-nginx --watch\n\nkubectl --kubeconfig=./kubeconfig.yml apply -f ./k8s/api.ingress.yml\n\nkubectl --kubeconfig=./kubeconfig.yml apply -f https://github.com/jetstack/cert-manager/releases/download/v1.7.1/cert-manager.yaml\n\nkubectl --kubeconfig=./kubeconfig.yml get pods --namespace cert-manager\n\nkubectl --kubeconfig=./kubeconfig.yml create -f k8s/prod-issuer.yml\n```\n\n### Useful K8S commands\n##### Set $POD as the name of the pod in K8s\n`export POD=\"$(kubectl --kubeconfig=kubeconfig.yml --namespace=s3-api get pods --field-selector=status.phase==Running --no-headers -o custom-columns=\":metadata.name\")\"`\n\n##### Execute bash script inside running container\n`kubectl --kubeconfig=kubeconfig.yml exec -ti $POD -- /bin/bash -c \"sequelize db:migrate\"`\n\n##### Get logs for $POD\n`kubectl --kubeconfig=kubeconfig.yml logs $POD`\n\n##### Create a cron job\n`kubectl --kubeconfig=kubeconfig.yml create job --from=cronjob/s3-api-cron-job s3-api-cron-job`\n\n##### Delete all faild cron jobs\n`kubectl --kubeconfig=kubeconfig.yml delete jobs --field-selector status.successful=0`\n"
},
{
"path": "api/k8s/api.deployment.yml",
"content": "apiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: s3-api\n namespace: s3-api\nspec:\n replicas: 1\n selector:\n matchLabels:\n app: s3-api\n template:\n metadata:\n labels:\n app: s3-api\n spec:\n volumes:\n - name: s3-api-jwt-secret\n secret:\n secretName: s3-api-jwt-secret\n - name: storage-cluster-config\n secret:\n secretName: storage-cluster-config \n containers:\n - name: s3-api\n image: gitlab.local:5050/anthonybudd/api:master\n imagePullPolicy: Always\n lifecycle:\n postStart:\n exec:\n command: [\"/bin/bash\", \"-c\", \"sequelize db:migrate\"]\n ports:\n - containerPort: 80\n volumeMounts:\n - name: s3-api-jwt-secret\n mountPath: \"/app/private.pem\"\n subPath: private.pem\n - name: s3-api-jwt-secret\n mountPath: \"/app/public.pem\"\n subPath: public.pem\n - name: storage-k8s-config\n mountPath: \"/app/config\"\n subPath: storage-config\n env:\n - name: S3_ROOT\n value: \"s3.anthonybudd.io\"\n - name: K8S_CONFIG_PATH\n value: \"/app/k8s-config\"\n - name: NODE_ENV\n value: \"production\"\n - name: FRONTEND_URL\n value: \"https://s3.anthonybudd.io\"\n - name: BACKEND_URL\n value: \"https://s3-api.anthonybudd.io/api/v1\"\n - name: PORT\n value: \"80\"\n - name: PRIVATE_KEY_PATH\n value: \"/app/private.pem\"\n - name: PUBLIC_KEY_PATH\n value: \"/app/public.pem\"\n - name: DB_HOST\n value: \"s3-db\"\n - name: DB_PORT\n value: \"3306\"\n - name: DB_USERNAME\n value: \"app\"\n - name: DB_DATABASE\n value: \"app\"\n - name: DB_PASSWORD\n valueFrom:\n secretKeyRef:\n name: s3-api-secrets\n key: DB_PASSWORD\n - name: HCAPTCHA_SECRET\n valueFrom:\n secretKeyRef:\n name: s3-api-secrets\n key: HCAPTCHA_SECRET\n"
},
{
"path": "api/k8s/api.ingress.yml",
"content": "apiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n namespace: s3-api\n name: s3-ingress\n annotations:\n kubernetes.io/ingress.class: \"traefik\"\nspec:\n rules:\n - host: api.local\n http:\n paths:\n - path: /\n pathType: Prefix\n backend:\n service:\n name: s3-api\n port:\n number: 80"
},
{
"path": "api/k8s/api.service.yml",
"content": "apiVersion: v1\nkind: Service\nmetadata:\n name: s3-api\n namespace: s3-api\nspec:\n ports:\n - port: 80\n targetPort: 80\n selector:\n app: s3-api"
},
{
"path": "api/k8s/api.ssl.ingress.yml",
"content": "apiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n namespace: s3-api\n name: s3-ingress\n annotations:\n cert-manager.io/cluster-issuer: \"letsencrypt-prod\"\n kubernetes.io/ingress.class: \"traefik\"\nspec:\n tls:\n - hosts:\n - s3-api.anthonybudd.io\n secretName: s3-api-anthonybudd-io-cert\n rules:\n - host: s3-api.anthonybudd.io\n http:\n paths:\n - path: /\n pathType: Prefix\n backend:\n service:\n name: s3-api\n port:\n number: 80"
},
{
"path": "api/k8s/db.yml",
"content": "apiVersion: v1\nkind: Service\nmetadata:\n name: s3-db\n namespace: s3-api\nspec:\n selector: \n app: s3-db \n ports: \n - protocol: TCP \n port: 80 \n targetPort: 3306\n---\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n name: s3-db\n namespace: s3-api\nspec:\n selector:\n matchLabels:\n app: s3-db\n strategy:\n type: Recreate\n template:\n metadata:\n labels:\n app: s3-db\n spec:\n containers:\n - image: mysql:8\n name: s3-db\n env:\n - name: MYSQL_ROOT_PASSWORD\n value: password\n - name: MYSQL_DATABASE\n value: app\n - name: MYSQL_USER\n value: app\n - name: MYSQL_PASSWORD\n value: password\n ports:\n - containerPort: 3306\n name: s3-db\n # volumeMounts:\n # - name: mysql-persistent-storage\n # mountPath: /var/lib/mysql\n # volumes:\n # - name: mysql-persistent-storage\n # persistentVolumeClaim:\n # claimName: mysql-pv-claim"
},
{
"path": "api/k8s/prod.clusterissuer.yml",
"content": "apiVersion: cert-manager.io/v1\nkind: ClusterIssuer\nmetadata:\n name: letsencrypt-prod\n namespace: cert-manager\nspec:\n acme:\n email: YOUR_EMAIL_ADDRESS\n server: https://acme-v02.api.letsencrypt.org/directory\n privateKeySecretRef:\n name: letsencrypt-prod\n solvers:\n - http01:\n ingress:\n class: traefik"
},
{
"path": "api/k8s/secrets.example.yml",
"content": "apiVersion: v1 \nkind: Secret\nmetadata:\n name: s3-api-secrets\n namespace: default\ntype: Opaque\ndata:\n DB_PASSWORD: \n "
},
{
"path": "api/k8s/sync.job.yml",
"content": "apiVersion: batch/v1\nkind: CronJob\nmetadata:\n name: sync\n namespace: s3-api\nspec:\n schedule: \"* * * * *\"\n jobTemplate:\n spec:\n template:\n spec:\n restartPolicy: OnFailure\n volumes:\n - name: storage-k8s-config\n secret:\n secretName: storage-k8s-config \n containers:\n - name: s3-api\n image: gitlab.local:5050/anthonybudd/api:main\n imagePullPolicy: Always\n command:\n - /bin/bash\n - -c\n - \"node /app/src/scripts/sync.js\"\n volumeMounts:\n - name: storage-k8s-config\n mountPath: \"/app/storage-config\"\n subPath: storage-config\n env:\n - name: S3_ROOT\n value: \"s3.anthonybudd.io\"\n - name: K8S_CONFIG_PATH\n value: \"/app/storage-config\"\n - name: NODE_ENV\n value: \"production\"\n - name: FRONTEND_URL\n value: \"https://s3.anthonybudd.io\"\n - name: BACKEND_URL\n value: \"https://s3-api.anthonybudd.io/api/v1\"\n - name: PORT\n value: \"80\"\n - name: DB_HOST\n value: \"s3-db\"\n - name: DB_PORT\n value: \"3306\"\n - name: DB_USERNAME\n value: \"app\"\n - name: DB_DATABASE\n value: \"app\"\n - name: DB_PASSWORD\n valueFrom:\n secretKeyRef:\n name: s3-api-secrets\n key: DB_PASSWORD\n - name: HCAPTCHA_SECRET\n valueFrom:\n secretKeyRef:\n name: s3-api-secrets\n key: HCAPTCHA_SECRET"
},
{
"path": "api/package.json",
"content": "{\n \"name\": \"s3-api-boilerplate\",\n \"version\": \"1.0.0\",\n \"main\": \"./src/index.js\",\n \"author\": \"Anthony Budd\",\n \"scripts\": {\n \"start\": \"node ./src/\",\n \"lint\": \"eslint src\",\n \"_lint\": \"docker exec -ti s3-api npm run lint\",\n \"jwt\": \"node ./src/scripts/jwt.js\",\n \"_jwt\": \"docker exec -ti s3-api npm run jwt\",\n \"env\": \"./src/scripts/env\",\n \"db:migrate\": \"sequelize db:migrate\",\n \"db:seed\": \"./src/scripts/seed\",\n \"db:refresh\": \"./src/scripts/refresh\",\n \"_db:refresh\": \"docker exec -ti s3-api npm run db:refresh\",\n \"db:refresh-test\": \"node_modules/.bin/sequelize db:migrate:undo:all --env test && node_modules/.bin/sequelize db:migrate --env test && node_modules/.bin/sequelize db:seed:all --env test\",\n \"test\": \"npm run db:refresh-test && mocha --exit --timeout 10000 tests\",\n \"_test\": \"docker exec -ti s3-api npm run test\"\n },\n \"eslintConfig\": {\n \"extends\": \"eslint:recommended\",\n \"parserOptions\": {\n \"ecmaVersion\": 8,\n \"sourceType\": \"module\"\n },\n \"env\": {\n \"node\": true,\n \"es6\": true\n },\n \"rules\": {\n \"no-console\": 0,\n \"no-unused-vars\": 1\n }\n },\n \"dependencies\": {\n \"axios\": \"^0.24.0\",\n \"bcrypt-nodejs\": \"0.0.3\",\n \"cors\": \"^2.4.1\",\n \"dotenv\": \"^10.0.0\",\n \"express\": \"^4.8.5\",\n \"express-fileupload\": \"^1.4.0\",\n \"express-jwt\": \"^6.1.0\",\n \"express-validator\": \"^6.13.0\",\n \"faker\": \"^4.1.0\",\n \"i\": \"^0.3.6\",\n \"install\": \"^0.12.1\",\n \"jsonwebtoken\": \"^5.7.0\",\n \"jwt-decode\": \"^2.2.0\",\n \"lodash\": \"^4.17.21\",\n \"minimist\": \"^1.2.6\",\n \"moment\": \"^2.30.1\",\n \"morgan\": \"^1.9.1\",\n \"mustache\": \"^3.2.1\",\n \"mysql2\": \"^2.2.5\",\n \"npm\": \"^7.20.6\",\n \"passport\": \"^0.4.0\",\n \"passport-jwt\": \"^4.0.0\",\n \"passport-local\": \"^1.0.0\",\n \"sequelize\": \"^6.11.0\",\n \"sequelize-cli\": \"^6.3.0\",\n \"sha256\": \"^0.2.0\",\n \"uuid\": \"^3.4.0\"\n },\n \"devDependencies\": {\n \"chai\": \"^3.2.0\",\n \"chai-http\": \"^4.3.0\",\n \"eslint\": \"^5.8.0\",\n \"mocha\": \"^9.1.3\",\n \"nyc\": \"^14.1.1\",\n \"prettier\": \"^1.18.2\"\n }\n}\n"
},
{
"path": "api/postman.json",
"content": "{\n\t\"info\": {\n\t\t\"_postman_id\": \"994858ef-e55e-425c-9aac-1cf12496a933\",\n\t\t\"name\": \"s3-api-Boilerplate\",\n\t\t\"schema\": \"https://schema.getpostman.com/json/collection/v2.1.0/collection.json\"\n\t},\n\t\"item\": [\n\t\t{\n\t\t\t\"name\": \"Auth\",\n\t\t\t\"item\": [\n\t\t\t\t{\n\t\t\t\t\t\"name\": \"/auth\",\n\t\t\t\t\t\"event\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"listen\": \"prerequest\",\n\t\t\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\t\t\"\"\n\t\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t],\n\t\t\t\t\t\"protocolProfileBehavior\": {\n\t\t\t\t\t\t\"disableBodyPruning\": true\n\t\t\t\t\t},\n\t\t\t\t\t\"request\": {\n\t\t\t\t\t\t\"method\": \"GET\",\n\t\t\t\t\t\t\"header\": [\n\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\"key\": \"Authorization\",\n\t\t\t\t\t\t\t\t\"value\": \"Bearer {{accessToken}}\",\n\t\t\t\t\t\t\t\t\"type\": \"text\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"body\": {\n\t\t\t\t\t\t\t\"mode\": \"urlencoded\",\n\t\t\t\t\t\t\t\"urlencoded\": []\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"url\": {\n\t\t\t\t\t\t\t\"raw\": \"{{hostname}}/_authcheck\",\n\t\t\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\t\t\"_authcheck\"\n\t\t\t\t\t\t\t]\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"description\": \"The body must have `username` and `password`. It returns `id_token` and `access_token` are signed with the secret located at the `config.json` file. The `id_token` will contain the `username` and the `extra` information sent, while the `access_token` will contain the `audience`, `jti`, `issuer` and `scope`.\"\n\t\t\t\t\t},\n\t\t\t\t\t\"response\": []\n\t\t\t\t},\n\t\t\t\t{\n\t\t\t\t\t\"name\": \"/auth/login\",\n\t\t\t\t\t\"event\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\t\t\"var jsonData = pm.response.json()\",\n\t\t\t\t\t\t\t\t\t\"pm.collectionVariables.set(\\\"accessToken\\\", jsonData.data.accessToken);\",\n\t\t\t\t\t\t\t\t\t\"\"\n\t\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t},\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"listen\": \"prerequest\",\n\t\t\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\t\t\"\"\n\t\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t],\n\t\t\t\t\t\"request\": {\n\t\t\t\t\t\t\"method\": \"POST\",\n\t\t\t\t\t\t\"header\": [\n\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\"key\": \"Content-Type\",\n\t\t\t\t\t\t\t\t\"value\": \"application/x-www-form-urlencoded\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"body\": {\n\t\t\t\t\t\t\t\"mode\": \"urlencoded\",\n\t\t\t\t\t\t\t\"urlencoded\": [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"key\": \"email\",\n\t\t\t\t\t\t\t\t\t\"value\": \"user@example.com\",\n\t\t\t\t\t\t\t\t\t\"type\": \"text\"\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"key\": \"password\",\n\t\t\t\t\t\t\t\t\t\"value\": \"password\",\n\t\t\t\t\t\t\t\t\t\"type\": \"text\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t]\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"url\": {\n\t\t\t\t\t\t\t\"raw\": \"{{hostname}}/auth/login\",\n\t\t\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\t\t\"auth\",\n\t\t\t\t\t\t\t\t\"login\"\n\t\t\t\t\t\t\t]\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"description\": \"The body must have `username` and `password`. It returns `id_token` and `access_token` are signed with the secret located at the `config.json` file. The `id_token` will contain the `username` and the `extra` information sent, while the `access_token` will contain the `audience`, `jti`, `issuer` and `scope`.\"\n\t\t\t\t\t},\n\t\t\t\t\t\"response\": []\n\t\t\t\t},\n\t\t\t\t{\n\t\t\t\t\t\"name\": \"/auth/sign-up\",\n\t\t\t\t\t\"event\": [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\"listen\": \"test\",\n\t\t\t\t\t\t\t\"script\": {\n\t\t\t\t\t\t\t\t\"exec\": [\n\t\t\t\t\t\t\t\t\t\"var jsonData = pm.response.json()\",\n\t\t\t\t\t\t\t\t\t\"pm.collectionVariables.set(\\\"accessToken\\\", jsonData.data.accessToken);\",\n\t\t\t\t\t\t\t\t\t\"\"\n\t\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\t\"type\": \"text/javascript\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t}\n\t\t\t\t\t],\n\t\t\t\t\t\"request\": {\n\t\t\t\t\t\t\"method\": \"POST\",\n\t\t\t\t\t\t\"header\": [\n\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\"warning\": \"This is a duplicate header and will be overridden by the Content-Type header generated by Postman.\",\n\t\t\t\t\t\t\t\t\"key\": \"Content-Type\",\n\t\t\t\t\t\t\t\t\"value\": \"application/json\"\n\t\t\t\t\t\t\t}\n\t\t\t\t\t\t],\n\t\t\t\t\t\t\"body\": {\n\t\t\t\t\t\t\t\"mode\": \"urlencoded\",\n\t\t\t\t\t\t\t\"urlencoded\": [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"key\": \"email\",\n\t\t\t\t\t\t\t\t\t\"value\": \"anthonybudd@example.com\",\n\t\t\t\t\t\t\t\t\t\"type\": \"text\"\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"key\": \"password\",\n\t\t\t\t\t\t\t\t\t\"value\": \"password\",\n\t\t\t\t\t\t\t\t\t\"type\": \"text\"\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"key\": \"firstName\",\n\t\t\t\t\t\t\t\t\t\"value\": \"Anthony\",\n\t\t\t\t\t\t\t\t\t\"type\": \"text\"\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"key\": \"lastName\",\n\t\t\t\t\t\t\t\t\t\"value\": \"Budd\",\n\t\t\t\t\t\t\t\t\t\"type\": \"text\"\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"key\": \"groupName\",\n\t\t\t\t\t\t\t\t\t\"value\": \"GitHub\",\n\t\t\t\t\t\t\t\t\t\"type\": \"text\"\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\t\"key\": \"tos\",\n\t\t\t\t\t\t\t\t\t\"value\": \"2021-21-19\",\n\t\t\t\t\t\t\t\t\t\"type\": \"text\"\n\t\t\t\t\t\t\t\t}\n\t\t\t\t\t\t\t]\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"url\": {\n\t\t\t\t\t\t\t\"raw\": \"{{hostname}}/auth/sign-up\",\n\t\t\t\t\t\t\t\"host\": [\n\t\t\t\t\t\t\t\t\"{{hostname}}\"\n\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\t\"path\": [\n\t\t\t\t\t\t\t\t\"auth\",\n\t\t\t\t\t\t\t\t\"sign-up\"\n\t\t\t\t\t\t\t]\n\t\t\t\t\t\t},\n\t\t\t\t\t\t\"description\": \"The body must have `username` and `password`. It returns `id_token` and `access_token` are signed with the secret located at the `config.json` file. The `id_token` will contain the `username` and the `extra` information sent, while the `access_token` will contain the `audience`, `jti`, `issuer` and `scope`.\"\n\t\t\t\t\t},\n\t\t\t\t\t\"response\": []\n\t\t\t\t}\n\t\t\t]\n\t\t}\n\t],\n\t\"event\": [\n\t\t{\n\t\t\t\"listen\": \"prerequest\",\n\t\t\t\"script\": {\n\t\t\t\t\"type\": \"text/javascript\",\n\t\t\t\t\"exec\": [\n\t\t\t\t\t\"\"\n\t\t\t\t]\n\t\t\t}\n\t\t},\n\t\t{\n\t\t\t\"listen\": \"test\",\n\t\t\t\"script\": {\n\t\t\t\t\"type\": \"text/javascript\",\n\t\t\t\t\"exec\": [\n\t\t\t\t\t\"\"\n\t\t\t\t]\n\t\t\t}\n\t\t}\n\t],\n\t\"variable\": [\n\t\t{\n\t\t\t\"key\": \"hostname\",\n\t\t\t\"value\": \"http://localhost:8888/api/v1\"\n\t\t},\n\t\t{\n\t\t\t\"key\": \"accessToken\",\n\t\t\t\"value\": \"\"\n\t\t}\n\t]\n}"
},
{
"path": "api/requests.http",
"content": "# Install VS Code extension rest-client \n# URL: https://marketplace.visualstudio.com/items?itemName=humao.rest-client\n\n@host=http://localhost:8888/api/v1\n# @host=http://api.local/api/v1\n@AccessToken=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzUxMiJ9.eyJpZCI6ImM0NjQ0NzMzLWRlZWEtNDdkOC1iMzVhLTg2ZjMwZmY5NjE4ZSIsImVtYWlsIjoidXNlckBleGFtcGxlLmNvbSIsImZpcnN0TmFtZSI6IlVzZXIiLCJsYXN0TmFtZSI6Ik9uZSIsImlhdCI6MTcxNTIxMzk0MiwiZXhwIjozNDMwNTE0MjgzfQ.khGH3zHxztsWmpwL9bWpwGr_VXcPFxGTCtgoCYJq9tz0H638kWKH_k_zLgjCQ1rD6N0fWh31pTE4l53RgUGz2iL8lAoYmq0ScwSgMmiWMKm6d1vxaN3UK0CivvZPku2Pn4MQ6p12xrfRxTUVCzxI_xP9hHEhG1VUbCA07JJnl-OJFQCwYVQWCmdK5daFe8wybddYLUCG0oAGpy7Kaf0_CBbJAeIccVCKI7fILgBxowVTwl7nqruzr3-k0biXuitkegNfHPyPwbs4AvIIYxdyLXZiT-Zz0JUazphQZncw4WBqB_PX4Eyoflf8xzQNRtgvdV3ANc6ZKeMG05jAp1IV3A\n\n###########################################\n# Auth\n\nPOST {{host}}/auth/login\ncontent-type: application/json\n\n{\n \"email\": \"user@example.com\",\n \"password\": \"Password@1234\"\n}\n\n### Check auth\nGET {{host}}/_authcheck\nAuthorization: Bearer {{AccessToken}}\n\n\n###########################################\n# Buckets\n\nGET {{host}}/buckets\nAuthorization: Bearer {{AccessToken}}\n\n\n### Create Bucket\nPOST {{host}}/buckets\nAuthorization: Bearer {{AccessToken}}\ncontent-type: application/json\n\n{\n \"namespace\": \"x--xxctest\",\n \"name\": \"x-0testx\"\n}\n\n### Delete Bucket\nDELETE {{host}}/buckets/fae8a1fb-bc90-4565-b567-1fe6846544de\nAuthorization: Bearer {{AccessToken}}\n"
},
{
"path": "api/src/database/migrations/20180726090304-create-Users.js",
"content": "module.exports = {\n up: (queryInterface, Sequelize) => queryInterface.createTable('Users', {\n id: {\n type: Sequelize.UUID,\n defaultValue: Sequelize.UUIDV4,\n primaryKey: true,\n allowNull: false,\n unique: true\n },\n\n email: {\n type: Sequelize.STRING,\n allowNull: false,\n unique: true\n },\n password: Sequelize.STRING,\n\n firstName: Sequelize.STRING,\n lastName: Sequelize.STRING,\n bio: Sequelize.TEXT,\n\n tos: Sequelize.STRING,\n inviteKey: Sequelize.STRING,\n passwordResetKey: Sequelize.STRING,\n emailVerificationKey: Sequelize.STRING,\n emailVerified: {\n type: Sequelize.BOOLEAN,\n defaultValue: false,\n allowNull: false,\n },\n\n lastLoginAt: {\n type: Sequelize.DATE,\n allowNull: true,\n },\n createdAt: {\n type: Sequelize.DATE,\n allowNull: true,\n },\n updatedAt: {\n type: Sequelize.DATE,\n allowNull: true,\n },\n }),\n down: (queryInterface, Sequelize) => queryInterface.dropTable('Users'),\n};\n"
},
{
"path": "api/src/database/migrations/20180726090404-create-Groups.js",
"content": "module.exports = {\n up: (queryInterface, Sequelize) => queryInterface.createTable('Groups', {\n id: {\n type: Sequelize.UUID,\n defaultValue: Sequelize.UUIDV4,\n primaryKey: true,\n allowNull: false,\n unique: true\n },\n\n name: Sequelize.STRING,\n ownerID: Sequelize.UUID,\n\n createdAt: {\n type: Sequelize.DATE,\n allowNull: true,\n },\n updatedAt: {\n type: Sequelize.DATE,\n allowNull: true,\n },\n deletedAt: {\n type: Sequelize.DATE,\n allowNull: true,\n },\n }),\n down: (queryInterface, Sequelize) => queryInterface.dropTable('Groups')\n};\n"
},
{
"path": "api/src/database/migrations/20180726090405-create-GroupsUsers.js",
"content": "module.exports = {\n up: (queryInterface, Sequelize) => queryInterface.createTable('GroupsUsers', {\n id: { // Not used. required by msq system var sql_require_primary_key\n type: Sequelize.UUID,\n defaultValue: Sequelize.UUIDV4,\n primaryKey: true,\n allowNull: false,\n unique: true\n },\n groupID: {\n type: Sequelize.UUID,\n },\n userID: {\n type: Sequelize.UUID,\n },\n\n createdAt: {\n type: Sequelize.DATE,\n allowNull: true,\n },\n }).then(() => queryInterface.addConstraint('GroupsUsers', {\n fields: ['groupID', 'userID'],\n type: 'unique',\n name: 'groupID_userID_index'\n })),\n down: (queryInterface, Sequelize) => queryInterface.dropTable('GroupsUsers'),\n};"
},
{
"path": "api/src/database/migrations/20240411041313-create-Buckets.js",
"content": "module.exports = {\n up: (queryInterface, Sequelize) => queryInterface.createTable('Buckets', {\n id: {\n type: Sequelize.UUID,\n defaultValue: Sequelize.UUIDV4,\n primaryKey: true,\n allowNull: false,\n unique: true\n },\n\n createdAt: {\n type: Sequelize.DATE,\n allowNull: true,\n },\n updatedAt: {\n type: Sequelize.DATE,\n allowNull: true,\n },\n deletedAt: {\n type: Sequelize.DATE,\n allowNull: true,\n },\n\n userID: {\n type: Sequelize.UUID,\n allowNull: true,\n },\n\n namespace: {\n type: Sequelize.STRING,\n allowNull: false,\n },\n name: {\n type: Sequelize.STRING,\n allowNull: false,\n },\n\n status: {\n type: Sequelize.STRING,\n allowNull: false,\n },\n bucketCreated: {\n type: Sequelize.BOOLEAN,\n allowNull: false,\n defaultValue: false,\n },\n endpoint: {\n type: Sequelize.STRING,\n allowNull: false,\n },\n\n stdout: {\n type: Sequelize.TEXT,\n allowNull: true,\n },\n stderr: {\n type: Sequelize.TEXT,\n allowNull: true,\n },\n }),\n down: (queryInterface, Sequelize) => queryInterface.dropTable('Buckets'),\n};\n"
},
{
"path": "api/src/database/migrations/20240430101608-create-Blacklist.js",
"content": "module.exports = {\n up: (queryInterface, Sequelize) => queryInterface.createTable('Blacklist', {\n id: {\n type: Sequelize.UUID,\n defaultValue: Sequelize.UUIDV4,\n primaryKey: true,\n allowNull: false,\n unique: true\n },\n\n value: {\n type: Sequelize.STRING,\n allowNull: false,\n },\n\n createdAt: {\n type: Sequelize.DATE,\n allowNull: true,\n },\n updatedAt: {\n type: Sequelize.DATE,\n allowNull: true,\n },\n }),\n down: (queryInterface, Sequelize) => queryInterface.dropTable('Blacklist'),\n};\n"
},
{
"path": "api/src/database/seeders/20180726092449-Users.js",
"content": "const bcrypt = require('bcrypt-nodejs');\nconst moment = require('moment');\nconst faker = require('faker');\n\nconst insert = [{\n id: 'c4644733-deea-47d8-b35a-86f30ff9618e',\n email: 'user@example.com',\n password: bcrypt.hashSync('Password@1234', bcrypt.genSaltSync(10)),\n firstName: 'User',\n lastName: 'One',\n tos: 'tos-version-2023-07-13',\n createdAt: moment().format('YYYY-MM-DD HH:mm:ss'),\n updatedAt: moment().format('YYYY-MM-DD HH:mm:ss'),\n}, {\n id: 'd700932c-4a11-427f-9183-d6c4b69368f9',\n email: 'other.user@foobar.com',\n password: bcrypt.hashSync('Password@1234', bcrypt.genSaltSync(10)),\n firstName: faker.name.firstName(),\n lastName: faker.name.lastName(),\n tos: 'tos-version-2023-07-13',\n inviteKey: '86f30ff9618e',\n createdAt: moment().format('YYYY-MM-DD HH:mm:ss'),\n updatedAt: moment().format('YYYY-MM-DD HH:mm:ss'),\n}];\n\n\nmodule.exports = {\n up: (queryInterface, Sequelize) => queryInterface.bulkInsert('Users', insert).catch(err => console.log(err)),\n down: (queryInterface, Sequelize) => { }\n};\n"
},
{
"path": "api/src/database/seeders/20180726093449-Group.js",
"content": "const moment = require('moment');\n\nconst insert = [{\n id: 'fdab7a99-2c38-444b-bcb3-f7cef61c275b',\n ownerID: 'c4644733-deea-47d8-b35a-86f30ff9618e',\n name: 'Group A',\n createdAt: moment().format('YYYY-MM-DD HH:mm:ss'),\n updatedAt: moment().format('YYYY-MM-DD HH:mm:ss'),\n}, {\n id: 'be1fcb4e-caf9-41c2-ac27-c06fa24da36a',\n ownerID: 'd700932c-4a11-427f-9183-d6c4b69368f9',\n name: 'Group B',\n createdAt: moment().format('YYYY-MM-DD HH:mm:ss'),\n updatedAt: moment().format('YYYY-MM-DD HH:mm:ss'),\n}];\n\n\nmodule.exports = {\n up: (queryInterface, Sequelize) => queryInterface.bulkInsert('Groups', insert).catch(err => console.log(err)),\n down: (queryInterface, Sequelize) => { }\n};\n"
},
{
"path": "api/src/database/seeders/20180726093449-GroupsUsers.js",
"content": "const moment = require('moment');\n\nconst insert = [\n {\n id: '1872dcde-b79d-4f28-a36b-a22af519ac23',\n userID: 'c4644733-deea-47d8-b35a-86f30ff9618e',\n groupID: 'fdab7a99-2c38-444b-bcb3-f7cef61c275b',\n createdAt: moment().format('YYYY-MM-DD HH:mm:ss'),\n },\n {\n id: 'f4444505-cec7-4f91-948f-cdf3d4471c9e',\n userID: 'c4644733-deea-47d8-b35a-86f30ff9618e',\n groupID: 'be1fcb4e-caf9-41c2-ac27-c06fa24da36a',\n createdAt: moment().add(1, 'min').format('YYYY-MM-DD HH:mm:ss'),\n },\n {\n id: 'ed748a2d-453b-4bc8-b80d-bf1056e2b920',\n userID: 'd700932c-4a11-427f-9183-d6c4b69368f9',\n groupID: 'be1fcb4e-caf9-41c2-ac27-c06fa24da36a',\n createdAt: moment().format('YYYY-MM-DD HH:mm:ss'),\n }\n];\n\n\nmodule.exports = {\n up: (queryInterface, Sequelize) => queryInterface.bulkInsert('GroupsUsers', insert).catch(err => console.log(err)),\n down: (queryInterface, Sequelize) => { }\n};\n"
},
{
"path": "api/src/database/seeders/20240411041313-Buckets.js",
"content": "const moment = require('moment');\n\nconst insert = [{\n id: 'fae8a1fb-bc90-4565-b567-1fe6846544de',\n createdAt: moment().format('YYYY-MM-DD HH:mm:ss'),\n updatedAt: moment().format('YYYY-MM-DD HH:mm:ss'),\n userID: 'c4644733-deea-47d8-b35a-86f30ff9618e',\n namespace: 'test-bucket',\n name: 'test-bucket',\n status: 'Provisioned',\n bucketCreated: 1,\n endpoint: `test-bucket.${process.env.S3_ROOT}`,\n}];\n\nmodule.exports = {\n up: (queryInterface, Sequelize) => queryInterface.bulkInsert('Buckets', insert).catch(err => console.log(err)),\n down: (queryInterface, Sequelize) => { }\n};\n"
},
{
"path": "api/src/database/seeders/20240430101608-Blacklist.js",
"content": "const { v4: uuidv4 } = require('uuid');\n\nconst blacklist = [\n 'about',\n 'aboutu',\n 'abuse',\n 'acme',\n 'ad',\n 'admanager',\n 'admin',\n 'admindashboard',\n 'administrator',\n 'ads',\n 'adsense',\n 'adult',\n 'adword',\n 'affiliate',\n 'affiliatepage',\n 'afp',\n 'alpha',\n 'anal',\n 'analytic',\n 'android',\n 'answer',\n 'anu',\n 'anus',\n 'ap',\n 'api',\n 'app',\n 'appengine',\n 'application',\n 'appnew',\n 'arse',\n 'asdf',\n 'a',\n 'as',\n 'ass',\n 'asset',\n 'asshole',\n 'atf',\n 'backup',\n 'ball',\n 'balls',\n 'ballsack',\n 'bank',\n 'base',\n 'bastard',\n 'beginner',\n 'beta',\n 'biatch',\n 'billing',\n 'binarie',\n 'binary',\n 'bitch',\n 'biz',\n 'blackberry',\n 'blog',\n 'blogsearch',\n 'bloody',\n 'blowjob',\n 'blowjobs',\n 'bollock',\n 'boner',\n 'boob',\n 'boobs',\n 'book',\n 'bugger',\n 'bum',\n 'butt',\n 'buttplug',\n 'buy',\n 'buzz',\n 'c',\n 'cache',\n 'calendar',\n 'cart',\n 'catalog',\n 'ceo',\n 'chart',\n 'chat',\n 'checkout',\n 'ci',\n 'cia',\n 'client',\n 'clitori',\n 'clitoris',\n 'cname',\n 'cnarne',\n 'cock',\n 'code',\n 'community',\n 'confirm',\n 'confirmation',\n 'contact',\n 'contact-u',\n 'contactu',\n 'content',\n 'controlpanel',\n 'coon',\n 'core',\n 'corp',\n 'countrie',\n 'country',\n 'cp',\n 'cpanel',\n 'crap',\n 'cs',\n 'cunt',\n 'cv',\n 'damn',\n 'dashboard',\n 'data',\n 'demo',\n 'deploy',\n 'deployment',\n 'desktop',\n 'dev',\n 'devel',\n 'developement',\n 'developer',\n 'development',\n 'dick',\n 'dike',\n 'dildo',\n 'dir',\n 'directory',\n 'discussion',\n 'dl',\n 'doc',\n 'document',\n 'donate',\n 'download',\n 'dyke',\n 'e',\n 'earth',\n 'email',\n 'enable',\n 'encrypted',\n 'engine',\n 'error',\n 'errorlog',\n 'fag',\n 'faggot',\n 'fbi',\n 'feature',\n 'feck',\n 'feed',\n 'feedburner',\n 'feedproxy',\n 'felching',\n 'fellate',\n 'fellatio',\n 'file',\n 'finance',\n 'flange',\n 'folder',\n 'forgotpassword',\n 'forum',\n 'friend',\n 'ftp',\n 'fuck',\n 'fudgepacker',\n 'fun',\n 'fusion',\n 'gadget',\n 'gear',\n 'geographic',\n 'gettingstarted',\n 'git',\n 'gitlab',\n 'gmail',\n 'go',\n 'goddamn',\n 'goto',\n 'gov',\n 'graph',\n 'group',\n 'hell',\n 'help',\n 'home',\n 'homo',\n 'html',\n 'htrnl',\n 'http',\n 'i',\n 'image',\n 'img',\n 'investor',\n 'invoice',\n 'io',\n 'ios',\n 'ipad',\n 'iphone',\n 'irnage',\n 'irng',\n 'item',\n 'j',\n 'jenkin',\n 'jerk',\n 'jira',\n 'jizz',\n 'job',\n 'join',\n 'js',\n 'knobend',\n 'lab',\n 'labia',\n 'legal',\n 'lesbo',\n 'list',\n 'lmao',\n 'lmfao',\n 'local',\n 'locale',\n 'location',\n 'log',\n 'login',\n 'logout',\n 'm',\n 'mail',\n 'manage',\n 'manager',\n 'map',\n 'marketing',\n 'me',\n 'media',\n 'message',\n 'misc',\n 'mm',\n 'mms',\n 'mobile',\n 'model',\n 'money',\n 'movie',\n 'muff',\n 'my',\n 'mystore',\n 'n',\n 'net',\n 'network',\n 'new',\n 'newsite',\n 'nigga',\n 'nigger',\n 'npm',\n 'ns',\n 'omg',\n 'online',\n 'order',\n 'org',\n 'other',\n 'p0rn',\n 'pack',\n 'packagist',\n 'page',\n 'partner',\n 'partnerpage',\n 'password',\n 'payment',\n 'peni',\n 'penis',\n 'people',\n 'person',\n 'pi',\n 'pis',\n 'piss',\n 'place',\n 'podcast',\n 'policy',\n 'poop',\n 'pop',\n 'pop3',\n 'popular',\n 'porn',\n 'pr0n',\n 'pricing',\n 'prick',\n 'print',\n 'privacy',\n 'private',\n 'prod',\n 'product',\n 'production',\n 'profile',\n 'promo',\n 'promotion',\n 'proxie',\n 'proxies',\n 'proxy',\n 'pube',\n 'public',\n 'purchase',\n 'pussy',\n 'queer',\n 'querie',\n 'queries',\n 'query',\n 'r',\n 'radio',\n 'random',\n 'reader',\n 'recover',\n 'redirect',\n 'register',\n 'registration',\n 'release',\n 'report',\n 'research',\n 'resolve',\n 'resolver',\n 'rnail',\n 'rnicrosoft',\n 'root',\n 'rs',\n 'rss',\n 'sale',\n 'sandbox',\n 'scholar',\n 'scrotum',\n 'search',\n 'secure',\n 'seminar',\n 'server',\n 'service',\n 'sex',\n 'sftp',\n 'sh1t',\n 'shit',\n 'shop',\n 'shopping',\n 'shortcut',\n 'signin',\n 'signup',\n 'site',\n 'sitemap',\n 'sitenew',\n 'sketchup',\n 'sky',\n 'slash',\n 'slashinvoice',\n 'slut',\n 'sm',\n 'smegma',\n 'sms',\n 'smtp',\n 'soap',\n 'software',\n 'sorry',\n 'spreadsheet',\n 'spunk',\n 'srntp',\n 'ssh',\n 'ssl',\n 'stage',\n 'staging',\n 'stat',\n 'static',\n 'statistic',\n 'statu',\n 'store',\n 'suggest',\n 'suggestquerie',\n 'suggestquery',\n 'support',\n 'survey',\n 'surveytool',\n 'svn',\n 'sync',\n 'sysadmin',\n 'talk',\n 'talkgadget',\n 'test',\n 'tester',\n 'testing',\n 'text',\n 'tit',\n 'tits',\n 'tool',\n 'toolbar',\n 'tosser',\n 'trac',\n 'translate',\n 'translation',\n 'translator',\n 'trend',\n 'turd',\n 'twat',\n 'txt',\n 'ul',\n 'upload',\n 'vagina',\n 'validation',\n 'vid',\n 'video',\n 'video-stat',\n 'voice',\n 'w',\n 'wank',\n 'wave',\n 'webdisk',\n 'webmail',\n 'webmaster',\n 'webrnail',\n 'whm',\n 'whoi',\n 'whore',\n 'wifi',\n 'wiki',\n 'wtf',\n 'ww',\n 'www',\n 'wwww',\n 'xhtml',\n 'xhtrnl',\n 'xml',\n 'xxx',\n];\n\n\nmodule.exports = {\n up: (queryInterface, Sequelize) => queryInterface.bulkInsert('Blacklist', blacklist.map((value) => ({\n id: uuidv4(),\n value,\n }))).catch(err => console.log(err)),\n down: (queryInterface, Sequelize) => { }\n};\n"
},
{
"path": "api/src/index.js",
"content": "require('dotenv').config();\nrequire('./providers/passport');\nconst fileUpload = require('express-fileupload');\nconst express = require('express');\nconst morgan = require('morgan');\nconst cors = require('cors');\n\n\nconsole.log('*************************************');\nconsole.log('* Express API Boilerplate');\nconsole.log('*');\nconsole.log('* ENV');\nconsole.log(`* NODE_ENV: ${process.env.NODE_ENV}`);\nconsole.log(`* TEMP_FILE_DIR: ${process.env.TEMP_FILE_DIR}`);\nif (!process.env.H_CAPTCHA_SECRET) console.log(`* H_CAPTCHA_SECRET: null ⚠️ Login/Sign-up requests will not require captcha validadation!`);\nconsole.log('*');\nconsole.log('*');\n\n\n////////////////////////////////////////////////\n// Express\nconst app = express();\napp.disable('x-powered-by');\napp.use(cors({\n origin: '*',\n credentials: true,\n allowedHeaders: ['Content-Type', 'Authorization']\n}));\napp.use(express.json());\napp.use(express.urlencoded({ extended: true }));\napp.use(fileUpload({\n limits: { fileSize: 50 * 1024 * 1024 },\n tempFileDir: process.env.TEMP_FILE_DIR,\n useTempFiles: true,\n parseNested: true,\n}));\napp.get('/_readiness', (req, res) => res.send('healthy'));\napp.get('/api/v1/_healthcheck', (req, res) => res.json({ status: 'ok' }));\nif (typeof global.it !== 'function') app.use(morgan('[:date[iso]] HTTP/:http-version :status :method :url :response-time ms'));\n\n\n////////////////////////////////////////////////\n// HTTP\napp.use('/api/v1/', require('./routes/auth'));\napp.use('/api/v1/', require('./routes/user'));\napp.use('/api/v1/', require('./routes/groups'));\napp.use('/api/v1/', require('./routes/Buckets')); // AB: gen\n\n\n////////////////////////////////////////////////\n// Listen\nlet port = process.env.PORT || 80;\nif (typeof global.it === 'function') port = 7777;\napp.listen(port, () => console.log(`* Listening: http://127.0.0.1:${port}`));\nmodule.exports = app;\n"
},
{
"path": "api/src/models/Blacklist.js",
"content": "const Sequelize = require('sequelize');\nconst db = require('./../providers/db');\n\nconst Blacklist = db.define('Blacklist', {\n id: {\n type: Sequelize.UUID,\n defaultValue: Sequelize.UUIDV4,\n primaryKey: true,\n allowNull: false,\n unique: true\n },\n\n value: {\n type: Sequelize.STRING,\n allowNull: false,\n },\n\n createdAt: {\n type: Sequelize.DATE,\n allowNull: true,\n },\n updatedAt: {\n type: Sequelize.DATE,\n allowNull: true,\n },\n\n}, {\n tableName: 'Blacklist',\n defaultScope: {\n attributes: {\n exclude: [\n\n ]\n }\n },\n});\n\nmodule.exports = Blacklist;"
},
{
"path": "api/src/models/Bucket.js",
"content": "const { exec } = require('child_process');\nconst db = require('./../providers/db');\nconst Sequelize = require('sequelize');\nconst tmp = require('tmp');\nconst fs = require('fs');\n\nconst Bucket = db.define('Bucket', {\n id: {\n type: Sequelize.UUID,\n defaultValue: Sequelize.UUIDV4,\n primaryKey: true,\n allowNull: false,\n unique: true\n },\n\n createdAt: {\n type: Sequelize.DATE,\n allowNull: true,\n },\n updatedAt: {\n type: Sequelize.DATE,\n allowNull: true,\n },\n deletedAt: {\n type: Sequelize.DATE,\n allowNull: true,\n },\n\n userID: {\n type: Sequelize.UUID,\n allowNull: true,\n },\n\n namespace: {\n type: Sequelize.STRING,\n allowNull: false,\n },\n name: {\n type: Sequelize.STRING,\n allowNull: false,\n },\n\n status: {\n type: Sequelize.STRING,\n allowNull: false,\n },\n bucketCreated: {\n type: Sequelize.BOOLEAN,\n allowNull: false,\n defaultValue: false,\n },\n endpoint: {\n type: Sequelize.STRING,\n allowNull: false,\n },\n\n stdout: {\n type: Sequelize.TEXT,\n allowNull: true,\n },\n stderr: {\n type: Sequelize.TEXT,\n allowNull: true,\n },\n}, {\n tableName: 'Buckets',\n paranoid: true,\n defaultScope: {\n attributes: {\n exclude: []\n }\n },\n});\n\nBucket.prototype.createK3sAssets = async function () {\n const generateAccessKeyID = () => {\n const charSet = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz23456789';\n const length = 20;\n let randomString = '';\n for (let i = 0; i < length; i++) {\n const randomIndex = Math.floor(Math.random() * charSet.length);\n randomString += charSet.charAt(randomIndex);\n }\n return randomString;\n };\n\n const generateSecretAccessKey = () => {\n const length = 40;\n const charset = 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789+/';\n let randomString = '';\n for (let i = 0; i < length; i++) {\n const randomIndex = Math.floor(Math.random() * charset.length);\n randomString += charset.charAt(randomIndex);\n }\n return randomString;\n };\n\n const accessKeyID = generateAccessKeyID();\n const secretAccessKey = generateSecretAccessKey();\n\n tmp.file((err, path) => {\n if (err) throw err;\n fs.readFile('/app/src/providers/bucket.yml', 'utf8', (err, data) => {\n if (err) throw err;\n\n const result = data.replace(/NAMESPACE_HERE/g, this.namespace)\n .replace(/BUCKETNAME_HERE/g, this.name)\n .replace(/ROOTUSER/g, accessKeyID)\n .replace(/ROOTPASSWORD/g, secretAccessKey);\n\n fs.writeFile(path, result, 'utf8', (err) => {\n if (err) throw err;\n\n exec(`kubectl --kubeconfig=${process.env.K8S_CONFIG_PATH} apply -f ${path}`, (err, stdout, stderr) => {\n if (err) console.error(err);\n\n console.log(`stdout: ${stdout}`);\n console.log(`stderr: ${stderr}`);\n\n let status = 'Provisioning';\n if (stderr) status = 'Error';\n\n this.update({\n status,\n stdout,\n stderr,\n });\n });\n });\n });\n });\n\n return {\n accessKeyID,\n secretAccessKey\n };\n};\n\nBucket.prototype.createBucket = async function () {\n const command = `kubectl --kubeconfig=${process.env.K8S_CONFIG_PATH} -n ${this.namespace} exec minio-pod -- ./s3-create-bucket-script/create-bucket.sh`;\n console.log(command);\n exec(command, (err, stdout, stderr) => {\n if (err) console.error(err);\n\n console.log(`stdout: ${stdout}`);\n console.log(`stderr: ${stderr}`);\n\n if (stderr) {\n this.update({\n status: 'Error',\n createStderr: `2: ${stderr}`,\n });\n } else {\n this.update({ bucketCreated: true });\n }\n });\n};\n\nBucket.prototype.sync = async function () {\n if (this.status !== 'Error') {\n exec(`kubectl --kubeconfig=${process.env.K8S_CONFIG_PATH} -n ${this.namespace} get pod minio-pod --no-headers -o custom-columns=\":status.phase\"`, (err, stdout, stderr) => {\n if (err) console.error(err);\n\n console.log(`stdout: ${stdout}`);\n console.log(`stderr: ${stderr}`);\n\n switch (stdout.trim()) {\n case 'Running':\n if (this.status !== 'Provisioned') this.update({ status: 'Provisioned' });\n if (!this.bucketCreated) this.createBucket();\n break;\n }\n });\n }\n};\n\nBucket.prototype.deleteK3sAssets = async function () {\n exec(`kubectl --kubeconfig=${process.env.K8S_CONFIG_PATH} -n ${this.namespace} delete pod/minio-pod service/minio-svc ingress/minio-ing persistentvolumeclaim/minio-pvc namespace/${this.namespace}`, (err, stdout, stderr) => {\n if (err) console.error(err);\n if (stderr) console.log(`stderr: ${stderr}`);\n console.log(`stdout: ${stdout}`);\n\n this.update({\n stdout,\n stderr,\n });\n });\n};\n\nmodule.exports = Bucket;"
},
{
"path": "api/src/models/Group.js",
"content": "const Sequelize = require('sequelize');\nconst db = require('./../providers/db');\n\nmodule.exports = db.define('Group', {\n id: {\n type: Sequelize.UUID,\n defaultValue: Sequelize.UUIDV4,\n primaryKey: true,\n allowNull: false,\n unique: true\n },\n\n name: Sequelize.STRING,\n ownerID: Sequelize.UUID,\n\n deletedAt: {\n type: Sequelize.DATE,\n allowNull: true,\n },\n}, {\n tableName: 'Groups',\n paranoid: true,\n});\n"
},
{
"path": "api/src/models/GroupsUsers.js",
"content": "const Sequelize = require('sequelize');\nconst db = require('./../providers/db');\n\nmodule.exports = db.define('GroupsUsers', {\n id: { // Not used. required by msq system var sql_require_primary_key\n type: Sequelize.UUID,\n defaultValue: Sequelize.UUIDV4,\n primaryKey: true,\n allowNull: false,\n unique: true\n },\n userID: Sequelize.UUID,\n groupID: Sequelize.UUID,\n}, {\n tableName: 'GroupsUsers',\n updatedAt: false,\n});\n"
},
{
"path": "api/src/models/User.js",
"content": "const Sequelize = require('sequelize');\nconst db = require('./../providers/db');\n\nmodule.exports = db.define('User', {\n id: {\n type: Sequelize.UUID,\n defaultValue: Sequelize.UUIDV4,\n primaryKey: true,\n allowNull: false,\n unique: true\n },\n\n email: {\n type: Sequelize.STRING,\n allowNull: false,\n unique: true\n },\n password: Sequelize.STRING,\n\n firstName: Sequelize.STRING,\n lastName: Sequelize.STRING,\n bio: Sequelize.TEXT,\n\n tos: Sequelize.STRING,\n inviteKey: Sequelize.STRING,\n passwordResetKey: Sequelize.STRING,\n emailVerificationKey: Sequelize.STRING,\n emailVerified: {\n type: Sequelize.BOOLEAN,\n defaultValue: false,\n allowNull: false,\n },\n\n lastLoginAt: {\n type: Sequelize.DATE,\n allowNull: true,\n },\n}, {\n tableName: 'Users',\n defaultScope: {\n attributes: {\n exclude: [\n 'password',\n 'passwordResetKey',\n ]\n }\n },\n});\n"
},
{
"path": "api/src/models/index.js",
"content": "const User = require('./User');\nconst Group = require('./Group');\nconst GroupsUsers = require('./GroupsUsers');\nconst Bucket = require('./Bucket');\nconst Blacklist = require('./Blacklist');\n\n\nUser.belongsToMany(Group, {\n through: GroupsUsers,\n foreignKey: 'userID',\n otherKey: 'groupID',\n});\nGroup.belongsToMany(User, {\n through: GroupsUsers,\n foreignKey: 'groupID',\n otherKey: 'userID',\n});\n\n\nmodule.exports = {\n User,\n Group,\n GroupsUsers,\n Bucket,\n Blacklist,\n};\n"
},
{
"path": "api/src/providers/bucket.yml",
"content": "apiVersion: v1\nkind: Namespace\nmetadata:\n name: NAMESPACE_HERE\n labels:\n name: NAMESPACE_HERE\n---\napiVersion: v1\nkind: Pod\nmetadata:\n labels:\n app: minio-pod\n name: minio-pod\n namespace: NAMESPACE_HERE\nspec:\n containers:\n - name: minio-pod\n image: quay.io/minio/minio:latest\n env:\n - name: MINIO_ROOT_USER\n value: ROOTUSER\n - name: MINIO_ROOT_PASSWORD\n value: ROOTPASSWORD\n - name: S3_NAMESPACE\n value: NAMESPACE_HERE\n - name: S3_BUCKET_NAME\n value: BUCKETNAME_HERE\n command:\n - /bin/bash\n - -c\n args: \n - minio server /data --console-address :9001\n ports:\n - name: http\n containerPort: 80\n - name: https\n containerPort: 443\n - name: console\n containerPort: 9001\n - name: api\n containerPort: 9000\n volumeMounts:\n - name: longhornvolume\n mountPath: /data\n - name: s3-create-bucket-script\n mountPath: /s3-create-bucket-script\n volumes:\n - name: s3-create-bucket-script\n configMap:\n name: s3-create-bucket-script\n defaultMode: 0777\n items:\n - key: create-bucket.sh\n path: create-bucket.sh\n - name: longhornvolume\n persistentVolumeClaim:\n claimName: minio-pvc\n---\napiVersion: v1\nkind: Service \nmetadata:\n name: minio-svc\n namespace: NAMESPACE_HERE \nspec:\n selector:\n app: minio-pod \n ports:\n - name: http\n protocol: TCP \n port: 80 \n targetPort: 9001\n - name: api\n port: 9000\n protocol: TCP\n---\napiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n namespace: NAMESPACE_HERE\n name: minio-ing\n annotations:\n cert-manager.io/cluster-issuer: \"letsencrypt-prod\"\n kubernetes.io/ingress.class: \"traefik\"\nspec:\n tls:\n - hosts:\n - NAMESPACE_HERE.s3.anthonybudd.io\n secretName: NAMESPACE_HERE-s3-anthonybudd-io-cert\n - hosts:\n - BUCKETNAME_HERE.NAMESPACE_HERE.s3.anthonybudd.io\n secretName: BUCKETNAME_HERE-NAMESPACE_HERE-s3-anthonybudd-io-cert\n rules:\n - host: NAMESPACE_HERE.s3.anthonybudd.io\n http:\n paths:\n - path: /\n pathType: Prefix\n backend:\n service:\n name: minio-svc\n port:\n number: 80\n - host: BUCKETNAME_HERE.NAMESPACE_HERE.s3.anthonybudd.io\n http:\n paths:\n - path: /\n pathType: Prefix\n backend:\n service:\n name: minio-svc\n port:\n number: 9000\n---\napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n name: minio-pvc\n namespace: NAMESPACE_HERE\nspec:\n accessModes:\n - ReadWriteOnce\n storageClassName: longhorn\n resources:\n requests:\n storage: 5Gi\n---\napiVersion: v1\nkind: ConfigMap\nmetadata:\n name: s3-create-bucket-script\n namespace: NAMESPACE_HERE\ndata:\n create-bucket.sh: |\n #!/bin/bash\n\n mc alias set local http://localhost:9000 \"$MINIO_ROOT_USER\" \"$MINIO_ROOT_PASSWORD\"\n mc mb local/\"$S3_BUCKET_NAME\""
},
{
"path": "api/src/providers/connections.js",
"content": "require('dotenv').config();\n\nmodule.exports = {\n development: {\n username: process.env.DB_USERNAME,\n password: process.env.DB_PASSWORD,\n database: process.env.DB_DATABASE,\n host: process.env.DB_HOST,\n port: process.env.DB_PORT || '3306',\n dialect: 'mysql',\n },\n production: {\n username: process.env.DB_USERNAME,\n password: process.env.DB_PASSWORD,\n database: process.env.DB_DATABASE,\n host: process.env.DB_HOST,\n port: process.env.DB_PORT || '3306',\n dialect: 'mysql',\n },\n test: {\n username: process.env.DB_USERNAME,\n password: process.env.DB_PASSWORD,\n database: process.env.DB_DATABASE,\n host: 's3-api-db-test',\n port: process.env.DB_PORT || '3306',\n dialect: 'mysql',\n }\n};\n"
},
{
"path": "api/src/providers/db.js",
"content": "const Sequelize = require('sequelize');\nconst connections = require('./connections');\nconst errorHandler = require('./errorHandler');\n\n\nconst connection = (typeof global.it === 'function') ? 'test' : (process.env.NODE_ENV || 'development');\nconst dbHost = connections[connection].host;\nconst dbPort = connections[connection].port;\nconst dbName = connections[connection].database;\nconst dbUser = connections[connection].username;\nconst dbPassword = connections[connection].password;\nconst dbDialect = connections[connection].dialect;\n\n\nconst sequelize = new Sequelize(dbName, dbUser, dbPassword, {\n port: dbPort,\n host: dbHost,\n dialect: dbDialect,\n logging: false,\n pool: {\n max: 5,\n min: 0,\n acquire: 30000,\n idle: 10000,\n },\n});\n\nsequelize.authenticate()\n .then(() => ((typeof global.it !== 'function') ? console.log('* Sequelize: Connected') : ''))\n .catch(err => errorHandler(err));\n\nmodule.exports = sequelize;\n"
},
{
"path": "api/src/providers/errorHandler.js",
"content": "const crypto = require('crypto');\n\nmodule.exports = (err, res) => {\n if (err.isAxiosError) {\n console.log(`Axios Error: ${err.request.path}`);\n if (err.response && err.response.data) {\n console.log(err.response.data);\n } else {\n console.error(err);\n }\n } else if (err.response && err.response.body) {\n console.error(err);\n console.error(err.response.body);\n } else {\n console.error(err);\n }\n\n if (res && !res.headersSent) res.status(500).json({\n msg: `Error`,\n code: crypto.randomBytes(32).toString('base64'),\n });\n};\n"
},
{
"path": "api/src/providers/generateJWT.js",
"content": "const jwt = require('jsonwebtoken');\nconst moment = require('moment');\nconst fs = require('fs');\n\nmodule.exports = (user, expires) => {\n const payload = {\n id: user.id,\n email: user.email,\n firstName: user.firstName,\n lastName: user.lastName,\n displayName: user.displayName,\n };\n\n let expiresIn = moment(new Date()).add(1, 'day').unix();\n if (Array.isArray(expires) && expires.length === 2) {\n expiresIn = moment(new Date()).add(expires[0], expires[1]).unix();\n } else if (typeof expires === 'string') {\n expiresIn = moment(expires, 'YYYY-MM-DD HH:mmZ').unix();\n }\n\n return jwt.sign(payload, fs.readFileSync(process.env.PRIVATE_KEY_PATH, 'utf8'), {\n expiresIn,\n algorithm: 'RS512',\n });\n};\n"
},
{
"path": "api/src/providers/hCaptcha.js",
"content": "const axios = require('axios');\nconst qs = require('qs');\n\nconst hCaptcha = axios.create({\n baseURL: 'https://hcaptcha.com',\n});\n\nmodule.exports = {\n verify: async (response) => await hCaptcha.post('/siteverify',\n qs.stringify({\n response,\n secret: process.env.H_CAPTCHA_SECRET,\n }),\n {\n headers: {\n 'Content-Type': 'application/x-www-form-urlencoded'\n }\n }\n ),\n\n axios: hCaptcha,\n};\n"
},
{
"path": "api/src/providers/passport.js",
"content": "const LocalStrategy = require('passport-local').Strategy;\nconst { User, Group } = require('./../models');\nconst passportJWT = require('passport-jwt');\nconst ExtractJWT = passportJWT.ExtractJwt;\nconst JWTStrategy = passportJWT.Strategy;\nconst bcrypt = require('bcrypt-nodejs');\nconst passport = require('passport');\nconst fs = require('fs');\n\n\npassport.use(new LocalStrategy({\n usernameField: 'email',\n passwordField: 'password'\n}, async (email, password, cb) => {\n\n const user = await User.unscoped().findOne({\n where: { email },\n include: [Group]\n });\n\n if (!user) return cb(null, false, { message: 'Incorrect email or password.' });\n\n return bcrypt.compare(password, user.password, (err, compare) => {\n if (compare) {\n return cb(null, user, { message: 'Logged in successfully' });\n } else {\n return cb(null, false, { message: 'Incorrect email or password.' });\n }\n });\n}));\n\n\npassport.use(new JWTStrategy({\n jwtFromRequest: ExtractJWT.fromExtractors([\n ExtractJWT.fromAuthHeaderAsBearerToken(),\n ExtractJWT.fromUrlQueryParameter('token'),\n ]),\n secretOrKey: fs.readFileSync(process.env.PUBLIC_KEY_PATH, 'utf8'),\n}, (jwtPayload, cb) => cb(null, jwtPayload)));\n\n\nmodule.exports = passport;\n"
},
{
"path": "api/src/routes/Buckets.js",
"content": "const { body, validationResult, matchedData } = require('express-validator');\nconst errorHandler = require('./../providers/errorHandler');\nconst { Bucket, Blacklist } = require('./../models');\nconst middleware = require('./middleware');\nconst passport = require('passport');\nconst express = require('express');\n\n\nconst app = (module.exports = express.Router());\n\n\n/**\n * GET /api/v1/buckets\n * \n */\napp.get('/buckets', [\n passport.authenticate('jwt', { session: false })\n], async (req, res) => {\n try {\n return res.json(await Bucket.findAll({\n where: {\n userID: req.user.id,\n }\n }));\n } catch (error) {\n errorHandler(error, res);\n }\n});\n\n\n/**\n * POST /api/v1/buckets\n * \n * Create Bucket\n */\napp.post('/buckets', [\n passport.authenticate('jwt', { session: false }),\n body('namespace')\n .exists()\n .notEmpty()\n .matches(/^[a-z0-9-_]+$/),\n\n body('namespace')\n .custom(async (value) => {\n const blacklist = await Blacklist.findOne({ where: { value } });\n if (blacklist) throw new Error('This namespace is not allowed');\n })\n .custom(async (namespace, { req }) => {\n const exists = await Bucket.findOne({\n where: {\n namespace: req.body.namespace\n }\n });\n\n if (exists) throw new Error('Namespace already exists.');\n }),\n\n body('name')\n .exists()\n .notEmpty()\n .matches(/^[a-z0-9-_]+$/),\n body('name')\n .custom(async (value) => {\n const blacklist = await Blacklist.findOne({ where: { value } });\n if (blacklist) throw new Error('This bucket name is not allowed');\n })\n .custom(async (name, { req }) => {\n const exists = await Bucket.findOne({\n where: {\n name: req.body.name\n }\n });\n\n if (exists) throw new Error('Bucket already exists.');\n }),\n], async (req, res) => {\n try {\n const errors = validationResult(req);\n if (!errors.isEmpty()) return res.status(422).json({ errors: errors.mapped() });\n const data = matchedData(req);\n\n const bucket = await Bucket.create({\n userID: req.user.id,\n namespace: data.namespace,\n name: data.name,\n status: 'Provisioning',\n endpoint: `${data.name}.${data.namespace}.${process.env.S3_ROOT}`,\n });\n\n const { accessKeyID, secretAccessKey } = await bucket.createK3sAssets();\n\n return res.json({\n ...bucket.get({ plain: true }),\n accessKeyID,\n secretAccessKey,\n });\n } catch (error) {\n return errorHandler(error, res);\n }\n});\n\n\n/**\n * DELETE /api/v1/buckets/:bucketID\n * \n * Delete Bucket\n */\napp.delete('/buckets/:bucketID', [\n passport.authenticate('jwt', { session: false }),\n middleware.canAccessBucket,\n], async (req, res) => {\n try {\n const bucket = await Bucket.findByPk(req.params.bucketID);\n bucket.deleteK3sAssets();\n await bucket.destroy();\n return res.json({ id: req.params.bucketID });\n } catch (error) {\n return errorHandler(error, res);\n }\n});\n"
},
{
"path": "api/src/routes/auth.js",
"content": "const { body, validationResult, matchedData } = require('express-validator');\nconst { User, Group, GroupsUsers } = require('./../models');\nconst errorHandler = require('./../providers/errorHandler');\nconst generateJWT = require('./../providers/generateJWT');\nconst middleware = require('./middleware');\nconst bcrypt = require('bcrypt-nodejs');\nconst passport = require('passport');\nconst express = require('express');\nconst uuidv4 = require('uuid/v4');\nconst moment = require('moment');\nconst crypto = require('crypto');\n\nconst app = (module.exports = express.Router());\n\n\n/**\n * GET /api/v1/_authcheck\n * \n * Helper route for testing auth status\n */\napp.get('/_authcheck', [\n passport.authenticate('jwt', { session: false })\n], (req, res) => res.json({\n auth: true,\n id: req.user.id,\n}));\n\n\n/**\n * POST api/v1/auth/login\n * \n */\napp.post('/auth/login', [\n body('email').notEmpty().toLowerCase(),\n body('password').notEmpty(),\n middleware.hCaptcha,\n], async (req, res) => {\n const errors = validationResult(req);\n if (!errors.isEmpty()) return res.status(422).json({ errors: errors.mapped() });\n\n passport.authenticate('local', { session: false }, (err, user) => {\n if (err) return errorHandler(err, res);\n if (!user) return res.status(401).json('Incorrect email or password');\n\n req.login(user, { session: false }, (err) => {\n if (err) return errorHandler(err, res);\n\n res.json({\n accessToken: generateJWT(user)\n });\n\n User.update({\n lastLoginAt: moment(new Date()).format('YYYY-MM-DD HH:mm:ss'),\n }, {\n where: {\n id: user.id\n }\n });\n });\n })(req, res);\n});\n\n\n/**\n * POST /api/v1/auth/sign-up\n * \n */\napp.post('/auth/sign-up', [\n body('email')\n .notEmpty()\n .isEmail()\n .trim()\n .toLowerCase()\n .custom(async (email) => {\n const user = await User.findOne({ where: { email } });\n if (user) throw new Error('This email address is taken');\n }),\n body('password', 'Your password must be atleast 7 characters long')\n .notEmpty()\n .isLength({ min: 7 }),\n body('firstName', 'You must provide your first name')\n .notEmpty()\n .exists(),\n body('lastName')\n .optional(),\n body('groupName')\n .optional(),\n body('tos', 'You must accept the Terms of Service to use this platform')\n .exists()\n .notEmpty(),\n middleware.hCaptcha,\n], async (req, res) => {\n try {\n const errors = validationResult(req);\n if (!errors.isEmpty()) return res.status(422).json({ errors: errors.mapped() });\n const data = matchedData(req);\n\n const userID = uuidv4();\n const groupID = uuidv4();\n const ucFirst = (string) => string.charAt(0).toUpperCase() + string.slice(1);\n const nameArr = data.firstName.split(' ');\n if (!data.lastName && nameArr.length >= 2) {\n data.firstName = nameArr[0];\n data.lastName = nameArr[1];\n }\n if (!data.lastName) data.lastName = '';\n if (!data.groupName) data.groupName = data.firstName.concat(\"'s Team\");\n\n await Group.create({\n id: groupID,\n name: data.groupName,\n ownerID: userID,\n });\n\n await GroupsUsers.create({ userID, groupID });\n\n const user = await User.create({\n id: userID,\n email: data.email,\n password: bcrypt.hashSync(data.password, bcrypt.genSaltSync(10)),\n firstName: ucFirst(data.firstName),\n lastName: ucFirst(data.lastName),\n lastLoginAt: moment().format(\"YYYY-MM-DD HH:mm:ss\"),\n tos: data.tos,\n emailVerificationKey: crypto.randomBytes(20).toString('hex'),\n });\n\n console.log(`\\n\\nEMAIL THIS TO THE USER\\nEMAIL VERIFICATION LINK: ${process.env.FRONTEND_URL}/validate-email/${user.emailVerificationKey}\\n\\n`);\n\n return passport.authenticate('local', { session: false }, (err, user) => {\n if (err) return errorHandler(err, res);\n req.login(user, { session: false }, (err) => {\n if (err) return errorHandler(err, res);\n res.json({\n accessToken: generateJWT(user)\n });\n });\n })(req, res);\n } catch (error) {\n return errorHandler(error, res);\n }\n});\n\n\n/**\n * GET /api/v1/auth/verify-email/:emailVerificationKey\n * \n * Verify Email\n */\napp.get('/auth/verify-email', async (req, res) => {\n const user = await User.findOne({\n where: {\n emailVerificationKey: req.params.emailVerificationKey\n }\n });\n\n if (!user) return res.status(404).json({\n msg: 'User not found',\n code: 40402\n });\n\n await user.update({\n emailVerified: true,\n emailVerificationKey: null,\n });\n\n return res.json({ success: true });\n});\n\n\n/**\n * POST /api/v1/auth/forgot\n * \n * Forgot Password\n */\napp.post('/auth/forgot', [\n body('email')\n .isEmail()\n .toLowerCase()\n .custom(async (email) => {\n const user = await User.findOne({ where: { email } });\n if (!user) throw new Error('This email address does not exist');\n }),\n middleware.hCaptcha,\n], async (req, res) => {\n\n const errors = validationResult(req);\n if (!errors.isEmpty()) return res.status(422).json({ errors: errors.mapped() });\n const { email } = matchedData(req);\n\n const user = await User.findOne({ where: { email } });\n if (!user) return res.status(404).json({\n msg: 'User not found',\n code: 40401\n });\n\n const passwordResetKey = crypto.randomBytes(32).toString('base64').replace(/[^a-zA-Z0-9]/g, '');\n\n await user.update({ passwordResetKey });\n\n console.log(`\\n\\nEMAIL THIS TO THE USER\\nPASSWORD RESET LINK: ${process.env.FRONTEND_URL}/reset/${passwordResetKey}\\n\\n`);\n\n return res.json({ success: true });\n});\n\n\n/**\n * GET /api/v1/auth/get-user-by-reset-key/:passwordResetKey\n * \n * Get users email\n */\napp.get('/auth/get-user-by-reset-key/:passwordResetKey', async (req, res) => {\n const user = await User.findOne({\n where: {\n passwordResetKey: req.params.passwordResetKey\n },\n });\n if (!user) return res.status(404).send('Not found');\n\n return res.json({\n id: user.id,\n email: user.email\n });\n});\n\n\n/**\n * POST /api/v1/auth/reset\n * \n * Update User's Password\n */\napp.post('/auth/reset', [\n body('email')\n .isEmail()\n .toLowerCase()\n .custom(async (email) => {\n const user = await User.findOne({ where: { email } });\n if (!user) throw new Error('This email address does not exist');\n }),\n body('password').exists().isLength({ min: 7 }),\n body('passwordResetKey', 'This link has expired')\n .custom(async (passwordResetKey) => {\n if (!passwordResetKey) throw new Error('This link has expired');\n const user = await User.findOne({ where: { passwordResetKey } });\n if (!user) throw new Error('This link has expired');\n }),\n middleware.hCaptcha,\n], async (req, res) => {\n const errors = validationResult(req);\n if (!errors.isEmpty()) return res.status(422).json({ errors: errors.mapped() });\n const { email, password, passwordResetKey } = matchedData(req);\n\n const user = await User.findOne({\n where: { email, passwordResetKey },\n include: [Group],\n });\n if (!user) return res.status(404).send('Not found');\n\n await user.update({\n password: bcrypt.hashSync(password, bcrypt.genSaltSync(10)),\n passwordResetKey: null,\n });\n\n return passport.authenticate('local', { session: false }, (err, user) => {\n if (err) return errorHandler(err, res);\n req.login(user, { session: false }, (err) => {\n if (err) return errorHandler(err, res);\n return res.json({ accessToken: generateJWT(user) });\n });\n })(req, res);\n});\n\n\n/**\n * GET /api/v1/auth/get-user-by-invite-key/:inviteKey\n * \n * Get users email\n */\napp.get('/auth/get-user-by-invite-key/:inviteKey', async (req, res) => {\n const user = await User.findOne({\n where: {\n inviteKey: req.params.inviteKey\n },\n });\n if (!user) return res.status(404).send('Not found');\n\n return res.json({\n id: user.id,\n email: user.email\n });\n});\n\n\n/**\n * POST /api/v1/auth/invite\n * \n */\napp.post('/auth/invite', [\n body('email', 'You must provide your email address')\n .exists({ checkFalsy: true })\n .isEmail()\n .toLowerCase(),\n body('password', 'Your password must be atleast 7 characters long')\n .isLength({ min: 7 }),\n body('firstName', 'You must provide your first name')\n .exists(),\n body('lastName'),\n body('tos', 'You must accept the Terms of Service to use this platform')\n .exists(),\n body('inviteKey').exists(),\n middleware.hCaptcha,\n], async (req, res) => {\n try {\n const errors = validationResult(req);\n if (!errors.isEmpty()) return res.status(422).json({ errors: errors.mapped() });\n const data = matchedData(req);\n\n const ucFirst = (string) => string.charAt(0).toUpperCase() + string.slice(1);\n const user = await User.findOne({ where: { inviteKey: data.inviteKey } });\n if (!user) return res.status(404).send('Not found');\n await user.update({\n password: bcrypt.hashSync(data.password, bcrypt.genSaltSync(10)),\n firstName: ucFirst(data.firstName),\n lastName: ucFirst(data.lastName),\n lastLoginAt: moment().format('YYYY-MM-DD HH:mm:ss'),\n tos: data.tos,\n inviteKey: null,\n emailVerified: true,\n emailVerificationKey: null,\n });\n\n return passport.authenticate('local', { session: false }, (err, user) => {\n if (err) return errorHandler(err, res);\n req.login(user, { session: false }, (err) => {\n if (err) return errorHandler(err, res);\n return res.json({ accessToken: generateJWT(user) });\n });\n })(req, res);\n } catch (error) {\n return errorHandler(error, res);\n }\n});\n"
},
{
"path": "api/src/routes/groups.js",
"content": "const { body, validationResult, matchedData } = require('express-validator');\nconst { User, Group, GroupsUsers } = require('./../models');\nconst errorHandler = require('./../providers/errorHandler');\nconst middleware = require('./middleware');\nconst passport = require('passport');\nconst express = require('express');\nconst crypto = require('crypto');\n\nconst app = (module.exports = express.Router());\n\n\n/**\n * GET /api/v1/groups/:groupID\n *\n */\napp.get('/groups/:groupID', [\n passport.authenticate('jwt', { session: false }),\n middleware.isInGroup,\n], async (req, res) => {\n try {\n const group = await Group.findByPk(req.params.groupID, {\n include: (req.query.with === 'users') ? [User] : [],\n });\n\n return res.json(group);\n } catch (error) {\n return errorHandler(error, res);\n }\n});\n\n\n/**\n * POST /api/v1/groups/:groupID\n *\n */\napp.post('/groups/:groupID', [\n passport.authenticate('jwt', { session: false }),\n middleware.isInGroup,\n body('name')\n], async (req, res) => {\n try {\n const errors = validationResult(req);\n if (!errors.isEmpty()) return res.status(422).json({ errors: errors.mapped() });\n const data = matchedData(req);\n\n await Group.update(data, {\n where: {\n id: req.params.groupID\n }\n });\n\n return res.json(\n await Group.findByPk(req.params.groupID)\n );\n } catch (error) {\n return errorHandler(error, res);\n }\n});\n\n\n/**\n * POST /api/v1/groups/:groupID/users/invite\n *\n */\napp.post('/groups/:groupID/users/invite', [\n passport.authenticate('jwt', { session: false }),\n middleware.isGroupOwner,\n body('email').isEmail().toLowerCase(),\n], async (req, res) => {\n try {\n const errors = validationResult(req);\n if (!errors.isEmpty()) return res.status(422).json({ errors: errors.mapped() });\n const { email } = matchedData(req);\n\n const groupID = req.params.groupID;\n\n let user = await User.findOne({\n where: { email }\n });\n\n if (user) {\n if (user.id === req.user.id) return res.status(401).json({\n msg: 'You cannot add yourself to a group',\n code: 98644,\n });\n\n // Check if relationship already exists\n const relationship = await GroupsUsers.findOne({\n where: {\n groupID,\n userID: user.id\n }\n });\n if (relationship) return res.json({\n groupID,\n userID: user.id\n });\n } else {\n try {\n user = await User.create({\n email,\n inviteKey: crypto.randomBytes(20).toString('hex'),\n emailVerificationKey: crypto.randomBytes(20).toString('hex'),\n });\n\n console.log(`\\n\\nEMAIL THIS TO THE USER\\nINVITE LINK: ${process.env.FRONTEND_URL}/invite/${user.inviteKey}\\n\\n`);\n } catch (error) {\n errorHandler(error);\n }\n }\n\n // Delete all first\n await GroupsUsers.destroy({\n where: {\n groupID,\n userID: user.id,\n }\n });\n\n await GroupsUsers.create({\n groupID,\n userID: user.id,\n });\n\n return res.json({\n groupID,\n userID: user.id,\n });\n } catch (error) {\n return errorHandler(error, res);\n }\n});\n\n\n/**\n * DELETE /api/v1/groups/:groupID/users/:userID\n *\n */\napp.delete('/groups/:groupID/users/:userID', [\n passport.authenticate('jwt', { session: false }),\n middleware.isGroupOwner,\n middleware.isNotSelf,\n], async (req, res) => {\n\n await GroupsUsers.destroy({\n where: {\n groupID: req.params.groupID,\n userID: req.params.userID,\n }\n });\n\n return res.json({\n userID: req.params.userID,\n groupID: req.params.groupID\n });\n});\n"
},
{
"path": "api/src/routes/middleware/canAccessBucket.js",
"content": "const { Bucket } = require('./../../models');\n\nmodule.exports = async (req, res, next) => {\n const bucketID = (req.params.bucketID || req.body.bucketID);\n const bucket = await Bucket.findByPk(bucketID);\n\n if (!bucket) return res.status(404).json({\n msg: `Bucket does not exists.`,\n code: 97924,\n });\n\n if (req.user.id === bucket.userID) {\n return next();\n } else {\n return res.status(401).json({\n msg: `You do not have access to bucket.id:${bucketID}`,\n code: 49390,\n });\n }\n};\n"
},
{
"path": "api/src/routes/middleware/checkPassword.js",
"content": "const errorHandler = require('./../../providers/errorHandler');\nconst { User } = require('./../../models');\nconst bcrypt = require('bcrypt-nodejs');\n\nmodule.exports = (req, res, next) => {\n if (!req.body.password) return res.status(422).json({\n errors: {\n components: {\n location: 'body',\n param: 'password',\n msg: 'Password must be provided'\n }\n }\n });\n\n User.unscoped().findOne({ where: { id: req.user.id } }).then(user => {\n if (!user) return res.status(401).json({\n msg: 'Incorrect password',\n code: 92294,\n });\n\n bcrypt.compare(req.body.password, user.password, (err, compare) => {\n if (err) return res.status(401).json({\n msg: 'Incorrect password',\n code: 96294,\n });\n\n if (compare) {\n return next();\n } else {\n return res.status(401).json({\n msg: 'Incorrect password',\n code: 92298,\n });\n }\n });\n }).catch(err => errorHandler(err, res));\n};\n"
},
{
"path": "api/src/routes/middleware/hCaptcha.js",
"content": "const hCaptcha = require('./../../providers/hCaptcha');\n\nmodule.exports = async (req, res, next) => {\n\n if (!process.env.H_CAPTCHA_SECRET) {\n console.log(`⚠️ Warning: H_CAPTCHA_SECRET not set, skipping captcha validadation`);\n return next();\n }\n\n if (!req.body.htoken) return res.status(422).json({\n errors: {\n htoken: {\n location: \"body\",\n param: \"htoken\",\n msg: \"You must complete the captcha\"\n }\n }\n });\n\n const { data } = await hCaptcha.verify(req.body.htoken);\n\n if (data.success) return next();\n\n return res.status(422).json({\n errors: {\n htoken: {\n location: \"body\",\n param: \"htoken\",\n msg: 'Captcha validation failed.'\n }\n }\n });\n};\n"
},
{
"path": "api/src/routes/middleware/index.js",
"content": "const checkPassword = require('./checkPassword');\nconst isInGroup = require('./isInGroup');\nconst isNotSelf = require('./isNotSelf');\nconst isGroupOwner = require('./isGroupOwner');\nconst canAccessBucket = require('./canAccessBucket');\nconst hCaptcha = require('./hCaptcha');\n\nmodule.exports = {\n checkPassword,\n isInGroup,\n isNotSelf,\n isGroupOwner,\n canAccessBucket,\n hCaptcha,\n};\n"
},
{
"path": "api/src/routes/middleware/isGroupOwner.js",
"content": "const { Group } = require('./../../models');\n\nmodule.exports = async (req, res, next) => {\n const groupID = (req.params.groupID || req.body.groupID);\n const group = await Group.findByPk(groupID);\n\n if (group.ownerID === req.user.id) {\n return next();\n } else {\n return res.status(401).json({\n msg: `You are not the owner of this group ${groupID}`,\n code: 55213,\n });\n }\n};\n"
},
{
"path": "api/src/routes/middleware/isInGroup.js",
"content": "const { User, Group } = require('./../../models');\n\nmodule.exports = async (req, res, next) => {\n const groupID = (req.params.groupID || req.body.groupID);\n const user = await User.findByPk(req.user.id, {\n include: [Group],\n });\n\n if (!user) return res.status(401).json({\n msg: `User not found`,\n code: 40120,\n });\n\n const groups = user.Groups.map(({ id }) => (id));\n\n if (Array.isArray(groups) && groups.includes(groupID)) {\n return next();\n } else {\n return res.status(401).json({\n msg: `You do not have access to group ${groupID} in [${groups.join(', ')}]`,\n code: 65196,\n });\n }\n};\n"
},
{
"path": "api/src/routes/middleware/isNotSelf.js",
"content": "module.exports = (req, res, next) => {\n if (!req.user || !req.user.id) return res.status(401).json({\n msg: 'Access error',\n code: 18196,\n });\n\n if (req.user.id === req.body.userID) return res.status(401).json({\n msg: 'Access error',\n code: 18196,\n });\n\n return next();\n};\n"
},
{
"path": "api/src/routes/user.js",
"content": "const { body, validationResult, matchedData } = require('express-validator');\nconst errorHandler = require('./../providers/errorHandler');\nconst { User, Group } = require('./../models');\nconst middleware = require('./middleware');\nconst bcrypt = require('bcrypt-nodejs');\nconst passport = require('passport');\nconst express = require('express');\n\nconst app = (module.exports = express.Router());\n\n\n/**\n * GET /api/v1/user\n * \n */\napp.get('/user', [\n passport.authenticate('jwt', { session: false })\n], async (req, res) => {\n try {\n const user = await User.findByPk(req.user.id, {\n include: [Group],\n });\n\n if (!user) return res.status(404).send('User not found');\n\n return res.json(user);\n } catch (error) {\n errorHandler(error, res);\n }\n});\n\n\n/**\n * POST /api/v1/user\n * \n */\napp.post('/user', [\n passport.authenticate('jwt', { session: false }),\n body('firstName').exists(),\n body('lastName').exists(),\n body('bio').exists(),\n], async (req, res) => {\n try {\n const errors = validationResult(req);\n if (!errors.isEmpty()) return res.status(422).json({ errors: errors.mapped() });\n const data = matchedData(req);\n\n await User.update(data, { where: { id: req.user.id } });\n\n return res.json(\n await User.findByPk(req.user.id)\n );\n } catch (error) {\n return errorHandler(error, res);\n }\n});\n\n\n/**\n * POST /api/v1/user/update-password\n * \n * Update Password\n */\napp.post('/user/update-password', [\n passport.authenticate('jwt', { session: false }),\n middleware.checkPassword,\n body('password').exists(),\n body('newPassword').exists(),\n body('newPassword', 'Your password must be atleast 7 characters long').isLength({ min: 7 }),\n], async (req, res) => {\n try {\n const errors = validationResult(req);\n if (!errors.isEmpty()) return res.status(422).json({ errors: errors.mapped() });\n const data = matchedData(req);\n\n await User.unscoped().update({\n password: bcrypt.hashSync(data.newPassword, bcrypt.genSaltSync(10)),\n }, {\n where: {\n id: req.user.id,\n }\n });\n\n return res.json({ success: true });\n } catch (error) {\n return errorHandler(error, res);\n }\n});\n"
},
{
"path": "api/src/scripts/blacklist.js",
"content": "\n/**\n * node ./src/scripts/blacklist.js --value=\"bad_word\"\n * docker exec -ti s3-api node ./src/scripts/blacklist.js --value=\"bad_word\"\n *\n */\nrequire('dotenv').config();\nconst argv = require('minimist')(process.argv.slice(2));\nconst { Blacklist } = require('./../models');\nconst db = require('./../providers/db');\n\nif (!argv['value']) throw Error('You must provide --value argument');\n\n(async function Main() {\n try {\n await Blacklist.create({\n value: argv['value']\n });\n console.log(`Word added to blacklist: ${argv['value']}`);\n } catch (err) {\n console.error(err);\n } finally {\n db.connectionManager.close();\n }\n})();\n"
},
{
"path": "api/src/scripts/buckets.js",
"content": "\n/**\n * node ./src/scripts/buckets.js\n * docker exec -ti s3-api node ./src/scripts/buckets.js\n *\n */\nrequire('dotenv').config();\nconst { Bucket } = require('./../models');\nconst db = require('./../providers/db');\n\n(async function Main() {\n try {\n const buckets = await Bucket.findAll();\n for (let i = 0; i < buckets.length; i++) {\n const bucket = buckets[i];\n console.log(`${i} - ${bucket.id} [${bucket.status}] ${bucket.name}.${bucket.namespace}`);\n }\n } catch (err) {\n console.error(err);\n } finally {\n db.connectionManager.close();\n }\n})();\n"
},
{
"path": "api/src/scripts/deleteUser.js",
"content": "\n/**\n * node ./src/scripts/deleteUser.js --userID=\"fdab7a99-2c38-444b-bcb3-f7cef61c275b\"\n * docker exec -ti s3-api node ./src/scripts/deleteUser.js --userID=\"fdab7a99-2c38-444b-bcb3-f7cef61c275b\"\n *\n */\nrequire('dotenv').config();\nconst argv = require('minimist')(process.argv.slice(2));\nconst { User } = require('./../models');\nconst db = require('./../providers/db');\n\nif (!argv['userID']) throw Error('You must provide --userID argument');\n\n(async function Main() {\n try {\n await User.destroy({\n where: {\n id: argv['userID'],\n }\n });\n\n console.log(`User ${argv['userID']} deleted`);\n } catch (err) {\n console.error(err);\n } finally {\n db.connectionManager.close();\n }\n})();\n"
},
{
"path": "api/src/scripts/env",
"content": "#!/bin/bash\n\nif [[ $NODE_ENV == \"production\" ]]\n then\n echo \"NODE_ENV=production ⚠️\"\n else\n echo \"NODE_ENV=$NODE_ENV\"\n fi\n"
},
{
"path": "api/src/scripts/forgotPassword.js",
"content": "\n/**\n * node ./src/scripts/forgotPassword.js --userID=\"c4644733-deea-47d8-b35a-86f30ff9618e\"\n * docker exec -ti s3-api node ./src/scripts/forgotPassword.js --userID=\"c4644733-deea-47d8-b35a-86f30ff9618e\"\n *\n */\nrequire('dotenv').config();\nconst generateJWT = require('./../providers/generateJWT');\nconst argv = require('minimist')(process.argv.slice(2));\nconst { User, Group } = require('./../models');\nconst db = require('./../providers/db');\nconst crypto = require('crypto');\n\nif (!argv['userID']) throw Error('You must provide --userID argument');\n\n(async function Main() {\n try {\n const user = await User.findByPk(argv['userID']);\n\n const passwordResetKey = crypto.randomBytes(32).toString('base64').replace(/[^a-zA-Z0-9]/g, '');\n\n await user.update({ passwordResetKey });\n\n console.log(`\\n\\nEMAIL THIS TO THE USER\\nPASSWORD RESET LINK: ${process.env.FRONTEND_URL}/reset/${passwordResetKey}\\n\\n`);\n } catch (err) {\n console.error(err);\n } finally {\n db.connectionManager.close();\n }\n})();\n"
},
{
"path": "api/src/scripts/generate.js",
"content": "\n/**\n * node ./src/scripts/generate.js --modelName=\"bucket\"\n * docker exec -ti s3-api node ./src/scripts/generate.js --modelName=\"bucket\"\n\n */\nrequire('dotenv').config();\nconst argv = require('minimist')(process.argv.slice(2));\nconst { v4: uuidv4 } = require('uuid');\nconst Mustache = require('mustache');\nconst moment = require('moment');\nconst path = require('path');\nconst fs = require('fs');\n\nif (!argv['modelName']) throw Error('You must provide --modelName argument');\n\n(async function Main() {\n const ucFirst = (string) => (string.charAt(0).toUpperCase().concat(string.slice(1)));\n\n const params = {\n modelname: argv['modelName'].toLowerCase(),\n modelName: argv['modelName'],\n ModelName: ucFirst(argv['modelName']),\n\n modelnames: argv['modelName'].toLowerCase().concat('s'),\n modelNames: argv['modelName'].concat('s'),\n ModelNames: ucFirst(argv['modelName']).concat('s'),\n UUID: uuidv4(),\n };\n\n if (argv['v']) console.log(params);\n\n const pathModel = path.resolve(`./src/models/${params.ModelName}.js`);\n fs.writeFileSync(pathModel, Mustache.render(fs.readFileSync(path.resolve('./src/scripts/generator/Model.js'), 'utf8'), params));\n console.log(`Created: ${pathModel}`);\n\n const pathRoute = path.resolve(`./src/routes/${params.ModelNames}.js`);\n fs.writeFileSync(pathRoute, Mustache.render(fs.readFileSync(path.resolve('./src/scripts/generator/Route.js'), 'utf8'), params));\n console.log(`Created: ${pathRoute}`);\n\n const pathMigration = path.resolve(`./src/database/migrations/${moment().format('YYYYMMDDHHmmss')}-create-${params.ModelNames}.js`);\n fs.writeFileSync(pathMigration, Mustache.render(fs.readFileSync(path.resolve('./src/scripts/generator/Migration.js'), 'utf8'), params));\n console.log(`Created: ${pathMigration}`);\n\n const pathSeeder = path.resolve(`./src/database/seeders/${moment().format('YYYYMMDDHHmmss')}-${params.ModelNames}.js`);\n fs.writeFileSync(pathSeeder, Mustache.render(fs.readFileSync(path.resolve('./src/scripts/generator/Seeder.js'), 'utf8'), params));\n console.log(`Created: ${pathSeeder}`);\n})();\n\n\n"
},
{
"path": "api/src/scripts/generator/Migration.js",
"content": "module.exports = {\n up: (queryInterface, Sequelize) => queryInterface.createTable('{{ ModelNames }}', {\n id: {\n type: Sequelize.UUID,\n defaultValue: Sequelize.UUIDV4,\n primaryKey: true,\n allowNull: false,\n unique: true\n },\n\n createdAt: {\n type: Sequelize.DATE,\n allowNull: true,\n },\n updatedAt: {\n type: Sequelize.DATE,\n allowNull: true,\n },\n deletedAt: {\n type: Sequelize.DATE,\n allowNull: true,\n },\n }),\n down: (queryInterface, Sequelize) => queryInterface.dropTable('{{ ModelNames }}'),\n};\n"
},
{
"path": "api/src/scripts/generator/Model.js",
"content": "const Sequelize = require('sequelize');\nconst db = require('./../providers/db');\n\nmodule.exports = db.define('{{ ModelName }}', {\n id: {\n type: Sequelize.UUID,\n defaultValue: Sequelize.UUIDV4,\n primaryKey: true,\n allowNull: false,\n unique: true\n },\n\n createdAt: {\n type: Sequelize.DATE,\n allowNull: true,\n },\n updatedAt: {\n type: Sequelize.DATE,\n allowNull: true,\n },\n deletedAt: {\n type: Sequelize.DATE,\n allowNull: true,\n },\n\n}, {\n tableName: '{{ ModelNames }}',\n paranoid: true,\n defaultScope: {\n attributes: {\n exclude: [\n\n ]\n }\n },\n});\n"
},
{
"path": "api/src/scripts/generator/Route.js",
"content": "const { body, validationResult, matchedData } = require('express-validator');\nconst errorHandler = require('./../providers/errorHandler');\nconst { {{ ModelName }}, Group } = require('./../models');\nconst middleware = require('./middleware');\nconst bcrypt = require('bcrypt-nodejs');\nconst passport = require('passport');\nconst express = require('express');\n\nconst app = (module.exports = express.Router());\n\n\n/**\n * GET /api/v1/{{ modelnames }}\n * \n */\napp.get('/{{ modelnames }}', [\n passport.authenticate('jwt', { session: false })\n], async (req, res) => {\n try {\n const {{ modelnames }} = await {{ ModelName }}.findAll();\n\n return res.json({{ modelnames }});\n } catch (error) {\n errorHandler(error, res);\n }\n});\n\n\n/**\n * GET /api/v1/{{ modelnames }}/:{{ modelName }}ID\n * \n */\napp.get('/{{ modelnames }}/:{{ modelName }}ID', [\n passport.authenticate('jwt', { session: false }),\n], async (req, res) => {\n try {\n return res.json(\n await {{ ModelName }}.findByPk(req.params.{{ modelName }}ID)\n );\n } catch (error) {\n return errorHandler(error, res);\n }\n});\n\n\n/**\n * POST /api/v1/{{ modelnames }}/:{{ modelName }}ID\n * \n * Update {{ ModelName }}\n */\napp.post('/{{ modelnames }}/:{{ modelName }}ID', [\n passport.authenticate('jwt', { session: false }),\n // body('field').exists(),\n], async (req, res) => {\n try {\n const errors = validationResult(req);\n if (!errors.isEmpty()) return res.status(422).json({ errors: errors.mapped() });\n const data = matchedData(req);\n\n await {{ ModelName }}.update(data, {\n where: {\n id: req.params.{{ modelName }}ID,\n }\n });\n\n return res.json({ success: true });\n } catch (error) {\n return errorHandler(error, res);\n }\n});\n"
},
{
"path": "api/src/scripts/generator/Seeder.js",
"content": "const moment = require('moment');\n\nconst insert = [{\n id: '{{ UUID }}',\n createdAt: moment().format('YYYY-MM-DD HH:mm:ss'),\n updatedAt: moment().format('YYYY-MM-DD HH:mm:ss'),\n}];\n\nmodule.exports = {\n up: (queryInterface, Sequelize) => queryInterface.bulkInsert('{{ ModelNames }}', insert).catch(err => console.log(err)),\n down: (queryInterface, Sequelize) => { }\n};\n"
},
{
"path": "api/src/scripts/inviteUser.js",
"content": "\n/**\n * node ./src/scripts/inviteUser.js --email=\"newuser@example.com\" --groupID=\"fdab7a99-2c38-444b-bcb3-f7cef61c275b\"\n * docker exec -ti s3-api node ./src/scripts/inviteUser.js --email=\"newuser@example.com\" --groupID=\"fdab7a99-2c38-444b-bcb3-f7cef61c275b\"\n *\n */\nrequire('dotenv').config();\nconst generateJWT = require('./../providers/generateJWT');\nconst argv = require('minimist')(process.argv.slice(2));\nconst { User, Group, GroupsUsers } = require('./../models');\nconst db = require('./../providers/db');\nconst crypto = require('crypto');\n\nif (!argv['email']) throw Error('You must provide --email argument');\nif (!argv['groupID']) throw Error('You must provide --groupID argument');\n\n(async function Main() {\n try {\n const email = argv['email'];\n const groupID = argv['groupID'];\n\n let user = await User.unscoped().findOne({\n where: { email }\n });\n\n if (user) {\n // Check if relationship already exists\n const relationship = await GroupsUsers.findOne({\n where: {\n groupID,\n userID: user.id\n }\n });\n if (relationship) return console.log(`User already in this group`);\n } else {\n try {\n user = await User.create({\n email,\n inviteKey: crypto.randomBytes(20).toString('hex')\n });\n\n console.log(user);\n console.log(user.get({ plain: true }));\n console.log(user.inviteKey);\n\n console.log(`\\n\\nEMAIL THIS TO THE USER\\nINVITE LINK: ${process.env.FRONTEND_URL}/invite/${user.inviteKey}\\n\\n`);\n } catch (error) {\n errorHandler(error);\n }\n }\n\n // Delete all first\n await GroupsUsers.destroy({\n where: {\n groupID,\n userID: user.id,\n }\n });\n\n await GroupsUsers.create({\n groupID,\n userID: user.id,\n });\n\n console.log(`User ${user.email} invited`);\n } catch (err) {\n console.error(err);\n } finally {\n db.connectionManager.close();\n }\n})();\n"
},
{
"path": "api/src/scripts/jwt.js",
"content": "\n/**\n * node ./src/scripts/jwt.js --userID=\"c4644733-deea-47d8-b35a-86f30ff9618e\"\n * docker exec -ti s3-api node ./src/scripts/jwt.js --userID=\"c4644733-deea-47d8-b35a-86f30ff9618e\"\n *\n */\nrequire('dotenv').config();\nconst generateJWT = require('./../providers/generateJWT');\nconst argv = require('minimist')(process.argv.slice(2));\nconst { User, Group } = require('./../models');\nconst db = require('./../providers/db');\n\nif (!argv['userID']) throw Error('You must provide --userID argument');\n\n(async function Main() {\n try {\n const user = await User.findByPk(argv['userID'], {\n include: [Group]\n });\n console.log(`\\n\\nJWT:\\n\\n${generateJWT(user)}\\n\\n`);\n } catch (err) {\n console.error(err);\n } finally {\n db.connectionManager.close();\n }\n})();\n"
},
{
"path": "api/src/scripts/refresh",
"content": "#!/bin/bash\n\nif [[ $NODE_ENV == \"production\" ]]\n then\n echo \"ERROR: Can not refresh while in production\"\n else\n sequelize db:migrate:undo:all\n sequelize db:migrate\n sequelize db:seed:all\n fi"
},
{
"path": "api/src/scripts/resetPassword.js",
"content": "\n/**\n * node ./src/scripts/resetPassword.js --userID=\"c4644733-deea-47d8-b35a-86f30ff9618e\" --password=\"password\"\n * docker exec -ti s3-api node ./src/scripts/resetPassword.js --userID=\"c4644733-deea-47d8-b35a-86f30ff9618e\" --password=\"password\"\n *\n */\nrequire('dotenv').config();\nconst generateJWT = require('./../providers/generateJWT');\nconst argv = require('minimist')(process.argv.slice(2));\nconst { User, Group } = require('./../models');\nconst db = require('./../providers/db');\nconst bcrypt = require('bcrypt-nodejs');\n\nif (!argv['userID']) throw Error('You must provide --userID argument');\nif (!argv['password']) throw Error('You must provide --password argument');\n\n(async function Main() {\n try {\n const user = await User.findByPk(argv['userID']);\n\n await user.update({\n password: bcrypt.hashSync(argv['password'], bcrypt.genSaltSync(10)),\n passwordResetKey: null\n });\n\n console.log(`Password updated`);\n } catch (err) {\n console.error(err);\n } finally {\n db.connectionManager.close();\n }\n})();\n"
},
{
"path": "api/src/scripts/seed",
"content": "#!/bin/bash\n\nif [[ $NODE_ENV == \"production\" ]]\n then\n echo \"ERROR: Can not seed while in production\"\n else\n sequelize db:seed:all\n fi"
},
{
"path": "api/src/scripts/sync.js",
"content": "/**\n * node ./src/scripts/sync.js\n * docker exec -ti s3-api node ./src/scripts/sync.js\n *\n */\n\nrequire('dotenv').config();\nconst argv = require('minimist')(process.argv.slice(2));\nconst { Bucket } = require('./../models');\nconst db = require('./../providers/db');\n\n(async function Main() {\n try {\n const buckets = await Bucket.findAll();\n for (const bucket of buckets) {\n await bucket.sync();\n }\n } catch (err) {\n console.error(err);\n }\n})();\n"
},
{
"path": "api/src/scripts/users.js",
"content": "\n/**\n * node ./src/scripts/users.js\n * docker exec -ti s3-api node ./src/scripts/users.js\n *\n */\nrequire('dotenv').config();\nconst { User } = require('./../models');\nconst db = require('./../providers/db');\n\n(async function Main() {\n try {\n const users = await User.findAll();\n for (let i = 0; i < users.length; i++) {\n const user = users[i];\n console.log(`${i} - ${user.id}: ${user.email}`);\n }\n } catch (err) {\n console.error(err);\n } finally {\n db.connectionManager.close();\n }\n})();\n"
},
{
"path": "api/tests/Auth.js",
"content": "const chai = require('chai');\nconst chaiHttp = require('chai-http');\nconst server = require('../src');\nconst should = chai.should();\nconst faker = require('faker');\n\nchai.use(chaiHttp);\n\n\ndescribe('Auth', () => {\n\n /**\n * GET api/v1/_authcheck\n * \n */\n describe('GET /api/v1/_authcheck', () => {\n it('Should check auth status', (done) => {\n chai.request(server)\n .post('/api/v1/auth/login')\n .send({\n email: 'user@example.com',\n password: 'Password@1234'\n })\n .end((err, res) => {\n\n chai.request(server)\n .get('/api/v1/_authcheck')\n .set({\n 'Authorization': `Bearer ${res.body.accessToken}`,\n })\n .end((err, res) => {\n res.should.have.status(200);\n res.body.should.have.property('id');\n done(err);\n });\n });\n });\n\n it('Should check bad headers', (done) => {\n chai.request(server)\n .get('/api/v1/_authcheck')\n .set({\n 'Authorization': 'Bearer xx.xx.xx',\n })\n .end((err, res) => {\n res.should.have.status(401);\n done(err);\n });\n });\n });\n\n\n /**\n * POST api/v1/auth/login\n * \n */\n describe('POST /api/v1/auth/login', () => {\n it('Should return auth access token', (done) => {\n chai.request(server)\n .post('/api/v1/auth/login')\n .send({\n email: 'user@example.com',\n password: 'Password@1234'\n })\n .end((err, res) => {\n res.should.have.status(200);\n res.should.be.json;\n res.body.should.be.a('object');\n res.body.should.have.property('accessToken');\n done(err);\n });\n });\n\n it('Should reject absent password', (done) => {\n chai.request(server)\n .post('/api/v1/auth/login')\n .send({\n email: 'user@example.com',\n })\n .end((err, res) => {\n res.should.have.status(422);\n done(err);\n });\n });\n\n it('Should reject wrong password', (done) => {\n chai.request(server)\n .post('/api/v1/auth/login')\n .send({\n email: 'user@example.com',\n password: 'BAD_PASSWORD'\n })\n .end((err, res) => {\n res.should.have.status(401);\n done(err);\n });\n });\n });\n\n\n /**\n * POST api/v1/auth/sign-up\n * \n */\n describe('POST /api/v1/auth/sign-up', () => {\n\n it('Should create a new user', (done) => {\n chai.request(server)\n .post('/api/v1/auth/sign-up')\n .send({\n email: faker.internet.email(),\n password: 'Password@1234',\n firstName: faker.name.firstName(),\n lastName: faker.name.lastName(),\n groupName: faker.company.bsBuzz(),\n tos: '2020-03-20'\n })\n .end((err, res) => {\n res.should.have.status(200);\n res.should.be.json;\n res.body.should.be.a('object');\n res.body.should.have.property('accessToken');\n done(err);\n });\n });\n\n it('Should reject bad data', (done) => {\n chai.request(server)\n .post('/api/v1/auth/sign-up')\n .send({\n email: faker.internet.email(),\n firstName: faker.name.firstName(),\n })\n .end((err, res) => {\n res.should.have.status(422);\n done(err);\n });\n });\n\n it('Should reject bad email', (done) => {\n chai.request(server)\n .post('/api/v1/auth/sign-up')\n .send({\n email: 'anthonybudd@',\n password: 'password',\n firstName: faker.name.firstName(),\n lastName: faker.name.lastName(),\n groupName: faker.company.bsBuzz()\n })\n .end((err, res) => {\n res.should.have.status(422);\n done(err);\n });\n });\n\n it('Should reject taken email', (done) => {\n chai.request(server)\n .post('/api/v1/auth/sign-up')\n .send({\n email: 'user@example.com',\n password: 'also_bad_password',\n firstName: faker.name.firstName(),\n lastName: faker.name.lastName(),\n groupName: faker.company.bsBuzz()\n })\n .end((err, res) => {\n res.should.have.status(422);\n done(err);\n });\n });\n\n it('Should reject bad password', (done) => {\n chai.request(server)\n .post('/api/v1/auth/sign-up')\n .send({\n email: 'user@example.com',\n password: '12345',\n firstName: faker.name.firstName(),\n lastName: faker.name.lastName(),\n groupName: faker.company.bsBuzz()\n })\n .end((err, res) => {\n res.should.have.status(422);\n done(err);\n });\n });\n });\n});\n"
},
{
"path": "api/tests/Group.js",
"content": "require('dotenv').config();\nconst chai = require('chai');\nconst chaiHttp = require('chai-http');\nconst server = require('../src');\nconst should = chai.should();\n\nchai.use(chaiHttp);\n\nconst GROUP_ID = 'fdab7a99-2c38-444b-bcb3-f7cef61c275b';\nconst OTHER_GROUP_ID = '190c8a70-34d1-4281-a775-850058453704';\n\ndescribe('Groups', () => {\n\n /**\n * GET /api/v1/groups/:groupID\n * \n */\n describe('GET /api/v1/groups/:groupID', () => {\n\n it('Should return the group', (done) => {\n chai.request(server)\n .get(`/api/v1/groups/${GROUP_ID}`)\n .set({\n 'Authorization': `Bearer ${process.env.TEST_JWT}`,\n })\n .end((err, res) => {\n res.should.have.status(200);\n res.should.be.json;\n res.body.should.be.a('object');\n res.body.should.have.property('id');\n res.body.should.have.property('name');\n done();\n });\n });\n\n it('Should reject bad group', (done) => {\n chai.request(server)\n .get(`/api/v1/groups/${OTHER_GROUP_ID}`)\n .set({\n 'Authorization': `Bearer ${process.env.TEST_JWT}`,\n })\n .end((err, res) => {\n res.should.have.status(401);\n done();\n });\n });\n });\n\n\n /**\n * POST /api/v1/groups/:groupID\n * \n */\n describe('POST /api/v1/groups/:groupID', () => {\n\n it('Should update the group name', done => {\n chai.request(server)\n .post(`/api/v1/groups/${GROUP_ID}`)\n .set({\n 'Authorization': `Bearer ${process.env.TEST_JWT}`,\n })\n .send({\n name: 'Test Group'\n })\n .end((err, res) => {\n res.should.have.status(200);\n res.should.be.json;\n res.body.should.be.a('object');\n res.body.should.have.property('id');\n res.body.should.have.property('name');\n res.body.name.should.equal('Test Group');\n done();\n });\n });\n\n it('Should reject bad group', done => {\n chai.request(server)\n .post(`/api/v1/groups/${OTHER_GROUP_ID}`)\n .set({\n 'Authorization': `Bearer ${process.env.TEST_JWT}`,\n })\n .send({\n name: 'Test Group'\n })\n .end((err, res) => {\n res.should.have.status(401);\n done();\n });\n });\n });\n\n\n /**\n * POST /api/v1/groups/:groupID/users/add\n * \n */\n describe('POST /api/v1/groups/:groupID/users/add', () => {\n it('Should add user to the group', done => {\n chai.request(server)\n .post(`/api/v1/groups/${GROUP_ID}/users/add`)\n .set({\n 'Authorization': `Bearer ${process.env.TEST_JWT}`,\n })\n .send({\n userID: 'd700932c-4a11-427f-9183-d6c4b69368f9',\n })\n .end((err, res) => {\n res.should.have.status(200);\n res.should.be.json;\n res.body.should.be.a('object');\n res.body.should.have.property('userID');\n res.body.should.have.property('groupID');\n done();\n });\n });\n\n it('Should reject bad userID', done => {\n chai.request(server)\n .post(`/api/v1/groups/${GROUP_ID}/users/add`)\n .set({\n 'Authorization': `Bearer ${process.env.TEST_JWT}`,\n })\n .send({\n userID: '00000000-0000-0000-0000-000000000000',\n })\n .end((err, res) => {\n res.should.have.status(422);\n done();\n });\n });\n });\n\n\n /**\n * DELETE /api/v1/groups/:groupID/users/:userID\n * \n */\n describe('DELETE /api/v1/groups/:groupID/users/:userID', () => {\n it('Should remove user from the group', done => {\n chai.request(server)\n .delete(`/api/v1/groups/${GROUP_ID}/users/d700932c-4a11-427f-9183-d6c4b69368f9`)\n .set({\n 'Authorization': `Bearer ${process.env.TEST_JWT}`,\n })\n .end((err, res) => {\n res.should.have.status(200);\n res.should.be.json;\n res.body.should.be.a('object');\n res.body.should.have.property('userID');\n res.body.should.have.property('groupID');\n done();\n });\n });\n });\n});\n"
},
{
"path": "api/tests/HealthCheck.js",
"content": "const chai = require('chai');\nconst chaiHttp = require('chai-http');\nconst server = require('../src');\nconst should = chai.should();\n\nchai.use(chaiHttp);\n\ndescribe('DevOps', () => {\n describe('GET /api/v1/_healthcheck', () => {\n it('Should return system status', (done) => {\n chai.request(server)\n .get('/api/v1/_healthcheck')\n .end((err, res) => {\n res.should.have.status(200);\n res.should.be.json;\n res.body.should.be.a('object');\n res.body.status.should.equal('ok');\n done();\n });\n });\n });\n});\n"
},
{
"path": "api/tests/User.js",
"content": "require('dotenv').config();\nconst chai = require('chai');\nconst chaiHttp = require('chai-http');\nconst server = require('../src');\nconst should = chai.should();\n\nchai.use(chaiHttp);\n\n\ndescribe('User', () => {\n\n /**\n * GET /api/v1/user\n * \n */\n describe('GET /api/v1/user', () => {\n\n it('Should return the user model', done => {\n chai.request(server)\n .get('/api/v1/user')\n .set({\n 'Authorization': `Bearer ${process.env.TEST_JWT}`,\n })\n .end((err, res) => {\n res.should.have.status(200);\n res.should.be.json;\n res.body.should.be.a('object');\n res.body.should.have.property('id');\n done();\n });\n });\n\n it('Should reject bad access token', done => {\n chai.request(server)\n .get('/api/v1/user')\n .set({\n 'Authorization': `Bearer BAD.TOKEN`,\n })\n .end((err, res) => {\n res.should.have.status(401);\n done();\n });\n });\n });\n\n\n /**\n * POST /api/v1/user\n * \n */\n describe('POST /api/v1/user', () => {\n it('Should update the current user', done => {\n chai.request(server)\n .post('/api/v1/user')\n .set({\n 'Authorization': `Bearer ${process.env.TEST_JWT}`,\n })\n .send({\n firstName: 'John',\n lastName: 'Smith'\n })\n .end((err, res) => {\n res.should.have.status(200);\n res.should.be.json;\n res.body.should.be.a('object');\n done();\n });\n });\n });\n\n\n /**\n * POST /api/v1/user/update-password\n * \n */\n describe('POST /api/v1/user/update-password', () => {\n it('Should update the current users password', (done) => {\n chai.request(server)\n .post('/api/v1/user/update-password')\n .set({\n 'Authorization': `Bearer ${process.env.TEST_JWT}`,\n })\n .send({\n password: 'password',\n newPassword: 'newpassword'\n })\n .end((err, res) => {\n res.should.have.status(200);\n res.should.be.json;\n res.body.should.be.a('object');\n done();\n });\n });\n });\n});"
},
{
"path": "automation-test/.gitlab-ci.yml",
"content": "stages:\n - build\n\nbuild-job:\n image: docker:dind\n stage: build\n services:\n - docker:dind\n variables:\n IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG\n script:\n - docker login $CI_REGISTRY -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD\n - docker build -t $IMAGE_TAG .\n - docker push $IMAGE_TAG"
},
{
"path": "automation-test/Dockerfile",
"content": "FROM ubuntu:noble\n\nRUN apt-get update && apt-get install -y curl\n\nRUN curl -LO \"https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/arm64/kubectl\"\n\nRUN install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl\n\nCOPY bucket.yml /root/bucket.yml"
},
{
"path": "automation-test/bucket.yml",
"content": "apiVersion: v1\nkind: Namespace\nmetadata:\n name: XXX\n labels:\n name: XXX\n---\napiVersion: v1\nkind: Pod\nmetadata:\n labels:\n app: XXX-pod\n name: XXX-pod\n namespace: XXX\nspec:\n containers:\n - name: XXX-pod\n image: quay.io/minio/minio:latest\n env:\n - name: MINIO_ROOT_USER\n value: root\n - name: MINIO_ROOT_PASSWORD\n value: password\n command:\n - /bin/bash\n - -c\n args: \n - minio server /data --console-address :9001\n ports:\n - containerPort: 9001\n volumeMounts:\n - name: longhornvolume\n mountPath: /data\n volumes:\n - name: longhornvolume\n persistentVolumeClaim:\n claimName: XXX-pvc\n---\napiVersion: v1\nkind: Service \nmetadata:\n name: XXX-svc\n namespace: XXX \nspec:\n selector: \n app: XXX-pod \n ports: \n - protocol: TCP \n port: 80 \n targetPort: 9001\n---\napiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n namespace: XXX\n name: XXX-ing\n annotations:\n kubernetes.io/ingress.class: \"traefik\"\nspec:\n rules:\n - host: XXX.minio.local\n http:\n paths:\n - path: /\n pathType: Prefix\n backend:\n service:\n name: XXX-svc\n port:\n number: 80\n---\napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n name: XXX-pvc\n namespace: XXX\nspec:\n accessModes:\n - ReadWriteOnce\n storageClassName: longhorn\n resources:\n requests:\n storage: 5Gi\n"
},
{
"path": "automation-test/deployment.yml",
"content": "kind: Deployment \napiVersion: apps/v1 \nmetadata: \n name: automation-test-deployment \n labels: \n app: automation-test \nspec: \n replicas: 1 \n selector: \n matchLabels: \n app: automation-test\n template: \n metadata: \n labels: \n app: automation-test\n spec: \n volumes:\n - name: storage-cluster-config\n secret:\n secretName: storage-cluster-config \n containers: \n - name: automation-test\n image: gitlab.local:5050/anthonybudd/automation-test:master\n imagePullPolicy: Always \n ports:\n - containerPort: 80\n command: [ \"/bin/bash\", \"-c\", \"--\" ]\n args: [ \"while true; do sleep 30; done;\" ]\n volumeMounts:\n - name: storage-cluster-config\n mountPath: \"/root/config\"\n subPath: config\n "
},
{
"path": "aws-sdk-test/.gitignore",
"content": "node_modules/"
},
{
"path": "aws-sdk-test/index.js",
"content": "const { S3Client, ListObjectsV2Command, PutObjectCommand } = require(\"@aws-sdk/client-s3\");\n\nconst Bucket = 'kjdoewl';\nconst Namespace = 'gdiwk';\nconst accessKeyId = \"kUoZRWyhUae7tFZNduTS\";\nconst secretAccessKey = \"qxYIqD/8WnvWYIkY7Rg7PSqSMrnxcVfBdpWgzz7z\";\n\n(async function () {\n const client = new S3Client({\n region: 'us-west-2',\n endpoint: `https://${Bucket}.${Namespace}.s3.anthonybudd.io`,\n forcePathStyle: true,\n sslEnabled: true,\n credentials: {\n accessKeyId,\n secretAccessKey\n },\n });\n\n const Key = `${Date.now().toString()}.txt`;\n await client.send(new PutObjectCommand({\n Bucket,\n Key,\n Body: `The time now is ${new Date().toLocaleString()}`,\n ACL: 'public-read',\n ContentType: 'text/plain',\n }));\n console.log(`New object successfully written to: ${Bucket}://${Key}\\n`);\n\n const { Contents } = await client.send(new ListObjectsV2Command({ Bucket }));\n console.log(\"Bucket Contents:\");\n console.log(Contents.map(({ Key }) => Key).join(\"\\n\"));\n})();\n"
},
{
"path": "aws-sdk-test/package.json",
"content": "{\n \"name\": \"aws-sdk-test\",\n \"version\": \"1.0.0\",\n \"main\": \"index.js\",\n \"scripts\": {\n \"test\": \"echo \\\"Error: no test specified\\\" && exit 1\"\n },\n \"author\": \"\",\n \"license\": \"ISC\",\n \"description\": \"\",\n \"dependencies\": {\n \"@aws-sdk/client-s3\": \"^3.575.0\",\n \"aws-sdk\": \"^2.1620.0\"\n }\n}\n"
},
{
"path": "deployment-test/.gitlab-ci.yml",
"content": "stages:\n - build\n\nbuild-job:\n image: docker:dind\n stage: build\n services:\n - docker:dind\n variables:\n IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG\n script:\n - docker login $CI_REGISTRY -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD\n - docker build -t $IMAGE_TAG .\n - docker push $IMAGE_TAG"
},
{
"path": "deployment-test/Dockerfile",
"content": "FROM nginx:alpine\nCOPY . /usr/share/nginx/html"
},
{
"path": "deployment-test/index.html",
"content": ""
},
{
"path": "frontend/default.conf",
"content": "server {\n listen 80;\n listen [::]:80;\n server_name localhost;\n\n #access_log /var/log/nginx/host.access.log main;\n\n location / {\n root /usr/share/nginx/html;\n index index.html index.htm;\n try_files $uri $uri/ /index.html =404;\n }\n\n #error_page 404 /404.html;\n\n # redirect server error pages to the static page /50x.html\n #\n error_page 500 502 503 504 /50x.html;\n location = /50x.html {\n root /usr/share/nginx/html;\n }\n\n # proxy the PHP scripts to Apache listening on 127.0.0.1:80\n #\n #location ~ \\.php$ {\n # proxy_pass http://127.0.0.1;\n #}\n\n # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000\n #\n #location ~ \\.php$ {\n # root html;\n # fastcgi_pass 127.0.0.1:9000;\n # fastcgi_index index.php;\n # fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;\n # include fastcgi_params;\n #}\n\n # deny access to .htaccess files, if Apache's document root\n # concurs with nginx's one\n #\n #location ~ /\\.ht {\n # deny all;\n #}\n}\n"
},
{
"path": "frontend/index.html",
"content": "\n\n\n\n \n \n \n \n Welcome to S3.anthonybudd.io (\"Service\" or \"API\"). These Terms of Service (\"Terms\")\n govern your access to and use of the Service offered by Anthony Budd (\"we,\" \"us,\" or \"our\"). By\n accessing or using the Service, you (\"you\" or \"your\") agree to be bound by these Terms. If you do not\n agree\n to all of the Terms, you are not authorized to access or use the Service.\n
\n\n\n Account: Your account with the Service.\n API Key: A unique identifier used to access the Service.\n API Request: A request sent to the Service to perform an operation, such as uploading or downloading an\n object.\n Content: Any data, information, or materials you upload, store, or transmit through the Service.\n Object: A unit of data stored in the Service, identified by a unique key.\n Service: The s3.anthonybudd.io service, including all features, functionalities,\n and\n documentation offered by us.\n
\n\n\n 3.1 You must be at least 18 years old to access or use the Service.\n 3.2 You are responsible for maintaining the confidentiality of your Account information, including your\n login credentials, and for all activities that occur under your Account.\n 3.3 You agree to notify us immediately of any unauthorized use of your Account or any other security\n breach.\n 3.4 We reserve the right to terminate or suspend your access to the Service at any time and for any\n reason,\n with or without notice.\n
\n\n\n 4.1 You are solely responsible for all Content you upload, store, or transmit through the Service.\n 4.2 You represent and warrant that you have all necessary rights, licenses, and permissions to use,\n upload,\n store, and transmit the Content.\n 4.3 You agree not to upload, store, or transmit any Content that is:\n * Illegal, obscene, defamatory, threatening, harassing, or abusive.\n * Infringes on the intellectual property rights of any third party.\n * Contains viruses or other malicious code.\n * Violates any applicable laws or regulations.\n
\n\n\n 5.1 You may use the Service only for lawful purposes and in accordance with these Terms.\n 5.2 You are responsible for ensuring that your use of the Service complies with all applicable laws and\n regulations.\n 5.3 You agree not to:\n * Reverse engineer, decompile, disassemble, or modify the Service.\n * Access or use the Service in a way that disrupts, overloads, or impairs the performance of the Service\n or\n any other user's use of the Service.\n * Use the Service to store or transmit any Content that violates these Terms.\n * Attempt to gain unauthorized access to the Service or any other user's Account.\n
\n\n\n 6.1 The Service may offer free and paid tiers. Pricing information will be available on our website or\n within the Service.\n 6.2 For paid tiers, you agree to pay the applicable fees on time and in accordance with our payment\n terms.\n 6.3 We reserve the right to change our pricing at any time, with or without notice.\n
\n\n\n\n There is no Service Level Agreement (SLA) for the Service. We do not guarantee any specific level of\n uptime.\n
\n\n\n 8.1 The Service and all underlying technology are protected by intellectual property rights, including\n copyrights, trademarks, and patents. You agree not to remove or alter any proprietary notices on the\n Service.\n 8.2 You grant us a non-exclusive, worldwide, royalty-free right to use, reproduce, modify, publish,\n distribute, and sublicense your Content solely for the purpose of providing the Service to you.\n
\n\n\n THE SERVICE IS PROVIDED \"AS IS\" AND \"AS AVAILABLE\" WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,\n INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE,\n NON-INFRINGEMENT, OR COURSE OF DEALING. WE DO NOT WARRANT THAT THE SERVICE WILL BE UNINTERRUPTED,\n ERROR-FREE, OR VIRUS-FREE.\n
\n\n\n IN NO EVENT SHALL WE, OUR AFFILIATES, OR OUR LICENSORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,\n SPECIAL, CONSEQUENTIAL, OR PUNITIVE DAMAGES ARISING OUT OF OR RELATING TO YOUR USE OF THE SERVICE,\n INCLUDING\n BUT NOT LIMITED TO DAMAGES FOR LOSS OF PROFITS, DATA, GOODWILL, OR USE, EVEN IF WE HAVE BEEN ADVISED\n
\n\n {{ user.firstName }} {{ user.lastName }}
\n {{ user.id.split('-')[0].toUpperCase() }}\n
Click Create Bucket to create a new bucket.
\n\n {{ item.name }}
\n \n {{ item.endpoint }}\n \n
{{ item.status }}
\n\n Are you sure you want to delete the bucket named {{ item.name }} and all\n of it's contents? This action cannot be undone.\n
\n\n\n\nAccessKeyID:
{{ item.accessKeyID }}\nSecretAccessKey:
{{ item.secretAccessKey }}\n
\n\n ![](\"https://raw.githubusercontent.com/anthonybudd/s3-from-scratch/master/_img/test-bucket-longhorn.png\")
\n![](\"https://raw.githubusercontent.com/anthonybudd/s3-from-scratch/master/_img/test-bucket-longhorn-volume.png\")
\n\n\n# Success!\nWhen we go to `test-bucket-100424-211606.minio.local` we should be greeted by the Minio login screen. You can login with `root / password`. \n\nYou can change the login details by editing the env vars `MINIO_ROOT_USER` and `MINIO_ROOT_PASSWORD` in [bucket.yml](automation-test/bucket.yml)\n\nThis demonstrates that we can create new buckets for our users on our storage cluster programmatically.\n\n![](\"https://raw.githubusercontent.com/anthonybudd/s3-from-scratch/master/_img/test-bucket.png\")
"
},
{
"path": "sections/console.md",
"content": "# Console\n\n\n\nThe console will be the device we use for interacting with the infrastructure. Whenever you are not working with the infrastructure unplug the power and disconnect the network adapter from the MacBook.\n\n_AB: USB networking, security reasons_\n\n### MacBook Set-up\nDo a standard MacBook Pro set-up.\n\n- Update to latest version of MacOS\n- Set-up full disk encryption\n- Enable firewall\n- Enable SSH - system prefernce -> secuity -> check \"Remote login\"\n- Confirm SSH is allowed by the firewall\n- Install Xcode\n\nAdditionally you should\n- Disable wifi\n- Prevent sleep while plugged-in\n\n### SSH\nSSH into the console to confirm everything is set-up correctly. \n\n```\n[Dev] ssh Console@10.0.0.XXX\n```\n\nCopy your SSH ID to the console so we can use public key authentication\n\n```sh\n[Dev] ssh-copy-id Console@10.0.0.XXX\n```\n\nSSH back into the console, this should not ask for a password.\n\n```sh\n[Dev] ssh Console@10.0.0.XXX\n```\n\n### Install Homebrew\n```sh\n[Console] /bin/bash -c \"$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)\"\n```\nSource: [https://brew.sh/](https://brew.sh/)\n\n### Install Helm\n```[Console] brew install helm```\n\n### Install Ansible\n```[Console] brew install ansible```\n\n### Install Kubectl\n```[Console] brew install kubectl```\n\n### Install Pass\nPass is a CLI password manager, we will use this to securley manage the secrets for the infrastructure.\n\n```[Console] brew install pass```\n\nSource: [https://www.passwordstore.org/](https://www.passwordstore.org/)\n\nWe can create passwords now using the following command\n```bash\npass generate node1/root 15\nAccIEuEvvTXNgaQ\n```\n\n### Alias & ENV\nTo make life easier, add an environment variable and an alias command for each of the nodes in your infrastructure.\n\n```sh\n[Console] nano ~/.zshrc\n\nexport N1IP=10.0.0.XXX\nalias sshn1=\"ssh node@$N1IP\"\n\nexport N2IP=10.0.0.XXX\nalias sshn2=\"ssh node@$N2IP\"\n...\n```\n"
},
{
"path": "sections/deploying-from-gitlab-to-k3s.md",
"content": "# Deploying From GitLab Registry To Local K3s\n\nWe need to be able to deploy the SaaS front-end and REST API for S3 from our private GitLab Repo to our prod cluster.\n\n\n### Make A New Repo\n\nMake a new repo in GitLab with the following structure, all of the files can be found in [./deployment-test](./../deployment-test)\n\n```sh\nnew-repo/\n├─ .gitlab-ci.yml\n├─ Dockerfile\n├─ k8s.yml\n└─ index.html\n```\n\n### Compile\nCommit the files to the repo which should trigger a build in GitLab.\n\n![](\"https://raw.githubusercontent.com/anthonybudd/s3-from-scratch/master/_img/test-builds.png\")
\n\nYou should see that a new image has been pushed to the container registy by going to __Deploy -> Container Registry__\n\n![](\"https://raw.githubusercontent.com/anthonybudd/s3-from-scratch/master/_img/test-container-registry.png\")
\n\n\n### Deploy\nFrom the Console run the following kubectl command to manually deploy the image from our private registry onto our cluster.\n\n```\n[Console] kubectl --kubeconfig=.kube/config apply -f ./deployment-test/k8s.yml\n\ndeployment.apps/website-deployment created\nservice/website-service created\ningress.networking.k8s.io/website-ingress created\n```\n\nUse `get pods` to see if the pod has deployed\n```sh\n[Console] kubectl --kubeconfig=.kube/config get pods\nNAME READY STATUS RESTARTS AGE\nwebsite-deployment-77c6cfb55c-9fzvq 0/1 ImagePullBackOff 0 43s\n```\n\nThis pod hasn't deployed and has the status `ImagePullBackOff` this is because Kubernetes (more specifically containerd) can't pull the image from our local GitLab container registry.\n\nTo find our more info about this we can use the `describe` command\n```sh\n[Console] kubectl --kubeconfig=.kube/config describe pod website-deployment-77c6cfb55c-9fzvq\n\nName: website-deployment-77c6cfb55c-9fzvq\nNamespace: default\nPriority: 0\nService Account: default\nNode: node-2/10.0.0.217\nStart Time: Wed, 10 Apr 2024 15:54:03 -0700\n ...\n\nEvents:\n Type Reason Age From Message\n ---- ------ ---- ---- -------\n Normal Scheduled 112s default-scheduler Successfully assigned default/website-deployment-77c6cfb55c-9fzvq to node-2\n Normal Pulling 21s (x4 over 111s) kubelet Pulling image \"gitlab.local:5050/anthonybudd/website:master\"\n Warning Failed 21s (x4 over 111s) kubelet Failed to pull image \"gitlab.local:5050/anthonybudd/website:master\": rpc error: code = Unknown desc = failed to pull and unpack image \"gitlab.local:5050/anthonybudd/website:master\": failed to resolve reference \"gitlab.local:5050/anthonybudd/website:master\": failed to do request: Head \"https://gitlab.local:5050/v2/anthonybudd/website/manifests/master\": dial tcp: lookup gitlab.local: no such host\n Warning Failed 21s (x4 over 111s) kubelet Error: ErrImagePull\n Normal BackOff 7s (x6 over 111s) kubelet Back-off pulling image \"gitlab.local:5050/anthonybudd/website:master\"\n Warning Failed 7s (x6 over 111s) kubelet Error: ImagePullBackOff\n```\n\nIt seems there is a problem connecting to gitlab.local from the node.\n\n_AB: Trying to debug the above issue_\n```\n[Node 1] sudo apt install nmap\nnmap -p 5050 gitlab.local\n\nPORT STATE SERVICE\n5050/tcp open mmcc\n```\n\nSolution\n```\n[Node X] sudo nano /etc/hosts\n\n10.0.0.XXX gitlab.local\n```\n\nUpdating out hosts file on each of the nodes solves the issue of not being able to reach gitlab.local\n\n_AB: This isn't a great solution, seems too much to update each node's host file. Figure out why https://gitlab.local works but kube/cd can't reach the registry on gitlab.local:5050_\n\nBut the image is still not pulling 🙃\n\n```\n[Node 1] kubectl --kubeconfig=.kube/config describe pod website-deployment-77c6cfb55c-9fzvq\n\nEvents:\n Type Reason Age From Message\n ---- ------ ---- ---- -------\n Normal Scheduled 13s default-scheduler Successfully assigned default/website-deployment-77c6cfb55c-skg69 to node-1\n Normal BackOff 12s kubelet Back-off pulling image \"gitlab.local:5050/anthonybudd/website:master\"\n Warning Failed 12s kubelet Error: ImagePullBackOff\n Normal Pulling 0s (x2 over 13s) kubelet Pulling image \"gitlab.local:5050/anthonybudd/website:master\"\n Warning Failed 0s (x2 over 13s) kubelet Failed to pull image \"gitlab.local:5050/anthonybudd/website:master\": rpc error: code = Unknown desc = failed to pull and unpack image \"gitlab.local:5050/anthonybudd/website:master\": failed to resolve reference \"gitlab.local:5050/anthonybudd/website:master\": failed to do request: Head \"https://gitlab.local:5050/v2/anthonybudd/website/manifests/master\": tls: failed to verify certificate: x509: certificate signed by unknown authority\n Warning Failed 0s (x2 over 13s) kubelet Error: ErrImagePull\n```\n\nThis is because we are using a self-signed cert on GitLab. \n\nWe can fix this by adding the self signed .crt file to our trust store.\n\n```\n[GitLab Node] cat /etc/gitlab/ssl/gitlab.local.crt \n-----BEGIN CERTIFICATE-----\nMIIEADCCAuigAwIBAgIUUewxBRQiVhwq/OATC/JBVGLGtNkwDQYJKoZIhvcNAQEL\n...\nuyrPmdQ04E6sqfwHUPvtDxxceqzgVS2J0MISbGKa3uDxyQneJnysliILDhNyO/Fg\neSidLK9LN0iPX+GKIL06ieAdSZs=\n-----END CERTIFICATE-----\n\n\n[Node X] sudo nano /usr/local/share/ca-certificates/gitlab.local.crt\n**Paste .crt**\n\n[Node X] sudo update-ca-certificates\n\n[Node X] openssl s_client -connect gitlab.local:5050\n\n...\nSSL handshake has read 1588 bytes and written 398 bytes\nVerification: OK\n```\n\n\n# It Works 🎉\n\n![](\"https://raw.githubusercontent.com/anthonybudd/s3-from-scratch/master/_img/test-deployment.jpg\")
\n\nYou have no idea how long that took me to resolve"
},
{
"path": "sections/gitlab.md",
"content": "# Installing GitLab on a Raspberry Pi 4\n\nFirst you will need to build a single node. Flash the SD card with a copy of 32-bit Raspberry Pi OS 11 (bullseye). At time of writing, GitLab-CE is not supported on later versions of Raspberry Pi OS or running on a 64-bit OS. Set the hostname to `gitlab`.\nAlso follow the [default node set-up instructions](/sections/node.md)\n\n![](\"https://raw.githubusercontent.com/anthonybudd/s3-from-scratch/master/_img/gitlab-32-bit.png?v=1\")
\n\nSource: [https://about.gitlab.com/install/#raspberry-pi-os](https://about.gitlab.com/install/#raspberry-pi-os)\n\n### Add `arm_64bit=0` to config.txt\nFor some insane reason when you select 32-bit Raspberry Pi OS in Raspberry Pi imager you actually get [a 32-bit userland on top of a 64-bit kernel.](https://github.com/raspberrypi/rpi-imager/issues/847#issuecomment-2035800759) This will cause issues with GitLab runner later. To get a full 32-bit OS add `arm_64bit=0` to the config.txt\n\n- [gitlab-org/gitlab-runner issue:37336](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/37336)\n- [raspberrypi/rpi-imager issue:847](https://github.com/raspberrypi/rpi-imager/issues/847)\n- [R Pi Docs: arm_64bit](https://www.raspberrypi.com/documentation/computers/config_txt.html#arm_64bit)\n\n### Boot\nInsert the SD card and boot the pi. From the console SSH into the GitLab node using your password. \n\n```[Console] ssh gitlab@10.0.0.XXX```\n\nIf you connect successfully, add the console's key to authorized key file onto the GitLab node to enable passwordless public-key authentication.\n\n```[Console] ssh-copy-id gitlab@10.0.0.XXX```\n\n__Hint:__ Save the root password for the GitLab node in pass \n\n```[Console] pass insert gitlab```\n\n__Hint:__ Make an alias\n```\n[Console] nano ~/.zshrc\n\nexport GLIP=10.0.0.175\nalias sshgl=\"ssh gitlab@$GLIP\"\n```\n\n_AB: Firewall Settings?_\n\n\n### Installing GitLab\nRun the below commands to install GitLab.\n```sh\n[GitLab Node] sudo apt update && sudo apt upgrade -y\nsudo apt-get install -y curl openssh-server ca-certificates apt-transport-https perl\ncurl https://packages.gitlab.com/gpg.key | sudo tee /etc/apt/trusted.gpg.d/gitlab.asc\nsudo curl -sS https://packages.gitlab.com/install/repositories/gitlab/raspberry-pi2/script.deb.sh | sudo bash\nsudo EXTERNAL_URL=\"https://gitlab.local\" apt-get install gitlab-ce\n```\n\n_AB: More on securing gitlab?_\n\n_AB:_ https://docs.gitlab.com/ee/security/index.html\n\n\n### Setting-up GitLab\nYou should be able to access GitLab at https://gitlab.local\n\nYou can get the root password by running this command on the GitLab node.\n\n```[GitLab Node] sudo cat /etc/gitlab/initial_root_password```\n\nUse pass to generate a new password and update the GitLab root account with the new password.\n\n```[Console] pass generate gitlab/gitlab-app-root 30```\n\n#### Create a new user\nCreate a new user by going to __Admin Area -> Overview -> Users -> New user__\n\nBecasue we have not set-up internet we will need to set this users password using the CLI\n\n```[GitLab Node] sudo gitlab-rake \"gitlab:password:reset[sidneyjones]\"```\n\n__Hint:__ This command might hang for 5mins before it prompts you to enter a password. IDK why this happens.\n\n\n### Installing Docker for GitLab Runner\nWe will need to use GitLabs CI/CD feature to compile our code and deploy it. I am experimenting using GitLab to trigger the rollout of the deployment in K3s. I think the best solution would be to use ArgoCD but I want to get the whole thing working first then I will address CD.\n\nFirst, install docker on the GitLab node\n```sh\nsudo apt-get update\n\n# ca-certificates curl should already be installed\nsudo apt-get install -y ca-certificates curl \nsudo install -m 0755 -d /etc/apt/keyrings\nsudo curl -fsSL https://download.docker.com/linux/raspbian/gpg -o /etc/apt/keyrings/docker.asc\nsudo chmod a+r /etc/apt/keyrings/docker.asc\n\n# Set up Docker's APT repository:\necho \\\n \"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/raspbian \\\n $(. /etc/os-release && echo \"$VERSION_CODENAME\") stable\" | \\\n sudo tee /etc/apt/sources.list.d/docker.list > /dev/null\n\nsudo apt-get update\nsudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin\nsudo docker run hello-world\n```\nSource: [https://docs.docker.com/engine/install/raspberry-pi-os/](https://docs.docker.com/engine/install/raspberry-pi-os/)\n\n#### Docker Linux Post-install\n```sh\nsudo groupadd docker\nsudo usermod -aG docker $USER\nnewgrp docker\ndocker run hello-world\n```\nSource: [https://docs.docker.com/engine/install/linux-postinstall/](https://docs.docker.com/engine/install/linux-postinstall/)\n\n### Installing GitLab Runner\nNext, install gitlab-runner on the GitLab node\n```sh\ncurl -L \"https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh\" | sudo bash\nsudo apt-get install -y gitlab-runner\n```\nSource: [https://docs.gitlab.com/runner/install/linux-repository.html#installing-gitlab-runner](https://docs.gitlab.com/runner/install/linux-repository.html#installing-gitlab-runner)\n\n\n#### Create a Runner\nCreate a new GitLab runner by going to __Admin Area -> CI/CD -> Runners -> New instance runner__\n\nMake a note of the token, you will need this to register the runner.\n\n![](\"https://raw.githubusercontent.com/anthonybudd/s3-from-scratch/master/_img/new-gitlab-runner.jpg\")
\n\nBecause we are using a local GitLab instance `gitlab-runner register` will produce the error.\n```\nx509: certificate signed by unknown authority\n```\n\nTo get around this we will need to create our own self signed certificate and pass the path to ther .crt file to the register command with `--tls-ca-file`\n\n```sh\napt-get install -y openssl ca-certificates\n\ncd /etc/gitlab/ssl/\n\nsudo mv gitlab.local.key /tmp\nsudo mv gitlab.local.crt /tmp\n12\nsudo openssl req -nodes -new -x509 -sha256 -keyout gitlab.local.key -out gitlab.local.crt -days 356 -subj \"/C=US/ST=State/L=City/O=Organization/OU=Department/CN=*.gitlab.local\" -addext \"subjectAltName = DNS:localhost,DNS:gitlab.local,DNS:registry.gitlab.local\"\n\nsudo gitlab-ctl restart\n```\nSource: [https://docs.gitlab.com/runner/configuration/tls-self-signed.html](https://docs.gitlab.com/runner/configuration/tls-self-signed.html)\n\nNotice `subjectAltName = ... DNS:registry.gitlab.local` without this Kubernetes will not be able to pull from images from the regsiett and will error with `ErrImagePull`\n\n```\nfailed to do request: Head \"https://registry.gitlab.local/v2/anthonybudd/website/manifests/main\": tls: failed to verify certificate: x509: certificate signed by unknown authority\n```\n\nUpdate hosts file with\n```sh\nsudo nano /etc/hosts\n\n127.0.0.1 gitlab.local\n```\n\nNow you can register the runner with GitLab.\n```sh\ngitlab-runner register \\\n --non-interactive \\\n --token TOKEN_HERE \\\n --url https://gitlab.local/ \\\n --executor docker \\\n --tls-ca-file /etc/gitlab/ssl/gitlab.local.crt\n```\n\n### Add `network_mode` and `volumes` to config.toml\nBecause we are using a local instance of GitLab you will need to add `network_mode = \"host\"` to the `[runners.docker]` section of the GitLab config file located at `/etc/gitlab-runner/config.toml`\n\nYou will also need to add `volumes = [\"/var/run/docker.sock:/var/run/docker.sock\", \"/cache\"]`\n\n```toml\nconcurrent = 1\ncheck_interval = 0\nconnection_max_age = \"15m0s\"\nshutdown_timeout = 0\n\n[session_server]\n session_timeout = 1800\n\n[[runners]]\n name = \"gitlab\"\n url = \"https://gitlab.local\"\n id = 1\n token = \"TOKEN_HERE\"\n token_obtained_at = 2024-03-31T22:43:20Z\n token_expires_at = 0001-01-01T00:00:00Z\n tls-ca-file = \"/etc/gitlab/ssl/gitlab.local.crt\"\n executor = \"docker\"\n [runners.cache]\n MaxUploadedArchiveSize = 0\n [runners.docker]\n tls_verify = false\n image = \"docker:dind\"\n privileged = false\n disable_entrypoint_overwrite = false\n oom_kill_disable = false\n disable_cache = true\n network_mode = \"host\"\n volumes = [\"/var/run/docker.sock:/var/run/docker.sock\", \"/cache\"]\n shm_size = 0\n network_mtu = 0\n```\nSource: [https://gitlab.com/gitlab-org/gitlab-runner/-/issues/305](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/305)\n\n\nFinally start GitLab runner with\n\n```[GitLab Node] sudo gitlab-runner run --config /etc/gitlab-runner/config.toml```\n\n_AB: auto-start runner on boot?_\n\n\n### Create a test repo\nLog-out of the root GitLab account and login as your user account.\n\nCreate a new repo called `test`.\n\nUsing the GitLab web UI create a file called `.gitlab-ci.yml` and add the below to the file.\n```\nstages:\n - build\n\nbuild-job:\n stage: build\n script:\n - echo \"Compiling the code...\"\n - pwd\n - ls\n```\n\nCommiting this file should trigger a build job. In the sidebar go to __Build -> Jobs__ and open the latest job. You should see that the `script` section in the ci file has successfully been called inside the repo. This shows that GitLab and GitLab runner are working as expected. You can delete the test repo.\n\n![](\"https://raw.githubusercontent.com/anthonybudd/s3-from-scratch/master/_img/gitlab-job.jpg\")
"
},
{
"path": "sections/internet.md",
"content": "# Connecting to the Internet\n\nIn a true datacenter we would request a static IP address from our ISP and point our DNS servers to that IP Address. However I cannot get a static IP at my office, to get around this and to make this project more useful I have decided to use OpenVPN, this will allow us to \"rent\" a static IP address from an existing cloud provider.\n\nTo do this we will set up a Digital Ocean Droplet running the latest version of Ubuntu.\n\nMake sure you add the consoles public key to the droplet\n_AB: add image_\n\n__Note:__ Do not request a \"Reserved IP\" this causes issues with OpenVPN for some reason.\n\nInstall OpenVPN\n```sh\n[OpenVPNServer] wget https://git.io/vpn -O openvpn-install.sh\nsudo chmod +x openvpn-install.sh\nsudo bash openvpn-install.sh\n```\n\nYou will see an interactive install script like this. When prompted name the clinet `node-1`\n```\nWelcome to this OpenVPN road warrior installer!\n\nWhich protocol should OpenVPN use?\n 1) UDP (recommended)\n 2) TCP\nProtocol [1]: 1\n\nWhat port should OpenVPN listen to?\nPort [1194]: \n\nSelect a DNS server for the clients:\n 1) Current system resolvers\n 2) Google\n 3) 1.1.1.1\n 4) OpenDNS\n 5) Quad9\n 6) AdGuard\nDNS server [1]: 2\n\nEnter a name for the first client:\nName [client]: node-1\n\nOpenVPN installation is ready to begin.\nPress any key to continue...\n```\n\n```sh\n[OpenVPNServer] sudo systemctl restart openvpn-server@server.service\n```\n\n### Node Set-up\n\nInstall OpenVPN onto the master node of each cluster\n```\n[Node X] apt-get install openvpn -y\n```\n\nMake a .ovpn config file on the VPN server and SCP it to the node\n```\n[Console] scp root@OPEN_VPN_SERVER_IP:/etc/openvpn/???/node-1.ovpn /tmp\nmv /tmp/node-1.ovpn /tmp/node-1.config\nscp /tmp/node-1.config node@$N1IP:/etc/openvpn\n```\n__Note:__ I'm deliberately renameing the file to .config, this is becasue the extension OpenVPN uses to autoload.\n_AB: Confirm file path_\n_AB: Add file to pass?_\n\n\n```\n[Node X] nano /etc/default/openvpn\n\n# Uncomment this line\nAUTOSTART=\"all\"\n```\n\n_AB: This is probbably not that good of a set-up, i should probably improve this by moving openVPN to the OpenWRT raspberry pi_\n\n\n### Install Nginx on the OpenVPN server\n\n```[OpenVPNServer] sudo apt install -y nginx```\n\n\n\n```sh\n[OpenVPNServer] sudo nano /etc/hosts\n\nPROD_CLUSTER_MASTER_NODE_IP app.YOUR_DOMAIN.com\nPROD_CLUSTER_MASTER_NODE_IP api.YOUR_DOMAIN.com\n```\n\n/etc/nginx/nginx.conf\n```\n...\nstream {\n map $ssl_preread_server_name $targetBackend {\n s3.anthonybudd.io 10.8.0.3:443;\n s3-api.anthonybudd.io 10.8.0.3:443;\n echo.s3.anthonybudd.io 10.8.0.4:443;\n } \n\n server {\n listen 443;\n proxy_pass $targetBackend; \n ssl_preread on;\n }\n}\n...\n```\n\naster.s3.YOUR_DOMAIN.com\n```sh\nserver {\n listen 80;\n server_name ~^(?\n\n### Default Set-up Procedure\nBy default always do the following set-up procedure when creating a new node.\n\nUnless otherwise specified always flash the SD card with 64-bit Raspberry Pi OS Lite.\n\n#### Public Key Auth\n```[Console] ssh-copy-id node@10.0.0.XXX```\n\n#### Enable cpuset\n```sh\n[Node X] sudo nano /boot/firmware/cmdline.txt\n\ncgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory \n```\n\nMore on cgroups: [Downey.io/blog/exploring-cgroups-raspberry-pi](https://downey.io/blog/exploring-cgroups-raspberry-pi)\n\n#### Disable WiFi & Bluetoooth\n```sh\n[Node X] sudo nano /boot/firmware/config.txt\n\ndtoverlay=disable-wifi\ndtoverlay=disable-bt\n```\n\n#### Run `node-config-script.sh`\nThis script will add some security changes to SSH and install Fail2Ban.\n\nSCP the [node-config-script.sh](./../node/node-config-script.sh) to the node and run it. \n\n_AB: Fail2Ban Config?_\n\n_AB: Test Fail2Ban_\n\n```sh\n[Node X] curl https://raw.githubusercontent.com/anthonybudd/s3-from-scratch/master/node/node-config-script.sh -sSL | sh\n\n# Or\n\n[Console] scp ./node/node-config-script.sh node@10.0.0.XXX:~\n[Console] ssh node@10.0.0.XXX\n[Node X] sudo ~/node-config-script.sh\n[Node X] sudo reboot\n```\n\n_AB: Is this enough? What more SSH changes should I make to improve security?_\n"
},
{
"path": "sections/production-cluster.md",
"content": "# K3S: Production cluster\n\nThis guide will cover how to set-up the production kubernetes cluster for hosting our public website, api and front-end.\n\n_AB: Make cluster HA_\n\n### Build nodes\nStart by building two nodes, [following the default node set-up procedure](./node.md), and the install them into the infrastructure.\n\nOnce both nodes have booted-up, confirm that you can SSH into the nodes from the console.\n\n```[Console] ssh node@10.0.0.XXX```\n\n```[Console] ssh node@10.0.0.YYY```\n\n\n### Install K3s\nClone this repo on the Console and make a copy of the `example` directory located at `./ansible/inventory/example` \n\n```sh\n[Console] cp -R ansible/inventory/example ansible/inventory/prod-cluster\n```\n\nEdit the `hosts.ini` located in `./ansible/inventory/prod-cluster` so node-1 IP address is the master list and node-2 is in the node list\n\n```\n[Console] nano ansible/inventory/prod-cluster/hosts.ini\n```\n\n```\n[master]\n10.0.0.XXX\n\n[node]\n10.0.0.YYY\n\n[k3s_cluster:children]\nmaster\nnode\n```\n\n#### Run the Ansible playbook\n\nRun the Ansible playbook to install K3s across all of the nodes in our cluster.\n\n```sh\n[Console] ansible-playbook ansible/site.yml -i ansible/inventory/prod-cluster/hosts.ini\n```\n\nIf the playbook completes successfully you should see output like this\n\n```sh\nPLAY RECAP ****************************************************************************************************\n10.0.0.XXX : ok=21 changed=12 unreachable=0 failed=0 skipped=10 rescued=0 ignored=0 \n10.0.0.YYY : ok=10 changed=5 unreachable=0 failed=0 skipped=10 rescued=0 ignored=0 \n```\n\n#### Test the nodes\nCopy the kubernetes config file from the master node to the Console.\n\n```\n[Console] scp node@10.0.0.XXX:~/.kube/config ~/.kube/config\n```\n\nTest that all of the nodes are up and running by running this command.\n\n```\n[Console] kubectl --kubeconfig=.kube/config get nodes\n```\n\nYou should get a response that looks like this.\n```sh\nNAME STATUS ROLES AGE VERSION\nnode-1 Ready control-plane,master 3m6s v1.26.9+k3s1\nnode-2 Ready ![](\"https://raw.githubusercontent.com/anthonybudd/s3-from-scratch/master/_img/echo.png\")
\n\n\n\n_AB: k8s dashboard_\n\n\n\n\n\n\n###\n\nkubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.1.1/cert-manager.yaml\n\nkubectl get pods --namespace cert-manager\n\nNAME READY STATUS RESTARTS AGE\ncert-manager-5c6866597-zw7kh 1/1 Running 0 2m\ncert-manager-cainjector-577f6d9fd7-tr77l 1/1 Running 0 2m\ncert-manager-webhook-787858fcdb-nlzsq 1/1 Running 0 \n\n"
},
{
"path": "sections/ssl.md",
"content": "# SSL\n\nkubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.10.1/deploy/static/provider/baremetal/deploy.yaml\n\n```sh\nhelm --kubeconfig=.kube/storage-config repo add jetstack https://charts.jetstack.io --force-update\n\nhelm --kubeconfig=.kube/storage-config repo update\n\nhelm --kubeconfig=.kube/storage-config install \\\n cert-manager jetstack/cert-manager \\\n --namespace cert-manager \\\n --create-namespace \\\n --version v1.14.4 \\\n --set installCRDs=true\n\nNAMESPACE: cert-manager\nSTATUS: deployed\nREVISION: 1\nTEST SUITE: None\nNOTES:\ncert-manager v1.14.4 has been deployed successfully!\n\nIn order to begin issuing certificates, you will need to set up a ClusterIssuer\nor Issuer resource (for example, by creating a 'letsencrypt-staging' issuer).\n\nMore information on the different types of issuers and how to configure them\ncan be found in our documentation:\n\nhttps://cert-manager.io/docs/configuration/\n\nFor information on how to configure cert-manager to automatically provision\nCertificates for Ingress resources, take a look at the `ingress-shim`\ndocumentation:\n\nhttps://cert-manager.io/docs/usage/ingress/\n\n\nkubectl --kubeconfig=.kube/storage-config -n cert-manager get pods\n\nNAME READY STATUS RESTARTS AGE\ncert-manager-cainjector-58c4f6d945-8thcs 1/1 Running 0 3m43s\ncert-manager-5bfc55c5c6-zhmvs 1/1 Running 0 3m43s\ncert-manager-webhook-7bd66d5b9c-dlqdr 1/1 Running 0 3m43s\n```\n\nkubectl --kubeconfig=.kube/storage-config get Issuers,ClusterIssuers,Certificates,CertificateRequests,Orders,Challenges --all-namespaces\n\nkprod\n\n```sh\napiVersion: cert-manager.io/v1\nkind: ClusterIssuer\nmetadata:\n name: letsencrypt-prod\n namespace: cert-manager\nspec:\n acme:\n # The ACME server URL\n server: https://acme-v02.api.letsencrypt.org/directory\n # Email address used for ACME registration\n email: your_email_address_here\n # Name of a secret used to store the ACME account private key\n privateKeySecretRef:\n name: letsencrypt-prod\n # Enable the HTTP-01 challenge provider\n solvers:\n - http01:\n ingress:\n class: traefik\n```\n\n[VPN] apt install -y libnginx-mod-stream\n\n[VPN] apt-get install nginx-extras\n\n\nnano /etc/nignx/nginx.conf\n\nstream {\n server {\n listen 443;\n proxy_pass 10.8.0.3:443;\n }\n}\n"
},
{
"path": "sections/storage-cluster.md",
"content": "# K3S: Storage Cluster\n\nThis section will cover how to set-up the storage kubernetes cluster which will store of the data for our buckets.\n\n\n### Build nodes\nStart by building three nodes and the install them into the infrastructure.\n\nOnce both nodes have booted-up, confirm that you can SSH into the nodes from the console.\n\n```[Console] ssh node@10.0.0.XXX```\n\n```[Console] ssh node@10.0.0.YYY```\n\n```[Console] ssh node@10.0.0.ZZZ```\n\n\n### Install K3s\nClone this repo on the Console and make a copy of the `example` directory located at `./ansible/inventory/example` \n\n```sh\n[Console] cp -R ansible/inventory/example ansible/inventory/storage-cluster\n```\n\nEdit the `hosts.ini` located in `./ansible/inventory/storage-cluster` so node-3 IP address is the master list and node-4 and node 5 are in the node list\n\n```\n[Console] nano ansible/inventory/storage-cluster/hosts.ini\n```\n\n```\n[master]\n10.0.0.7\n\n[node]\n10.0.0.8\n10.0.0.9\n\n[k3s_cluster:children]\nmaster\nnode\n```\n\n#### Run the Ansible playbook\n\nRun the Ansible playbook to install K3s across all of the nodes in our cluster.\n\n```sh\n[Console] ansible-playbook ansible/site.yml -i ansible/inventory/storage-cluster/hosts.ini\n```\n\nIf the playbook completes successfully you should see output like this\n\n```sh\nPLAY RECAP ****************************************************************************************************\n10.0.0.XXX : ok=21 changed=12 unreachable=0 failed=0 skipped=10 rescued=0 ignored=0 \n10.0.0.YYY : ok=10 changed=5 unreachable=0 failed=0 skipped=10 rescued=0 ignored=0 \n10.0.0.ZZZ : ok=10 changed=5 unreachable=0 failed=0 skipped=10 rescued=0 ignored=0 \n```\n\n#### Test the nodes\nCopy the kubernetes config file from the master node to the Console. Remember to give the config file another name so it doesn't overwrite the prod-cluster config file.\n\n```\n[Console] scp node@10.0.0.XXX:.kube/config ~/.kube/storage-config\n```\n\nTest that all of the nodes are up and running by running this command.\n\n```\n[Console] kubectl --kubeconfig=.kube/storage-config get nodes\n```\n\nYou should get a response that looks like this.\n```sh\nNAME STATUS ROLES AGE VERSION\nnode-1 Ready control-plane,master 1m49s v1.26.9+k3s1\nnode-2 Ready ![](\"https://raw.githubusercontent.com/anthonybudd/s3-from-scratch/master/_img/longhorn-ui.png\")
\n\n\n### Configure Longhorn\nBy default Longhorn will save data to `/var/lib/longhorn` which is our SD card. To make our Longhorn nodes save to our SSD go to __Node__ in tha top menu.\n\nFor each of the Longhorn nodes click on __Edit node and Disks__ in the far right. Set Scheduling to Disable and then delete the existing disk. Click __Add Disk__ set the Name to `ssd` and the Path to `/ssd`. Click save.\n\n![](\"https://raw.githubusercontent.com/anthonybudd/s3-from-scratch/master/_img/longhorn-ssd.png\")
\n\n### Set Longhorn to the default storageclass\nYou can set Longhorn as the default storage class by running the following\n\n```\n[Console] kubectl --kubeconfig=.kube/storage-config get storageclass \nNAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE\nlocal-path (default) rancher.io/local-path Delete WaitForFirstConsumer false 2d\nlonghorn (default) driver.longhorn.io Delete Immediate true 2d\n\n[Console] kubectl --kubeconfig=.kube/storage-config patch storageclass local-path -p '{\"metadata\": {\"annotations\":{\"storageclass.kubernetes.io/is-default-class\":\"false\"}}}' \nstorageclass.storage.k8s.io/local-path patched\n\n[Console] kubectl --kubeconfig=.kube/storage-config get storageclass \nNAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE\nlonghorn (default) driver.longhorn.io Delete Immediate true 2d\nlocal-path rancher.io/local-path Delete WaitForFirstConsumer false 2d\n```\n\n"
}
]