From 08c21cafe4839ffa0654f6066c1a25dd42744220 Mon Sep 17 00:00:00 2001 From: Karol Wypchlo Date: Tue, 13 Sep 2022 13:39:00 +0200 Subject: [PATCH] remove outdated setup scripts --- setup-scripts/README.md | 155 -------------------- setup-scripts/setup-docker-services.sh | 48 ------ setup-scripts/setup-health-check-scripts.sh | 11 -- setup-scripts/setup-server.sh | 48 ------ 4 files changed, 262 deletions(-) delete mode 100644 setup-scripts/README.md delete mode 100755 setup-scripts/setup-docker-services.sh delete mode 100755 setup-scripts/setup-health-check-scripts.sh delete mode 100755 setup-scripts/setup-server.sh diff --git a/setup-scripts/README.md b/setup-scripts/README.md deleted file mode 100644 index d5237e09..00000000 --- a/setup-scripts/README.md +++ /dev/null @@ -1,155 +0,0 @@ -# Skynet Portal Setup Scripts - -> :warning: This documentation is outdated and should be used for reference -only. Portal setup documentation is located at -https://portal-docs.skynetlabs.com/. - -This directory contains a setup guide and scripts that will install and -configure some basic requirements for running a Skynet Portal. The assumption is -that we are working with a Debian Buster Minimal system or similar. - -## Latest Setup Documentation - -Latest Skynet Webportal setup documentation and the setup process Skynet Labs -supports is located at https://docs.siasky.net/webportal-management/overview. - -Some of the scripts and setup documentation contained in this repository -(`skynet-webportal`) can be outdated and generally should not be used. - -## Initial Setup - -You may want to fork this repository and replace ssh keys in -`setup-scripts/support/authorized_keys` and optionally edit the `setup-scripts/support/tmux.conf` and `setup-scripts/support/bashrc` configurations to fit your needs. - -### Step 0: stack overview - -- dockerized services inside `docker-compose.yml` - - [sia](https://sia.tech) ([docker hub](https://hub.docker.com/r/nebulouslabs/sia)): storage provider, heart of the portal setup - - [caddy](https://caddyserver.com) ([docker hub](https://hub.docker.com/r/caddy/caddy)): reverse proxy (similar to nginx) that handles ssl out of a box and acts as a transparent entry point - - [openresty](https://openresty.org) ([docker hub](https://hub.docker.com/r/openresty/openresty)): nginx custom build, acts as a cached proxy to siad and exposes all api endpoints - - [health-check](https://github.com/SkynetLabs/skynet-webportal/tree/master/packages/health-check): simple service that runs periodically and collects health data about the server (status and response times) - [read more](https://github.com/SkynetLabs/skynet-webportal/blob/master/packages/health-check/README.md) - - [handshake](https://handshake.org) ([github](https://github.com/handshake-org/hsd)): full handshake node - - [handshake-api](https://github.com/SkynetLabs/skynet-webportal/tree/master/packages/handshake-api): simple API talking to the handshake node - [read more](https://github.com/SkynetLabs/skynet-webportal/blob/master/packages/handshake-api/README.md) - - [website](https://github.com/SkynetLabs/skynet-webportal/tree/master/packages/website): portal frontend application - [read more](https://github.com/SkynetLabs/skynet-webportal/blob/master/packages/website/README.md) -- discord integration - - [funds-checker](funds-checker.py): script that checks wallet balance and sends status messages to discord periodically - - [health-checker](health-checker.py): script that monitors health-check service for server health issues and reports them to discord periodically - - [log-checker](log-checker.py): script that scans siad logs for critical errors and reports them to discord periodically -- [blocklist-skylink](../scripts/blocklist-skylink.sh): script that can be run locally from a machine that has access to all your skynet portal servers that blocklists provided skylink and prunes nginx cache to ensure it's not available any more (that is a bit much but that's the best we can do right now without paid nginx version) - if you want to use it, make sure to adjust the server addresses - -### Step 1: setting up server user - -1. SSH in a freshly installed Debian machine on a user with sudo access (can be root) -1. `apt-get update && apt-get install sudo libnss3-tools -y` to make sure `sudo` is available -1. `adduser user` to create user called `user` (creates `/home/user` directory) -1. `usermod -aG sudo user` to add this new user to sudo group -1. `sudo groupadd docker` to create a group for docker (it might already exist) -1. `sudo usermod -aG docker user` to add your user to that group -1. Quit the ssh session with `exit` command - -You can now ssh into your machine as the user `user`. - -### Step 2: setting up environment - -1. On your local machine: `ssh-copy-id user@ip-addr` to copy over your ssh key to server -1. On your local machine: `ssh user@ip-addr` to log in to server as user `user` -1. You are now logged in as `user` - -**Following step will be executed on remote host logged in as a `user`:** - -1. `sudo apt-get install git -y` to install git -1. `git clone https://github.com/SkynetLabs/skynet-webportal` -1. `cd skynet-webportal` -1. run setup scripts in the exact order and provide sudo password when asked (if one of them fails, you can retry just this one before proceeding further) - 1. `/home/user/skynet-webportal/setup-scripts/setup-server.sh` - 1. `/home/user/skynet-webportal/setup-scripts/setup-docker-services.sh` - 1. `/home/user/skynet-webportal/setup-scripts/setup-health-check-scripts.sh` (optional) - -### Step 3: configuring siad - -At this point we have almost everything running, we just need to set up your wallet and allowance: - -1. Create a new wallet (remember to save the seed) - > `docker exec -it sia siac wallet init` -1. Unlock the wallet (use the seed as password) - > `docker exec -it sia siac wallet unlock` -1. Generate a new wallet address (save it for later to transfer the funds) - > `docker exec -it sia siac wallet address` -1. Set up allowance - > `docker exec -it sia siac renter setallowance` - 1. 10 KS (keep 25 KS in your wallet) - 1. default period - 1. default number of hosts - 1. 4 week renewal time - 1. 500 GB expected storage - 1. 500 GB expected upload - 1. 5 TB expected download - 1. default redundancy -1. Set a maximum storage price - > `docker exec -it sia siac renter setallowance --max-storage-price 100SC` -1. Instruct siad to start making 10 contracts per block with many hosts to potentially view the whole network's files - > `docker exec -it sia siac renter setallowance --payment-contract-initial-funding 10SC` - -### Step 4: configuring docker services - -1. edit `/home/user/skynet-webportal/.env` and configure following environment variables - - - `PORTAL_DOMAIN` (required) is a skynet portal domain (ex. siasky.net) - - `SERVER_DOMAIN` (optional) is an optional direct server domain (ex. eu-ger-1.siasky.net) - leave blank unless it is different than PORTAL_DOMAIN - - `EMAIL_ADDRESS` is your email address used for communication regarding SSL certification (required if you're using http-01 challenge) - - `SIA_WALLET_PASSWORD` is your wallet password (or seed if you did not set a password) - - `HSD_API_KEY` this is a random security key for a handshake integration that gets generated automatically - - `CLOUDFLARE_AUTH_TOKEN` (optional) if using cloudflare as dns loadbalancer (need to change it in Caddyfile too) - - `AWS_ACCESS_KEY_ID` (optional) if using route53 as a dns loadbalancer - - `AWS_SECRET_ACCESS_KEY` (optional) if using route53 as a dns loadbalancer - - `DISCORD_WEBHOOK_URL` (required if using Discord notifications) discord webhook url (generate from discord app) - - `DISCORD_MENTION_USER_ID` (optional) add `/cc @user` mention to important messages from webhook (has to be id not user name) - - `DISCORD_MENTION_ROLE_ID` (optional) add `/cc @role` mention to important messages from webhook (has to be id not role name) - - `SKYNET_DB_USER` (optional) if using `accounts` this is the MongoDB username - - `SKYNET_DB_PASS` (optional) if using `accounts` this is the MongoDB password - - `SKYNET_DB_HOST` (optional) if using `accounts` this is the MongoDB address or container name - - `SKYNET_DB_PORT` (optional) if using `accounts` this is the MongoDB port - - `COOKIE_DOMAIN` (optional) if using `accounts` this is the domain to which your cookies will be issued - - `COOKIE_HASH_KEY` (optional) if using `accounts` hashing secret, at least 32 bytes - - `COOKIE_ENC_KEY` (optional) if using `accounts` encryption key, at least 32 bytes - - `S3_BACKUP_PATH` (optional) is using `accounts` and backing up the databases to S3. This path should be an S3 bucket - with path to the location in the bucket where we want to store the daily backups. - -1. `docker-compose up -d` to restart the services so they pick up new env variables - -## Subdomains - -It might prove useful for certain skapps to be accessible through a custom subdomain. So instead of being accessed through `https://portal.com/[skylink]`, it would be accessible through `https://[skylink_base32].portal.com`. We call this "subdomain access" and it is made possible by encoding Skylinks using a base32 encoding. We have to use a base32 encoding scheme because subdomains have to be all lower case and the base64 encoded Skylink is case sensitive and thus might contain uppercase characters. - -You can convert Skylinks using this [converter skapp](https://convert-skylink.hns.siasky.net). To see how the encoding and decoding works, please follow the link to the repo in the application itself. - -There is also an option to access handshake domain through the subdomain using `https://[domain_name].hns.portal.com`. - -To configure this on your portal, you have to make sure to configure the following: - -## Useful Commands - -- Starting the whole stack - > `docker-compose up -d` -- Stopping the whole stack - > `docker-compose down` -- Accessing siac - > `docker exec -it sia siac` -- Portal maintenance - - Pulling portal out for maintenance - > `scripts/portal-down.sh` - - Putting portal back into place after maintenance - > `scripts/portal-up.sh` - - Upgrading portal containers (takes care of pulling it and putting it back) - > `scripts/portal-upgrade.sh` -- Restarting caddy gracefully after making changes to Caddyfile (no downtime) - > `docker exec caddy caddy reload --config /etc/caddy/Caddyfile` -- Restarting nginx gracefully after making changes to nginx configs (no downtime) - > `docker exec nginx openresty -s reload` -- Checking siad service logs (since last hour) - > `docker logs --since 1h $(docker ps -q --filter "name=^sia$")` -- Checking caddy logs (for example in case ssl certificate fails) - > `docker logs caddy -f` -- Checking nginx logs (nginx handles all communication to siad instances) - > `tail -n 50 docker/data/nginx/logs/access.log` to follow last 50 lines of access log - > `tail -n 50 docker/data/nginx/logs/error.log` to follow last 50 lines of error log diff --git a/setup-scripts/setup-docker-services.sh b/setup-scripts/setup-docker-services.sh deleted file mode 100755 index 76c203be..00000000 --- a/setup-scripts/setup-docker-services.sh +++ /dev/null @@ -1,48 +0,0 @@ -#! /usr/bin/env bash - -set -e # exit on first error - -# Install docker (cleans up old docker installation) -# sudo apt-get remove -y docker docker-engine docker.io containerd runc # fails if it is the first installation -sudo apt-get update -sudo apt-get install -y apt-transport-https ca-certificates curl gnupg-agent software-properties-common -curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - -sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable" -sudo apt-get update -sudo apt-get install -y docker-ce docker-ce-cli containerd.io -docker --version # sanity check - -# add user to docker group to avoid having to use sudo for every docker command -sudo usermod -aG docker user - -# Install docker-compose -sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose -sudo chmod +x /usr/local/bin/docker-compose -docker-compose --version # sanity check - -# Create dummy .env file for docker-compose usage with variables -# * PORTAL_DOMAIN - (required) is a skynet portal domain (ex. siasky.net) -# * SERVER_DOMAIN - (optional) is an optional direct server domain (ex. eu-ger-1.siasky.net) - leave blank unless it is different than PORTAL_DOMAIN -# * EMAIL_ADDRESS - this is the administrator contact email you need to supply for communication regarding SSL certification -# * HSD_API_KEY - this is auto generated secure key for your handshake service integration -# * CLOUDFLARE_AUTH_TOKEN - (optional) if using cloudflare as dns loadbalancer (need to change it in Caddyfile too) -# * AWS_ACCESS_KEY_ID - (optional) if using route53 as a dns loadbalancer -# * AWS_SECRET_ACCESS_KEY - (optional) if using route53 as a dns loadbalancer -# * API_PORT - (optional) the port on which siad is listening, defaults to 9980 -# * DISCORD_WEBHOOK_URL - (required if using Discord notifications) discord webhook url (generate from discord app) -# * DISCORD_MENTION_USER_ID - (optional) add `/cc @user` mention to important messages from webhook (has to be id not user name) -# * DISCORD_MENTION_ROLE_ID - (optional) add `/cc @role` mention to important messages from webhook (has to be id not role name) -# * SKYNET_DB_USER - (optional) if using `accounts` this is the MongoDB username -# * SKYNET_DB_PASS - (optional) if using `accounts` this is the MongoDB password -# * SKYNET_DB_HOST - (optional) if using `accounts` this is the MongoDB address or container name -# * SKYNET_DB_PORT - (optional) if using `accounts` this is the MongoDB port -# * COOKIE_DOMAIN - (optional) if using `accounts` this is the domain to which your cookies will be issued -# * COOKIE_HASH_KEY - (optional) if using `accounts` hashing secret, at least 32 bytes -# * COOKIE_ENC_KEY - (optional) if using `accounts` encryption key, at least 32 bytes -if ! [ -f /home/user/skynet-webportal/.env ]; then - HSD_API_KEY=$(openssl rand -base64 32) # generate safe random key for handshake - printf "PORTAL_DOMAIN=siasky.net\nSERVER_DOMAIN=\nEMAIL_ADDRESS=email@example.com\nSIA_WALLET_PASSWORD=\nHSD_API_KEY=${HSD_API_KEY}\nCLOUDFLARE_AUTH_TOKEN=\nAWS_ACCESS_KEY_ID=\nAWS_SECRET_ACCESS_KEY=\nDISCORD_WEBHOOK_URL=\nDISCORD_MENTION_USER_ID=\nDISCORD_MENTION_ROLE_ID=\n" > /home/user/skynet-webportal/.env -fi - -# Start docker container with nginx and client -docker-compose -f docker-compose.yml up --build -d diff --git a/setup-scripts/setup-health-check-scripts.sh b/setup-scripts/setup-health-check-scripts.sh deleted file mode 100755 index 6cac2c9e..00000000 --- a/setup-scripts/setup-health-check-scripts.sh +++ /dev/null @@ -1,11 +0,0 @@ -#! /usr/bin/env bash - -set -e # exit on first error - -sudo apt-get update -sudo apt-get -y install python3-pip - -pip3 install discord-webhook python-dotenv requests elasticsearch-curator - -# add cron entries to user crontab -crontab -u user /home/user/skynet-webportal/setup-scripts/support/crontab diff --git a/setup-scripts/setup-server.sh b/setup-scripts/setup-server.sh deleted file mode 100755 index c2899d7d..00000000 --- a/setup-scripts/setup-server.sh +++ /dev/null @@ -1,48 +0,0 @@ -#! /usr/bin/env bash - -set -e # exit on first error - -# Copy over basic configuration files -cp /home/user/skynet-webportal/setup-scripts/support/tmux.conf /home/user/.tmux.conf -cp /home/user/skynet-webportal/setup-scripts/support/bashrc /home/user/.bashrc -source /home/user/.bashrc - -# Add SSH keys and set SSH configs -sudo cp /home/user/skynet-webportal/setup-scripts/support/ssh_config /etc/ssh/ssh_config -mkdir -p /home/user/.ssh -# cat /home/user/skynet-webportal/setup-scripts/support/authorized_keys >> /home/user/.ssh/authorized_keys - -# Install apt packages -sudo apt-get update -sudo apt-get -y install ufw tmux ranger htop nload gcc g++ make git vim unzip curl awscli - -# Setup GIT credentials (so commands like git stash would work) -git config --global user.email "devs@nebulous.tech" -git config --global user.name "Sia Dev" - -# Setup firewall -sudo ufw --force enable # --force to make it non-interactive -sudo ufw logging low # enable logging for debugging purpose: tail -f /var/log/ufw.log -sudo ufw allow ssh # allow ssh connection to server -sudo ufw allow 80,443/tcp # allow http and https ports - -# Block outgoing traffic to local networks -# https://community.hetzner.com/tutorials/block-outgoing-traffic-to-private-networks -sudo ufw deny out from any to 10.0.0.0/8 -sudo ufw deny out from any to 172.16.0.0/12 -sudo ufw deny out from any to 192.168.0.0/16 -sudo ufw deny out from any to 100.64.0.0/10 -sudo ufw deny out from any to 198.18.0.0/15 -sudo ufw deny out from any to 169.254.0.0/16 - -# OPTIONAL: terminfo for alacritty terminal via ssh -# If you don't use the alacritty terminal you can remove this step. -wget -c https://raw.githubusercontent.com/alacritty/alacritty/master/extra/alacritty.info -sudo tic -xe alacritty,alacritty-direct alacritty.info -rm alacritty.info - -# Set up file limits - siad uses a lot so we need to adjust so it doesn't choke up -sudo cp /home/user/skynet-webportal/setup-scripts/support/limits.conf /etc/security/limits.conf - -# Set UTC timezone so all of the servers report the same time -sudo timedatectl set-timezone UTC