skynet-webportal/setup-scripts

Skynet Portal Setup Scripts

This directory contains a setup guide and scripts that will install and configure some basic requirements for running a Skynet Portal. The assumption is that we are working with a Debian Buster Minimal system or similar.

Initial Setup

You may want to fork this repository and replace ssh keys in setup-scripts/support/authorized_keys and optionally edit the setup-scripts/support/tmux.conf and setup-scripts/support/bashrc configurations to fit your needs.

Step 0: stack overview

• dockerized services inside docker-compose.yml
  • docker-host (docker hub): service that exposes server ip to docker container so we could access siad from within the nginx container
  • caddy (docker hub): reverse proxy (similar to nginx) that handles ssl out of a box and acts as an entry point
  • openresty (docker hub): nginx custom build, acts as a cached proxy to siad (we only use it because caddy doesn't support proxy caching, otherwise we could drop it)
  • health-check: this is a simple service that runs periodically and collects health data about the server (status and response times) and exposes /health-check api endpoint that is deliberately delayed based on the response times of the server so potential load balancer could prioritize servers based on that (we use it with cloudflare)
• siad setup: we use "double siad" setup that has one node solely for download and one for upload to improve performance
  • we use systemd to manage siad services
  • siad is not installed as docker service for improved performance
• discord integration
  • funds-checker: script that checks wallet balance and sends status messages to discord periodically
  • log-checker: script that scans siad logs for critical errors and reports them to discord periodically
• blacklist-skylink: script that can be run locally from a machine that has access to all your skynet portal servers that blacklists provided skylink and prunes nginx cache to ensure it's not available any more (that is a bit much but that's the best we can do right now without paid nginx version) - if you want to use it, make sure to adjust the server addresses

Step 1: setting up server user

1. SSH in a freshly installed Debian machine on a user with sudo access (can be root)
2. apt-get update && apt-get install sudo to make sure sudo is available
3. adduser user to create user called user (creates /home/user directory)
4. usermod -a -G sudo user to add this new user to sudo group
5. usermod -a -G systemd-journal user to add this new user to systemd-journal group
6. Quit the ssh session with exit command

You a can now ssh into your machine as the user user.

Step 2: setting up environment

1. On your local machine: ssh-copy-id user@ip-addr to copy over your ssh key to server
2. On your local machine: ssh user@ip-addr to log in to server as user user
3. You are now logged in as user

Following step will be executed on remote host logged in as a user:

1. sudo apt-get install git to install git
2. git clone https://github.com/NebulousLabs/skynet-webportal
3. run setup scripts in the exact order and provide sudo password when asked (if one of them fails, you can retry just this one before proceeding further)
   1. /home/user/skynet-webportal/setup-scripts/setup-server.sh
   2. /home/user/skynet-webportal/setup-scripts/setup-siad.sh
   3. /home/user/skynet-webportal/setup-scripts/setup-docker-services.sh
   4. /home/user/skynet-webportal/setup-scripts/setup-health-check-scripts.sh (optional)

Step 3: configuring siad

At this point we have almost everything set up. We have 2 siad instances running as services and we need to set up the wallets and allowance on those.

1. Create new wallet for both siad instances (remember to save the seeds)
   1. siac wallet init to init download node wallet
   2. siac-upload wallet init to init upload node wallet
2. Unlock both wallets
   1. siac wallet unlock to unlock download node wallet (use seed as password)
   2. siac-upload wallet unlock to unlock upload node wallet (use seed as password)
3. Generate wallet addresses for both siad instances (save them for later to transfer the funds)
   1. siac wallet address to generate address for download node wallet
   2. siac-upload wallet address to generate address for upload node wallet
4. Set up allowance on both siad instances
   1. siac renter setallowance to set allowance on download node
      1. 10 KS (keep 25 KS in your wallet)
      2. default period
      3. default number of hosts
      4. 8 week renewal time
      5. 500 GB expected storage
      6. 500 GB expected upload
      7. 5 TB expected download
      8. default redundancy
   2. siac-upload renter setallowance to set allowance on upload node
      1. use the same allowance settings as download node
5. Run siac renter setallowance --payment-contract-initial-funding 10SC so that your download node will start making 10 contracts per block with many hosts to potentially view the whole network's files
6. Copy over apipassword from /home/user/.sia/apipassword and save it for the next step
7. Edit environment files for both siad instances
   1. /home/user/.sia/sia.env for the download node
      1. SIA_API_PASSWORD to previously copied apipassword (same for both instances)
      2. SIA_WALLET_PASSWORD to be the wallet seed
      3. PORTAL_NAME xxxxed part to some meaningful name like warsaw.siasky.net
      4. DISCORD_BOT_TOKEN for discord health check scripts integration
   2. /home/user/.sia/sia-upload.env for the upload node
      1. SIA_API_PASSWORD to previously copied apipassword (same for both instances)
      2. SIA_WALLET_PASSWORD to be the wallet seed
      3. PORTAL_NAME xxxxed part to some meaningful name like warsaw.siasky.net
      4. DISCORD_BOT_TOKEN for discord health check scripts integration

Step 4: configuring docker services

1. generate and copy sia api token printf ":$(cat /home/user/.sia/apipassword)" | base64
2. edit /home/user/skynet-webportal/.env and configure following environment variables
   • DOMAIN_NAME (optional) is your domain name if you have it
   • EMAIL_ADDRESS (required) is your email address used for communication regarding SSL certification (required)
   • SIA_API_AUTHORIZATION (required) is token you just generated in the previous point
   • CLOUDFLARE_AUTH_TOKEN (optional) if using cloudflare as dns loadbalancer (it's just for siasky.net configuration)
   • HSD_API_KEY (optional) this is a random security key for an optional handshake integration that gets generated automatically
3. if you have a custom domain and you configured it in DOMAIN_NAME, edit /home/user/skynet-webportal/docker/caddy/Caddyfile and uncomment import custom.domain
4. only for siasky.net domain instances: edit /home/user/skynet-webportal/docker/caddy/Caddyfile, uncomment import siasky.net
5. sudo docker-compose up -d to restart the services so they pick up new env variables
6. sudo docker exec caddy caddy reload --config /etc/caddy/Caddyfile to reload Caddyfile configuration

Useful Commands

• Accessing siac for both nodes
  • siac for download node
  • siac-upload for upload node
• Checking status of siad service
  • systemctl --user status siad for download node
  • systemctl --user status siad-upload for upload node
• Stopping siad service
  • systemctl --user stop siad for download node
  • systemctl --user stop siad-upload for upload node
• Starting siad service
  • systemctl --user start siad for download node
  • systemctl --user start siad-upload for upload node
• Restarting siad service
  • systemctl --user restart siad for download node
  • systemctl --user restart siad-upload for upload node
• Restarting caddy gracefully after making changes to Caddyfile
  • sudo docker exec caddy caddy reload --config /etc/caddy/Caddyfile
• Restarting nginx gracefully after making changes to nginx configs
  • sudo docker exec nginx openresty -s reload
• Checking siad service logs (follow last 50 lines)
  • journalctl -f -n 50 --user-unit siad for download node
  • journalctl -f -n 50 --user-unit siad-upload for upload node
• Checking caddy logs (for example in case ssl certificate fails)
  • sudo docker logs caddy -f
• Checking nginx logs (nginx handles all communication to siad instances)
  • tail -n 50 docker/data/nginx/logs/access.log to follow last 50 lines of access log
  • tail -n 50 docker/data/nginx/logs/error.log to follow last 50 lines of error log