This repository has been archived on 2022-10-07. You can view files and clone it, but cannot push or open issues or pull requests.
skynet-webportal/setup-scripts
Karol Wypchlo 6acd3309f8 fix blacklist command 2020-09-14 14:45:06 +02:00
..
support drop Marcin's key 2020-08-28 14:56:01 +02:00
README.md Remove unneeded comment and example commands. 2020-08-19 16:13:56 +03:00
blacklist-skylink.sh fix blacklist command 2020-09-14 14:45:06 +02:00
bot_utils.py Mention role in discord, rather than online users 2020-04-14 14:50:25 -04:00
funds-checker.py Add a health checker script to Gollum. 2020-09-04 16:12:20 +02:00
health-checker.py fix health checks 2020-09-10 15:16:31 +02:00
log-checker.py Add free disk space check to health-checker.py. 2020-09-07 17:56:47 +02:00
setup-docker-services.sh Switch from `/home/user/.sia/sia.env` to `/home/user/skynet-webportal/.env`. 2020-08-19 19:14:55 +03:00
setup-health-check-scripts.sh Make the time comparisons in the health checker timezone-aware. 2020-09-08 18:07:33 +02:00
setup-server.sh docker setup 2020-07-31 00:50:42 +02:00
stats-logger.sh Improve server infrastructure setup and scripts (#231) 2020-06-22 11:54:01 +02:00

README.md

Skynet Portal Setup Scripts

This directory contains a setup guide and scripts that will install and configure some basic requirements for running a Skynet Portal. The assumption is that we are working with a Debian Buster Minimal system or similar.

Initial Setup

You may want to fork this repository and replace ssh keys in setup-scripts/support/authorized_keys and optionally edit the setup-scripts/support/tmux.conf and setup-scripts/support/bashrc configurations to fit your needs.

Step 0: stack overview

  • dockerized services inside docker-compose.yml
    • sia (docker hub): storage provider, heart of the portal setup
    • caddy (docker hub): reverse proxy (similar to nginx) that handles ssl out of a box and acts as a transparent entry point
    • openresty (docker hub): nginx custom build, acts as a cached proxy to siad and exposes all api endpoints
    • health-check: this is a simple service that runs periodically and collects health data about the server (status and response times) and exposes /health-check api endpoint that is deliberately delayed based on the response times of the server so potential load balancer could prioritize servers based on that (we use it with cloudflare)
  • siad setup: we use "double siad" setup that has one node solely for download and one for upload to improve performance
    • we use systemd to manage siad service
    • siad is not installed as docker service for improved performance
  • discord integration
    • funds-checker: script that checks wallet balance and sends status messages to discord periodically
    • log-checker: script that scans siad logs for critical errors and reports them to discord periodically
  • blacklist-skylink: script that can be run locally from a machine that has access to all your skynet portal servers that blacklists provided skylink and prunes nginx cache to ensure it's not available any more (that is a bit much but that's the best we can do right now without paid nginx version) - if you want to use it, make sure to adjust the server addresses

Step 1: setting up server user

  1. SSH in a freshly installed Debian machine on a user with sudo access (can be root)
  2. apt-get update && apt-get install sudo to make sure sudo is available
  3. adduser user to create user called user (creates /home/user directory)
  4. usermod -a -G sudo user to add this new user to sudo group
  5. Quit the ssh session with exit command

You a can now ssh into your machine as the user user.

Step 2: setting up environment

  1. On your local machine: ssh-copy-id user@ip-addr to copy over your ssh key to server
  2. On your local machine: ssh user@ip-addr to log in to server as user user
  3. You are now logged in as user

Following step will be executed on remote host logged in as a user:

  1. sudo apt-get install git to install git
  2. git clone https://github.com/NebulousLabs/skynet-webportal
  3. run setup scripts in the exact order and provide sudo password when asked (if one of them fails, you can retry just this one before proceeding further)
    1. /home/user/skynet-webportal/setup-scripts/setup-server.sh
    2. /home/user/skynet-webportal/setup-scripts/setup-docker-services.sh
    3. /home/user/skynet-webportal/setup-scripts/setup-health-check-scripts.sh (optional)

Step 3: configuring siad

At this point we have almost everything running, we just need to set up your wallet and allowance:

  1. Create new wallet (remember to save the seeds)

    docker exec -it sia siac wallet init

  2. Unlock wallet (use seed as password)

    docker exec -it sia siac wallet unlock

  3. Generate wallet addresse (save them for later to transfer the funds)

    docker exec -it sia siac wallet address

  4. Set up allowance

    docker exec -it sia siac renter setallowance

    1. 10 KS (keep 25 KS in your wallet)
    2. default period
    3. default number of hosts
    4. 8 week renewal time
    5. 500 GB expected storage
    6. 500 GB expected upload
    7. 5 TB expected download
    8. default redundancy
  5. Instruct siad to start making 10 contracts per block with many hosts to potentially view the whole network's files

    docker exec -it sia siac renter setallowance --payment-contract-initial-funding 10SC

Step 4: configuring docker services

  1. generate and copy sia api token printf ":$(cat /home/user/.sia/apipassword)" | base64
  2. edit /home/user/skynet-webportal/.env and configure following environment variables
    • DOMAIN_NAME (optional) is your domain name if you have it
    • EMAIL_ADDRESS (required) is your email address used for communication regarding SSL certification (required)
    • SIA_WALLET_PASSWORD (required) is your wallet password (or seed if you did not set a password)
    • HSD_API_KEY (optional) this is a random security key for an optional handshake integration that gets generated automatically
    • CLOUDFLARE_AUTH_TOKEN (optional) if using cloudflare as dns loadbalancer (need to change it in Caddyfile too)
    • AWS_ACCESS_KEY_ID (optional) if using route53 as a dns loadbalancer
    • AWS_SECRET_ACCESS_KEY (optional) if using route53 as a dns loadbalancer
  3. if you have a custom domain and you configured it in DOMAIN_NAME, edit /home/user/skynet-webportal/docker/caddy/Caddyfile and uncomment import custom.domain
  4. only for siasky.net domain instances: edit /home/user/skynet-webportal/docker/caddy/Caddyfile, uncomment import siasky.net
  5. docker-compose up -d to restart the services so they pick up new env variables
  6. docker exec caddy caddy reload --config /etc/caddy/Caddyfile to reload Caddyfile configuration

Useful Commands

  • Accessing siac

    docker exec -it sia siac

  • Checking status of siad service

    systemctl --user status siad

  • Stopping siad service

    systemctl --user stop siad

  • Starting siad service

    systemctl --user start siad

  • Restarting siad service

    systemctl --user restart siad

  • Restarting caddy gracefully after making changes to Caddyfile

    docker exec caddy caddy reload --config /etc/caddy/Caddyfile

  • Restarting nginx gracefully after making changes to nginx configs

    docker exec nginx openresty -s reload

  • Checking siad service logs (last hour)

    docker logs --since 1h $(docker ps -q --filter "name=^sia$")

  • Checking caddy logs (for example in case ssl certificate fails)

    docker logs caddy -f

  • Checking nginx logs (nginx handles all communication to siad instances)

    tail -n 50 docker/data/nginx/logs/access.log to follow last 50 lines of access log tail -n 50 docker/data/nginx/logs/error.log to follow last 50 lines of error log