e53a8ba46a | ||
---|---|---|
.. | ||
support | ||
README.md | ||
bot_utils.py | ||
disk-usage-dump.sh | ||
funds-checker.py | ||
health-checker.py | ||
log-checker.py | ||
serverload.service | ||
serverload.sh | ||
setup-docker-services.sh | ||
setup-health-check-scripts.sh | ||
setup-server.sh |
README.md
Skynet Portal Setup Scripts
⚠️ This documentation is outdated and should be used for reference only. Portal setup documentation is located at https://portal-docs.skynetlabs.com/.
This directory contains a setup guide and scripts that will install and configure some basic requirements for running a Skynet Portal. The assumption is that we are working with a Debian Buster Minimal system or similar.
Latest Setup Documentation
Latest Skynet Webportal setup documentation and the setup process Skynet Labs supports is located at https://docs.siasky.net/webportal-management/overview.
Some of the scripts and setup documentation contained in this repository
(skynet-webportal
) can be outdated and generally should not be used.
Initial Setup
You may want to fork this repository and replace ssh keys in
setup-scripts/support/authorized_keys
and optionally edit the setup-scripts/support/tmux.conf
and setup-scripts/support/bashrc
configurations to fit your needs.
Step 0: stack overview
- dockerized services inside
docker-compose.yml
- sia (docker hub): storage provider, heart of the portal setup
- caddy (docker hub): reverse proxy (similar to nginx) that handles ssl out of a box and acts as a transparent entry point
- openresty (docker hub): nginx custom build, acts as a cached proxy to siad and exposes all api endpoints
- health-check: simple service that runs periodically and collects health data about the server (status and response times) - read more
- handshake (github): full handshake node
- handshake-api: simple API talking to the handshake node - read more
- website: portal frontend application - read more
- discord integration
- funds-checker: script that checks wallet balance and sends status messages to discord periodically
- health-checker: script that monitors health-check service for server health issues and reports them to discord periodically
- log-checker: script that scans siad logs for critical errors and reports them to discord periodically
- blocklist-skylink: script that can be run locally from a machine that has access to all your skynet portal servers that blocklists provided skylink and prunes nginx cache to ensure it's not available any more (that is a bit much but that's the best we can do right now without paid nginx version) - if you want to use it, make sure to adjust the server addresses
Step 1: setting up server user
- SSH in a freshly installed Debian machine on a user with sudo access (can be root)
apt-get update && apt-get install sudo libnss3-tools -y
to make suresudo
is availableadduser user
to create user calleduser
(creates/home/user
directory)usermod -aG sudo user
to add this new user to sudo groupsudo groupadd docker
to create a group for docker (it might already exist)sudo usermod -aG docker user
to add your user to that group- Quit the ssh session with
exit
command
You can now ssh into your machine as the user user
.
Step 2: setting up environment
- On your local machine:
ssh-copy-id user@ip-addr
to copy over your ssh key to server - On your local machine:
ssh user@ip-addr
to log in to server as useruser
- You are now logged in as
user
Following step will be executed on remote host logged in as a user
:
sudo apt-get install git -y
to install gitgit clone https://github.com/SkynetLabs/skynet-webportal
cd skynet-webportal
- run setup scripts in the exact order and provide sudo password when asked (if one of them fails, you can retry just this one before proceeding further)
/home/user/skynet-webportal/setup-scripts/setup-server.sh
/home/user/skynet-webportal/setup-scripts/setup-docker-services.sh
/home/user/skynet-webportal/setup-scripts/setup-health-check-scripts.sh
(optional)
Step 3: configuring siad
At this point we have almost everything running, we just need to set up your wallet and allowance:
- Create a new wallet (remember to save the seed)
docker exec -it sia siac wallet init
- Unlock the wallet (use the seed as password)
docker exec -it sia siac wallet unlock
- Generate a new wallet address (save it for later to transfer the funds)
docker exec -it sia siac wallet address
- Set up allowance
docker exec -it sia siac renter setallowance
- 10 KS (keep 25 KS in your wallet)
- default period
- default number of hosts
- 4 week renewal time
- 500 GB expected storage
- 500 GB expected upload
- 5 TB expected download
- default redundancy
- Set a maximum storage price
docker exec -it sia siac renter setallowance --max-storage-price 100SC
- Instruct siad to start making 10 contracts per block with many hosts to potentially view the whole network's files
docker exec -it sia siac renter setallowance --payment-contract-initial-funding 10SC
Step 4: configuring docker services
-
edit
/home/user/skynet-webportal/.env
and configure following environment variablesPORTAL_DOMAIN
(required) is a skynet portal domain (ex. siasky.net)SERVER_DOMAIN
(optional) is an optional direct server domain (ex. eu-ger-1.siasky.net) - leave blank unless it is different than PORTAL_DOMAINEMAIL_ADDRESS
is your email address used for communication regarding SSL certification (required if you're using http-01 challenge)SIA_WALLET_PASSWORD
is your wallet password (or seed if you did not set a password)HSD_API_KEY
this is a random security key for a handshake integration that gets generated automaticallyCLOUDFLARE_AUTH_TOKEN
(optional) if using cloudflare as dns loadbalancer (need to change it in Caddyfile too)AWS_ACCESS_KEY_ID
(optional) if using route53 as a dns loadbalancerAWS_SECRET_ACCESS_KEY
(optional) if using route53 as a dns loadbalancerDISCORD_WEBHOOK_URL
(required if using Discord notifications) discord webhook url (generate from discord app)DISCORD_MENTION_USER_ID
(optional) add/cc @user
mention to important messages from webhook (has to be id not user name)DISCORD_MENTION_ROLE_ID
(optional) add/cc @role
mention to important messages from webhook (has to be id not role name)SKYNET_DB_USER
(optional) if usingaccounts
this is the MongoDB usernameSKYNET_DB_PASS
(optional) if usingaccounts
this is the MongoDB passwordSKYNET_DB_HOST
(optional) if usingaccounts
this is the MongoDB address or container nameSKYNET_DB_PORT
(optional) if usingaccounts
this is the MongoDB portCOOKIE_DOMAIN
(optional) if usingaccounts
this is the domain to which your cookies will be issuedCOOKIE_HASH_KEY
(optional) if usingaccounts
hashing secret, at least 32 bytesCOOKIE_ENC_KEY
(optional) if usingaccounts
encryption key, at least 32 bytesS3_BACKUP_PATH
(optional) is usingaccounts
and backing up the databases to S3. This path should be an S3 bucket with path to the location in the bucket where we want to store the daily backups.
-
docker-compose up -d
to restart the services so they pick up new env variables
Subdomains
It might prove useful for certain skapps to be accessible through a custom subdomain. So instead of being accessed through https://portal.com/[skylink]
, it would be accessible through https://[skylink_base32].portal.com
. We call this "subdomain access" and it is made possible by encoding Skylinks using a base32 encoding. We have to use a base32 encoding scheme because subdomains have to be all lower case and the base64 encoded Skylink is case sensitive and thus might contain uppercase characters.
You can convert Skylinks using this converter skapp. To see how the encoding and decoding works, please follow the link to the repo in the application itself.
There is also an option to access handshake domain through the subdomain using https://[domain_name].hns.portal.com
.
To configure this on your portal, you have to make sure to configure the following:
Useful Commands
- Starting the whole stack
docker-compose up -d
- Stopping the whole stack
docker-compose down
- Accessing siac
docker exec -it sia siac
- Portal maintenance
- Pulling portal out for maintenance
scripts/portal-down.sh
- Putting portal back into place after maintenance
scripts/portal-up.sh
- Upgrading portal containers (takes care of pulling it and putting it back)
scripts/portal-upgrade.sh
- Pulling portal out for maintenance
- Restarting caddy gracefully after making changes to Caddyfile (no downtime)
docker exec caddy caddy reload --config /etc/caddy/Caddyfile
- Restarting nginx gracefully after making changes to nginx configs (no downtime)
docker exec nginx openresty -s reload
- Checking siad service logs (since last hour)
docker logs --since 1h $(docker ps -q --filter "name=^sia$")
- Checking caddy logs (for example in case ssl certificate fails)
docker logs caddy -f
- Checking nginx logs (nginx handles all communication to siad instances)
tail -n 50 docker/data/nginx/logs/access.log
to follow last 50 lines of access logtail -n 50 docker/data/nginx/logs/error.log
to follow last 50 lines of error log