LumeWeb
/
skynet-webportal
Archived
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge branch 'master' into add-shell-scripts-lint

This commit is contained in:
Karol Wypchło 2022-09-20 15:21:10 +02:00 committed by GitHub
parent a43cddb334 e4cd4bf991
commit 4da5c6c696
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
11 changed files with 5 additions and 360 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

33
README.md
View File

@ -8,23 +8,6 @@ supports is located at https://portal-docs.skynetlabs.com/.
Some scripts and setup documentation contained in this repository Some scripts and setup documentation contained in this repository
(`skynet-webportal`) may be outdated and generally should not be used. (`skynet-webportal`) may be outdated and generally should not be used.
## Web application
Change current directory with `cd packages/website`.
Use `yarn start` to start the development server.
Use `yarn build` to compile the application to `/public` directory.
You can use the below build parameters to customize your web application.
- development example `GATSBY_API_URL=https://siasky.dev yarn start`
- production example `GATSBY_API_URL=https://siasky.net yarn build`
List of available parameters:
- `GATSBY_API_URL`: override api url (defaults to location origin)
## License ## License
Skynet uses a custom [License](./LICENSE.md). The Skynet License is a source code license that allows you to use, modify Skynet uses a custom [License](./LICENSE.md). The Skynet License is a source code license that allows you to use, modify
@ -33,19 +16,3 @@ and distribute the software, but you must preserve the payment mechanism in the
For the purposes of complying with our code license, you can use the following Siacoin address: For the purposes of complying with our code license, you can use the following Siacoin address:
`fb6c9320bc7e01fbb9cd8d8c3caaa371386928793c736837832e634aaaa484650a3177d6714a` `fb6c9320bc7e01fbb9cd8d8c3caaa371386928793c736837832e634aaaa484650a3177d6714a`
## Running a Portal
For those interested in running a Webportal, head over to our developer docs [here](https://portal-docs.skynetlabs.com/) to learn more.
## Contributing
### Testing Your Code
Before pushing your code, you should verify that it will pass our online test suite.
**Cypress Tests**
Verify the Cypress test suite by doing the following:
1. In one terminal screen run `GATSBY_API_URL=https://siasky.net website serve`
1. In a second terminal screen run `yarn cypress run`

6
docker-compose.jaeger.yml
View File

@ -21,7 +21,7 @@ services:
- JAEGER_REPORTER_LOG_SPANS=false - JAEGER_REPORTER_LOG_SPANS=false
jaeger-agent: jaeger-agent:
image: jaegertracing/jaeger-agent:1.37.0 image: jaegertracing/jaeger-agent:1.38.0
command: command:
[ [
"--reporter.grpc.host-port=jaeger-collector:14250", "--reporter.grpc.host-port=jaeger-collector:14250",
@ -43,7 +43,7 @@ services:
- jaeger-collector - jaeger-collector
jaeger-collector: jaeger-collector:
image: jaegertracing/jaeger-collector:1.37.0 image: jaegertracing/jaeger-collector:1.38.0
entrypoint: /wait_to_start.sh entrypoint: /wait_to_start.sh
container_name: jaeger-collector container_name: jaeger-collector
restart: on-failure restart: on-failure
@ -68,7 +68,7 @@ services:
- elasticsearch - elasticsearch
jaeger-query: jaeger-query:
image: jaegertracing/jaeger-query:1.37.0 image: jaegertracing/jaeger-query:1.38.0
entrypoint: /wait_to_start.sh entrypoint: /wait_to_start.sh
container_name: jaeger-query container_name: jaeger-query
restart: on-failure restart: on-failure

2
docker-compose.pinner.yml
View File

@ -10,7 +10,7 @@ services:
pinner: pinner:
# uncomment "build" and comment out "image" to build from sources # uncomment "build" and comment out "image" to build from sources
# build: https://github.com/SkynetLabs/pinner.git#main # build: https://github.com/SkynetLabs/pinner.git#main
image: skynetlabs/pinner:0.7.5 image: skynetlabs/pinner:0.7.6
container_name: pinner container_name: pinner
restart: unless-stopped restart: unless-stopped
logging: *default-logging logging: *default-logging

2
docker-compose.yml
View File

@ -21,7 +21,7 @@ services:
# dockerfile: scratch/Dockerfile # dockerfile: scratch/Dockerfile
# args: # args:
# branch: master # branch: master
image: skynetlabs/skyd:1.6.8 image: skynetlabs/skyd:1.6.9
command: --disable-api-security --api-addr :9980 --modules gctwra command: --disable-api-security --api-addr :9980 --modules gctwra
container_name: sia container_name: sia
restart: unless-stopped restart: unless-stopped

16
scripts/maintenance-upgrade.sh
View File

@ -1,16 +0,0 @@
#! /usr/bin/env bash
###############################################################
# this script is an automation for restarting docker containers
# on maintenance nodes strictly built for purpose of siasky.net
###############################################################
set -e # exit on first error
docker build --no-cache --quiet --build-arg branch=master -t sia-master /home/user/sia-dockerfile
for container in `docker container ls --format '{{.Names}}'`; do
docker stop $container
docker rm $container
docker run -d -v /home/user/nodes/$container/sia-data:/sia-data --env-file /home/user/nodes/$container/.env --name $container --log-opt max-size=100m --log-opt max-file=3 sia-master
done

18
scripts/portal-restart.sh
View File

@ -1,18 +0,0 @@
#!/bin/bash
set -e # exit on first error
# get current working directory (pwd doesn't cut it)
cwd=$(cd -P -- "$(dirname -- "$0")" && pwd -P)
# put the server down for maintenance
. ${cwd}/portal-down.sh
# stop the docker services
docker-compose down
# start the docker services
docker-compose up -d
# enable the server again
. ${cwd}/portal-up.sh

26
scripts/portal-upgrade.sh
View File

@ -1,26 +0,0 @@
#!/bin/bash
set -e # exit on first error
# get current working directory (pwd doesn't cut it)
cwd=$(cd -P -- "$(dirname -- "$0")" && pwd -P)
# put the server down for maintenance
. ${cwd}/portal-down.sh
# build all container without cache
docker-compose build --no-cache --parallel --pull --quiet
# stop the docker services
docker-compose down -v
# clear unused docker containers so we don't run into out of disk space
# it should be done after the container have been stopped and before
# building them again
docker system prune --force
# start the docker services
docker-compose up -d
# enable the server again
. ${cwd}/portal-up.sh

155
setup-scripts/README.md
View File

@ -1,155 +0,0 @@
# Skynet Portal Setup Scripts
> :warning: This documentation is outdated and should be used for reference
only. Portal setup documentation is located at
https://portal-docs.skynetlabs.com/.
This directory contains a setup guide and scripts that will install and
configure some basic requirements for running a Skynet Portal. The assumption is
that we are working with a Debian Buster Minimal system or similar.
## Latest Setup Documentation
Latest Skynet Webportal setup documentation and the setup process Skynet Labs
supports is located at https://docs.siasky.net/webportal-management/overview.
Some of the scripts and setup documentation contained in this repository
(`skynet-webportal`) can be outdated and generally should not be used.
## Initial Setup
You may want to fork this repository and replace ssh keys in
`setup-scripts/support/authorized_keys` and optionally edit the `setup-scripts/support/tmux.conf` and `setup-scripts/support/bashrc` configurations to fit your needs.
### Step 0: stack overview
- dockerized services inside `docker-compose.yml`
- [sia](https://sia.tech) ([docker hub](https://hub.docker.com/r/nebulouslabs/sia)): storage provider, heart of the portal setup
- [caddy](https://caddyserver.com) ([docker hub](https://hub.docker.com/r/caddy/caddy)): reverse proxy (similar to nginx) that handles ssl out of a box and acts as a transparent entry point
- [openresty](https://openresty.org) ([docker hub](https://hub.docker.com/r/openresty/openresty)): nginx custom build, acts as a cached proxy to siad and exposes all api endpoints
- [health-check](https://github.com/SkynetLabs/skynet-webportal/tree/master/packages/health-check): simple service that runs periodically and collects health data about the server (status and response times) - [read more](https://github.com/SkynetLabs/skynet-webportal/blob/master/packages/health-check/README.md)
- [handshake](https://handshake.org) ([github](https://github.com/handshake-org/hsd)): full handshake node
- [handshake-api](https://github.com/SkynetLabs/skynet-webportal/tree/master/packages/handshake-api): simple API talking to the handshake node - [read more](https://github.com/SkynetLabs/skynet-webportal/blob/master/packages/handshake-api/README.md)
- [website](https://github.com/SkynetLabs/skynet-webportal/tree/master/packages/website): portal frontend application - [read more](https://github.com/SkynetLabs/skynet-webportal/blob/master/packages/website/README.md)
- discord integration
- [funds-checker](funds-checker.py): script that checks wallet balance and sends status messages to discord periodically
- [health-checker](health-checker.py): script that monitors health-check service for server health issues and reports them to discord periodically
- [log-checker](log-checker.py): script that scans siad logs for critical errors and reports them to discord periodically
- [blocklist-skylink](../scripts/blocklist-skylink.sh): script that can be run locally from a machine that has access to all your skynet portal servers that blocklists provided skylink and prunes nginx cache to ensure it's not available any more (that is a bit much but that's the best we can do right now without paid nginx version) - if you want to use it, make sure to adjust the server addresses
### Step 1: setting up server user
1. SSH in a freshly installed Debian machine on a user with sudo access (can be root)
1. `apt-get update && apt-get install sudo libnss3-tools -y` to make sure `sudo` is available
1. `adduser user` to create user called `user` (creates `/home/user` directory)
1. `usermod -aG sudo user` to add this new user to sudo group
1. `sudo groupadd docker` to create a group for docker (it might already exist)
1. `sudo usermod -aG docker user` to add your user to that group
1. Quit the ssh session with `exit` command
You can now ssh into your machine as the user `user`.
### Step 2: setting up environment
1. On your local machine: `ssh-copy-id user@ip-addr` to copy over your ssh key to server
1. On your local machine: `ssh user@ip-addr` to log in to server as user `user`
1. You are now logged in as `user`
**Following step will be executed on remote host logged in as a `user`:**
1. `sudo apt-get install git -y` to install git
1. `git clone https://github.com/SkynetLabs/skynet-webportal`
1. `cd skynet-webportal`
1. run setup scripts in the exact order and provide sudo password when asked (if one of them fails, you can retry just this one before proceeding further)
1. `/home/user/skynet-webportal/setup-scripts/setup-server.sh`
1. `/home/user/skynet-webportal/setup-scripts/setup-docker-services.sh`
1. `/home/user/skynet-webportal/setup-scripts/setup-health-check-scripts.sh` (optional)
### Step 3: configuring siad
At this point we have almost everything running, we just need to set up your wallet and allowance:
1. Create a new wallet (remember to save the seed)
> `docker exec -it sia siac wallet init`
1. Unlock the wallet (use the seed as password)
> `docker exec -it sia siac wallet unlock`
1. Generate a new wallet address (save it for later to transfer the funds)
> `docker exec -it sia siac wallet address`
1. Set up allowance
> `docker exec -it sia siac renter setallowance`
1. 10 KS (keep 25 KS in your wallet)
1. default period
1. default number of hosts
1. 4 week renewal time
1. 500 GB expected storage
1. 500 GB expected upload
1. 5 TB expected download
1. default redundancy
1. Set a maximum storage price
> `docker exec -it sia siac renter setallowance --max-storage-price 100SC`
1. Instruct siad to start making 10 contracts per block with many hosts to potentially view the whole network's files
> `docker exec -it sia siac renter setallowance --payment-contract-initial-funding 10SC`
### Step 4: configuring docker services
1. edit `/home/user/skynet-webportal/.env` and configure following environment variables
- `PORTAL_DOMAIN` (required) is a skynet portal domain (ex. siasky.net)
- `SERVER_DOMAIN` (optional) is an optional direct server domain (ex. eu-ger-1.siasky.net) - leave blank unless it is different than PORTAL_DOMAIN
- `EMAIL_ADDRESS` is your email address used for communication regarding SSL certification (required if you're using http-01 challenge)
- `SIA_WALLET_PASSWORD` is your wallet password (or seed if you did not set a password)
- `HSD_API_KEY` this is a random security key for a handshake integration that gets generated automatically
- `CLOUDFLARE_AUTH_TOKEN` (optional) if using cloudflare as dns loadbalancer (need to change it in Caddyfile too)
- `AWS_ACCESS_KEY_ID` (optional) if using route53 as a dns loadbalancer
- `AWS_SECRET_ACCESS_KEY` (optional) if using route53 as a dns loadbalancer
- `DISCORD_WEBHOOK_URL` (required if using Discord notifications) discord webhook url (generate from discord app)
- `DISCORD_MENTION_USER_ID` (optional) add `/cc @user` mention to important messages from webhook (has to be id not user name)
- `DISCORD_MENTION_ROLE_ID` (optional) add `/cc @role` mention to important messages from webhook (has to be id not role name)
- `SKYNET_DB_USER` (optional) if using `accounts` this is the MongoDB username
- `SKYNET_DB_PASS` (optional) if using `accounts` this is the MongoDB password
- `SKYNET_DB_HOST` (optional) if using `accounts` this is the MongoDB address or container name
- `SKYNET_DB_PORT` (optional) if using `accounts` this is the MongoDB port
- `COOKIE_DOMAIN` (optional) if using `accounts` this is the domain to which your cookies will be issued
- `COOKIE_HASH_KEY` (optional) if using `accounts` hashing secret, at least 32 bytes
- `COOKIE_ENC_KEY` (optional) if using `accounts` encryption key, at least 32 bytes
- `S3_BACKUP_PATH` (optional) is using `accounts` and backing up the databases to S3. This path should be an S3 bucket
with path to the location in the bucket where we want to store the daily backups.
1. `docker-compose up -d` to restart the services so they pick up new env variables
## Subdomains
It might prove useful for certain skapps to be accessible through a custom subdomain. So instead of being accessed through `https://portal.com/[skylink]`, it would be accessible through `https://[skylink_base32].portal.com`. We call this "subdomain access" and it is made possible by encoding Skylinks using a base32 encoding. We have to use a base32 encoding scheme because subdomains have to be all lower case and the base64 encoded Skylink is case sensitive and thus might contain uppercase characters.
You can convert Skylinks using this [converter skapp](https://convert-skylink.hns.siasky.net). To see how the encoding and decoding works, please follow the link to the repo in the application itself.
There is also an option to access handshake domain through the subdomain using `https://[domain_name].hns.portal.com`.
To configure this on your portal, you have to make sure to configure the following:
## Useful Commands
- Starting the whole stack
> `docker-compose up -d`
- Stopping the whole stack
> `docker-compose down`
- Accessing siac
> `docker exec -it sia siac`
- Portal maintenance
- Pulling portal out for maintenance
> `scripts/portal-down.sh`
- Putting portal back into place after maintenance
> `scripts/portal-up.sh`
- Upgrading portal containers (takes care of pulling it and putting it back)
> `scripts/portal-upgrade.sh`
- Restarting caddy gracefully after making changes to Caddyfile (no downtime)
> `docker exec caddy caddy reload --config /etc/caddy/Caddyfile`
- Restarting nginx gracefully after making changes to nginx configs (no downtime)
> `docker exec nginx openresty -s reload`
- Checking siad service logs (since last hour)
> `docker logs --since 1h $(docker ps -q --filter "name=^sia$")`
- Checking caddy logs (for example in case ssl certificate fails)
> `docker logs caddy -f`
- Checking nginx logs (nginx handles all communication to siad instances)
> `tail -n 50 docker/data/nginx/logs/access.log` to follow last 50 lines of access log
> `tail -n 50 docker/data/nginx/logs/error.log` to follow last 50 lines of error log

48
setup-scripts/setup-docker-services.sh
View File

@ -1,48 +0,0 @@
#! /usr/bin/env bash
set -e # exit on first error
# Install docker (cleans up old docker installation)
# sudo apt-get remove -y docker docker-engine docker.io containerd runc # fails if it is the first installation
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl gnupg-agent software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable"
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io
docker --version # sanity check
# add user to docker group to avoid having to use sudo for every docker command
sudo usermod -aG docker user
# Install docker-compose
sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
docker-compose --version # sanity check
# Create dummy .env file for docker-compose usage with variables
# * PORTAL_DOMAIN - (required) is a skynet portal domain (ex. siasky.net)
# * SERVER_DOMAIN - (optional) is an optional direct server domain (ex. eu-ger-1.siasky.net) - leave blank unless it is different than PORTAL_DOMAIN
# * EMAIL_ADDRESS - this is the administrator contact email you need to supply for communication regarding SSL certification
# * HSD_API_KEY - this is auto generated secure key for your handshake service integration
# * CLOUDFLARE_AUTH_TOKEN - (optional) if using cloudflare as dns loadbalancer (need to change it in Caddyfile too)
# * AWS_ACCESS_KEY_ID - (optional) if using route53 as a dns loadbalancer
# * AWS_SECRET_ACCESS_KEY - (optional) if using route53 as a dns loadbalancer
# * API_PORT - (optional) the port on which siad is listening, defaults to 9980
# * DISCORD_WEBHOOK_URL - (required if using Discord notifications) discord webhook url (generate from discord app)
# * DISCORD_MENTION_USER_ID - (optional) add `/cc @user` mention to important messages from webhook (has to be id not user name)
# * DISCORD_MENTION_ROLE_ID - (optional) add `/cc @role` mention to important messages from webhook (has to be id not role name)
# * SKYNET_DB_USER - (optional) if using `accounts` this is the MongoDB username
# * SKYNET_DB_PASS - (optional) if using `accounts` this is the MongoDB password
# * SKYNET_DB_HOST - (optional) if using `accounts` this is the MongoDB address or container name
# * SKYNET_DB_PORT - (optional) if using `accounts` this is the MongoDB port
# * COOKIE_DOMAIN - (optional) if using `accounts` this is the domain to which your cookies will be issued
# * COOKIE_HASH_KEY - (optional) if using `accounts` hashing secret, at least 32 bytes
# * COOKIE_ENC_KEY - (optional) if using `accounts` encryption key, at least 32 bytes
if ! [ -f /home/user/skynet-webportal/.env ]; then
HSD_API_KEY=$(openssl rand -base64 32) # generate safe random key for handshake
printf "PORTAL_DOMAIN=siasky.net\nSERVER_DOMAIN=\nEMAIL_ADDRESS=email@example.com\nSIA_WALLET_PASSWORD=\nHSD_API_KEY=${HSD_API_KEY}\nCLOUDFLARE_AUTH_TOKEN=\nAWS_ACCESS_KEY_ID=\nAWS_SECRET_ACCESS_KEY=\nDISCORD_WEBHOOK_URL=\nDISCORD_MENTION_USER_ID=\nDISCORD_MENTION_ROLE_ID=\n" > /home/user/skynet-webportal/.env
fi
# Start docker container with nginx and client
docker-compose -f docker-compose.yml up --build -d

11
setup-scripts/setup-health-check-scripts.sh
View File

@ -1,11 +0,0 @@
#! /usr/bin/env bash
set -e # exit on first error
sudo apt-get update
sudo apt-get -y install python3-pip
pip3 install discord-webhook python-dotenv requests elasticsearch-curator
# add cron entries to user crontab
crontab -u user /home/user/skynet-webportal/setup-scripts/support/crontab

48
setup-scripts/setup-server.sh
View File

@ -1,48 +0,0 @@
#! /usr/bin/env bash
set -e # exit on first error
# Copy over basic configuration files
cp /home/user/skynet-webportal/setup-scripts/support/tmux.conf /home/user/.tmux.conf
cp /home/user/skynet-webportal/setup-scripts/support/bashrc /home/user/.bashrc
source /home/user/.bashrc
# Add SSH keys and set SSH configs
sudo cp /home/user/skynet-webportal/setup-scripts/support/ssh_config /etc/ssh/ssh_config
mkdir -p /home/user/.ssh
# cat /home/user/skynet-webportal/setup-scripts/support/authorized_keys >> /home/user/.ssh/authorized_keys
# Install apt packages
sudo apt-get update
sudo apt-get -y install ufw tmux ranger htop nload gcc g++ make git vim unzip curl awscli
# Setup GIT credentials (so commands like git stash would work)
git config --global user.email "devs@nebulous.tech"
git config --global user.name "Sia Dev"
# Setup firewall
sudo ufw --force enable # --force to make it non-interactive
sudo ufw logging low # enable logging for debugging purpose: tail -f /var/log/ufw.log
sudo ufw allow ssh # allow ssh connection to server
sudo ufw allow 80,443/tcp # allow http and https ports
# Block outgoing traffic to local networks
# https://community.hetzner.com/tutorials/block-outgoing-traffic-to-private-networks
sudo ufw deny out from any to 10.0.0.0/8
sudo ufw deny out from any to 172.16.0.0/12
sudo ufw deny out from any to 192.168.0.0/16
sudo ufw deny out from any to 100.64.0.0/10
sudo ufw deny out from any to 198.18.0.0/15
sudo ufw deny out from any to 169.254.0.0/16
# OPTIONAL: terminfo for alacritty terminal via ssh
# If you don't use the alacritty terminal you can remove this step.
wget -c https://raw.githubusercontent.com/alacritty/alacritty/master/extra/alacritty.info
sudo tic -xe alacritty,alacritty-direct alacritty.info
rm alacritty.info
# Set up file limits - siad uses a lot so we need to adjust so it doesn't choke up
sudo cp /home/user/skynet-webportal/setup-scripts/support/limits.conf /etc/security/limits.conf
# Set UTC timezone so all of the servers report the same time
sudo timedatectl set-timezone UTC