diff --git a/.gitignore b/.gitignore index fa7c2589..566afecb 100644 --- a/.gitignore +++ b/.gitignore @@ -89,7 +89,10 @@ __pycache__ /venv* # CockroachDB certificates -.cr_certs/ +docker/cockroach/certs/*.crt +docker/cockroach/certs/*.key +docker/kratos/cr_certs/*.crt +docker/kratos/cr_certs/*.key # Oathkeeper JWKS signing token docker/kratos/oathkeeper/id_token.jwks.json diff --git a/README.md b/README.md index 865ffcd2..6deccfe3 100644 --- a/README.md +++ b/README.md @@ -19,20 +19,21 @@ List of available parameters: Mongo needs a couple of extra steps in order to start a secure cluster. -* Open port 27017 on all nodes that will take part in the cluster. Ideally, you -would only open the port for the other nodes in the cluster. -* Manually run an initialisation `docker run` with extra environment variables -that will initialise the admin user with a password (example below). -* Manually add a `mgkey` file under `./docker/data/mongo` with the respective -secret (see [Mongo's keyfile access control](https://docs.mongodb.com/manual/tutorial/enforce-keyfile-access-control-in-existing-replica-set/) for details). -* During the initialisation run mentioned above, we need to make two extra steps -within the container: +* Open port 27017 on all nodes that will take part in the cluster. Ideally, you would only open the port for the other + nodes in the cluster. +* Manually run an initialisation `docker run` with extra environment variables that will initialise the admin user with + a password (example below). +* Manually add a `mgkey` file under `./docker/data/mongo` with the respective secret ( + see [Mongo's keyfile access control](https://docs.mongodb.com/manual/tutorial/enforce-keyfile-access-control-in-existing-replica-set/) + for details). +* During the initialisation run mentioned above, we need to make two extra steps within the container: * Change the ownership of `mgkey` to `mongodb:mongodb` * Change its permissions to 400 -* After these steps are done we can open a mongo shell on the master node and -run `rs.add()` in order to add the new node to the cluster. +* After these steps are done we can open a mongo shell on the master node and run `rs.add()` in order to add the new + node to the cluster. Example initialisation docker run command: + ``` docker run \ --rm \ @@ -44,7 +45,9 @@ docker run \ -v /home/user/skynet-webportal/docker/data/mongo/mgkey:/data/mgkey \ mongo --keyFile=/data/mgkey --replSet=skynet ``` + Regular docker run command: + ``` docker run \ --rm \ @@ -53,7 +56,9 @@ docker run \ -v /home/user/skynet-webportal/docker/data/mongo/db:/data/db \ -v /home/user/skynet-webportal/docker/data/mongo ``` + Cluster initialisation mongo command: + ``` rs.initiate( { @@ -68,19 +73,19 @@ rs.initiate( ### Kratos & Oathkeeper Setup -[Kratos](https://www.ory.sh/kratos) is our user management system of choice and +[Kratos](https://www.ory.sh/kratos) is our user management system of choice and [Oathkeeper](https://www.ory.sh/oathkeeper) is the identity and access proxy. -Most of the needed config is already under `docker/kratos`. The only two things -that need to be changed are the config for Kratos that might contain you email -server password, and the JWKS Oathkeeper uses to sign its JWT tokens. +Most of the needed config is already under `docker/kratos`. The only two things that need to be changed are the config +for Kratos that might contain you email server password, and the JWKS Oathkeeper uses to sign its JWT tokens. -To override the default `kratos.yml` you can create ` .kratos.yml` in the root -directory of the project, alongside the `.env` file. +To override the default `kratos.yml` you can create ` .kratos.yml` in the root directory of the project, alongside +the `.env` file. + +To override the JWKS you will need to directly edit +`docker/kratos/oathkeeper/id_token.jwks.json` and replace it with your generated key set. If you don't know how to +generate a key set you can use this code: -To override the JWKS you will need to directly edit -`docker/kratos/oathkeeper/id_token.jwks.json` and replace it with your generated -key set. If you don't know how to generate a key set you can use this code: ```go package main @@ -107,52 +112,56 @@ func main() { os.Stdout.Write(jsonbuf) } ``` -While you can directly put the output of this programme into the file mentioned -above, you can also remove the public key from the set and change the `kid` of -the private key to not include the prefix `private:`. +While you can directly put the output of this programme into the file mentioned above, you can also remove the public +key from the set and change the `kid` of the private key to not include the prefix `private:`. ### CockroachDB Setup -Kratos uses CockroachDB to store its data. For that data to be shared across all -nodes that comprise your portal cluster setup, we need to set up a CockroachDB -cluster, complete with secure communication. +Kratos uses CockroachDB to store its data. For that data to be shared across all nodes that comprise your portal cluster +setup, we need to set up a CockroachDB cluster, complete with secure communication. #### Generate the certificates for secure communication -For a detailed walk-through, please check [this guide](https://www.cockroachlabs.com/docs/v20.2/secure-a-cluster.html) out. +For a detailed walk-through, please check [this guide](https://www.cockroachlabs.com/docs/v20.2/secure-a-cluster.html) +out. Steps: -1. Start a local cockroach docker instance: + +1. Start a local cockroach docker instance: `docker run -d -v ":/cockroach/cockroach-secure" --name=crdb cockroachdb/cockroach start --insecure` 1. Get a shall into that instance: `docker exec -it crdb /bin/bash` 1. Go to the directory we which we mapped to a local dir: `cd /cockroach/cockroach-secure` 1. Create the subdirectories in which to create certificates and keys: `mkdir certs my-safe-directory` -1. Create the CA (Certificate Authority) certificate and key pair: `cockroach cert create-ca --certs-dir=certs --ca-key=my-safe-directory/ca.key` -1. Create a client certificate and key pair for the root user: `cockroach cert create-client root --certs-dir=certs --ca-key=my-safe-directory/ca.key` -1. Create the certificate and key pair for your nodes: `cockroach cert create-node mynode.siasky.net --certs-dir=certs --ca-key=my-safe-directory/ca.key` - 1. If you want to create certificates for more nodes, just delete the `node.*` - files (after you've finished the next step!) and re-run the above - command with the new node name. -1. Put the contents of the `certs` folder in `.cr_certs/` under your portal's root - dir and store the content of `my-safe-directory` somewhere safe. - +1. Create the CA (Certificate Authority) certificate and key + pair: `cockroach cert create-ca --certs-dir=certs --ca-key=my-safe-directory/ca.key` +1. Create a client certificate and key pair for the root + user: `cockroach cert create-client root --certs-dir=certs --ca-key=my-safe-directory/ca.key` +1. Create the certificate and key pair for your + nodes: `cockroach cert create-node cockroach mynode.siasky.net --certs-dir=certs --ca-key=my-safe-directory/ca.key`. + Don't forget the `cockroach` node name - it's needed by our docker-compose setup. If you want to create certificates + for more nodes, just delete the `node.*` files (after you've finished the next steps for this node!) and re-run the + above command with the new node name. +1. Put the contents of the `certs` folder under `docker/cockroach/certs/*` under your portal's root dir and store the + content of `my-safe-directory` somewhere safe. +1. Put *another copy* of those certificates under `docker/kratos/cr_certs` and change permissions of the `*.key` files, + so they can be read by anyone (644). + #### Configure your CockroachDB node There is some configuration that needs to be added to your `.env`file, namely: + 1. CR_NODE - the name of your node 1. CR_IP - the public IP of your node -1. CR_CLUSTER_NODES - a list of IPs and ports which make up your cluster, e.g. - `95.216.13.185:26257,147.135.37.21:26257,144.76.136.122:26257`. This will be - the list of nodes that will make up your cluster, so make sure those are - accurate. +1. CR_CLUSTER_NODES - a list of IPs and ports which make up your cluster, e.g. + `95.216.13.185:26257,147.135.37.21:26257,144.76.136.122:26257`. This will be the list of nodes that will make up your + cluster, so make sure those are accurate. ## Contributing ### Testing Your Code -Before pushing your code you should verify that it will pass our online test -suite. +Before pushing your code, you should verify that it will pass our online test suite. **Cypress Tests** Verify the Cypress test suite by doing the following: diff --git a/docker-compose.yml b/docker-compose.yml index 88047ebb..769063b6 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -208,11 +208,12 @@ services: restart: on-failure logging: *default-logging environment: - - DSN=cockroach://root@cockroach:26257/defaultdb?sslmode=disable&max_conns=20&max_idle_conns=4 + - DSN=cockroach://root@cockroach:26257/defaultdb?max_conns=20&max_idle_conns=4&sslmode=verify-full&sslcert=/certs/node.crt&sslkey=/certs/node.key&sslrootcert=/certs/ca.crt - SQA_OPT_OUT=true volumes: - ./docker/kratos/config:/etc/config/kratos - ./docker/data/cockroach/sqlite:/var/lib/sqlite + - ./docker/kratos/cr_certs:/certs command: -c /etc/config/kratos/kratos.yml migrate sql -e --yes networks: shared: @@ -230,7 +231,7 @@ services: - 4433 # public - 4434 # admin environment: - - DSN=cockroach://root@cockroach:26257/defaultdb?sslmode=disable&max_conns=20&max_idle_conns=4 + - DSN=cockroach://root@cockroach:26257/defaultdb?max_conns=20&max_idle_conns=4&sslmode=verify-full&sslcert=/certs/node.crt&sslkey=/certs/node.key&sslrootcert=/certs/ca.crt - LOG_LEVEL=trace - SERVE_PUBLIC_BASE_URL=/.ory/kratos/public/ - SQA_OPT_OUT=true @@ -239,6 +240,7 @@ services: - ./docker/kratos/config:/etc/config/kratos - ./docker/data/cockroach/sqlite:/var/lib/sqlite - ./.kratos.yml:/etc/config/kratos/kratos.yml + - ./docker/kratos/cr_certs:/certs networks: shared: ipv4_address: 10.10.10.81 @@ -293,10 +295,10 @@ services: cockroach: image: cockroachdb/cockroach:v20.1.10 container_name: cockroach - command: start --advertise-addr=$CR_IP --join=$CR_CLUSTER_NODES --certs-dir=/certs --listen-addr=0.0.0.0:26257 --http-addr=0.0.0.0:8080 + command: start --advertise-addr=${CR_IP} --join=${CR_CLUSTER_NODES} --certs-dir=/certs --listen-addr=0.0.0.0:26257 --http-addr=0.0.0.0:8080 volumes: - ./docker/data/cockroach/sqlite:/cockroach/cockroach-data - - ./.cr_certs:/certs + - ./docker/cockroach/certs:/certs expose: - 8080 - 26257 diff --git a/docker/cockroach/certs/README.md b/docker/cockroach/certs/README.md new file mode 100644 index 00000000..fd957133 --- /dev/null +++ b/docker/cockroach/certs/README.md @@ -0,0 +1,2 @@ +This directory needs to contain all certificates needed by this cockroachdb node. Those can be generated by the steps +outlined in the README in the root directory, under "Setting up CockroachDB". diff --git a/docker/kratos/cr_certs/README.md b/docker/kratos/cr_certs/README.md new file mode 100644 index 00000000..47c143eb --- /dev/null +++ b/docker/kratos/cr_certs/README.md @@ -0,0 +1,7 @@ +This directory needs to contain all certificates needed by this cockroachdb node. Those can be generated by the steps +outlined in the README in the root directory, under "Setting up CockroachDB". + +The only difference between the files here and those under +`docker/cockroach/certs` is that the files here need to be readable by anyone, while the files under `cockroach` need to +have their original access rights +(all *.key files should be 600 instead of 644 there).