LumeWeb
/
skynet-webportal
Archived
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Switch to fully secured cockroachdb.

This commit is contained in:
Ivaylo Novakov 2021-01-08 17:40:25 +01:00
parent 97aac10fb3
commit 4e6b4b36a3
No known key found for this signature in database
GPG Key ID: 06B9354AB08BE9C6
5 changed files with 70 additions and 47 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
.gitignore vendored
View File

@ -89,7 +89,10 @@ __pycache__
/venv*
# CockroachDB certificates
.cr_certs/
docker/cockroach/certs/*.crt
docker/cockroach/certs/*.key
docker/kratos/cr_certs/*.crt
docker/kratos/cr_certs/*.key
# Oathkeeper JWKS signing token
docker/kratos/oathkeeper/id_token.jwks.json

93
README.md
View File

@ -19,20 +19,21 @@ List of available parameters:
Mongo needs a couple of extra steps in order to start a secure cluster.
* Open port 27017 on all nodes that will take part in the cluster. Ideally, you
would only open the port for the other nodes in the cluster.
* Manually run an initialisation `docker run` with extra environment variables
that will initialise the admin user with a password (example below).
* Manually add a `mgkey` file under `./docker/data/mongo` with the respective
secret (see [Mongo's keyfile access control](https://docs.mongodb.com/manual/tutorial/enforce-keyfile-access-control-in-existing-replica-set/) for details).
* During the initialisation run mentioned above, we need to make two extra steps
within the container:
* Open port 27017 on all nodes that will take part in the cluster. Ideally, you would only open the port for the other
nodes in the cluster.
* Manually run an initialisation `docker run` with extra environment variables that will initialise the admin user with
a password (example below).
* Manually add a `mgkey` file under `./docker/data/mongo` with the respective secret (
see [Mongo's keyfile access control](https://docs.mongodb.com/manual/tutorial/enforce-keyfile-access-control-in-existing-replica-set/)
for details).
* During the initialisation run mentioned above, we need to make two extra steps within the container:
* Change the ownership of `mgkey` to `mongodb:mongodb`
* Change its permissions to 400
* After these steps are done we can open a mongo shell on the master node and
run `rs.add()` in order to add the new node to the cluster.
* After these steps are done we can open a mongo shell on the master node and run `rs.add()` in order to add the new
node to the cluster.
Example initialisation docker run command:
```
docker run \
--rm \
@ -44,7 +45,9 @@ docker run \
-v /home/user/skynet-webportal/docker/data/mongo/mgkey:/data/mgkey \
mongo --keyFile=/data/mgkey --replSet=skynet
```
Regular docker run command:
```
docker run \
--rm \
@ -53,7 +56,9 @@ docker run \
-v /home/user/skynet-webportal/docker/data/mongo/db:/data/db \
-v /home/user/skynet-webportal/docker/data/mongo
```
Cluster initialisation mongo command:
```
rs.initiate(
{
@ -68,19 +73,19 @@ rs.initiate(
### Kratos & Oathkeeper Setup
[Kratos](https://www.ory.sh/kratos) is our user management system of choice and
[Kratos](https://www.ory.sh/kratos) is our user management system of choice and
[Oathkeeper](https://www.ory.sh/oathkeeper) is the identity and access proxy.
Most of the needed config is already under `docker/kratos`. The only two things
that need to be changed are the config for Kratos that might contain you email
server password, and the JWKS Oathkeeper uses to sign its JWT tokens.
Most of the needed config is already under `docker/kratos`. The only two things that need to be changed are the config
for Kratos that might contain you email server password, and the JWKS Oathkeeper uses to sign its JWT tokens.
To override the default `kratos.yml` you can create ` .kratos.yml` in the root
directory of the project, alongside the `.env` file.
To override the default `kratos.yml` you can create ` .kratos.yml` in the root directory of the project, alongside
the `.env` file.
To override the JWKS you will need to directly edit
`docker/kratos/oathkeeper/id_token.jwks.json` and replace it with your generated key set. If you don't know how to
generate a key set you can use this code:
To override the JWKS you will need to directly edit
`docker/kratos/oathkeeper/id_token.jwks.json` and replace it with your generated
key set. If you don't know how to generate a key set you can use this code:
```go
package main
@ -107,52 +112,56 @@ func main() {
os.Stdout.Write(jsonbuf)
}
```
While you can directly put the output of this programme into the file mentioned
above, you can also remove the public key from the set and change the `kid` of
the private key to not include the prefix `private:`.
While you can directly put the output of this programme into the file mentioned above, you can also remove the public
key from the set and change the `kid` of the private key to not include the prefix `private:`.
### CockroachDB Setup
Kratos uses CockroachDB to store its data. For that data to be shared across all
nodes that comprise your portal cluster setup, we need to set up a CockroachDB
cluster, complete with secure communication.
Kratos uses CockroachDB to store its data. For that data to be shared across all nodes that comprise your portal cluster
setup, we need to set up a CockroachDB cluster, complete with secure communication.
#### Generate the certificates for secure communication
For a detailed walk-through, please check [this guide](https://www.cockroachlabs.com/docs/v20.2/secure-a-cluster.html) out.
For a detailed walk-through, please check [this guide](https://www.cockroachlabs.com/docs/v20.2/secure-a-cluster.html)
out.
Steps:
1. Start a local cockroach docker instance:
1. Start a local cockroach docker instance:
`docker run -d -v "<local dir>:/cockroach/cockroach-secure" --name=crdb cockroachdb/cockroach start --insecure`
1. Get a shall into that instance: `docker exec -it crdb /bin/bash`
1. Go to the directory we which we mapped to a local dir: `cd /cockroach/cockroach-secure`
1. Create the subdirectories in which to create certificates and keys: `mkdir certs my-safe-directory`
1. Create the CA (Certificate Authority) certificate and key pair: `cockroach cert create-ca --certs-dir=certs --ca-key=my-safe-directory/ca.key`
1. Create a client certificate and key pair for the root user: `cockroach cert create-client root --certs-dir=certs --ca-key=my-safe-directory/ca.key`
1. Create the certificate and key pair for your nodes: `cockroach cert create-node mynode.siasky.net --certs-dir=certs --ca-key=my-safe-directory/ca.key`
1. If you want to create certificates for more nodes, just delete the `node.*`
files (after you've finished the next step!) and re-run the above
command with the new node name.
1. Put the contents of the `certs` folder in `.cr_certs/` under your portal's root
dir and store the content of `my-safe-directory` somewhere safe.
1. Create the CA (Certificate Authority) certificate and key
pair: `cockroach cert create-ca --certs-dir=certs --ca-key=my-safe-directory/ca.key`
1. Create a client certificate and key pair for the root
user: `cockroach cert create-client root --certs-dir=certs --ca-key=my-safe-directory/ca.key`
1. Create the certificate and key pair for your
nodes: `cockroach cert create-node cockroach mynode.siasky.net --certs-dir=certs --ca-key=my-safe-directory/ca.key`.
Don't forget the `cockroach` node name - it's needed by our docker-compose setup. If you want to create certificates
for more nodes, just delete the `node.*` files (after you've finished the next steps for this node!) and re-run the
above command with the new node name.
1. Put the contents of the `certs` folder under `docker/cockroach/certs/*` under your portal's root dir and store the
content of `my-safe-directory` somewhere safe.
1. Put *another copy* of those certificates under `docker/kratos/cr_certs` and change permissions of the `*.key` files,
so they can be read by anyone (644).
#### Configure your CockroachDB node
There is some configuration that needs to be added to your `.env`file, namely:
1. CR_NODE - the name of your node
1. CR_IP - the public IP of your node
1. CR_CLUSTER_NODES - a list of IPs and ports which make up your cluster, e.g.
`95.216.13.185:26257,147.135.37.21:26257,144.76.136.122:26257`. This will be
the list of nodes that will make up your cluster, so make sure those are
accurate.
1. CR_CLUSTER_NODES - a list of IPs and ports which make up your cluster, e.g.
`95.216.13.185:26257,147.135.37.21:26257,144.76.136.122:26257`. This will be the list of nodes that will make up your
cluster, so make sure those are accurate.
## Contributing
### Testing Your Code
Before pushing your code you should verify that it will pass our online test
suite.
Before pushing your code, you should verify that it will pass our online test suite.
**Cypress Tests**
Verify the Cypress test suite by doing the following:

10
docker-compose.yml
View File

@ -208,11 +208,12 @@ services:
restart: on-failure
logging: *default-logging
environment:
- DSN=cockroach://root@cockroach:26257/defaultdb?sslmode=disable&max_conns=20&max_idle_conns=4
- DSN=cockroach://root@cockroach:26257/defaultdb?max_conns=20&max_idle_conns=4&sslmode=verify-full&sslcert=/certs/node.crt&sslkey=/certs/node.key&sslrootcert=/certs/ca.crt
- SQA_OPT_OUT=true
volumes:
- ./docker/kratos/config:/etc/config/kratos
- ./docker/data/cockroach/sqlite:/var/lib/sqlite
- ./docker/kratos/cr_certs:/certs
command: -c /etc/config/kratos/kratos.yml migrate sql -e --yes
networks:
shared:
@ -230,7 +231,7 @@ services:
- 4433 # public
- 4434 # admin
environment:
- DSN=cockroach://root@cockroach:26257/defaultdb?sslmode=disable&max_conns=20&max_idle_conns=4
- DSN=cockroach://root@cockroach:26257/defaultdb?max_conns=20&max_idle_conns=4&sslmode=verify-full&sslcert=/certs/node.crt&sslkey=/certs/node.key&sslrootcert=/certs/ca.crt
- LOG_LEVEL=trace
- SERVE_PUBLIC_BASE_URL=/.ory/kratos/public/
- SQA_OPT_OUT=true
@ -239,6 +240,7 @@ services:
- ./docker/kratos/config:/etc/config/kratos
- ./docker/data/cockroach/sqlite:/var/lib/sqlite
- ./.kratos.yml:/etc/config/kratos/kratos.yml
- ./docker/kratos/cr_certs:/certs
networks:
shared:
ipv4_address: 10.10.10.81
@ -293,10 +295,10 @@ services:
cockroach:
image: cockroachdb/cockroach:v20.1.10
container_name: cockroach
command: start --advertise-addr=$CR_IP --join=$CR_CLUSTER_NODES --certs-dir=/certs --listen-addr=0.0.0.0:26257 --http-addr=0.0.0.0:8080
command: start --advertise-addr=${CR_IP} --join=${CR_CLUSTER_NODES} --certs-dir=/certs --listen-addr=0.0.0.0:26257 --http-addr=0.0.0.0:8080
volumes:
- ./docker/data/cockroach/sqlite:/cockroach/cockroach-data
- ./.cr_certs:/certs
- ./docker/cockroach/certs:/certs
expose:
- 8080
- 26257

2
docker/cockroach/certs/README.md Normal file
View File

@ -0,0 +1,2 @@
This directory needs to contain all certificates needed by this cockroachdb node. Those can be generated by the steps
outlined in the README in the root directory, under "Setting up CockroachDB".

7
docker/kratos/cr_certs/README.md Normal file
View File

@ -0,0 +1,7 @@
This directory needs to contain all certificates needed by this cockroachdb node. Those can be generated by the steps
outlined in the README in the root directory, under "Setting up CockroachDB".
The only difference between the files here and those under
`docker/cockroach/certs` is that the files here need to be readable by anyone, while the files under `cockroach` need to
have their original access rights
(all *.key files should be 600 instead of 644 there).