Creating Services and Containers
- Create a new project in geostreams. Instructions are here.
Configure the docker-compose file for the services that need to be added. The docker-compose file linked below The docker-compose file for this project is based on the Default Geostreams file and
...
traefik v2. This way minimal configuration is required on the machine.
Code Block language yml title docker-compose.yml collapse true version: "3.3" services: traefik: image: traefik:latest networks:
...
- geostreams volumes: - ./traefik:/traefik/ - /var/run/docker.sock:/var/run/docker.sock:ro ports: - 80:80 - 443:443
...
- 8001:8001 command: - --entrypoints.private.address=:8001 - --entrypoints.web.address=:80 - --entrypoints.websecure.address=:443 - --entrypoints.web.http.redirections.entryPoint.to=websecure - --entrypoints.web.http.redirections.entryPoint.scheme=https - "--providers.docker=true" - "--api=true" - --providers.docker.exposedbydefault=false # SSL Certificate Generation using Lets Encrypt # - "--certificatesresolvers.le.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory" - --certificatesResolvers.le.acme.email=dev@mbclab.ncsa.illinois.edu - --certificatesResolvers.le.acme.storage=/traefik/acme.json - --certificatesResolvers.le.acme.httpChallenge=true - --certificatesResolvers.le.acme.httpChallenge.entryPoint=web labels: - "traefik.enable=true" - "traefik.http.routers.dashboard.service=api@internal" # Route setting - "traefik.http.routers.dashboard.rule=Host(`${TRAEFIK_HOST:-localhost}`)" # set TLS (https) - "traefik.http.routers.dashboard.tls=true" - "traefik.http.routers.dashboard.entrypoints=private" - "traefik.http.routers.dashboard.tls.certresolver=le" # ---------------------------------------------------------------------- # GEOSTREAMS STACK # ---------------------------------------------------------------------- geodashboard: image: hub.ncsa.illinois.edu/geostreams/gd-smartfarm:
...
develop networks: - geostreams labels: - traefik.enable=true - traefik.http.services.geodashboard.loadbalancer.server.port=80 - traefik.http.routers.geodashboard.rule=Host(`${TRAEFIK_HOST:-localhost}`) && (PathPrefix(`${GD_PREFIX_PATH:-/}`)) - traefik.http.routers.geodashboard.entrypoints=websecure - traefik.http.routers.geodashboard.tls=true - traefik.http.routers.geodashboard.tls.certresolver=le restart: unless-stopped geostreams: image: geostreams/geostreams env_file: - ./geostreams.env networks: - geostreams labels: - traefik.enable=true - traefik.http.services.geostreams.loadbalancer.server.port=9000 - traefik.http.routers.geostreams.rule=Host(`${TRAEFIK_HOST:-localhost}`) && (PathPrefix(`${GEOSTREAMS_PREFIX_PATH:-/}`)) - traefik.http.routers.geostreams.entrypoints=
...
websecure - traefik.http.routers.
...
geostreams.tls=true
...
- traefik.http.routers.
...
geostreams.tls.certresolver=le volumes: - ./application.conf:/home/geostreams/conf/application.conf - ./messages.en:/home/geostreams/conf/messages.en
...
healthcheck: test: ["CMD", "curl", "-s", "--fail", "http://localhost:9000/geostreams/api/status"]
...
...
...
...
...
...
restart: unless-stopped
...
...
...
...
...
postgres:
...
...
...
...
...
image:
...
mdillon/
...
The following .env file sets up the routes:
Code Block | ||
---|---|---|
| ||
CLOWDER_PREFIX_PATH=/clowder
CLOWDER_CONTEXT=/clowder/
RABBITMQ_CLOWDERURL=http://clowder:9000/clowder
TRAEFIK_HOST=localhost
GEOSTREAMS_PREFIX_PATH=/geostreams
GD_PREFIX_PATH=/ |
...
Setting up the machine
...
postgis:9.5 networks: - geostreams ports: - 5432:5432 volumes: - ./postgres:/var/lib/postgresql/data restart: unless-stopped networks: geostreams:
The following .env file sets up the routes:
Code Block language bash TRAEFIK_HOST=localhost GEOSTREAMS_PREFIX_PATH=/geostreams GD_PREFIX_PATH=/
- For geostreams, use the example application.conf https://github.com/geostreams/geostreams/blob/master/conf/reference.conf .
- If using clowder, follow the link to initialize clowder https://github.com/clowder-framework/clowder#initializing-clowder.
Setting up the VM
Create a VM
- Install python-openstackclient
Create an rc file using the example below used for gltg nebula
Code Block export OS_AUTH_URL=http://nebula.ncsa.illinois.edu:5000/v2.0 export OS_TENANT_ID=c4121a001a8240d4a8b701d664ef4bf0 # Project ID in nebula export OS_TENANT_NAME="GLTG" export OS_PROJECT_NAME="GLTG" export OS_USERNAME=<nebula_username> export OS_PASSWORD=<nebula_password> export OS_REGION_NAME="RegionOne" export PS1="(openstack)[\e]0;\a]\h:\W \u [${OS_PROJECT_NAME}] $ "
- Run `bash --rcfile <rc_file_name>` to create a virtual environment
- Download and run the makevm-script (updated version in the comments) to create the vm.
- Use one of the available Floating IP Addresses on Nebula and provide it to the script.
- Email help+neteng@ncsa.illinois.edu to setup domain name for machine.
...
- The docker-compose file is setup to use Lets Encrypt for SSL certificate generation and renewal.
Setup Docker
- Install docker and docker-compose on the machine. Additionally, install pass and gnupg2 without which logging into docker will give an error.
- Docker Installation Instructions
- Docker-compose installation Instructions
Code Block sudo apt-get install pass gnupg2
- Install docker and docker-compose on the machine. Additionally, install pass and gnupg2 without which logging into docker will give an error.
...
- Login to hub.ncsa.illinois.edu using the robot account login. The account information is stored in lastpass folder.
Code Block docker login -u `robot$githubgeostreams` -p `<TOKEN>` hub.ncsa.illinois.edu
...
Set up Backup
Initialize Geostreams
- Move the docker-compose file
...
- and
...
- other relevant files
...
- to the machine.
- Run docker-compose up -d.
- Initialize the database by running the geostreams container with command `initialize` once.
Configuring Build and Deployment Pipeline for Project
Add the following Github Actions public key to the machine.
`cat PATH_TO_KEY | ssh -i PATH_TO_ID_KEY user@hostname "cat >> ~/.ssh/authorized_keys"`
Code Block
...
title GitHub_Repo_Public_key collapse true ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCL/mqWBK1BTwZxKnZ546F9IbFlYY6qgFzh9xTM6eY+DxagKVV1BAr4kbqamnLYDzrOLC6zY+8k3xGS1HSxp8UAWXLnPzDkb13uXj+neGty7DwMIVWRVSc0JNa0cRaEKI1wC9AK1utKEU7aaGu6fZsmExmXNzIzxLIYVUFdW8G2GVoK9wNSba3OT2rneutgOUrb5PR6ADpfBEO8h48CcP6edw5A2HoJ0ZXySeadvnInOhp3yisO3khaZ7t4ZPRtRVRM+M+V9H+1JpOmsulfAZUyEdntzU1MkduFAz+X5T/h9IhHYlplqJ00GEjc/zIPS9y39Be5XqgvMadapupmeGpZWU6K/xvluATcP1xGqy7ytytAr6ZIbsCyKWGJXeUYsR8K1MdNC8zAVB+2Cnu3Df0TUf7xIV6sSx66MtehAADIGxtik4KGl5DZiWENgf0aCSeJHAjRkxGx4mxb/Cp9iercoKs+6uIAKd7fU/pYK9kw3z8WlkOsTcQcII9hzY6bQs= Smartfarm Key for Github Actions
Update github deployment action and add the details for the new project under strategy->matrix.
Code Block language yml name: <project
...
_name> gd-name:
...
<lerna_project_name> prod-host: <production_host_name> dev-host: <dev_host_name> file-path:
...
<docker-compose
...
file
...
path
...
on VM> description:
- For development machines, an image with a develop tag will be created in hub.ncsa.illinois.edu and for production machines the latest tag will be used. The docker-compose files should be updated accordingly.