Local Setup Instructions

Creating Services and Containers

Create a new project in geostreams. Instructions are here.
Configure the docker-compose file for the services that need to be added. The docker-compose file linked below The docker-compose file for this project is based on the Default Geostreams file and the Default Clowder file and uses traefik v2. This way minimal configuration is required on the machine.

Wiki Markup

Code Block

language	yml
title	docker-compose.yml
collapse	true

version: "3.3"

services:
  traefik:
    image: traefik:latest
    networks:
- clowder
      - geostreams
    volumes:
      - ./traefik:/traefik/
      - /var/run/docker.sock:/var/run/docker.sock:ro
    ports:
      - 80:80
      - 443:443
labels:
- traefik.enable=true
- traefik.http.routers.traefik.rule=Host(`${TRAEFIK_HOST:-localhost}`) && (PathPrefix(`/traefik`) || PathPrefix(`/api`))
- traefik.http.routers.traefik.service=api@internal
- traefik.http.routers.traefik.entrypoints=http
- traefik.http.routers.traefik.middlewares=traefik-strip
- traefik.http.middlewares.traefik-strip.stripprefix.prefixes=/traefik
# Redirect all HTTP to HTTPS permanently
# - traefik.http.routers.http_catchall.rule=HostRegexp(`{any:.+}`)
# - traefik.http.routers.http_catchall.entrypoints=web
# - traefik.http.routers.http_catchall.middlewares=https_redirect
# - traefik.http.middlewares.https_redirect.redirectscheme.scheme=https
# - traefik.http.middlewares.https_redirect.redirectscheme.permanent=false
command:
--log.level=DEBUG
--entrypoints.http.address=:80
--entrypoints.https.address=:443
--api.dashboard
--providers.docker.exposedbydefault=false
# "      - 8001:8001

    command:
      - --log.level=DEBUG
      - --entrypoints.private.address=:8001
      - --entrypoints.web.address=:80
      - --entrypoints.websecure.address=:443
      - --entrypoints.web.http.redirections.entryPoint.to=websecure
      - --entrypoints.web.http.redirections.entryPoint.scheme=https
      - "--providers.docker=true"
      - "--api=true"
      - "--api.dashboard=true"
      - --providers.docker.exposedbydefault=false

	  # SSL Certificate Generation using Lets Encrypt
      # - "--certificatesresolvers.le.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
      - --certificatesResolvers.le.acme.email=<YOUR_EMAIL>"
# "dev@mbclab.ncsa.illinois.edu
      - --certificatesResolvers.le.acme.storage=/traefik/acme.json"
# "
      - --certificatesResolvers.le.acme.tlsChallengehttpChallenge=true"
# "
      - --certificatesResolvers.le.acme.httpChallenge.entryPoint=web

    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.dashboard.service=api@internal"

      # Route setting
      - "traefik.http.routers.dashboard.rule=Host(`${TRAEFIK_HOST:--certificatesResolvers.le.acme.httpChallenge.entryPoint=web"

localhost}`)"

      # set TLS (https)
      - "traefik.http.routers.dashboard.tls=true"
      - "traefik.http.routers.dashboard.entrypoints=private"
      - "traefik.http.routers.dashboard.tls.certresolver=le"


  # ----------------------------------------------------------------------
  # GEOSTREAMS STACK
  # ----------------------------------------------------------------------
  geodashboard:
    image: hub.ncsa.illinois.edu/geostreams/gd-smartfarm:dev-latest
develop
    networks:
      - geostreams
    labels:
      - traefik.enable=true
      - traefik.http.services.geodashboard.loadbalancer.server.port=80
      - traefik.http.routers.geodashboard.rule=Host(`${TRAEFIK_HOST:-localhost}`) && (PathPrefix(`${GD_PREFIX_PATH:-/}`))
      - traefik.http.routers.geodashboard.entrypoints=websecure
      - traefik.http.routers.geodashboard.tls=true
      - traefik.http.routers.geodashboard.tls.certresolver=le
    restart: unless-stopped
    
  geostreams:
    image: geostreams/geostreams
    env_file:
      - ./geostreams.env
    networks:
      - geostreams
    labels:
      - traefik.enable=true
      - traefik.http.services.geostreams.loadbalancer.server.port=9000
      - traefik.http.routers.geostreams.rule=Host(`${TRAEFIK_HOST:-localhost}`) && (PathPrefix(`${GEOSTREAMS_PREFIX_PATH:-/}`))
      - traefik.http.routers.geostreams.entrypoints=http
#websecure
      - traefik.http.routers.whoamigeostreams.tls=true
#      - traefik.http.routers.whoamigeostreams.tls.certresolver=le

    volumes:
      - ./application.conf:/home/geostreams/conf/application.conf
      - ./messages.en:/home/geostreams/conf/messages.en
restart: unless-stopped
    healthcheck:
       test: ["CMD", "curl", "-s", "--fail", "http://localhost:9000/geostreams/api/status"]
postgres:
image: mdillon/postgis:9.5
networks:
- geostreams
ports:
- 5432:5432
volumes:
- postgres:/var/lib/postgresql/data
restart: unless-stopped 


# ----------------------------------------------------------------------
# CLOWDER APPLICATIONpostgres:
# ----------------------------------------------------------------------

# main clowder application
clowder:
image: clowdermdillon/clowder:${CLOWDER_VERSION:-latest}
restart: unless-stopped
networks:
- clowder
depends_on:
- mongo
environment:
- CLOWDER_ADMINS=${CLOWDER_ADMINS:-admin@example.com}
- CLOWDER_REGISTER=${CLOWDER_REGISTER:-false}
- CLOWDER_CONTEXT=${CLOWDER_CONTEXT:-/}
- CLOWDER_SSL=${CLOWDER_SSL:-false}
- RABBITMQ_URI=${RABBITMQ_URI:-amqp://guest:guest@rabbitmq/%2F}
- RABBITMQ_EXCHANGE=${RABBITMQ_EXCHANGE:-clowder}
- RABBITMQ_CLOWDERURL=${RABBITMQ_CLOWDERURL:-http://clowder:9000}
- SMTP_MOCK=${SMTP_MOCK:-true}
- SMTP_SERVER=${SMTP_SERVER:-smtp}
labels:
- traefik.enable=true
- traefik.http.services.clowder.loadbalancer.server.port=9000
- traefik.http.routers.clowder.rule=Host(`${TRAEFIK_HOST:-localhost}`) && (PathPrefix(`${CLOWDER_PREFIX_PATH:-/}`))
- traefik.http.routers.clowder.entrypoints=http
volumes:
- clowder-custom:/home/clowder/custom
- clowder-data:/home/clowder/data
healthcheck:
test: ["CMD", "curl", "-s", "--fail", "http://localhost:9000/clowder/api/status"]

# ----------------------------------------------------------------------
# CLOWDER DEPENDENCIES
# ----------------------------------------------------------------------

# database to hold metadata (required)
mongo:
image: mongo:3.6
restart: unless-stopped
networks:
- clowder
volumes:
- mongo:/data/db

# message broker (optional but needed for extractors)
rabbitmq:
image: rabbitmq:management-alpine
restart: unless-stopped
networks:
- clowder
environment:
- RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS=-rabbitmq_management path_prefix "/rabbitmq"
- RABBITMQ_DEFAULT_USER=${RABBITMQ_DEFAULT_USER:-guest}
- RABBITMQ_DEFAULT_PASS=${RABBITMQ_DEFAULT_PASS:-guest}
labels:
- traefik.enable=true
- traefik.http.services.rabbitmq.loadbalancer.server.port=15672
- traefik.http.routers.rabbitmq.rule=Host(`${TRAEFIK_HOST:-localhost}`) && (PathPrefix(`/rabbitmq`))
- traefik.http.routers.rabbitmq.entrypoints=http
# - "traefik.website.frontend.whiteList.sourceRange=${TRAEFIK_IPFILTER:-172.16.0.0/12}"
volumes:
- rabbitmq:/var/lib/rabbitmq

# search index (optional, needed for search and sorting future) 
elasticsearch:
image: elasticsearch:2
command: elasticsearch -Des.cluster.name="clowder"
networks:
- clowder
restart: unless-stopped
environment:
- cluster.name=clowder
volumes:
- elasticsearch:/usr/share/elasticsearch/data

# monitor clowder extractors
monitor:
image: clowder/monitor:${CLOWDER_VERSION:-latest}
restart: unless-stopped
networks:
- clowder
depends_on:
- rabbitmq
environment:
- RABBITMQ_URI=${RABBITMQ_URI:-amqp://guest:guest@rabbitmq/%2F}
- RABBITMQ_MGMT_PORT=15672
- RABBITMQ_MGMT_PATH=/rabbitmq
labels:
- traefik.enable=true
- traefik.http.services.monitor.loadbalancer.server.port=9999
- traefik.frontend.rule=${TRAEFIK_FRONTEND_RULE:-}PathPrefixStrip:/monitor
- traefik.http.routers.monitor.rule=Host(`${TRAEFIK_HOST:-localhost}`) && (PathPrefix(`/monitor`))
- traefik.http.routers.monitor.entrypoints=http

networks:
clowder:
geostreams:

volumes:
traefik:
clowder-data:
clowder-custom:
mongo:
rabbitmq:
elasticsearch:
postgres:

postgis:9.5
    networks:
      - geostreams
    ports:
      - 5432:5432
    volumes:
      - ./postgres:/var/lib/postgresql/data
    restart: unless-stopped    



networks:
  geostreams:

The following .env file sets up the routes:

Code Block

language	bash

CLOWDER_PREFIX_PATH=/clowder
CLOWDER_CONTEXT=/clowder/
RABBITMQ_CLOWDERURL=http://clowder:9000/clowder
TRAEFIK_HOST=localhost
GEOSTREAMS_PREFIX_PATH=/geostreams
GD_PREFIX_PATH=/

For geostreams, use the example application.conf https://github.com/geostreams/geostreams/blob/master/conf/reference.conf .
If using clowder, follow the link to initialize clowder https://github.com/clowder-framework/clowder#initializing-clowder.

Setting up the

machine

VM

Create a VM

Install python-openstackclient

Create an rc file using the example below used for gltg nebula

Code Block

export OS_AUTH_URL=http://nebula.ncsa.illinois.edu:5000/v2.0
export OS_TENANT_ID=c4121a001a8240d4a8b701d664ef4bf0    # Project ID in nebula
export OS_TENANT_NAME="GLTG"
export OS_PROJECT_NAME="GLTG"
export OS_USERNAME=<nebula_username>
export OS_PASSWORD=<nebula_password>
export OS_REGION_NAME="RegionOne"

export PS1="(openstack)[\e]0;\a]\h:\W \u [${OS_PROJECT_NAME}] $ "

Run `bash --rcfile <rc_file_name>` to create a virtual environment
Download and run the makevm-script (updated version in the comments) to create the vm.
Use one of the available Floating IP Addresses on Nebula and provide it to the script.

Once its created, install

Email help+neteng@ncsa.illinois.edu to setup domain name for machine.
The docker-compose file is setup to use Lets Encrypt for SSL certificate generation and renewal.

Setup Docker

- Install docker and docker-compose on the machine. Additionally, install pass and gnupg2 without which logging into docker will give an error.

`sudo

- - Docker Installation Instructions
  - Docker-compose installation Instructions
  - Code Block
    sudo apt-get install pass

gnupg2`

- - gnupg2
- Login to hub.ncsa.illinois.edu using the robot account login. The account information is stored in lastpass folder.
  - Code Block
    docker login -u `robot$githubgeostreams` -p `<TOKEN>` hub.ncsa.illinois.edu
Add Move the docker-compose previously setup file and all other relevant files it depends on to the machine. Update host names in the docker-compose file and application.conf
Run docker-compose up -d.
Initialize the database by running the geostreams container with command `initialize` once.

Configuring Build and Deployment Pipeline for Project

Add the following Github Actions public key to the machine.

`cat PATH_TO_KEY | ssh -i PATH_TO_ID_KEY user@hostname "cat >> ~/.ssh/authorized_keys"`

Code Block

language	bash
title	GitHub_Repo_Public_key
collapse	true

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCL/mqWBK1BTwZxKnZ546F9IbFlYY6qgFzh9xTM6eY+DxagKVV1BAr4kbqamnLYDzrOLC6zY+8k3xGS1HSxp8UAWXLnPzDkb13uXj+neGty7DwMIVWRVSc0JNa0cRaEKI1wC9AK1utKEU7aaGu6fZsmExmXNzIzxLIYVUFdW8G2GVoK9wNSba3OT2rneutgOUrb5PR6ADpfBEO8h48CcP6edw5A2HoJ0ZXySeadvnInOhp3yisO3khaZ7t4ZPRtRVRM+M+V9H+1JpOmsulfAZUyEdntzU1MkduFAz+X5T/h9IhHYlplqJ00GEjc/zIPS9y39Be5XqgvMadapupmeGpZWU6K/xvluATcP1xGqy7ytytAr6ZIbsCyKWGJXeUYsR8K1MdNC8zAVB+2Cnu3Df0TUf7xIV6sSx66MtehAADIGxtik4KGl5DZiWENgf0aCSeJHAjRkxGx4mxb/Cp9iercoKs+6uIAKd7fU/pYK9kw3z8WlkOsTcQcII9hzY6bQs= Smartfarm Key for Github Actions

Update github deployment action and add the details for the new project under strategy->matrix.

Code Block

language	yml

name: <project_name>
gd-name: <lerna_project_name>
prod-host: <production_host_name>
dev-host: <dev_host_name>
file-path: <docker-compose file path on VM>
description:

For development machines, an image with a develop tag will be created in hub.ncsa.illinois.edu and for production machines the latest tag will be used. The docker-compose files should be updated accordingly.

Page tree

Versions Compared

Old Version 7

New Version 8

Key

Creating Services and Containers

Setting up the

VM

Create a VM

Setup Docker

Configuring Build and Deployment Pipeline for Project

Page tree

Page History

Versions Compared

Old Version 7

New Version 8

Key

Creating Services and Containers

Setting up the

VM

Create a VM

Setup Docker

Configuring Build and Deployment Pipeline for Project