Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Creating Services and Containers

  1. Create a new project in geostreams. Instructions are here.
  2. Configure the docker-compose file for the services that need to be added. The docker-compose file linked below The docker-compose file for this project is based on the Default Geostreams file and traefik v2. This way minimal configuration is required on the machine.



    1. Code Block
      languageyml
      titledocker-compose.yml
      collapsetrue
      version: "3.3"
      
      services:
        traefik:
          image: traefik:latest
          networks:
            - geostreams
          volumes:
            - ./traefik:/traefik/
            - /var/run/docker.sock:/var/run/docker.sock:ro
          ports:
            - 80:80
            - 443:443
            - 8001:8001
      
          command:
            - --entrypoints.private.address=:8001
            - --entrypoints.web.address=:80
            - --entrypoints.websecure.address=:443
            - --entrypoints.web.http.redirections.entryPoint.to=websecure
            - --entrypoints.web.http.redirections.entryPoint.scheme=https
            - "--providers.docker=true"
            - "--api=true"
            - --providers.docker.exposedbydefault=false
      
      	  # SSL Certificate Generation using Lets Encrypt
            # - "--certificatesresolvers.le.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
            - --certificatesResolvers.le.acme.email=dev@mbclab.ncsa.illinois.edu
            - --certificatesResolvers.le.acme.storage=/traefik/acme.json
            - --certificatesResolvers.le.acme.httpChallenge=true
            - --certificatesResolvers.le.acme.httpChallenge.entryPoint=web
      
          labels:
            - "traefik.enable=true"
            - "traefik.http.routers.dashboard.service=api@internal"
      
            # Route setting
            - "traefik.http.routers.dashboard.rule=Host(`${TRAEFIK_HOST:-localhost}`)"
      
            # set TLS (https)
            - "traefik.http.routers.dashboard.tls=true"
            - "traefik.http.routers.dashboard.entrypoints=private"
            - "traefik.http.routers.dashboard.tls.certresolver=le"
      
      
        # ----------------------------------------------------------------------
        # GEOSTREAMS STACK
        # ----------------------------------------------------------------------
        geodashboard:
          image: hub.ncsa.illinois.edu/geostreams/gd-smartfarm:develop
          networks:
            - geostreams
          labels:
            - traefik.enable=true
            - traefik.http.services.geodashboard.loadbalancer.server.port=80
            - traefik.http.routers.geodashboard.rule=Host(`${TRAEFIK_HOST:-localhost}`) && (PathPrefix(`${GD_PREFIX_PATH:-/}`))
            - traefik.http.routers.geodashboard.entrypoints=websecure
            - traefik.http.routers.geodashboard.tls=true
            - traefik.http.routers.geodashboard.tls.certresolver=le
          restart: unless-stopped
          
        geostreams:
          image: geostreams/geostreams
          env_file:
            - ./geostreams.env
          networks:
            - geostreams
          labels:
            - traefik.enable=true
            - traefik.http.services.geostreams.loadbalancer.server.port=9000
            - traefik.http.routers.geostreams.rule=Host(`${TRAEFIK_HOST:-localhost}`) && (PathPrefix(`${GEOSTREAMS_PREFIX_PATH:-/}`))
            - traefik.http.routers.geostreams.entrypoints=websecure
            - traefik.http.routers.geostreams.tls=true
            - traefik.http.routers.geostreams.tls.certresolver=le
      
          volumes:
            - ./application.conf:/home/geostreams/conf/application.conf
            - ./messages.en:/home/geostreams/conf/messages.en
          healthcheck:
             test: ["CMD", "curl", "-s", "--fail", "http://localhost:9000/geostreams/api/status"]
          restart: unless-stopped 
        postgres:
          image: mdillon/postgis:9.5
          networks:
            - geostreams
          ports:
            - 5432:5432
          volumes:
            - ./postgres:/var/lib/postgresql/data
          restart: unless-stopped    
      
      
      
      networks:
        geostreams:
      



    2. The following .env file sets up the routes:

      Code Block
      languagebash
      TRAEFIK_HOST=localhost
      GEOSTREAMS_PREFIX_PATH=/geostreams
      GD_PREFIX_PATH=/
      
      


  3. For geostreams, use the example application.conf https://github.com/geostreams/geostreams/blob/master/conf/reference.conf .
  4. If using clowder, follow the link to initialize clowder https://github.com/clowder-framework/clowder#initializing-clowder.


Setting up the VM

  • Create a VM

    • Install python-openstackclient
    • Create an rc file using the example below used for gltg nebula

      Code Block
      export OS_AUTH_URL=http://nebula.ncsa.illinois.edu:5000/v2.0
      export OS_TENANT_ID=c4121a001a8240d4a8b701d664ef4bf0    # Project ID in nebula
      export OS_TENANT_NAME="GLTG"
      export OS_PROJECT_NAME="GLTG"
      export OS_USERNAME=<nebula_username>
      export OS_PASSWORD=<nebula_password>
      export OS_REGION_NAME="RegionOne"
      
      export PS1="(openstack)[\e]0;\a]\h:\W \u [${OS_PROJECT_NAME}] $ "


    • Run `bash --rcfile <rc_file_name>` to create a virtual environment
    • Download and run the makevm-script (updated version in the comments) to create the vm.
    • Use one of the available Floating IP Addresses on Nebula and provide it to the script.
    • Email help+neteng@ncsa.illinois.edu to setup domain name for machine.
    • The docker-compose file is setup to use Lets Encrypt for SSL certificate generation and renewal.


  • Setup Docker

  • Set up Backup


  • Initialize Geostreams

    • Move the docker-compose file and other relevant files to the machine.
    • Run docker-compose up -d.
    • Initialize the database by running the geostreams container with command `initialize` once.


Configuring Build and Deployment Pipeline for Project

  • Add the following Github Actions public key to the machine.

    `cat PATH_TO_KEY | ssh -i PATH_TO_ID_KEY user@hostname "cat >> ~/.ssh/authorized_keys"`

    Code Block
    titleGitHub_Repo_Public_key
    collapsetrue
    ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDCL/mqWBK1BTwZxKnZ546F9IbFlYY6qgFzh9xTM6eY+DxagKVV1BAr4kbqamnLYDzrOLC6zY+8k3xGS1HSxp8UAWXLnPzDkb13uXj+neGty7DwMIVWRVSc0JNa0cRaEKI1wC9AK1utKEU7aaGu6fZsmExmXNzIzxLIYVUFdW8G2GVoK9wNSba3OT2rneutgOUrb5PR6ADpfBEO8h48CcP6edw5A2HoJ0ZXySeadvnInOhp3yisO3khaZ7t4ZPRtRVRM+M+V9H+1JpOmsulfAZUyEdntzU1MkduFAz+X5T/h9IhHYlplqJ00GEjc/zIPS9y39Be5XqgvMadapupmeGpZWU6K/xvluATcP1xGqy7ytytAr6ZIbsCyKWGJXeUYsR8K1MdNC8zAVB+2Cnu3Df0TUf7xIV6sSx66MtehAADIGxtik4KGl5DZiWENgf0aCSeJHAjRkxGx4mxb/Cp9iercoKs+6uIAKd7fU/pYK9kw3z8WlkOsTcQcII9hzY6bQs= Smartfarm Key for Github Actions


  • Update github deployment action and add the details for the new project under strategy->matrix.

    Code Block
    languageyml
    name: <project_name>
    gd-name: <lerna_project_name>
    prod-host: <production_host_name>
    dev-host: <dev_host_name>
    file-path: <docker-compose file path on VM>
    description: 


  • For development machines, an image with a develop tag will be created in hub.ncsa.illinois.edu and for production machines the latest tag will be used. The docker-compose files should be updated accordingly.