Page History

Monitoring

Qualys

Qualys is used by NCSA IT for vulnerability assessment and management.  Qualys will require SSH access to any public-facing host or service.  

NCSA Security has opened a ticket for this: https://jira.ncsa.illinois.edu/browse/SECOPS-340. We need to:

• Provide a list of IPs that we want scanned (in general they try to scan one system of each type)
• Security will provide SSH public key to use to login to local qualys user account.
• Instructions for setting up Qualys user: https://wiki.ncsa.illinois.edu/pages/viewpage.action?pageId=41461115
• Provide email address for reports.
• We will also need to do this to public-facing containers (e.g., Nginx controller)

Associated tickets:

• Jira
  server JIRA serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key NDS-565

Nagios

Nagios is an open source monitoring system. In general, the Nagios server is installed in one location and the Nagios Remote Plugin Executor (NRPE) on each node to be monitored. Nagios provides public service monitoring through standard plugins (e.g., DNS, HTTP, SMTP, etc).  It provides private service monitoring throug NRPE (CPU, memory, disk, etc).

For NDS Labs, we'll do the following:

• Evaluate using  https://github.com/QuantumObject/docker-nagios
• Create Nagios server Docker image if docker-nagios is not acceptable,  following the instructions in 
  • https://www.digitalocean.com/community/tutorials/how-to-install-nagios-4-and-monitor-your-servers-on-ubuntu-14-04
• Create Nagios daemonset for NRPE following the instructions in 
  • https://www.digitalocean.com/community/tutorials/how-to-install-nagios-4-and-monitor-your-servers-on-ubuntu-14-04
• Provision VM to run Nagios server at remote site (TACC)
• Create nagios configuration github repository to maintain versioned nagios monitoring per-cluster (starting with beta) configurations
• Configure Nagios contacts
• Configure Nagios hosts for priority systems.  This includes;
  • Ingress/Nginx
  • Web UI/API including Kube API/Etcd availability
  • Kube system (GFS, LMA tools, etc)
  • Openstack
  • Backups
  • NOTE: nagios server will not be able to directly access cluster servers which currently live in private network without going through ingress loadbalancer.  Monitoring should be direct if possible, which is addressed by NDS-581

Additionally, we will want to add health checks (healthz) to all system services.

Associated tickets:

• Jira
  server JIRA serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key NDS-566

Usage monitoring

We will use the Kubernetes addons, specifically ELK and Grafana, to monitor usage during the beta periodSee NDS Labs Monitoring.

Backup/Disaster Recovery

...