Table of Contents

Objective

1. Generate load on the system for a given number of users
2. Monitor the system's resource utilization using Grafana
   • This will give us a benchmark of the expected "load" on the cluster
3. Take user feedback regarding general usability of the system under the desired load conditions
   • This will let us know if user performance has degraded due to any stress on the system
4. Take note of how any node additions / removals affect resource constraint, and to what degree
   • If the system's resources become constrained, add a node to the cluster alleviating the resource constraint
   • If the system is far over-saturated with resources, remove a node from the cluster to simulate a downed node

Resulting Actions

1 hour group testing
1 hour writing new issues

Phase 1: Labs Workbench + Management

Current Cluster Configuration

See inventory at: https://github.com/nds-org/ndslabs-deploy-tools/commit/d8d8ef30dac74b1fe84185c7abc6136516d60e7b

Resulting Actions

1 hour group testing
1 hour writing new issues

Phase 1: Labs Workbench + Management

Participants

• Craig Willis
• David Raila
• Mike Lambert

...

• https://kubedash.workbench.nationaldataservice.org/#!/
• https://grafana.workbench.nationaldataservice.org/dashboard/db/cluster

...

Results

• Mike: API server crashed with an unknown error shortly after beginning the test
  • I started owncloud + cloudcmd + postgres x2 + mysql + dspace simultaneously
  • Craig: server.go 1200 on latest (but which latest?) - changed to 1.0.5
  • no stack trace, so no ticket filed... if we see it again we will address it
• David: Catalog links do not seem to work in Firefox
  • See https://files.slack.com/files-pri/T16F0Q17E-F2MK2PPT9/firefox_error.txt
  • Cached page?
  • See 

    Jira
    server JIRA serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key NDS-173

• David: File Manager occasionally refuses to start in Chrome
  
  • Popup blocker?
• David: HTTP Basic is old-timey and gross
  • I agree.
• Mike: Redis has an HTTP endpoint?
  • See 

    Jira
    server JIRA serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key NDS-621

• Mike: Jenkins encountered the "no data available error"

  • See https://nationaldataservice.slack.com/files/bodom0015/F2MKNMY2E/jenkins_error.txt

  • See 

    Jira
    server JIRA serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key NDS-464

• Craig: pyCharm encountered "no data available" error
  • Process was hung and would not shut down
  • See https://nationaldataservice.slack.com/files/craig-willis/F2MK9HJKY/example_error.txt
  • See 

    Jira
    server JIRA serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key NDS-464

• Mike: Fedora Commons encountered SSL errors on the REST endpoint
  • SSL errors prevented CSS from rendering
  • New ticket: 

    Jira
    server JIRA serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key NDS-644

  • See https://nationaldataservice.slack.com/files/bodom0015/F2MKG62R3/fedora-error.txt
• Mike: Clowder Digest Extractor label missing from dropdown
  • Fixed on master: https://github.com/nds-org/ndslabs-specs/commit/e37ab4946f9a14eb305ac803aeec7b924e901443
• Mike: Clowder starts slower than it used to - need to bump up the readinessProbe to accommodate
  • New ticket: 

    Jira
    server JIRA serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key NDS-645

• Mike: Clowder extractors / toolserver fails due to 401 (HTTP basic auth)
  • PlantCV failed (the TERRA demo from NDSC5)
  • See https://nationaldataservice.slack.com/files/bodom0015/F2MLHT2AZ/extractors-error.txt
  • New ticket: 

    Jira
    server JIRA serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key NDS-646

• Craig: Somehow the endpoints are being returned as the home page?
  • I have seen this intermittently, but am unable
• Craig: Rstudio has a default password
  • See 

    Jira
    server JIRA serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key NDS-575

  • New ticket: 

    Jira
    server JIRA serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key NDS-648

• Craig: Cloud9 needs java 8 to build dataverse, also killed with OOM
  • New ticket: 

    Jira
    server JIRA serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key NDS-640

• Craig: Redis endpoint shouldn't be external
  • See 

    Jira
    server JIRA serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key NDS-621

• David: No numpy in JupyterLab
  • New ticket: 

    Jira
    server JIRA serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key NDS-647

• Mike: Kibana redirects to Grafana
  • New ticket: 
• Craig: Chisel didn't work as expected.
  • See 

    Jira
    server JIRA serverId b14d4ad9-eb00-3a94-88ac-a843fb6fa1ca key NDS-646

...

Phase 2: Bug Party

Participants

• Craig Willis
• David Raila
• Mike Lambert
• ???

Measurement Utilities

• https://kubedash.workbench.nationaldataservice.org/#!/
• https://grafana.workbench.nationaldataservice.org/dashboard/db/cluster

Results

Prognosis

Resulting Actions