Versions Compared

Old Version 12

changes.mady.by.user Sara Lambert

Saved on

compared with

New Version Current

changes.mady.by.user Sara Lambert

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Overview

This is a page to house the results of the manual load testing done on the NDS Labs Workbench (Beta)

Table of Contents

Objective

  1. Generate load on the system for a given number of users
  2. Monitor the system's resource utilization using Grafana
    • This will give us a benchmark of the expected "load" on the cluster
  3. Take user feedback regarding general usability of the system under the desired load conditions
    • This will let us know if user performance has degraded due to any stress on the system
  4. Take note of how any node additions / removals affect resource constraint, and to what degree
    • If the system's resources become constrained, add a node to the cluster alleviating the resource constraint
    • If the system is far over-saturated with resources, remove a node from the cluster to simulate a downed node

...

1 hour group testing
1 hour writing new issues

Phase 1: Labs Workbench + Management

Current Cluster Configuration

See inventory at: https://github.com/nds-org/ndslabs-deploy-tools/commit/d8d8ef30dac74b1fe84185c7abc6136516d60e7b

Resulting Actions

1 hour group testing
1 hour writing new issues

Phase 1: Labs Workbench + Management

Workbench Version

  • 1.0.5

Participants

  • Craig Willis
  • David Raila
  • Mike Lambert

...

...

Results

  • Mike: API server crashed with an unknown error shortly after beginning the test
    • I started owncloud + cloudcmd + postgres x2 + mysql + dspace simultaneously
    • Craig: server.go 1200 on latest (but which latest?) - changed to 1.0.5
    • no stack trace, so no ticket filed... if we see it again we will address it
  • David: Catalog links do not seem to work in Firefox
  • David: File Manager occasionally refuses to start in Chrome
    • Popup blocker?
  • David: HTTP Basic is old-timey and gross
    • I agree.
  • Mike: Redis has an HTTP endpoint?
    • See 
      Jira
      serverJIRA
      serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
      keyNDS-621
  • Mike: Jenkins encountered the "no data available error"
  • Craig: pyCharm encountered "no data available" error
  • Mike: Fedora Commons encountered SSL errors on the REST endpoint
    • SSL errors prevented CSS from rendering
    • New ticket: 
      Jira
      serverJIRA
      serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
      keyNDS-644
  • Mike: Clowder Digest Extractor label missing from dropdown
  • Mike: Clowder starts slower than it used to - need to bump up the readinessProbe to accommodate
    • New ticket: 
      Jira
      serverJIRA
      serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
      keyNDS-645
  • Mike: Clowder extractors / toolserver fails due to 401 (HTTP basic auth)
  • Craig: Somehow the endpoints are being returned as the home page?
    • I have seen this intermittently, but am unable
  • Craig: Rstudio has a default password
    • See 
      Jira
      serverJIRA
      serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
      keyNDS-575
    • New ticket: 
      Jira
      serverJIRA
      serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
      keyNDS-648
  • Craig: Cloud9 needs java 8 to build dataverse, also killed with OOM
    • New ticket: 
      Jira
      serverJIRA
      serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
      keyNDS-640
  • Craig: Redis endpoint shouldn't be external
    • See 
      Jira
      serverJIRA
      serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
      keyNDS-621
  • David: No numpy in JupyterLab
    • New ticket: 
      Jira
      serverJIRA
      serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
      keyNDS-647
  • Mike: Kibana redirects to Grafana
    • New ticket: 
      Jira
      serverJIRA
      serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
      keyNDS-649
  • Craig: Chisel didn't work as expected.
    • See 
      Jira
      serverJIRA
      serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
      keyNDS-646

...

  • Jira
    serverJIRA
    serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
    keyNDS-646
  • Jira
    serverJIRA
    serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
    keyNDS-647
  • Jira
    serverJIRA
    serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
    keyNDS-645
  • Jira
    serverJIRA
    serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
    keyNDS-644
  • Jira
    serverJIRA
    serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
    keyNDS-649

Phase 2: Bug Party

Workbench Version

  • 1.0.6

Participants

  • Craig Willis
  • David Raila
  • Mike Lambert
  • Michal
  • Jing
  • Sandeep
  • Qiyue
  • Marcus

Measurement Utilities

Results

  • Michal: No indication of which fields are required for registration
    • New ticket: 
      Jira
      serverJIRA
      serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
      keyNDS-661
  • Michal: Needs to know what they are doing (i.e., Quickstart)
    • See 
      Jira
      serverJIRA
      serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
      keyNDS-485
  • David: Recommend whitelisting our site for / disabling pop-ups - can we detect this and make a recommendation to users without correct settings?
    • New ticket: 
      Jira
      serverJIRA
      serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
      keyNDS-662
  • Michal: couldn't sign up for DSpace - address in use
    • This is a more general problem with any service that generates admin credentials... user should be directed to the Config page
    • See 
      Jira
      serverJIRA
      columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
      serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
      keyNDS-560
  • Jing: Docker image name validation is incomplete
    • Underscore should be among accepted characters
    • New ticket: 
      Jira
      serverJIRA
      serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
      keyNDS-663
  • Jing: No indication of required fields during spec create?
    • See 
      Jira
      serverJIRA
      serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
      keyNDS-661
  • Mike: Saw a failure adding Sufia, only one time... next time it added properly
    • Was not able to reliably recreate, and no error message given... will file a ticket if I see it again
  • Jing: Custom service failed to start
    • See JSON: https://nationaldataservice.slack.com/files/bodom0015/F2NG4BRHS/jings_service_error.txt
    • This was due to her container running a single command and then stopping
    • To the user, this appears to be a CrashLoop, even though the command has successfully run
    • Need to discuss how to handle non-service container... perhaps Kubernetes jobs instead of pods?
    • New ticket: 
      Jira
      serverJIRA
      serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
      keyNDS-664
  • Qiyue: No indication of which fields are required for registration
    • See 
      Jira
      serverJIRA
      columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
      serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
      keyNDS-661
  • Qiyue: How do we use different versions... for example: Cloud9 Java7 vs Cloud9 Java8
    • New ticket: 
      Jira
      serverJIRA
      serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
      keyNDS-665
  • Qiyue: What is the storage quota? 20GB
    • See 
      Jira
      serverJIRA
      serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
      keyNDS-201
  • Jing: Redis is missing an info link
  • Marcus: NDS Confluence went down, as a result icons could not load
    • See 
      Jira
      serverJIRA
      serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
      keyNDS-591
  • Jing: Error messages are confusing - need to translate the error messages (or document them)
    • New ticket: 
      Jira
      serverJIRA
      serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
      keyNDS-666
  • Michal: would it be better to have a pre-populated instances?
    • This would be nice, but may be difficult to handle programmatically in a general way
  • Qiyue: Any plans to support Fortran?
    • New ticket: 
      Jira
      serverJIRA
      serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
      keyNDS-667
  • Mike: Kibana caused the following nagios alerts to come from the LMA node:
    • "workbench-lma/Load is WARNING:"
    • "WARNING - load average: 8.94, 7.92, 6.52"
    • New ticket: 
      Jira
      serverJIRA
      serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
      keyNDS-668
  • Jing: Order of top menu – Catalog then Applications?
    • New ticket: 
      Jira
      serverJIRA
      serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
      keyNDS-669
  • Michal: Can I use this framework to compare montecarlo simulations?
    • See 
      Jira
      serverJIRA
      serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
      keyNDS-664
  • David: Green/red bars are too big or other parts of application UI are too small.
    • I would be happy to look over any UI mockups that you would be willing to provide
  • David: Stopped "X" is confusing – thought it was delete
    • New ticket: 
      Jira
      serverJIRA
      serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
      keyNDS-670
  • Sandeep: Better way of differentiating user versus system specs (little icon isn't readily apparent)
  • Sandeep: Help pages as Wiki isn't great – should be part of application
    • See 
      Jira
      serverJIRA
      serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
      keyNDS-485
  • Marcus: Not sure what to do (quickstart/tutorial)
    • See 
      Jira
      serverJIRA
      serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
      keyNDS-485
  • Marcus: Documentation isn't clear
    • See 
      Jira
      serverJIRA
      serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
      keyNDS-485
  • Marcus: Can I use this to launch Jupyter notebooks for BrownDog users?
    • Labs Workbench is more for testing and development - publically-accessible services with real users are highly discouraged
    • That being said, if users did want to use Workbench to spin up personal notebook for their own private analysis, that would be highly encouraged
  • Craig: iRODS problems (multiple volumes; CloudBrowser Zone)
    • See 
      Jira
      serverJIRA
      serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
      keyNDS-654
  • Craig: Multiple port problem
    • See 
      Jira
      serverJIRA
      serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
      keyNDS-655

Prognosis

Aside from a slew of UX problems, the platform itself performed rather well!

Usage from 8 users peaked at:

  • ~10% Memory
  • ~6% CPU

This means that we should be able to easily support our target of 50 users.

Optimistically, assuming that gluster doesn't fall over and that our usage scales fairly linearly with increasing users, these results mean that we might be able to support upward of 60 or 70 users simultaneously using the Beta cluster without needing to resize it.

Resulting Actions

  • Jira
    serverJIRA
    serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
    keyNDS-201
  • Jira
    serverJIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
    keyNDS-560
  • Jira
    serverJIRA
    serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
    keyNDS-591
  • Jira
    serverJIRA
    serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
    keyNDS-485
  • Jira
    serverJIRA
    serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
    keyNDS-654
  • Jira
    serverJIRA
    serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
    keyNDS-655
  • Jira
    serverJIRA
    serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
    keyNDS-661
  • Jira
    serverJIRA
    columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
    serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
    keyNDS-662
  • Jira
    serverJIRA
    serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
    keyNDS-663
  • Jira
    serverJIRA
    serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
    keyNDS-664
  • Jira
    serverJIRA
    serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
    keyNDS-665
  • Jira
    serverJIRA
    serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
    keyNDS-666
  • Jira
    serverJIRA
    serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
    keyNDS-667
  • Jira
    serverJIRA
    serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
    keyNDS-668
  • Jira
    serverJIRA
    serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
    keyNDS-669
  • Jira
    serverJIRA
    serverIdb14d4ad9-eb00-3a94-88ac-a843fb6fa1ca
    keyNDS-670