In detail, what is the performance test scope? What subsystems, interfaces, components, etc. are in and out of scope for this test?

  • UI webserver / js scale
  • REST API / server scale
  • LoadBalancer scale
  • GlusterFS scale
    • i.e. Can 50+ users read / write to globalfs simultaneously?
    • Needs to be heavily exercised / stress tested
  • Are there other components or resiliency exercises that might be helpful?


NOTE: We should be able to easily scale up the LB / UI / API server using a replication controller to run multiple pods behind the service. We can even consider grouping them into the same RC, if we find some benefit there, although it is nice to be able to start / stop the UI and API server separately for development.

For the user interfaces (UIs) involved, how many concurrent users are expected for each (specify peak vs. nominal)?

  • Load test of ~30 users
  • Continuous load of ~50 users
  • Spikes of 50+ users
  • Configuration tests to tune cluster performance once stability is established

What does the target system (hardware) look like?

We don't actually know, as we do not yet control the underlying hardware

What we do know is this:

  • NDS Labs v1.0.X
  • Accessed through an Ingress LoadBalancer v0.8.2 based on NGINX
  • Running on Kubernetes v1.2.4
  • Running on CoreOS Stable v1122
  • Installed from an image on OpenStack Nebula
  • Running on OpenStack Nebula VMs

What is the Application Workload Mix of each system component?

Component%

Registration

10

Login

5

Searching the Application Catalog

20

Installing, configuring, and running applications from the catalog

35

Developing new applications to add to the catalog

30

What is the System Workload Mix? [Multiple workloads may be simulated in a single performance test].

  1. Benchmark A: ~10 users
    1. equal load (10% each)
  2. Benchmark B: ~30 users
    1. equal load (3.33% each)
  3. Benchmark C: ~50 users
    1. equal load (2% each)
  4. Benchmark D: ~70 users
    1. equal load (1.43% each)

Other scenarios worth taking a look at might for each benchmark include:

  • 50% heavy / 20% medium / 30% light
  • 75% heavy / 10% medium / 15% light
  • CPU / MEM / Disk / Network hog?
  • load spikes?

What are the time requirements for any/all back-end batch processes (specify peak vs. nominal)?

We do not have any batch processes. Start / stop time for services is our biggest time sink, but it is also highly variable. Our second biggest time sink is likely the time it takes to pull / extract the services' Docker images.

By what metrics do we judge pass / fail? What constitutes a failure by this metric?

Viable performance metrics might include:

  • total end-to-end request time => requests per second
  • maximum continuous load => number of simultaneous users
  • others?

What is the process by which load testing will be performed?

Option A

  1. Record one or more Selenium macros exercising the UI
  2. Playback the Selenium macro(s) on multiple threads simultaneously

Option B

  1. Script out some APi calls
  2. Playback the API calls using a program like felt, which renders the client-side js

Option C

  1. Use boom / hey / JMeter to test load API server

NOTE: although JMeter and friends have been a long accepted solution for server load testing, it does not work as well for client-side JavaScript frameworks like AngularJS. This means the test would not include the time taken to process and render the UI.

  • No labels