Resulting from the discussions surrounding NDS-310, we have a slightly better idea of how we plan to approach performance testing.

We now know:

1. GFS is our primary point of failure, and should be stress tested fairly heavily (NDS-262)
2. we need to do quite a bit more exploration on how many concurrent users we plan to support plan for 50 users initially, will scale as needed/possible
3. once since we know our potential capacity, we should revisit the approach that willis8 used to profile the system for the IASSIST workshop in May

Now we just need to explore the process itself.

Some unanswered questions:

• How do we learn our limits?
  • Capacity planning / monitoring - plan for 50 users initially
  • what is the workload?
  • what resource limits per user?
• What is the process for testing that we can handle the given workload
  • Write a test plan for testable components (see comment below)
  • Revisit IASSIST profiling techniques used by willis8
  • See NDS Labs Test Plan
• By what performance metric(s) do we judge pass/fail?
  • Given the above metric(s), what constitutes a failure?
  • Critical services crashing into "CrashLoopBackoff"? "Error"?
  • Dead openstack node? i.e. blackholed?
• What happens when we need to:
  • restart CoreOS? (NDS-346)
  • add GFS bricks? (NDS-529)
  • add kubernetes nodes? (NDS-528)
  • remove a kubernetes node for maintenance (NDS-528)

This ticket is complete when we have laid out how we plan to approach performance testing, including the metrics that we plan to use for pass/fail and service-specific, as well as an estimate of resource requirements to Kenton based on the capacity analysis.