Load Balancer
This section documents the results of NDS-239 - Getting issue details... STATUS . The goal of this ticket was to determine whether the Nginx ingress controller would be a performance bottleneck for the NDS Labs system.
Baseline service: Nginx
Load generation: boom
Use the boom load test generator to scale up concurrent requests on a Nebula m1.medium VM:
for i in `seq 1 10` do req=$((100*$i)) echo "bin/boom -cpus 4 -n 1000 -c $req http://perf-nginx.iassist.ndslabs.org/" bin/boom -cpus 4 -n 1000 -c $req http://perf-nginx.iassist.ndslabs.org/ sleep 1 done
Measuring latency and resource usage
Measuring latency: boom
Boom produces response time output, for example
bin/boom -cpus 4 -n 1000 -c 500 http://perf-nginx.iassist.ndslabs.org/ Summary: Total: 0.1539 secs Slowest: 0.1335 secs Fastest: 0.0193 secs Average: 0.0685 secs Requests/sec: 4842.2840 Status code distribution: [200] 745 responses Response time histogram: 0.019 [1] | 0.031 [28] |∎∎∎∎∎∎ 0.042 [110] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎ 0.054 [69] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎ 0.065 [161] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎ 0.076 [157] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎ 0.088 [60] |∎∎∎∎∎∎∎∎∎∎∎∎∎∎ 0.099 [38] |∎∎∎∎∎∎∎∎∎ 0.111 [49] |∎∎∎∎∎∎∎∎∎∎∎∎ 0.122 [37] |∎∎∎∎∎∎∎∎∎ 0.134 [35] |∎∎∎∎∎∎∎∎ Latency distribution: 10% in 0.0394 secs 25% in 0.0502 secs 50% in 0.0652 secs 75% in 0.0808 secs 90% in 0.1103 secs 95% in 0.1217 secs 99% in 0.1293 secs
Measuring latency: netperf
Measure latency and throughput to services inside kubernetes
Measuring CPU/Memory/IO utilization
Results
minflt/s | majflt/s | VSZ | RSS | %MEM | ||||||
proc1 | proc2 | proc1 | proc2 | proc1 | proc2 | proc1 | proc2 | proc1 | proc2 | |
15:56:52 | 0 | 0 | 0 | 0 | 326132 | 325992 | 15208 | 15068 | 0.38 | 0.37 |
15:56:53 | 0 | 0 | 0 | 0 | 326132 | 325992 | 15208 | 15068 | 0.38 | 0.37 |
15:56:54 | 3 | 0 | 0 | 0 | 326132 | 325992 | 15208 | 15068 | 0.38 | 0.37 |
15:56:55 | 29 | 299 | 0 | 0 | 325328 | 325992 | 14404 | 15068 | 0.36 | 0.37 |
15:56:56 | 0 | 477 | 0 | 0 | 325328 | 327576 | 14404 | 16360 | 0.36 | 0.4 |
15:56:57 | 0 | 0 | 0 | 0 | 325328 | 325768 | 14404 | 14844 | 0.36 | 0.37 |
15:56:58 | 0 | 648 | 0 | 0 | 325328 | 328416 | 14404 | 17216 | 0.36 | 0.42 |
15:56:59 | 0 | 0 | 0 | 0 | 325328 | 325328 | 14404 | 14404 | 0.36 | 0.36 |
15:57:00 | 0 | 1021 | 0 | 0 | 325328 | 329420 | 14404 | 18360 | 0.36 | 0.45 |
15:57:01 | 0 | 0 | 0 | 0 | 325328 | 326140 | 14404 | 15216 | 0.36 | 0.38 |
15:57:02 | 0 | 0 | 0 | 0 | 325328 | 326140 | 14404 | 15216 | 0.36 | 0.38 |
15:57:03 | 0 | 630 | 0 | 0 | 325328 | 326764 | 14404 | 15840 | 0.36 | 0.39 |
15:57:04 | 0 | 0 | 0 | 0 | 325328 | 325808 | 14404 | 14884 | 0.36 | 0.37 |
15:57:05 | 0 | 1002 | 0 | 0 | 325328 | 329908 | 14404 | 18840 | 0.36 | 0.46 |
15:57:06 | 0 | 47 | 0 | 0 | 325328 | 325628 | 14404 | 14704 | 0.36 | 0.36 |
15:57:07 | 1 | 1275 | 0 | 0 | 325328 | 330784 | 14404 | 19716 | 0.36 | 0.49 |
15:57:08 | 0 | 0 | 0 | 0 | 325328 | 325884 | 14404 | 14960 | 0.36 | 0.37 |
15:57:09 | 0 | 1502 | 0 | 0 | 325328 | 331960 | 14404 | 20756 | 0.36 | 0.51 |
15:57:10 | 0 | 0 | 0 | 0 | 325328 | 325328 | 14404 | 14404 | 0.36 | 0.36 |
15:57:11 | 0 | 1258 | 0 | 0 | 325328 | 329128 | 14404 | 18204 | 0.36 | 0.45 |
15:57:12 | 0 | 0 | 0 | 0 | 325328 | 325328 | 14404 | 14404 | 0.36 | 0.36 |
15:57:13 | 0 | 0 | 0 | 0 | 325328 | 325328 | 14404 | 14404 | 0.36 | 0.36 |
%usr | %system | %guest | %CPU | |||||
proc1 | proc2 | proc1 | proc2 | proc1 | proc2 | proc1 | proc2 | |
15:56:10 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
15:56:11 | 0 | 6 | 0 | 6 | 0 | 0 | 0 | 12 |
15:56:12 | 3 | 0 | 3 | 0 | 0 | 0 | 6 | 0 |
15:56:13 | 3 | 0 | 3 | 0 | 0 | 0 | 6 | 0 |
15:56:14 | 5 | 0 | 5 | 0 | 0 | 0 | 10 | 0 |
15:56:15 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 |
15:56:16 | 4 | 0 | 4 | 0 | 0 | 0 | 8 | 0 |
15:56:17 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 |
15:56:18 | 5 | 0 | 6 | 0 | 0 | 0 | 11 | 0 |
15:56:19 | 1 | 0 | 1 | 0 | 0 | 0 | 2 | 0 |
15:56:20 | 2 | 0 | 4 | 0 | 0 | 0 | 6 | 0 |
15:56:21 | 1 | 0 | 1 | 0 | 0 | 0 | 2 | 0 |
15:56:22 | 3 | 0 | 4 | 0 | 0 | 0 | 7 | 0 |
15:56:23 | 1 | 0 | 0 | 0 | 0 | 0 | 1 | 0 |
15:56:24 | 0 | 4 | 0 | 5 | 0 | 0 | 0 | 9 |
15:56:25 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | 2 |
15:56:26 | 0 | 5 | 1 | 6 | 0 | 0 | 1 | 11 |
15:56:27 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
15:56:28 | 4 | 0 | 6 | 0 | 0 | 0 | 10 | 0 |
15:56:29 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 |
15:56:30 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
Concurrent connections
Scaling services
Large-file upload/download
Killing the loadbalancer