ab压测报告

测试条件

• ab加-r参数（Don't exit on socket receive errors.）
• ab加-s参数指定超时时间（Seconds to max. wait for each response），避免aprpollsetpoll: The timeout specified has expired (70007)错误.参考链接
• 删除待测deployment的hpa，部署一个pod，避免扩容影响测试结果
• cpu 8000m， 内存10G，避免因为资源的问题干扰测试结果
• 压测页面为nginx 404页面
• 压测时间300s
• 待测服务配置ingress，并加域名
• 容器 net.ipv4.iplocalport_range配置为32768 61000

ab无k参数

Pod IP

/ # ab -c100 -t300 -n5000000 http://172.20.42.94/b; ss -ant |wc -l
This is ApacheBench, Version 2.3 <$Revision: 1826891 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 172.20.42.94 (be patient)
Completed 500000 requests
Completed 1000000 requests
Completed 1500000 requests
Completed 2000000 requests
Finished 2470218 requests


Server Software:        nginx
Server Hostname:        172.20.42.94
Server Port:            80

Document Path:          /b
Document Length:        162 bytes

Concurrency Level:      100
Time taken for tests:   300.002 seconds
Complete requests:      2470218
Failed requests:        0
Non-2xx responses:      2470284
Total transferred:      753436620 bytes
HTML transferred:       400186008 bytes
Requests per second:    8234.00 [#/sec] (mean)
Time per request:       12.145 [ms] (mean)
Time per request:       0.121 [ms] (mean, across all concurrent requests)
Transfer rate:          2452.57 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    6   9.5      6    1010
Processing:     1    6   4.3      6     412
Waiting:        0    4   4.2      4     412
Total:          2   12  10.5     12    1021

Percentage of the requests served within a certain time (ms)
  50%     12
  66%     13
  75%     14
  80%     14
  90%     16
  95%     17
  98%     17
  99%     18
 100%   1021 (longest request)
25

域名

/ # ab -c100 -t300 -n5000000 http://k8s.dev.example.com/b; ss -ant |wc -l
This is ApacheBench, Version 2.3 <$Revision: 1826891 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking k8s.dev.example.com (be patient)
Completed 500000 requests
Completed 1000000 requests
Completed 1500000 requests
Completed 2000000 requests
Completed 2500000 requests
Finished 2507606 requests


Server Software:        
Server Hostname:        k8s.dev.example.com
Server Port:            80

Document Path:          /b
Document Length:        162 bytes

Concurrency Level:      100
Time taken for tests:   300.002 seconds
Complete requests:      2507606
Failed requests:        0
Non-2xx responses:      2507635
Total transferred:      727214150 bytes
HTML transferred:       406236870 bytes
Requests per second:    8358.62 [#/sec] (mean)
Time per request:       11.964 [ms] (mean)
Time per request:       0.120 [ms] (mean, across all concurrent requests)
Transfer rate:          2367.21 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    3   3.0      3    1008
Processing:     0    9  13.9      6     221
Waiting:        0    8  13.9      5     203
Total:          1   12  14.1     10    1019

Percentage of the requests served within a certain time (ms)
  50%     10
  66%     11
  75%     12
  80%     13
  90%     18
  95%     41
  98%     67
  99%     77
 100%   1019 (longest request)
46

ClusterIP

/ # ab -r -c100 -t300 -n5000000 -s300 http://169.169.14.147/b; ss -ant |wc -l
This is ApacheBench, Version 2.3 <$Revision: 1826891 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 169.169.14.147 (be patient)
apr_pollset_poll: The timeout specified has expired (70007)
Total of 34163 requests completed
103

总结

• 容器ip: 完成247万请求，平均12.1ms, 99% 18ms以内， 容器cpu使用约3500m
• 域名： 完成250万请求，平均11.9ms，90% 18ms以内，容器cpu使用约700m（可能是因为nginx upstream配置了keepalive）
• clusterIP: 完成3.4万请求，报错 aprpollsetpoll: The timeout specified has expired (70007)退出

ab加k参数

Pod IP

/ # ab -r -c100 -t300 -n20000000 -k http://172.20.42.94/b; ss -ant |wc -l
This is ApacheBench, Version 2.3 <$Revision: 1826891 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 172.20.42.94 (be patient)
Completed 2000000 requests
Completed 4000000 requests
Completed 6000000 requests
Completed 8000000 requests
Completed 10000000 requests
Completed 12000000 requests
Finished 13954519 requests


Server Software:        nginx
Server Hostname:        172.20.42.94
Server Port:            80

Document Path:          /b
Document Length:        162 bytes

Concurrency Level:      100
Time taken for tests:   300.001 seconds
Complete requests:      13954519
Failed requests:        0
Non-2xx responses:      13954519
Keep-Alive requests:    13815022
Total transferred:      4325203405 bytes
HTML transferred:       2260632078 bytes
Requests per second:    46514.89 [#/sec] (mean)
Time per request:       2.150 [ms] (mean)
Time per request:       0.021 [ms] (mean, across all concurrent requests)
Transfer rate:          14079.39 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.3      0      14
Processing:     0    2   0.8      2      18
Waiting:        0    2   0.8      2      18
Total:          0    2   0.9      2      27

Percentage of the requests served within a certain time (ms)
  50%      2
  66%      2
  75%      2
  80%      3
  90%      4
  95%      4
  98%      5
  99%      5
 100%     27 (longest request)
3

域名

/ # ab -r -c100 -t300 -n20000000 -k http://k8s.dev.example.com/b; ss -ant |wc -l
This is ApacheBench, Version 2.3 <$Revision: 1826891 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking k8s.dev.example.com (be patient)
Completed 2000000 requests
Completed 4000000 requests
Completed 6000000 requests
Completed 8000000 requests
Completed 10000000 requests
Finished 11002187 requests


Server Software:        
Server Hostname:        k8s.dev.example.com
Server Port:            80

Document Path:          /b
Document Length:        162 bytes

Concurrency Level:      100
Time taken for tests:   300.000 seconds
Complete requests:      11002187
Failed requests:        0
Non-2xx responses:      11002187
Keep-Alive requests:    10892208
Total transferred:      3245095270 bytes
HTML transferred:       1782354294 bytes
Requests per second:    36673.93 [#/sec] (mean)
Time per request:       2.727 [ms] (mean)
Time per request:       0.027 [ms] (mean, across all concurrent requests)
Transfer rate:          10563.45 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0      15
Processing:     0    3   4.9      2     323
Waiting:        0    3   4.9      2     323
Total:          0    3   4.9      2     324

Percentage of the requests served within a certain time (ms)
  50%      2
  66%      2
  75%      3
  80%      3
  90%      5
  95%      8
  98%     15
  99%     22
 100%    324 (longest request)
3

ClusterIP

/ # ab -r -c100 -t300 -n20000000 -k http://169.169.14.147/b; ss -ant |wc -l
This is ApacheBench, Version 2.3 <$Revision: 1826891 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 169.169.14.147 (be patient)
Completed 2000000 requests
Completed 4000000 requests
Completed 6000000 requests
Completed 8000000 requests
Completed 10000000 requests
Finished 11123175 requests


Server Software:        nginx
Server Hostname:        169.169.14.147
Server Port:            80

Document Path:          /b
Document Length:        162 bytes

Concurrency Level:      100
Time taken for tests:   300.000 seconds
Complete requests:      11123175
Failed requests:        0
Non-2xx responses:      11123176
Keep-Alive requests:    11011978
Total transferred:      3447628570 bytes
HTML transferred:       1801954512 bytes
Requests per second:    37077.23 [#/sec] (mean)
Time per request:       2.697 [ms] (mean)
Time per request:       0.027 [ms] (mean, across all concurrent requests)
Transfer rate:          11222.74 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1  33.1      0    1012
Processing:     0    2   0.9      1     205
Waiting:        0    2   0.9      1     205
Total:          0    3  33.1      1    1019
WARNING: The median and mean for the processing time are not within a normal deviation
        These results are probably not that reliable.
WARNING: The median and mean for the waiting time are not within a normal deviation
        These results are probably not that reliable.

Percentage of the requests served within a certain time (ms)
  50%      1
  66%      2
  75%      2
  80%      2
  90%      3
  95%      3
  98%      4
  99%      5
 100%   1019 (longest request)
34

ipvsadm -ln

TCP  169.169.14.147:80 rr
  -> 172.20.42.94:80              Masq    1      55         28169     
TCP  169.169.31.173:80 rr
TCP  169.169.14.147:80 rr
  -> 172.20.42.94:80              Masq    1      49         28177     
TCP  169.169.31.173:80 rr
TCP  169.169.14.147:80 rr
  -> 172.20.42.94:80              Masq    1      44         28179     
TCP  169.169.31.173:80 rr
TCP  169.169.14.147:80 rr
  -> 172.20.42.94:80              Masq    1      68         28156     
TCP  169.169.31.173:80 rr
TCP  169.169.14.147:80 rr
  -> 172.20.42.94:80              Masq    1      61         28162     
TCP  169.169.31.173:80 rr
TCP  169.169.14.147:80 rr
  -> 172.20.42.94:80              Masq    1      55         28169 

总结

• 容器IP: 完成 1395.4 万， 平均2.1ms，99% 5ms以内，容器cpu使用约3200m
• 域名： 完成1100.2 万， 平均2.7ms, 90% 5ms以内， 容器cpu使用约2548m
• clusterIP: 完成 1112.3 万，平均 2.69ms，99% 5ms以内，容器cpu使用约2500m

obsolete

短连接（无-k参数）时，ss -ant |wc -l 看到的数量很少，可能是因为短连接时主动断开连接的是服务端，客户端没有TIME-WAIT状态。

/ # ab -c100 -t30 -q “http://169.169.14.147/b”;ss -ant |wc -l This is ApacheBench, Version 2.3 <$Revision: 1843412 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 169.169.14.147 (be patient)…..done

Server Software: nginx Server Hostname: 169.169.14.147 Server Port: 80

Document Path: /b Document Length: 162 bytes

Concurrency Level: 100 Time taken for tests: 30.020 seconds Complete requests: 15988 Failed requests: 0 Non-2xx responses: 16050 Total transferred: 4895250 bytes HTML transferred: 2600100 bytes Requests per second: 532.58 [#/sec] (mean) Time per request: 187.766 [ms] (mean) Time per request: 1.878 [ms] (mean, across all concurrent requests) Transfer rate: 159.24 [Kbytes/sec] received

Connection Times (ms)

            min  mean[+/-sd] median   max

Connect: 0 116 319.1 0 1009 Processing: 0 4 16.0 1 225 Waiting: 0 3 12.3 0 224 Total: 0 120 318.5 1 1056

Percentage of the requests served within a certain time (ms)

50%      1
66%      2
75%      7
80%     11
90%   1002
95%   1003
98%   1004
99%   1004

100% 1056 (longest request)

ss -ant: 1431

[root@k8s-node-199 ~]# ipvsadm -ln |grep -A1 “14.147” TCP 169.169.14.147:80 rr

1. > 172.20.36.44:80 Masq 1 0 8976

/ # ab -c100 -t30 -q “http://k8s.dev.scloud.k8s-demo.cn/b”;ss -ant |wc -l This is ApacheBench, Version 2.3 <$Revision: 1843412 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking k8s.dev.scloud.k8s-demo.cn (be patient)…..done

Server Software:
Server Hostname: k8s.dev.scloud.k8s-demo.cn Server Port: 80

Document Path: /b Document Length: 162 bytes

Concurrency Level: 100 Time taken for tests: 7.659 seconds Complete requests: 50000 Failed requests: 0 Non-2xx responses: 50000 Total transferred: 14500000 bytes HTML transferred: 8100000 bytes Requests per second: 6528.06 [#/sec] (mean) Time per request: 15.318 [ms] (mean) Time per request: 0.153 [ms] (mean, across all concurrent requests) Transfer rate: 1848.77 [Kbytes/sec] received

Connection Times (ms)

            min  mean[+/-sd] median   max

Connect: 0 6 2.8 5 24 Processing: 1 10 6.9 8 86 Waiting: 1 8 6.4 6 86 Total: 2 15 7.2 13 88

Percentage of the requests served within a certain time (ms)

50%     13
66%     16
75%     18
80%     20
90%     24
95%     29
98%     37
99%     42

100% 88 (longest request) 55

[root@k8s-node-199 kubernetes]# ipvsadm -ln |grep -A1 “14.147” TCP 169.169.14.147:80 rr

1. > 172.20.36.44:80 Masq 1 0 0

/ # ab -c500 -t30 -q “http://k8s.dev.scloud.k8s-demo.cn/b”;ss -ant |wc -l This is ApacheBench, Version 2.3 <$Revision: 1843412 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking k8s.dev.scloud.k8s-demo.cn (be patient)…..done

Server Software:
Server Hostname: k8s.dev.scloud.k8s-demo.cn Server Port: 80

Document Path: /b Document Length: 162 bytes

Concurrency Level: 500 Time taken for tests: 8.952 seconds Complete requests: 50000 Failed requests: 0 Non-2xx responses: 50000 Total transferred: 14500000 bytes HTML transferred: 8100000 bytes Requests per second: 5585.19 [#/sec] (mean) Time per request: 89.522 [ms] (mean) Time per request: 0.179 [ms] (mean, across all concurrent requests) Transfer rate: 1581.74 [Kbytes/sec] received

Connection Times (ms)

            min  mean[+/-sd] median   max

Connect: 0 20 49.8 16 1034 Processing: 4 57 203.2 32 5013 Waiting: 2 50 202.5 26 5012 Total: 10 77 207.7 53 5025

Percentage of the requests served within a certain time (ms)

50%     53
66%     63
75%     70
80%     75
90%     86
95%    100
98%    268
99%   1028

100% 5025 (longest request) 200

/ # ab -c500 -t30 -q -k “http://k8s.dev.scloud.k8s-demo.cn/b”;ss -ant |wc -l This is ApacheBench, Version 2.3 <$Revision: 1843412 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking k8s.dev.scloud.k8s-demo.cn (be patient)…..done

Server Software:
Server Hostname: k8s.dev.scloud.k8s-demo.cn Server Port: 80

Document Path: /b Document Length: 162 bytes

Concurrency Level: 500 Time taken for tests: 8.930 seconds Complete requests: 50000 Failed requests: 0 Non-2xx responses: 50000 Keep-Alive requests: 49656 Total transferred: 14748280 bytes HTML transferred: 8100000 bytes Requests per second: 5599.03 [#/sec] (mean) Time per request: 89.301 [ms] (mean) Time per request: 0.179 [ms] (mean, across all concurrent requests) Transfer rate: 1612.81 [Kbytes/sec] received

Connection Times (ms)

            min  mean[+/-sd] median   max

Connect: 0 0 3.6 0 48 Processing: 1 79 352.7 81 8014 Waiting: 1 79 352.7 81 8014 Total: 1 79 353.6 82 8047

Percentage of the requests served within a certain time (ms)

50%     82
66%     89
75%     93
80%     95
90%    102
95%    107
98%    115
99%    287

100% 8047 (longest request) 499

端口不够用？

/ # ab -c500 -t300 -q "http://169.169.14.147/b";ss -ant |wc -l
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 169.169.14.147 (be patient)...apr_pollset_poll: The timeout specified has expired (70007)
Total of 49966 requests completed
1848

/ # ab -c100 -t300 -q “http://169.169.14.147/b”;ss -ant |wc -l This is ApacheBench, Version 2.3 <$Revision: 1843412 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 169.169.14.147 (be patient)…aprsocketrecv: Connection reset by peer (104) Total of 31099 requests completed 1517

TCP 169.169.14.147:80 rr

1. > 172.20.36.44:80 Masq 1 83 26090
   TCP 169.169.14.147:80 rr
2. > 172.20.36.44:80 Masq 1 85 26160
   TCP 169.169.14.147:80 rr
3. > 172.20.36.44:80 Masq 1 85 26179
   TCP 169.169.14.147:80 rr
4. > 172.20.36.44:80 Masq 1 85 26179
   TCP 169.169.14.147:80 rr
5. > 172.20.36.44:80 Masq 1 85 26216
   TCP 169.169.14.147:80 rr
6. > 172.20.36.44:80 Masq 1 87 26231

/ # ab -c100 -t300 -n 500000 -q “http://k8s.dev.scloud.k8s-demo.cn/b”;ss -ant |wc -l This is ApacheBench, Version 2.3 <$Revision: 1843412 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking k8s.dev.scloud.k8s-demo.cn (be patient)…..done

Server Software:
Server Hostname: k8s.dev.scloud.k8s-demo.cn Server Port: 80

Document Path: /b Document Length: 162 bytes

Concurrency Level: 100 Time taken for tests: 80.888 seconds Complete requests: 500000 Failed requests: 0 Non-2xx responses: 500000 Total transferred: 145000000 bytes HTML transferred: 81000000 bytes Requests per second: 6181.36 [#/sec] (mean) Time per request: 16.178 [ms] (mean) Time per request: 0.162 [ms] (mean, across all concurrent requests) Transfer rate: 1750.58 [Kbytes/sec] received

Connection Times (ms)

            min  mean[+/-sd] median   max

Connect: 0 6 5.8 5 1018 Processing: 1 10 6.8 8 249 Waiting: 0 8 6.3 6 226 Total: 1 16 8.9 15 1056

Percentage of the requests served within a certain time (ms)

50%     15
66%     17
75%     19
80%     20
90%     24
95%     29
98%     38
99%     44

100% 1056 (longest request) 72

宿主机上测试clusterIP，local port从2000到65000，可用端口6万多

TCP  169.169.14.147:80 rr
  -> 172.20.36.44:80              Masq    1      0          62945     
TCP  169.169.14.147:80 rr
  -> 172.20.36.44:80              Masq    1      0          62943     
TCP  169.169.14.147:80 rr
  -> 172.20.36.44:80              Masq    1      0          62941     
TCP  169.169.14.147:80 rr
  -> 172.20.36.44:80              Masq    1      0          62941     
TCP  169.169.14.147:80 rr
  -> 172.20.36.44:80              Masq    1      0          62941     
TCP  169.169.14.147:80 rr
  -> 172.20.36.44:80              Masq    1      0          62941 

keepalive可用完成clusterIP测试 [root@k8s-node-199 ~]# ab -c100 -t300 -n 500000 -q “http://169.169.14.147/b”;ss -ant |wc -l This is ApacheBench, Version 2.3 <$Revision: 1430300 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 169.169.14.147 (be patient)…aprsocketrecv: Connection reset by peer (104) Total of 63443 requests completed 102 [root@k8s-node-199 ~]# ab -c100 -t300 -n 500000 -q -k “http://169.169.14.147/b”;ss -ant |wc -l This is ApacheBench, Version 2.3 <$Revision: 1430300 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 169.169.14.147 (be patient)…..done

Server Software: nginx Server Hostname: 169.169.14.147 Server Port: 80

Document Path: /b Document Length: 162 bytes

Concurrency Level: 100 Time taken for tests: 70.406 seconds Complete requests: 500000 Failed requests: 0 Write errors: 0 Non-2xx responses: 500000 Keep-Alive requests: 495045 Total transferred: 154975225 bytes HTML transferred: 81000000 bytes Requests per second: 7101.62 [#/sec] (mean) Time per request: 14.081 [ms] (mean) Time per request: 0.141 [ms] (mean, across all concurrent requests) Transfer rate: 2149.56 [Kbytes/sec] received

Connection Times (ms)

            min  mean[+/-sd] median   max

Connect: 0 0 0.5 0 18 Processing: 0 14 25.4 4 220 Waiting: 0 14 25.4 4 220 Total: 0 14 25.4 4 220

Percentage of the requests served within a certain time (ms)

50%      4
66%      5
75%      6
80%      7
90%     72
95%     80
98%     85
99%     87

100% 220 (longest request) 190