kafka单机性能测试和集群测试性能为什么没区别?

孩子,你真逗 发表于: 2022-07-28   最后更新时间: 2022-07-29 14:03:18   1,167 游览

kafka-producer-perf-test --producer-props bootstrap.servers=mycc-data-kafka-broker-1.broker-headless.default.svc.cluster.local:9095,mycc-data-kafka-broker-0.broker-headless.default.svc.cluster.local:9095 --topic haha128_22 --throughput 600000 --record-size 1024 --num-records 80000000 --print-metrics

kafka 单机 8 partition 每条数据1k, 吞吐量17w条每秒
kafka 集群 8 partition 每条数据1k, 吞吐量也是17w条每秒左右, 增加partition也没用

Metric Name                                                                          Value
app-info:commit-id:{client-id=producer-1}                                          : b06c3873f2b9afbd
app-info:version:{client-id=producer-1}                                            : 2.2.2-cp1
kafka-metrics-count:count:{client-id=producer-1}                                   : 125.000
producer-metrics:batch-size-avg:{client-id=producer-1}                             : 5346.717
producer-metrics:batch-size-max:{client-id=producer-1}                             : 15563.000
producer-metrics:batch-split-rate:{client-id=producer-1}                           : 0.000
producer-metrics:batch-split-total:{client-id=producer-1}                          : 0.000
producer-metrics:buffer-available-bytes:{client-id=producer-1}                     : 33554432.000
producer-metrics:buffer-exhausted-rate:{client-id=producer-1}                      : 0.000
producer-metrics:buffer-exhausted-total:{client-id=producer-1}                     : 0.000
producer-metrics:buffer-total-bytes:{client-id=producer-1}                         : 33554432.000
producer-metrics:bufferpool-wait-ratio:{client-id=producer-1}                      : 0.095
producer-metrics:bufferpool-wait-time-total:{client-id=producer-1}                 : 3381669934.000
producer-metrics:compression-rate-avg:{client-id=producer-1}                       : 1.000
producer-metrics:connection-close-rate:{client-id=producer-1}                      : 0.000
producer-metrics:connection-close-total:{client-id=producer-1}                     : 0.000
producer-metrics:connection-count:{client-id=producer-1}                           : 4.000
producer-metrics:connection-creation-rate:{client-id=producer-1}                   : 0.111
producer-metrics:connection-creation-total:{client-id=producer-1}                  : 4.000
producer-metrics:failed-authentication-rate:{client-id=producer-1}                 : 0.000
producer-metrics:failed-authentication-total:{client-id=producer-1}                : 0.000
producer-metrics:failed-reauthentication-rate:{client-id=producer-1}               : 0.000
producer-metrics:failed-reauthentication-total:{client-id=producer-1}              : 0.000
producer-metrics:incoming-byte-rate:{client-id=producer-1}                         : 133929.136
producer-metrics:incoming-byte-total:{client-id=producer-1}                        : 4798547.000
producer-metrics:io-ratio:{client-id=producer-1}                                   : 0.024
producer-metrics:io-time-ns-avg:{client-id=producer-1}                             : 64457.348
producer-metrics:io-wait-ratio:{client-id=producer-1}                              : 0.040
producer-metrics:io-wait-time-ns-avg:{client-id=producer-1}                        : 109651.400
producer-metrics:io-waittime-total:{client-id=producer-1}                          : 1452771400.000
producer-metrics:iotime-total:{client-id=producer-1}                               : 853995405.000
producer-metrics:metadata-age:{client-id=producer-1}                               : 5.821
producer-metrics:network-io-rate:{client-id=producer-1}                            : 178.997
producer-metrics:network-io-total:{client-id=producer-1}                           : 6414.000
producer-metrics:outgoing-byte-rate:{client-id=producer-1}                         : 23370227.156
producer-metrics:outgoing-byte-total:{client-id=producer-1}                        : 837355239.000
producer-metrics:produce-throttle-time-avg:{client-id=producer-1}                  : 0.000
producer-metrics:produce-throttle-time-max:{client-id=producer-1}                  : 0.000
producer-metrics:reauthentication-latency-avg:{client-id=producer-1}               : NaN
producer-metrics:reauthentication-latency-max:{client-id=producer-1}               : NaN
producer-metrics:record-error-rate:{client-id=producer-1}                          : 0.000
producer-metrics:record-error-total:{client-id=producer-1}                         : 0.000
producer-metrics:record-queue-time-avg:{client-id=producer-1}                      : 60.351
producer-metrics:record-queue-time-max:{client-id=producer-1}                      : 437.000
producer-metrics:record-retry-rate:{client-id=producer-1}                          : 0.000
producer-metrics:record-retry-total:{client-id=producer-1}                         : 0.000
producer-metrics:record-send-rate:{client-id=producer-1}                           : 22378.248
producer-metrics:record-send-total:{client-id=producer-1}                          : 800000.000
producer-metrics:record-size-avg:{client-id=producer-1}                            : 1110.000
producer-metrics:record-size-max:{client-id=producer-1}                            : 1110.000
producer-metrics:records-per-request-avg:{client-id=producer-1}                    : 249.844
producer-metrics:request-latency-avg:{client-id=producer-1}                        : 8.624
producer-metrics:request-latency-max:{client-id=producer-1}                        : 130.000
producer-metrics:request-rate:{client-id=producer-1}                               : 89.501
producer-metrics:request-size-avg:{client-id=producer-1}                           : 261102.351
producer-metrics:request-size-max:{client-id=producer-1}                           : 996574.000
producer-metrics:request-total:{client-id=producer-1}                              : 3207.000
producer-metrics:requests-in-flight:{client-id=producer-1}                         : 0.000
producer-metrics:response-rate:{client-id=producer-1}                              : 89.508
producer-metrics:response-total:{client-id=producer-1}                             : 3207.000
producer-metrics:select-rate:{client-id=producer-1}                                : 367.864
producer-metrics:select-total:{client-id=producer-1}                               : 13249.000
producer-metrics:successful-authentication-no-reauth-total:{client-id=producer-1}  : 0.000
producer-metrics:successful-authentication-rate:{client-id=producer-1}             : 0.000
producer-metrics:successful-authentication-total:{client-id=producer-1}            : 0.000
producer-metrics:successful-reauthentication-rate:{client-id=producer-1}           : 0.000
producer-metrics:successful-reauthentication-total:{client-id=producer-1}          : 0.000
producer-metrics:waiting-threads:{client-id=producer-1}                            : 0.000
producer-node-metrics:incoming-byte-rate:{client-id=producer-1, node-id=node--1}   : 135.557
producer-node-metrics:incoming-byte-rate:{client-id=producer-1, node-id=node--2}   : 7.872
producer-node-metrics:incoming-byte-rate:{client-id=producer-1, node-id=node-0}    : 111222.070
producer-node-metrics:incoming-byte-rate:{client-id=producer-1, node-id=node-1}    : 22838.939
producer-node-metrics:incoming-byte-total:{client-id=producer-1, node-id=node--1}  : 4857.000
producer-node-metrics:incoming-byte-total:{client-id=producer-1, node-id=node--2}  : 282.000
producer-node-metrics:incoming-byte-total:{client-id=producer-1, node-id=node-0}   : 3977190.000
producer-node-metrics:incoming-byte-total:{client-id=producer-1, node-id=node-1}   : 816218.000
producer-node-metrics:outgoing-byte-rate:{client-id=producer-1, node-id=node--1}   : 1.814
producer-node-metrics:outgoing-byte-rate:{client-id=producer-1, node-id=node--2}   : 0.670
producer-node-metrics:outgoing-byte-rate:{client-id=producer-1, node-id=node-0}    : 11809252.943
producer-node-metrics:outgoing-byte-rate:{client-id=producer-1, node-id=node-1}    : 11607373.640
producer-node-metrics:outgoing-byte-total:{client-id=producer-1, node-id=node--1}  : 65.000
producer-node-metrics:outgoing-byte-total:{client-id=producer-1, node-id=node--2}  : 24.000
producer-node-metrics:outgoing-byte-total:{client-id=producer-1, node-id=node-0}   : 422287076.000
producer-node-metrics:outgoing-byte-total:{client-id=producer-1, node-id=node-1}   : 415068074.000
producer-node-metrics:request-latency-avg:{client-id=producer-1, node-id=node--1}  : NaN
producer-node-metrics:request-latency-avg:{client-id=producer-1, node-id=node--2}  : NaN
producer-node-metrics:request-latency-avg:{client-id=producer-1, node-id=node-0}   : 6.690
producer-node-metrics:request-latency-avg:{client-id=producer-1, node-id=node-1}   : 21.505
producer-node-metrics:request-latency-max:{client-id=producer-1, node-id=node--1}  : NaN
producer-node-metrics:request-latency-max:{client-id=producer-1, node-id=node--2}  : NaN
producer-node-metrics:request-latency-max:{client-id=producer-1, node-id=node-0}   : 130.000
producer-node-metrics:request-latency-max:{client-id=producer-1, node-id=node-1}   : 91.000
producer-node-metrics:request-rate:{client-id=producer-1, node-id=node--1}         : 0.056
producer-node-metrics:request-rate:{client-id=producer-1, node-id=node--2}         : 0.028
producer-node-metrics:request-rate:{client-id=producer-1, node-id=node-0}          : 77.880
producer-node-metrics:request-rate:{client-id=producer-1, node-id=node-1}          : 11.717
producer-node-metrics:request-size-avg:{client-id=producer-1, node-id=node--1}     : 32.500
producer-node-metrics:request-size-avg:{client-id=producer-1, node-id=node--2}     : 24.000
producer-node-metrics:request-size-avg:{client-id=producer-1, node-id=node-0}      : 151629.112
producer-node-metrics:request-size-avg:{client-id=producer-1, node-id=node-1}      : 990615.928
producer-node-metrics:request-size-max:{client-id=producer-1, node-id=node--1}     : 41.000
producer-node-metrics:request-size-max:{client-id=producer-1, node-id=node--2}     : 24.000
producer-node-metrics:request-size-max:{client-id=producer-1, node-id=node-0}      : 996446.000
producer-node-metrics:request-size-max:{client-id=producer-1, node-id=node-1}      : 996574.000
producer-node-metrics:request-total:{client-id=producer-1, node-id=node--1}        : 2.000
producer-node-metrics:request-total:{client-id=producer-1, node-id=node--2}        : 1.000
producer-node-metrics:request-total:{client-id=producer-1, node-id=node-0}         : 2785.000
producer-node-metrics:request-total:{client-id=producer-1, node-id=node-1}         : 419.000
producer-node-metrics:response-rate:{client-id=producer-1, node-id=node--1}        : 0.056
producer-node-metrics:response-rate:{client-id=producer-1, node-id=node--2}        : 0.028
producer-node-metrics:response-rate:{client-id=producer-1, node-id=node-0}         : 77.882
producer-node-metrics:response-rate:{client-id=producer-1, node-id=node-1}         : 11.724
producer-node-metrics:response-total:{client-id=producer-1, node-id=node--1}       : 2.000
producer-node-metrics:response-total:{client-id=producer-1, node-id=node--2}       : 1.000
producer-node-metrics:response-total:{client-id=producer-1, node-id=node-0}        : 2785.000
producer-node-metrics:response-total:{client-id=producer-1, node-id=node-1}        : 419.000
producer-topic-metrics:byte-rate:{client-id=producer-1, topic=haha128_22}          : 23384838.728
producer-topic-metrics:byte-total:{client-id=producer-1, topic=haha128_22}         : 835937830.000
producer-topic-metrics:compression-rate:{client-id=producer-1, topic=haha128_22}   : 1.000
producer-topic-metrics:record-error-rate:{client-id=producer-1, topic=haha128_22}  : 0.000
producer-topic-metrics:record-error-total:{client-id=producer-1, topic=haha128_22} : 0.000
producer-topic-metrics:record-retry-rate:{client-id=producer-1, topic=haha128_22}  : 0.000
producer-topic-metrics:record-retry-total:{client-id=producer-1, topic=haha128_22} : 0.000
producer-topic-metrics:record-send-rate:{client-id=producer-1, topic=haha128_22}   : 22379.500
producer-topic-metrics:record-send-total:{client-id=producer-1, topic=haha128_22}  : 800000.000
发表于 2022-07-28
添加评论

压力不够,多开2个压力请求

你用脚本测试的不具有代表性吧,还是结合实际业务测试的,我用过这样的命令测试,但是测试和实际业务差距较大。

你的答案

查看kafka相关的其他问题或提一个您自己的问题