配置
集群配置
- 3个Node 3个节点；
- cpu: 40c
- memory: 256G
- 带宽: 4G
JVM配置
-Xmx30G -Xms30G -XX:MaxPermSize=48m -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35
关键因素测试
Producer测试
1 和线程数的关系(threads)
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 4096 --batch-size 10000 --threads 1
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 4096 --batch-size 10000 --threads 3 --hide-header
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 4096 --batch-size 10000 --threads 5 --hide-header
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 4096 --batch-size 10000 --threads 7 --hide-header
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 4096 --batch-size 10000 --threads 10 --hide-header
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 4096 --batch-size 10000 --threads 15 --hide-header
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 4096 --batch-size 10000 --threads 20 --hide-header
| Threads | MB/S | nMsg/S |
|---|
| 1 | 81.7960 | 20939.7772 |
| 3 | 172.1875 | 44080.0053 |
| 5 | 219.7115 | 56246.1331 |
| 7 | 252.8478 | 64729.0440 |
| 10 | 258.6749 | 66607801 |
| 15 | 283.6163 | 72605.0969 |
| 20 | 257.4644 | 65910.8885 |
2 和批大小的关系(batch-size)
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 4096 --batch-size 1000 --threads 10
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 4096 --batch-size 5000 --threads 10 --hide-header
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 4096 --batch-size 10000 --threads 10 --hide-header
| Batch-size | MB/S | nMsg/S |
|---|
| 1000 | 294.6779 | 214776.6323 |
| 5000 | 282.9386 | 72432.2758 |
| 10000 | 251.3108 | 32167.7872 |
3 和日志大小的关系(message-size)
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 1024 --batch-size 10000 --threads 10
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 4096 --batch-size 10000 --threads 10 --hide-header
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 8192 --batch-size 10000 --threads 10 --hide-header
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 10240 --batch-size 10000 --threads 10 --hide-header
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 20480 --batch-size 10000 --threads 10 --hide-header
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 40960 --batch-size 10000 --threads 10 --hide-header
| Message-size | MB/S | nMsg/S |
|---|
| 1024 | 209.7428 | 214776.6323 |
| 4096 | 282.9386 | 72432.2758 |
| 8192 | 251.3108 | 32167.7872 |
| 10240 | 233.6218 | 23922.8727 |
| 20480 | 220.0903 | 11268.6214 |
4 和响应方式的关系(acks)
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 4096 --batch-size 10000 --threads 10 --request-num-acks 0
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 4096 --batch-size 10000 --threads 10 --request-num-acks 1 --hide-header
| ack | MB/S | nMsg/S |
|---|
| 不响应 | 285.8999 | 73190.3681 |
| leader响应 | 276.9408 | 70896.8451 |
5 和压缩方式的关系(compression-codec)
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 4096 --batch-size 10000 --threads 10 --compression-codec 0
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 4096 --batch-size 10000 --threads 10 --compression-codec 1 --hide-header
./bin/kafka-producer-perf-test2.sh --broker-list cmjx-slave232:9092 --topics s1 --messages 1000000 --message-size 4096 --batch-size 10000 --threads 10 --compression-codec 2 --hide-header
| compresson-codec | MB/S | nMsg/S |
|---|
| 非压缩 | 274.5853 | 70293.8282 |
| Gzip | 267.6981 | 68530.7018 |
| Snappy | 827.5953 | 211864.4068 |
6 Producer配置总结
在特定资源情况下， 在日志大小为4kb, 批大小为5000, 使用15个线程，使用Snapy压缩方式，用不确认的响应方式的情况下，性能最好;
Cosumer测试
1. 和线程数的关系(threads)
./bin/kafka-consumer-perf-test.sh --broker-list cmjx-slave232:9092 --zookeeper cmjx-slave236:2181 --topic s1 --messages 1000000 --fetch-size 10000 --threads 1
./bin/kafka-consumer-perf-test.sh --broker-list cmjx-slave232:9092 --zookeeper cmjx-slave236:2181 --topic s1 --messages 1000000 --fetch-size 10000 --threads 3 --hide-header
./bin/kafka-consumer-perf-test.sh --broker-list cmjx-slave232:9092 --zookeeper cmjx-slave236:2181 --topic s1 --messages 1000000 --fetch-size 10000 --threads 6 --hide-header
./bin/kafka-consumer-perf-test.sh --broker-list cmjx-slave232:9092 --zookeeper cmjx-slave236:2181 --topic s1 --messages 1000000 --fetch-size 10000 --threads 10 --hide-header
| Threads | MB/S | nMsg/S |
|---|
| 1 | 61.9631 | 64973.0362 |
| 3 | 70.4807 | 73904.3677 |
| 6 | 72.2006 | 75707.8686 |
| 10 | 73.1439 | 76696.9193 |
2. 和抓取数据量的关系(fetch-size)
./bin/kafka-consumer-perf-test.sh --broker-list cmjx-slave232:9092 --zookeeper cmjx-slave236:2181 --topic s1 --messages 1000000 --fetch-size 1000 --threads 10
./bin/kafka-consumer-perf-test.sh --broker-list cmjx-slave232:9092 --zookeeper cmjx-slave236:2181 --topic s1 --messages 1000000 --fetch-size 2000 --threads 6 --hide-header
./bin/kafka-consumer-perf-test.sh --broker-list cmjx-slave232:9092 --zookeeper cmjx-slave236:2181 --topic s1 --messages 1000000 --fetch-size 5000 --threads 6 --hide-header
./bin/kafka-consumer-perf-test.sh --broker-list cmjx-slave232:9092 --zookeeper cmjx-slave236:2181 --topic s1 --messages 1000000 --fetch-size 10000 --threads 6 --hide-header
./bin/kafka-consumer-perf-test.sh --broker-list cmjx-slave232:9092 --zookeeper cmjx-slave236:2181 --topic s1 --messages 1000000 --fetch-size 15000 --threads 6 --hide-header
| Fetch-Size | MB/S | nMsg/S |
|---|
| 2000 | 13.2777 | 13922.7289 |
| 5000 | 46.4211 | 48676.0125 |
| 10000 | 75.8128 | 79495.4688 |
| 15000 | 92.6737 | 97175.4341 |
3. cosumer配置总结
使用6-8个线程, 抓取大小为10000左右，性能相对好;
