说明
测试各种虚拟化系统下的虚拟机性能。
测试使用sysbench。
CPU采用如下指令测试。
sysbench --test=cpu --num-threads=2 --cpu-max-prime=50000 run
文件IO采用如下指令测试。
sysbench --test=fileio --file-total-size=10G prepare
sysbench --test=fileio --file-total-size=10G --file-test-mode=rndrw --init-rng=on --max-time=300 --max-requests=0 run
内存采用如下指令测试。
sysbench --test=memory --num-threads=2 --memory-access-mode=seq run
sysbench --test=memory --num-threads=2 --memory-access-mode=rnd run
线程采用如下指令。
sysbench --test=threads --num-threads=2 run
sysbench --test=mutex --num-threads=2 --mutex-locks=1000000 run
裸硬盘测试采用如下指令。
hdparm -tT <dev>
物理机上有三个文件系统，ext4/xfs/btrfs，前两者仅做fileio测试以对比性能。
另外做两个特殊文件系统对比，aufs带复制和aufs无复制。前者在只读层上准备好测试文件，而后进行随机读写测试。其中就附带了文件复制开销。后者在aufs建立后初始化测试文件，因此消除了文件复制开销。
所有测试都是测试数次，取最高者（因为低者可能受到各种干扰）。一般是2-3次。
物理机是一台DELL Intel 64位桌面系统，支持硬件虚拟化，有4G内存。系统采用debian jessie，测试于2014年1月17日-20日执行，内核3.12.6-2 (2013-12-29) x86_64。 虚拟机lxc是使用lxc切分的一台虚拟机，没有做资源限制。
虚拟机vbox是使用virtualbox切分的一台虚拟机，分配了所有CPU，打开了硬件虚拟化，分配了1G内存。
文件系统
ext4
Operations performed: 21311 Read, 14207 Write, 45440 Other = 80958
Total Read 332.98Mb Written 221.98Mb
Total transferred 554.97Mb (1.8499Mb/sec)
118.39 Requests/sec executed
Test execution summary:
total time: 300.0044s
total number of events: 35518
total time taken by event execution: 168.4761
per-request statistics:
min: 0.00ms
avg: 4.74ms
max: 118.67ms
approx. 95 percentile: 12.48ms
Threads fairness:
events (avg/stddev): 35518.0000/0.00
execution time (avg/stddev): 168.4761/0.00
xfs
Operations performed: 20789 Read, 13859 Write, 44288 Other = 78936
Total Read 324.83Mb Written 216.55Mb
Total transferred 541.38M (1.8046Mb/sec)
115.49 Requests/sec executed
Test execution summary:
total time: 300.0018s
total number of events: 34648
total time taken by event execution: 172.0475
per-request statistics:
min: 0.00ms
avg: 4.97ms
max: 96.11ms
approx. 95 percentile: 12.30ms
Threads fairness:
events (avg/stddev): 34648.0000/0.00
execution time (avg/stddev): 172.0475/0.00
btrfs
Operations performed: 6180 Read, 4120 Write, 13105 Other = 23405 Total
Read 96.562Mb Written 64.375Mb
Total transferred 160.94M (549.23Kb/sec)
34.33 Requests/sec executed
Test execution summary:
total time: 300.0556s
total number of events: 10300
total time taken by event execution: 65.8914
per-request statistics:
min: 0.00ms
avg: 6.40ms
max: 337.28ms
approx. 95 percentile: 17.01ms
Threads fairness:
events (avg/stddev): 10300.0000/0.00
execution time (avg/stddev): 65.8914/0.00
aufs透明
Operations performed: 5340 Read, 3560 Write, 11279 Other = 20179
Total Read 83.438Mb Written 55.625Mb
Total transferred 139.06M (474.65Kb/sec)
29.67 Requests/sec executed
Test execution summary:
total time: 300.0084s
total number of events: 8900
Test execution summary:
total time: 300.0084s
total number of events: 8900
total time taken by event execution: 32.3634
per-request statistics:
min: 0.00ms
avg: 3.64ms
max: 1037.04ms
approx. 95 percentile: 0.02ms
Threads fairness:
events (avg/stddev): 8900.0000/0.00
execution time (avg/stddev): 32.3634/0.00
aufs非透明
Operations performed: 20320 Read, 13546 Write, 43264 Other = 77130
Total Read 317.5Mb Written 211.66Mb
Total transferred 529.16Mb (1.7638Mb/sec)
112.88 Requests/sec executed
Test execution summary:
total time: 300.0054s
total number of events: 33866
total time taken by event execution: 170.7252
per-request statistics:
min: 0.00ms
avg: 5.04ms
max: 143.86ms
approx. 95 percentile: 12.62ms
Threads fairness:
events (avg/stddev): 33866.0000/0.00
execution time (avg/stddev): 170.7252/0.00
物理机
hdparm
Timing cached reads: 11980 MB in 2.00 seconds = 5992.56 MB/sec
Timing buffered disk reads: 366 MB in 3.01 seconds = 121.52 MB/sec
cpu
Test execution summary:
total time: 51.4463s
total number of events: 10000
total time taken by event execution: 102.8828
per-request statistics:
min: 9.93ms
avg: 10.29ms
max: 36.11ms
approx. 95 percentile: 11.29ms
Threads fairness:
events (avg/stddev): 5000.0000/30.00
execution time (avg/stddev): 51.4414/0.00
memory
Operations performed: 104857600 (4545662.48 ops/sec)
102400.00 MB transferred (4439.12 MB/sec)
Test execution summary:
total time: 23.0676s
total number of events: 104857600
total time taken by event execution: 34.1991
per-request statistics:
min: 0.00ms
avg: 0.00ms
max: 18.05ms
approx. 95 percentile: 0.00ms
Threads fairness:
events (avg/stddev): 52428800.0000/69790.00
execution time (avg/stddev): 17.0995/0.01
Operations performed: 104857600 (5407739.22 ops/sec)
102400.00 MB transferred (5281.00 MB/sec)
Test execution summary:
total time: 19.3903s
total number of events: 104857600
total time taken by event execution: 26.8579
per-request statistics:
min: 0.00ms
avg: 0.00ms
max: 22.46ms
approx. 95 percentile: 0.00ms
Threads fairness:
events (avg/stddev): 52428800.0000/7211.00
execution time (avg/stddev): 13.4289/0.14
threads
Test execution summary:
total time: 1.0112s
total number of events: 10000
total time taken by event execution: 2.0210
per-request statistics:
min: 0.15ms
avg: 0.20ms
max: 11.19ms
approx. 95 percentile: 0.24ms
Threads fairness:
events (avg/stddev): 5000.0000/3.00
execution time (avg/stddev): 1.0105/0.00
Test execution summary:
total time: 0.1665s
total number of events: 2
total time taken by event execution: 0.3238
per-request statistics:
min: 157.39ms
avg: 161.90ms
max: 166.41ms
approx. 95 percentile: 10000000.00ms
Threads fairness:
events (avg/stddev): 1.0000/0.00
execution time (avg/stddev): 0.1619/0.00
nginx
Concurrency Level: 10000
Time taken for tests: 5.745 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Total transferred: 172100000 bytes
HTML transferred: 160000000 bytes
Requests per second: 17404.93 [\#/sec] (mean)
Time per request: 574.550 [ms] (mean)
Time per request: 0.057 [ms] (mean, across all concurrent requests)
Transfer rate: 29251.85 [Kbytes/sec] received
lxc
cpu
Test execution summary:
total time: 51.4368s
total number of events: 10000
total time taken by event execution: 102.8619
per-request statistics:
min: 9.92ms
avg: 10.29ms
max: 35.08ms
approx. 95 percentile: 11.68ms
Threads fairness:
events (avg/stddev): 5000.0000/5.00
execution time (avg/stddev): 51.4310/0.00
fileio
Operations performed: 5548 Read, 3698 Write, 11776 Other = 21022
Total Read 86.688Mb Written 57.781Mb
Total transferred 144.47Mb (493.07Kb/sec)
30.82 Requests/sec executed
Test execution summary:
total time: 300.0294s
total number of events: 9246
total time taken by event execution: 84.4687
per-request statistics:
min: 0.01ms
avg: 9.14ms
max: 394.13ms
approx. 95 percentile: 36.20ms
Threads fairness:
events (avg/stddev): 9246.0000/0.00
execution time (avg/stddev): 84.4687/0.00
memory
Operations performed: 104857600 (4456398.83 ops/sec)
102400.00 MB transferred (4351.95 MB/sec)
Test execution summary:
total time: 23.5297s
total number of events: 104857600
total time taken by event execution: 34.8417
per-request statistics:
min: 0.00ms
avg: 0.00ms
max: 20.22ms
approx. 95 percentile: 0.00ms
Threads fairness:
events (avg/stddev): 52428800.0000/155952.00
execution time (avg/stddev): 17.4208/0.06
Operations performed: 104857600 (5327923.43 ops/sec)
102400.00 MB transferred (5203.05 MB/sec)
Test execution summary:
total time: 19.6808s
total number of events: 104857600
total time taken by event execution: 27.3010
per-request statistics:
min: 0.00ms
avg: 0.00ms
max: 16.48ms
approx. 95 percentile: 0.00ms
Threads fairness:
events (avg/stddev): 52428800.0000/297738.00
execution time (avg/stddev): 13.6505/0.09
threads
Test execution summary:
total time: 1.2490s
total number of events: 10000
total time taken by event execution: 2.4954
per-request statistics:
min: 0.21ms
avg: 0.25ms
max: 7.39ms
approx. 95 percentile: 0.28ms
Threads fairness:
events (avg/stddev): 5000.0000/7.00
execution time (avg/stddev): 1.2477/0.00
Test execution summary:
total time: 0.1222s
total number of events: 2
total time taken by event execution: 0.2275
per-request statistics:
min: 107.53ms
avg: 113.77ms
max: 120.02ms
approx. 95 percentile: 10000000.00ms
Threads fairness:
events (avg/stddev): 1.0000/0.00
execution time (avg/stddev): 0.1138/0.01
nginx
Concurrency Level: 10000
Time taken for tests: 16.976 seconds
Complete requests: 100000
Failed requests: 0
Write errors: 0
Total transferred: 1551500000 bytes
HTML transferred: 1539400000 bytes
Requests per second: 5890.69 [#/sec] (mean)
Time per request: 1697.594 [ms] (mean)
Time per request: 0.170 [ms] (mean, across all concurrent requests)
Transfer rate: 89252.02 [Kbytes/sec] received
vbox
hdparm
Timing cached reads: 10122 MB in 1.99 seconds = 5088.70 MB/sec
Timing buffered disk reads: 300 MB in 3.00 seconds = 99.87 MB/sec
cpu
Test execution summary:
total time: 54.0469s
total number of events: 10000
total time taken by event execution: 108.0595
per-request statistics:
min: 9.03ms
avg: 10.81ms
max: 61.39ms
approx. 95 percentile: 14.87ms
Threads fairness:
events (avg/stddev): 5000.0000/87.00
execution time (avg/stddev): 54.0297/0.00
fileio
Operations performed: 68153 Read, 45435 Write, 145280 Other = 258868
Total Read 1.0399Gb Written 709.92Mb
Total transferred 1.7332Gb (5.916Mb/sec)
378.62 Requests/sec executed
Test execution summary:
total time: 300.0045s
total number of events: 113588
total time taken by event execution: 177.2314
per-request statistics:
min: 0.01ms
avg: 1.56ms
max: 579.66ms
approx. 95 percentile: 13.10ms
Threads fairness:
events (avg/stddev): 113588.0000/0.00
execution time (avg/stddev): 177.2314/0.00
memory
一直报错，测试不出来。
threads
Test execution summary:
total time: 16.0377s
total number of events: 10000
total time taken by event execution: 32.0421
per-request statistics:
min: 1.19ms
avg: 3.20ms
max: 38.39ms
approx. 95 percentile: 5.74ms
Threads fairness:
events (avg/stddev): 5000.0000/15.00
execution time (avg/stddev): 16.0210/0.00
Test execution summary:
total time: 0.3023s
total number of events: 2
total time taken by event execution: 0.5990
per-request statistics:
min: 297.52ms
avg: 299.50ms
max: 301.47ms
approx. 95 percentile: 10000000.00ms
Threads fairness:
events (avg/stddev): 1.0000/0.00
execution time (avg/stddev): 0.2995/0.00
nginx
Concurrency Level: 5000
Time taken for tests: 41.758 seconds
Complete requests: 50000
Failed requests: 0
Write errors: 0
Keep-Alive requests: 0
Total transferred: 890350000 bytes
HTML transferred: 884250000 bytes
Requests per second: 1197.38 [#/sec] (mean)
Time per request: 4175.799 [ms] (mean)
Time per request: 0.835 [ms] (mean, across all concurrent requests)
Transfer rate: 20821.94 [Kbytes/sec] received
vbox on lvm
在物理机上开辟一个lvm卷，然后用vmdk引用物理卷的功能挂到vbox上使用，格式化为ext4文件格式。
hdparm
Timing cached reads: 10034 MB in 1.99 seconds = 5044.30 MB/sec
Timing buffered disk reads: 270 MB in 3.01 seconds = 89.73 MB/sec
fileio
Operations performed: 22765 Read, 15176 Write, 48512 Other = 86453
Total Read 355.7Mb Written 237.12Mb Total transferred 592.83Mb (1.976Mb/sec)
126.47 Requests/sec executed
Test execution summary:
total time: 300.0080s
total number of events: 37941
total time taken by event execution: 286.5402
per-request statistics:
min: 0.01ms
avg: 7.55ms
max: 336.67ms
approx. 95 percentile: 20.49ms
Threads fairness:
events (avg/stddev): 37941.0000/0.00
execution time (avg/stddev): 286.5402/0.00
vbox on vdi
基本同vbox on lvm，不过使用vdi作为存储。和vbox测试相比，内部系统换为主系统同样的debian。
hdparm
Timing cached reads: 9152 MB in 1.99 seconds = 4598.80 MB/sec
Timing buffered disk reads: 352 MB in 3.00 seconds = 117.16 MB/sec
fileio
Operations performed: 151020 Read, 100680 Write, 322060 Other = 573760
Total Read 2.3044Gb Written 1.5363Gb
Total transferred 3.8406Gb (13.109Mb/sec)
839.00 Requests/sec executed
Test execution summary:
total time: 300.0011s
total number of events: 251700
total time taken by event execution: 60.6296
per-request statistics:
min: 0.01ms
avg: 0.24ms
max: 106.94ms
approx. 95 percentile: 0.26ms
Threads fairness:
events (avg/stddev): 251700.0000/0.00
execution time (avg/stddev): 60.6296/0.00
分析
计算
从CPU上说，vbox的消耗大约是5%，而lxc的消耗基本是0%。
内存测试上，vbox无法测量。lxc的性能和物理机十分相近，两者的差异在1.5%-2%之间。
在threads测试和mutex测试上，vbox显示出远远低于lxc的性能。这些基本都是内核陷入类的事务，也是预期vbox会发生性能下降的地方。
IO
从硬件设备IO上来分析，lxc和主机是共用一套物理设备的。vbox的裸设备IO比物理机低了15-18%。但是文件系统IO则表现出完全相反的景象。
lxc底层使用的是btrfs，因此性能和物理机上的btrfs性能十分相近，误差在10%以内。这一性能比物理机上的ext4/xfs低了70%以上。这个表现出btrfs的性能和ext4/xfs性能的差异(注：怎么会差这么多？)。
如果使用aufs的话，则要视复制特性而定。如果引发复制，性能会跌到和btrfs相近。而不引发复制的话，则和aufs下面的文件系统相近(误差在5%以内)。
从上面可以看出，对于lxc这种小消耗的虚拟方案，与其在意虚拟机的性能消耗，不如更在意文件系统的性能消耗。但是比较倒霉的是，lxc是不能在虚拟机内自行配置文件系统的，需要从主机内分配挂载。
- ext4：适用于大部分情况，iops很高，小文件下吞吐量很不错。大部分虚拟化场景都是小系统简单服务的时候适合。
- xfs：适用于大设备内部的大虚拟机。每个虚机的服务复杂，或者是有大文件。
- btrfs：这个东西性能很低，但是写时复制的能力对快速clone很重要。适合用于产生一堆临时生成的环境，用于程序员测试或者测试工程师搭环境，测试完了就删除的。
- aufs：和btrfs情况差不多，快速clone不错。不过安全起见，clone后原image不可以启动实例或者修改image。在没有COW写入时性能很高，这点比btrfs好。而在COW时瞬时开销很高。
by shell909090
under CC-BY-SA 3.0
