云硬盘压测

1. 背景

需对测试环境云硬盘进行压测，才能申请正式环境资源

2. 测试步骤

2.1 安装测试工具

yum install libaio -y
yum install libaio-devel -y
yum install fio -y

2.2 确认是否4KB对齐

[root@ecs ~]#  fdisk -lu

Disk /dev/sda: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000a9444

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048    41929649    20963801   83  Linux

Disk /dev/sdb: 214.7 GB, 214748364800 bytes, 419430400 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x649e0a7f

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048   419430399   209714176   83  Linux

返回的结果中，若Start值能被8整除则是4KB对齐。否则，请完成4KB对齐后再继续性能测试。

2.3 测试命令

测试随机写IOPS：fio -direct=1 -iodepth=128 -rw=randwrite -ioengine=libaio -bs=4k -size=10G -numjobs=1 -runtime=600 -group_reporting -filename=/opt/fiotest/fiotest.txt -name=Rand_Write_IOPS_Test
测试随机读IOPS：fio -direct=1 -iodepth=128 -rw=randread -ioengine=libaio -bs=4k -size=10G -numjobs=1 -runtime=600 -group_reporting -filename=/opt/fiotest/fiotest.txt -name=Rand_Read_IOPS_Test
测试写吞吐量：fio -direct=1 -iodepth=32 -rw=write -ioengine=libaio -bs=1024k -size=10G -numjobs=1 -runtime=600 -group_reporting -filename=/opt/fiotest/fiotest.txt -name=Write_BandWidth_Test
测试读吞吐量：fio -direct=1 -iodepth=32 -rw=read -ioengine=libaio -bs=1024k -size=10G -numjobs=1 -runtime=600 -group_reporting -filename=/opt/fiotest/fiotest.txt -name=Read_BandWidth_Test
测试单队列随机读时延：fio -direct=1 -iodepth=1 -rw=randread -ioengine=libaio -bs=4k -size=10G -numjobs=1 -runtime=60 -group_reporting -filename=/opt/fiotest/fiotest.txt -name=Rand_Read_LATE_Test

测试随机写结果

Rand_Write_IOPS_Test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=22.2MiB/s][r=0,w=5673 IOPS][eta 00m:00s]
Rand_Write_IOPS_Test: (groupid=0, jobs=1): err= 0: pid=25946: Thu Jun  9 17:01:37 2022
  write: IOPS=4860, BW=18.0MiB/s (19.9MB/s)(10.0GiB/539304msec)
    slat (nsec): min=1638, max=85494k, avg=198254.45, stdev=1380120.03
    clat (usec): min=753, max=1051.2k, avg=26133.75, stdev=15538.05
     lat (usec): min=760, max=1051.2k, avg=26332.21, stdev=15637.63
    clat percentiles (usec):
     |  1.00th=[  1958],  5.00th=[  3359], 10.00th=[  5276], 20.00th=[ 15795],
     | 30.00th=[ 19006], 40.00th=[ 21627], 50.00th=[ 24511], 60.00th=[ 27657],
     | 70.00th=[ 31327], 80.00th=[ 35914], 90.00th=[ 43254], 95.00th=[ 51643],
     | 99.00th=[ 80217], 99.50th=[ 92799], 99.90th=[120062], 99.95th=[131597],
     | 99.99th=[156238]
   bw (  KiB/s): min= 4608, max=115704, per=99.99%, avg=19440.61, stdev=14427.26, samples=1078
   iops        : min= 1152, max=28926, avg=4860.14, stdev=3606.82, samples=1078
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=1.07%, 4=5.89%, 10=7.33%, 20=19.12%, 50=60.92%
  lat (msec)   : 100=5.31%, 250=0.35%, 500=0.01%, 1000=0.01%
  cpu          : usr=0.51%, sys=1.99%, ctx=280227, majf=0, minf=32
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=0,2621440,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs):
  WRITE: bw=18.0MiB/s (19.9MB/s), 18.0MiB/s-18.0MiB/s (19.9MB/s-19.9MB/s), io=10.0GiB (10.7GB), run=539304-539304msec

Disk stats (read/write):
  sdb: ios=1409686/2614664, merge=18/6370, ticks=63417041/14319261, in_queue=77814075, util=100.00%

测试随机读结果

Rand_Read_IOPS_Test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=128
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=56.6MiB/s,w=0KiB/s][r=14.5k,w=0 IOPS][eta 00m:00s]
Rand_Read_IOPS_Test: (groupid=0, jobs=1): err= 0: pid=27410: Thu Jun  9 17:02:56 2022
   read: IOPS=3536, BW=13.8MiB/s (14.5MB/s)(8295MiB/600413msec)
    slat (nsec): min=1177, max=86973k, avg=138114.28, stdev=1100678.07
    clat (usec): min=500, max=4602.7k, avg=36050.90, stdev=133543.65
     lat (usec): min=506, max=4602.7k, avg=36189.22, stdev=133570.18
    clat percentiles (usec):
     |  1.00th=[    799],  5.00th=[    947], 10.00th=[   1074],
     | 20.00th=[   1483], 30.00th=[   2311], 40.00th=[   4686],
     | 50.00th=[   7767], 60.00th=[  11731], 70.00th=[  16909],
     | 80.00th=[  26346], 90.00th=[  57934], 95.00th=[ 133694],
     | 99.00th=[ 599786], 99.50th=[ 926942], 99.90th=[1736442],
     | 99.95th=[2197816], 99.99th=[3103785]
   bw (  KiB/s): min=  904, max=62352, per=100.00%, avg=14155.13, stdev=10720.65, samples=1200
   iops        : min=  226, max=15588, avg=3538.77, stdev=2680.16, samples=1200
  lat (usec)   : 750=0.37%, 1000=6.87%
  lat (msec)   : 2=19.81%, 4=10.62%, 10=18.37%, 20=18.17%, 50=14.45%
  lat (msec)   : 100=4.99%, 250=3.46%, 500=1.61%, 750=0.56%, 1000=0.28%
  cpu          : usr=0.58%, sys=2.20%, ctx=1259097, majf=0, minf=162
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=2123542,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=128

Run status group 0 (all jobs):
   READ: bw=13.8MiB/s (14.5MB/s), 13.8MiB/s-13.8MiB/s (14.5MB/s-14.5MB/s), io=8295MiB (8698MB), run=600413-600413msec

Disk stats (read/write):
  sdb: ios=2123523/2204816, merge=18/6656, ticks=73592674/12031849, in_queue=85624697, util=100.00%

测试写吞吐量结果

Write_BandWidth_Test: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 00m:00s]    
Write_BandWidth_Test: (groupid=0, jobs=1): err= 0: pid=17911: Thu Jun  9 17:09:49 2022
  write: IOPS=41, BW=41.9MiB/s (43.9MB/s)(10.0GiB/244403msec)
    slat (usec): min=35, max=679582, avg=8873.35, stdev=36678.73
    clat (usec): min=871, max=6094.4k, avg=754628.69, stdev=1034740.83
     lat (msec): min=3, max=6094, avg=763.50, stdev=1035.67
    clat percentiles (msec):
     |  1.00th=[    5],  5.00th=[    7], 10.00th=[    9], 20.00th=[   14],
     | 30.00th=[   24], 40.00th=[   48], 50.00th=[  100], 60.00th=[  405],
     | 70.00th=[ 1167], 80.00th=[ 1636], 90.00th=[ 2265], 95.00th=[ 2903],
     | 99.00th=[ 4111], 99.50th=[ 4597], 99.90th=[ 5000], 99.95th=[ 5336],
     | 99.99th=[ 5873]
   bw (  KiB/s): min= 2048, max=147456, per=100.00%, avg=45056.37, stdev=29199.32, samples=464
   iops        : min=    2, max=  144, avg=43.97, stdev=28.50, samples=464
  lat (usec)   : 1000=0.06%
  lat (msec)   : 2=0.27%, 4=0.52%, 10=12.81%, 20=13.59%, 50=13.56%
  lat (msec)   : 100=9.23%, 250=7.28%, 500=3.75%, 750=2.57%, 1000=3.02%
  cpu          : usr=0.33%, sys=0.18%, ctx=6769, majf=0, minf=31
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=99.7%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,10240,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
  WRITE: bw=41.9MiB/s (43.9MB/s), 41.9MiB/s-41.9MiB/s (43.9MB/s-43.9MB/s), io=10.0GiB (10.7GB), run=244403-244403msec

Disk stats (read/write):
  sdb: ios=30720/30914, merge=0/3, ticks=15514682/16218108, in_queue=31739430, util=99.99%

测试读吞吐量

Read_BandWidth_Test: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=53.0MiB/s,w=0KiB/s][r=53,w=0 IOPS][eta 00m:00s]
Read_BandWidth_Test: (groupid=0, jobs=1): err= 0: pid=19961: Thu Jun  9 17:09:48 2022
   read: IOPS=47, BW=47.2MiB/s (49.5MB/s)(10.0GiB/216751msec)
    slat (usec): min=27, max=467261, avg=8985.34, stdev=32463.76
    clat (usec): min=297, max=5347.7k, avg=668104.45, stdev=824242.86
     lat (usec): min=1653, max=5347.7k, avg=677090.35, stdev=828305.63
    clat percentiles (usec):
     |  1.00th=[   1942],  5.00th=[   2507], 10.00th=[   3195],
     | 20.00th=[   5800], 30.00th=[  12911], 40.00th=[ 104334],
     | 50.00th=[ 421528], 60.00th=[ 700449], 70.00th=[ 918553],
     | 80.00th=[1182794], 90.00th=[1686111], 95.00th=[2264925],
     | 99.00th=[3741320], 99.50th=[3942646], 99.90th=[4529849],
     | 99.95th=[4731175], 99.99th=[5066720]
   bw (  KiB/s): min= 2048, max=184320, per=100.00%, avg=48395.71, stdev=29546.48, samples=432
   iops        : min=    2, max=  180, avg=47.23, stdev=28.87, samples=432
  lat (usec)   : 500=0.18%, 750=0.04%, 1000=0.01%
  lat (msec)   : 2=1.10%, 4=12.33%, 10=14.04%, 20=4.82%, 50=4.11%
  lat (msec)   : 100=3.06%, 250=5.68%, 500=7.55%, 750=9.38%, 1000=11.32%
  cpu          : usr=0.03%, sys=0.27%, ctx=8656, majf=0, minf=8226
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=99.7%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: bw=47.2MiB/s (49.5MB/s), 47.2MiB/s-47.2MiB/s (49.5MB/s-49.5MB/s), io=10.0GiB (10.7GB), run=216751-216751msec

Disk stats (read/write):
  sdb: ios=30714/29123, merge=0/3, ticks=15505249/13683678, in_queue=29207916, util=100.00%

测试单队列随机读时延结果

Rand_Read_LATE_Test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=200KiB/s,w=0KiB/s][r=50,w=0 IOPS][eta 00m:00s]
Rand_Read_LATE_Test: (groupid=0, jobs=1): err= 0: pid=15780: Thu Jun  9 17:13:42 2022
   read: IOPS=59, BW=236KiB/s (242kB/s)(13.8MiB/60015msec)
    slat (nsec): min=4222, max=50572, avg=10247.06, stdev=4076.79
    clat (usec): min=748, max=751149, avg=16916.47, stdev=21660.92
     lat (usec): min=760, max=751159, avg=16926.97, stdev=21660.90
    clat percentiles (usec):
     |  1.00th=[   930],  5.00th=[  5276], 10.00th=[  7046], 20.00th=[  9241],
     | 30.00th=[ 10683], 40.00th=[ 11994], 50.00th=[ 13435], 60.00th=[ 15008],
     | 70.00th=[ 16909], 80.00th=[ 20579], 90.00th=[ 27657], 95.00th=[ 39060],
     | 99.00th=[ 67634], 99.50th=[ 87557], 99.90th=[265290], 99.95th=[517997],
     | 99.99th=[750781]
   bw (  KiB/s): min=   16, max=  328, per=100.00%, avg=238.19, stdev=48.60, samples=119
   iops        : min=    4, max=   82, avg=59.50, stdev=12.15, samples=119
  lat (usec)   : 750=0.03%, 1000=1.55%
  lat (msec)   : 2=1.55%, 4=0.42%, 10=21.41%, 20=53.68%, 50=18.79%
  lat (msec)   : 100=2.17%, 250=0.25%, 500=0.08%, 750=0.03%, 1000=0.03%
  cpu          : usr=0.03%, sys=0.09%, ctx=3548, majf=0, minf=35
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=3545,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=236KiB/s (242kB/s), 236KiB/s-236KiB/s (242kB/s-242kB/s), io=13.8MiB (14.5MB), run=60015-60015msec

Disk stats (read/write):
  sdb: ios=3536/43, merge=0/1, ticks=59834/122, in_queue=59961, util=99.77%

3. 引用

如何测试云硬盘性能
怎样测试云硬盘的性能