Test Broadcom MegaRAID 9460-16i storage controller

[geerlingguy]

Now this is an interesting one. Full story to come soon, but I've been testing Broadcom's MegaRAID 9460-16i for a bit (thanks to @joshsinykin at Broadcom for helping me get set up!), and (spoilers) I actually have SAS RAID storage working with the Pi now.

Here's a picture of the card:

I'll be posting the basic details soon, and once I can polish up the process for getting drivers working, I'll also add in that detail (right now it requires a bit of tweaking that I might not be necessary soon!).

I also have it on good authority the 9560-16i also works with the Pi :)

[geerlingguy]

$ sudo lspci -vvvv
...
01:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID Tri-Mode SAS3516 (rev 01)
	Subsystem: LSI Logic / Symbios Logic MegaRAID Tri-Mode SAS3516
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 255
	Region 0: Memory at 600200000 (64-bit, prefetchable) [disabled] [size=1M]
	Region 2: Memory at 600300000 (64-bit, prefetchable) [disabled] [size=1M]
	Region 4: Memory at 600000000 (32-bit, non-prefetchable) [disabled] [size=1M]
	Region 5: I/O ports at <unassigned> [disabled]
	[virtual] Expansion ROM at 600100000 [disabled] [size=1M]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
		Address: 0000000000000000  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [70] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 1024 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited, L1 <64us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [b0] MSI-X: Enable- Count=128 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [100 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [148 v1] Power Budgeting <?>
	Capabilities: [158 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [168 v1] #19
	Capabilities: [254 v1] #16
	Capabilities: [284 v1] Vendor Specific Information: ID=0002 Rev=1 Len=100 <?>
	Capabilities: [384 v1] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
	Capabilities: [3bc v1] #15

[geerlingguy]

All right, you must be joking.

Spoilers—everything's already working, and I'm just taking some time to compile the info to put on this issue at some point... but I've been scratching my head as to why I've only been able to get ~20 MB/sec for network copies (mostly testing with SMB and NFS) through the Pi with either the SSD or SAS drives attached.

It turns out, I had both WiFi and Wired networks enabled on my Mac, and even though I have all the network priority set up for wired, for some reason SMB, NFS, and rsync/SSH copies were all being performed over WiFi! It was when I noticed all copies were about the same speed (~144 Mbps) that I started suspecting this to be the problem.

Ugh, why can't multiple network interfaces ever be easy?!

[geerlingguy]

Documenting here the entire bringup process:

Prerequisite: You'll need a driver from Broadcom directly that supports ARMv7 / ARM64—the changes in the driver I used won't be available from the public download pages for a while.

1. Flash Pi OS 64-bit to the CM4 eMMC or a microSD card.

2. Boot the Pi.

3. Recompile the Pi OS kernel with CONFIG_IRQ_POLL=y:

   sudo apt-get install -y git bc bison flex libssl-dev make
   git clone --depth=1 https://github.com/raspberrypi/linux
   cd linux
   make ARCH=arm64 bcm2711_defconfig
   nano .config  # Set CONFIG_IRQ_POLL=y in here...
   make -j4 ARCH=arm64 Image modules dtbs  # This took ~1 hour on my CM4...
   sudo make ARCH=arm64 modules_install
   sudo cp arch/arm64/boot/Image /boot/kernel8.img
   sudo cp arch/arm64/boot/dts/broadcom/*.dtb /boot/
   sudo cp arch/arm64/boot/dts/overlays/*.dtb* /boot/overlays/
   sudo cp arch/arm64/boot/dts/overlays/README /boot/overlays/
   

4. Install ctags (prerequisite to compiling the Broadcom driver): sudo apt-get install -y ctags

5. Expand and compile the driver:

   tar -xvf megaraid_sas-07.718.00.00-src.tar.gz
   cd megaraid_sas-07.718.00.00/
   ./compile.sh
   

6. Enable the module: sudo insmod megaraid_sas.ko (assuming you have the card plugged in and a backplane or storage devices plugged into the card already).

At this point, you can use StorCLI to work on volumes and drives connected to the card—if it initialized correctly.

Here's what it looks like in dmesg once the card is successfully initialized:

[  138.129230] megaraid_sas: loading out-of-tree module taints kernel.
[  138.135261] megasas: 07.718.00.00
[  138.135732] pci 0000:00:00.0: enabling device (0000 -> 0002)
[  138.135767] megaraid_sas 0000:01:00.0: enabling device (0000 -> 0002)
[  138.136656] megaraid_sas 0000:01:00.0: BAR:0x0  BAR's base_addr(phys):0x0x0000000600200000  mapped virt_addr:0x00000000238916d8
[  138.136677] megaraid_sas 0000:01:00.0: Waiting for FW to come to ready state
[  139.400492] megaraid_sas 0000:01:00.0: FW now in Ready state
[  139.400515] megaraid_sas 0000:01:00.0: 63 bit DMA mask and 63 bit consistent mask
[  139.402988] megaraid_sas 0000:01:00.0: firmware supports msix	: (128)
[  139.403516] megaraid_sas 0000:01:00.0: requested/available msix 5/5
[  139.403531] megaraid_sas 0000:01:00.0: current msix/online cpus	: (5/4)
[  139.403543] megaraid_sas 0000:01:00.0: RDPQ mode	: (enabled)
[  139.403559] megaraid_sas 0000:01:00.0: Current firmware supports maximum commands: 3053	 LDIO threshold: 0
[  139.605418] megaraid_sas 0000:01:00.0: Performance mode :Latency (latency index = 1)
[  139.605440] megaraid_sas 0000:01:00.0: FW supports sync cache	: Yes
[  139.605452] megaraid_sas 0000:01:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
[  139.716509] megaraid_sas 0000:01:00.0: FW provided supportMaxExtLDs: 1	max_lds: 64
[  139.716518] megaraid_sas 0000:01:00.0: controller type	: MR(4096MB)
[  139.716523] megaraid_sas 0000:01:00.0: Online Controller Reset(OCR)	: Enabled
[  139.716530] megaraid_sas 0000:01:00.0: Secure JBOD support	: Yes
[  139.716530] megaraid_sas 0000:01:00.0: NVMe passthru support	: Yes
[  139.716539] megaraid_sas 0000:01:00.0: FW provided TM TaskAbort/Reset timeout	: 6 secs/60 secs
[  139.716543] megaraid_sas 0000:01:00.0: PCI Lane Margining support	: No
[  139.716550] megaraid_sas 0000:01:00.0: JBOD sequence map support	: Yes
[  139.745384] megaraid_sas 0000:01:00.0: NVME page size	: (4096)
[  139.745532] megaraid_sas 0000:01:00.0: megasas_enable_intr_fusion is called outbound_intr_mask:0x40000000
[  139.745532] megaraid_sas 0000:01:00.0: INIT adapter done
[  139.747717] megaraid_sas 0000:01:00.0: Snap dump wait time	: 15
[  139.747735] megaraid_sas 0000:01:00.0: pci id		: (0x1000)/(0x0014)/(0x1000)/(0x9460)
[  139.747740] megaraid_sas 0000:01:00.0: unevenspan support	: no
[  139.747740] megaraid_sas 0000:01:00.0: firmware crash dump	: no
[  139.747740] megaraid_sas 0000:01:00.0: JBOD sequence map	: enabled
[  139.747898] megaraid_sas 0000:01:00.0: Max firmware commands: 3052 shared with nr_hw_queues = 4
[  139.747904] scsi host0: Broadcom SAS based MegaRAID driver
[  139.769816] scsi 0:0:97:0: Enclosure         LSI      VirtualSES       03   PQ: 0 ANSI: 7
[  139.778076] scsi 0:0:97:0: Attached scsi generic sg0 type 13
[  139.789480] scsi 0:2:0:0: Direct-Access     AVAGO    MR9460-16i       5.16 PQ: 0 ANSI: 5
[  139.790170] sd 0:2:0:0: [sda] 700121088 512-byte logical blocks: (358 GB/334 GiB)
[  139.790216] sd 0:2:0:0: [sda] Write Protect is off
[  139.790223] sd 0:2:0:0: [sda] Mode Sense: 1f 00 00 08
[  139.790295] sd 0:2:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[  139.790410] sd 0:2:0:0: Attached scsi generic sg1 type 0
[  139.790606] sd 0:2:0:0: [sda] Optimal transfer size 196608 bytes
[  139.791735] scsi 0:2:1:0: Direct-Access     AVAGO    MR9460-16i       5.16 PQ: 0 ANSI: 5
[  139.792089] sd 0:2:0:0: [sda] Attached SCSI disk
[  139.792341] sd 0:2:1:0: Attached scsi generic sg2 type 0
[  139.792558] sd 0:2:1:0: [sdb] 1754529792 512-byte logical blocks: (898 GB/837 GiB)
[  139.793181] sd 0:2:1:0: [sdb] Write Protect is off
[  139.793196] sd 0:2:1:0: [sdb] Mode Sense: 1f 00 00 08
[  139.793281] sd 0:2:1:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[  139.793594] sd 0:2:1:0: [sdb] Optimal transfer size 786432 bytes
[  139.799745] sd 0:2:1:0: [sdb] Attached SCSI disk

You'll note this log is from after I had already used StorCLI to build two volumes, which are attached as sda and sdb once the driver loads.

We were originally having issues where we'd get the following messages after a long delay midway through initialization:

[  372.867846] megaraid_sas 0000:01:00.0: Init cmd return status FAILED for SCSI host 0
[  373.054122] megaraid_sas 0000:01:00.0: Failed from megasas_init_fw 6747

[geerlingguy]

I should also mention the exact same process is required if you want to run this card on 32-bit Raspberry Pi OS.

[geerlingguy]

Just to shed a little more light on the problems we had with the Pi OS 64-bit kernel:

• It seems like the BAR allocations and memory access is a little strange, where some accesses are 32-bit for 64-bit-configured BAR space.
• On 64-bit Pi, writeq/readq exist, but seem to be disabled/not working, so that causes confusion in some drivers
• Other drivers and Pi-based projects that run into this issue seem to implement Pi-specific workarounds
• It might be something that can be more easily solved in the Pi's kernel fork, by adding a config option or implementing some form of writeq that works correctly.

[geerlingguy]

Linking back to the kernel bug report: raspberrypi/linux#4158

[geerlingguy]

fio command and config for 1MB random read:

$ sudo fio --time_based=1 --thinktime=0 --runtime=45 --iodepth=64 --ramp_time=10 --group_reporting 1024K-rand-read.fio

$ cat 1024K-rand-read.fio 
[global]
buffered=0
random_generator=tausworthe64
scramble_buffers=1
sync=0
direct=1
percentile_list=1:5:10:20:30:40:50:60:70:80:90:95:99:99.5:99.9:99.95:99.99
offset=0
ioengine=libaio
cpus_allowed=0-3
norandommap
random_distribution=random:100
do_verify=0
randrepeat=0
continue_on_error=none
nice=-5
blockalign=4096
blocksize=1024k
readwrite=randread
rwmixread=100
percentage_random=100

[Target0]
filename=/dev/sda
numjobs=1
flow_id=0

sda (Kingston SSDs)

$ sudo fio --time_based=1 --thinktime=0 --runtime=45 --iodepth=64 --ramp_time=10 --group_reporting 1024K-rand-read.fio
Target0: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=399MiB/s][r=399 IOPS][eta 00m:00s]
Target0: (groupid=0, jobs=1): err= 0: pid=760: Fri Feb 26 17:38:50 2021
  read: IOPS=395, BW=397MiB/s (417MB/s)(17.5GiB/45106msec)
    slat (nsec): min=0, max=2519.9k, avg=177039.27, stdev=56358.83
    clat (nsec): min=0, max=833010k, avg=161201258.61, stdev=81075354.71
     lat (nsec): min=0, max=833306k, avg=161379074.55, stdev=81077781.78
    clat percentiles (msec):
     |  1.00th=[   69],  5.00th=[   87], 10.00th=[   96], 20.00th=[  110],
     | 30.00th=[  124], 40.00th=[  148], 50.00th=[  157], 60.00th=[  161],
     | 70.00th=[  165], 80.00th=[  174], 90.00th=[  207], 95.00th=[  317],
     | 99.00th=[  550], 99.50th=[  634], 99.90th=[  751], 99.95th=[  810],
     | 99.99th=[  818]
   bw (  KiB/s): min=    0, max=423936, per=99.87%, avg=406278.97, stdev=9308.35, samples=90
   iops        : min=    0, max=  414, avg=396.74, stdev= 9.10, samples=90
  lat (msec)   : 50=0.08%, 100=12.66%, 250=80.93%, 500=5.34%, 750=1.25%
  lat (msec)   : 1000=0.10%
  cpu          : usr=0.57%, sys=6.69%, ctx=30338, majf=0, minf=17
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=17856,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=397MiB/s (417MB/s), 397MiB/s-397MiB/s (417MB/s-417MB/s), io=17.5GiB (18.8GB), run=45106-45106msec

Disk stats (read/write):
  sda: ios=131038/0, merge=0/0, ticks=16892098/0, in_queue=16892098, util=99.93%

sdb (HP Proliant 10K SAS HDDs)

$ sudo fio --time_based=1 --thinktime=0 --runtime=45 --iodepth=64 --ramp_time=10 --group_reporting 1024K-rand-read.fio
Target0: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [r(1)][0.9%][r=106MiB/s][r=106 IOPS][eta 01h:38m:56s]
Target0: (groupid=0, jobs=1): err= 0: pid=780: Fri Feb 26 17:45:24 2021
  read: IOPS=144, BW=146MiB/s (153MB/s)(6650MiB/45448msec)
    slat (nsec): min=0, max=512241, avg=271826.49, stdev=31943.28
    clat (nsec): min=0, max=1416.9M, avg=439770563.63, stdev=276100098.44
     lat (nsec): min=0, max=1417.1M, avg=440043348.41, stdev=276100397.68
    clat percentiles (msec):
     |  1.00th=[  129],  5.00th=[  155], 10.00th=[  176], 20.00th=[  209],
     | 30.00th=[  247], 40.00th=[  292], 50.00th=[  355], 60.00th=[  426],
     | 70.00th=[  518], 80.00th=[  651], 90.00th=[  852], 95.00th=[ 1028],
     | 99.00th=[ 1334], 99.50th=[ 1351], 99.90th=[ 1385], 99.95th=[ 1385],
     | 99.99th=[ 1418]
   bw (  KiB/s): min=    0, max=174080, per=99.99%, avg=149818.16, stdev=12032.10, samples=90
   iops        : min=    0, max=  170, avg=146.19, stdev=11.76, samples=90
  lat (msec)   : 4=0.06%, 10=0.02%, 100=0.02%, 250=30.83%, 500=38.00%
  lat (msec)   : 750=17.58%, 1000=8.87%
  cpu          : usr=0.31%, sys=4.70%, ctx=6795, majf=0, minf=17
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=6587,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=146MiB/s (153MB/s), 146MiB/s-146MiB/s (153MB/s-153MB/s), io=6650MiB (6973MB), run=45448-45448msec

Disk stats (read/write):
  sdb: ios=15751/0, merge=0/0, ticks=5444861/0, in_queue=5444861, util=99.87%

[geerlingguy]

fio command, config, and results for 1MB random write:

$ sudo fio --time_based=1 --thinktime=0 --runtime=45 --iodepth=64 --ramp_time=10 --group_reporting 1024K-rand-write.fio

$ cat 1024K-rand-write.fio 
[global]
buffered=0
random_generator=tausworthe64
scramble_buffers=1
sync=0
direct=1
percentile_list=1:5:10:20:30:40:50:60:70:80:90:95:99:99.5:99.9:99.95:99.99
offset=0
ioengine=libaio
cpus_allowed=0-3
norandommap
random_distribution=random:100
do_verify=0
randrepeat=0
continue_on_error=none
nice=-5
blockalign=4096
blocksize=1024k
readwrite=randwrite
rwmixread=0
percentage_random=100

[Target0]
filename=/dev/sda
numjobs=1
flow_id=0

sda (Kingston SSDs)

$ sudo fio --time_based=1 --thinktime=0 --runtime=45 --iodepth=64 --ramp_time=10 --group_reporting 1024K-rand-write.fio
Target0: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=0): [f(1)][100.0%][w=311MiB/s][w=311 IOPS][eta 00m:00s]
Target0: (groupid=0, jobs=1): err= 0: pid=786: Fri Feb 26 17:46:48 2021
  write: IOPS=274, BW=276MiB/s (289MB/s)(12.2GiB/45185msec); 0 zone resets
    slat (nsec): min=0, max=1126.6k, avg=426122.06, stdev=102943.94
    clat (nsec): min=0, max=1605.8M, avg=232241941.35, stdev=206265049.39
     lat (nsec): min=0, max=1606.3M, avg=232669227.52, stdev=206267504.39
    clat percentiles (msec):
     |  1.00th=[   58],  5.00th=[   65], 10.00th=[   71], 20.00th=[   88],
     | 30.00th=[  113], 40.00th=[  161], 50.00th=[  188], 60.00th=[  203],
     | 70.00th=[  220], 80.00th=[  275], 90.00th=[  527], 95.00th=[  701],
     | 99.00th=[ 1003], 99.50th=[ 1116], 99.90th=[ 1502], 99.95th=[ 1569],
     | 99.99th=[ 1569]
   bw (  KiB/s): min=    0, max=372736, per=99.86%, avg=281749.71, stdev=89010.67, samples=90
   iops        : min=    0, max=  364, avg=275.07, stdev=86.94, samples=90
  lat (msec)   : 50=0.01%, 100=26.46%, 250=50.74%, 500=12.33%, 750=7.00%
  lat (msec)   : 1000=2.95%
  cpu          : usr=7.33%, sys=4.14%, ctx=25666, majf=0, minf=17
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,12387,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=276MiB/s (289MB/s), 276MiB/s-276MiB/s (289MB/s-289MB/s), io=12.2GiB (13.1GB), run=45185-45185msec

Disk stats (read/write):
  sda: ios=15/97990, merge=0/0, ticks=5/20626166, in_queue=20626172, util=99.68%

sdb (HP Proliant 10K SAS HDDs)

$ sudo fio --time_based=1 --thinktime=0 --runtime=45 --iodepth=64 --ramp_time=10 --group_reporting 1024K-rand-write.fio
Target0: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=64
fio-3.12
Starting 1 process
Jobs: 1 (f=1): [w(1)][0.6%][w=68.0MiB/s][w=68 IOPS][eta 02h:30m:02s]
Target0: (groupid=0, jobs=1): err= 0: pid=803: Fri Feb 26 17:48:17 2021
  write: IOPS=95, BW=96.8MiB/s (102MB/s)(4418MiB/45623msec); 0 zone resets
    slat (nsec): min=0, max=735129, avg=475278.54, stdev=77991.79
    clat (nsec): min=0, max=1285.4M, avg=664899695.04, stdev=94807677.18
     lat (nsec): min=0, max=1285.7M, avg=665376073.45, stdev=94807409.77
    clat percentiles (msec):
     |  1.00th=[  510],  5.00th=[  558], 10.00th=[  584], 20.00th=[  600],
     | 30.00th=[  617], 40.00th=[  634], 50.00th=[  651], 60.00th=[  667],
     | 70.00th=[  684], 80.00th=[  709], 90.00th=[  776], 95.00th=[  852],
     | 99.00th=[ 1020], 99.50th=[ 1133], 99.90th=[ 1267], 99.95th=[ 1267],
     | 99.99th=[ 1284]
   bw (  KiB/s): min=    0, max=129024, per=98.82%, avg=97987.56, stdev=16416.15, samples=91
   iops        : min=    0, max=  126, avg=95.59, stdev=16.03, samples=91
  lat (msec)   : 500=0.99%, 750=88.79%, 1000=10.54%
  cpu          : usr=2.43%, sys=2.92%, ctx=4587, majf=0, minf=17
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=0,4355,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=96.8MiB/s (102MB/s), 96.8MiB/s-96.8MiB/s (102MB/s-102MB/s), io=4418MiB (4633MB), run=45623-45623msec

Disk stats (read/write):
  sdb: ios=53/13528, merge=0/0, ticks=133/6968760, in_queue=6968894, util=99.87%

Those SAS drives are getting slower :D