Test LSI 9405W-16i SAS/NVMe HBA

[geerlingguy]

The LSI 9405W-16i HBA Should be similar to the 9460-16i, and should hopefully be supported on ARM (to some extent) unlike older cards like the 9305-16i (see #195). (Adding the term 9405 so this will also pop up in search.)

[geerlingguy]

Supposedly it's working on 32-bit Pi OS but not 64-bit. More to come.

[joshuaboud]

lspci

Output of sudo lspci -vvv on a custom 32-Bit Kernel:

01:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3616 Fusion-MPT Tri-Mode I/O Controller Chip (IOC) (rev 02)
        Subsystem: LSI Logic / Symbios Logic SAS3616 Fusion-MPT Tri-Mode I/O Controller Chip (IOC)
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 64
        Region 0: Memory at 600200000 (64-bit, prefetchable) [size=1M]
        Region 2: Memory at 600300000 (64-bit, prefetchable) [size=1M]
        Region 4: Memory at 600000000 (32-bit, non-prefetchable) [size=1M]
        Region 5: I/O ports at <unassigned> [disabled]
        [virtual] Expansion ROM at 600100000 [disabled] [size=1M]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [70] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 1024 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [b0] MSI-X: Enable+ Count=128 Masked-
                Vector table: BAR=0 offset=00002000
                PBA: BAR=0 offset=00003000
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
        Capabilities: [148 v1] Power Budgeting <?>
        Capabilities: [158 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 0
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [168 v1] #19
        Capabilities: [264 v1] #16
        Capabilities: [294 v1] Vendor Specific Information: ID=0002 Rev=2 Len=100 <?>
        Capabilities: [394 v1] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
        Capabilities: [3cc v0] Virtual Channel
                Caps:   LPEVC=1 RefClk=100ns PATEntryBits=1
                Arb:    Fixed- WRR32- WRR64- WRR128- ??6+
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32+ WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable- ID=0 ArbSelect=Fixed TC/VC=00
                        Status: NegoPending- InProgress-
        Kernel driver in use: mpt3sas

Kernel

Our custom kernel

Repo for custom kernel: https://github.com/45Drives/linux
Steps for building:

• clone repo
• cd linux/cross_compile
• ./build_cm4.sh 32 (no to customize config)
• copy files from ./boot to CM4 boot partition
• copy files from ./rootfs to CM4 root partition

Customizing your own

Follow Jeff's instructions, and in the .config, set:

• CONFIG_LOCALVERSION="-v7l-STORINATOR_JR"
• CONFIG_IRQ_POLL=y
• CONFIG_RAID_ATTRS=y
• CONFIG_SCSI_SAS_ATTRS=y
• # CONFIG_BLK_DEV_SD=y already set by default
• CONFIG_CHR_DEV_SG=y
• CONFIG_SCSI_MPT3SAS=y
  (Not sure if this route will work, as we replaced the mpt3sas driver in the kernel source with the latest from Broadcom's downloads for the 9405W. It will need further testing to see if the stock mpt3sas kernel driver will work.)

Using the card

We had a few 250 G SSDs around so we built a ZFS raid with it. Unfortunately, ZFS only works with 64-bit kernels, but ZFS Fuse works fine with 32-bit.
lsblk:

NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda            8:0    0 232.9G  0 disk 
sdb            8:16   0 232.9G  0 disk 
sdc            8:32   0 232.9G  0 disk 
sdd            8:48   0 232.9G  0 disk 
sde            8:64   0 232.9G  0 disk 
sdf            8:80   0 232.9G  0 disk 
sdg            8:96   0 232.9G  0 disk 
sdh            8:112  0 232.9G  0 disk 
sdi            8:128  0 232.9G  0 disk 
sdj            8:144  0 232.9G  0 disk 
mmcblk0      179:0    0  29.1G  0 disk 
├─mmcblk0p1  179:1    0   256M  0 part /boot
└─mmcblk0p2  179:2    0  28.9G  0 part /
mmcblk0boot0 179:32   0     4M  1 disk 
mmcblk0boot1 179:64   0     4M  1 disk

zpool status:

  pool: weenie-hut-jr
 state: ONLINE
 scrub: scrub completed after 0h0m with 0 errors on Tue Aug 24 14:44:40 2021
config:

        NAME                                                         STATE     READ WRITE CKSUM
        weenie-hut-jr                                                ONLINE       0     0     0
          raidz2-0                                                   ONLINE       0     0     0
            disk/by-id/wwn-0x5001b448b856e403                        ONLINE       0     0     0
            disk/by-id/wwn-0x588891410069f515                        ONLINE       0     0     0
            disk/by-id/ata-HDSTOR_-_HSAV25ST250AX_HS2001211005ECA56  ONLINE       0     0     0
            disk/by-id/wwn-0x5001b448b6affbfa                        ONLINE       0     0     0
            disk/by-id/wwn-0x5001b448b670ea7d                        ONLINE       0     0     0
          raidz2-1                                                   ONLINE       0     0     0
            disk/by-id/wwn-0x5001b448b670d475                        ONLINE       0     0     0
            disk/by-id/ata-WDC_WDS250G2B0A-00SM50_191540800565       ONLINE       0     0     0
            disk/by-id/wwn-0x5001b448b630fedb                        ONLINE       0     0     0
            disk/by-id/ata-WDC_WDS250G2B0A_191934804052              ONLINE       0     0     0
            disk/by-id/wwn-0x5001b448b8f9b97c                        ONLINE       0     0     0

errors: No known data errors

[joshuaboud]

As for 64-bit usage, the kernel was built the same way as above, but use ./build-cm4.sh 64. This kernel works fine with nothing plugged into the PCIe slot, but when the 9405W is plugged in, there is a kernel panic at boot. Here is a screenshot of said kernel panic:

The error seems to happen in pci_generic_config_read(), seemingly part of the PCI driver, not the mpt3sas driver.

[geerlingguy]

@joshuaboud - I seem to remember this bit of code leading to that crash: https://github.com/raspberrypi/linux/blob/2697f7403187bb2bb61cc716f33ee9f6cfb9af7c/drivers/scsi/megaraid/megaraid_sas_fusion.c#L262-L265

Can you try the following patch and see if that helps on 64-bit?

diff --git a/drivers/scsi/megaraid/megaraid_sas_fusion.c b/drivers/scsi/megaraid/megaraid_sas_fusion.c
index b0c01cf0428f..c4accee42e84 100644
--- a/drivers/scsi/megaraid/megaraid_sas_fusion.c
+++ b/drivers/scsi/megaraid/megaraid_sas_fusion.c
@@ -259,7 +259,8 @@ static void
 megasas_write_64bit_req_desc(struct megasas_instance *instance,
 		union MEGASAS_REQUEST_DESCRIPTOR_UNION *req_desc)
 {
-#if defined(writeq) && defined(CONFIG_64BIT)
+//#if defined(writeq) && defined(CONFIG_64BIT)
+#if 0
 	u64 req_data = (((u64)le32_to_cpu(req_desc->u.high) << 32) |
 		le32_to_cpu(req_desc->u.low));
 	writeq(req_data, &instance->reg_set->inbound_low_queue_port);

See: raspberrypi/linux#4158

[joshuaboud]

That specific patch won't do much as the megaraid driver isn't being compiled for my kernel, however - there is a very similar chunk of code in the mpt3sas driver:
drivers/scsi/mpt3sas/mpt3sas_base.c starting at line 5809:

/**
 * _base_writeq - 64 bit write to MMIO
 * @ioc: per adapter object
 * @b: data payload
 * @addr: address in MMIO space
 * @writeq_lock: spin lock
 *
 * Glue for handling an atomic 64 bit word to MMIO. This special handling takes
 * care of 32 bit environment where its not quarenteed to send the entire word
 * in one transfer.
 */
#if defined(writeq) && defined(CONFIG_64BIT)
static inline void
_base_writeq(__u64 b, volatile void __iomem *addr, spinlock_t *writeq_lock)
{
	writeq(b, addr);
}
#else
static inline void
_base_writeq(__u64 b, volatile void __iomem *addr, spinlock_t *writeq_lock)
{
	unsigned long flags;
	__u64 data_out = b;

	spin_lock_irqsave(writeq_lock, flags);
	writel((u32)(data_out), addr);
	writel((u32)(data_out >> 32), (addr + 4));
	spin_unlock_irqrestore(writeq_lock, flags);
}
#endif

I suppose if I apply the same change here, disabling writeq(), it may fix the issue. I will report back after trying.

[geerlingguy]

Ah yes, sorry was looking at what I was working on for the other card and forgot it was using a different module. Let me know if it helps!

[joshuaboud]

just tried out that fix with the patch:

@@ -5817,7 +5817,8 @@ _base_mpi_ep_writeq(__u64 b, volatile void __iomem *addr, spinlock_t *writeq_loc
  * care of 32 bit environment where its not quarenteed to send the entire word
  * in one transfer.
  */
-#if defined(writeq) && defined(CONFIG_64BIT)
+//#if defined(writeq) && defined(CONFIG_64BIT)
+#if 0
 static inline void
 _base_writeq(__u64 b, volatile void __iomem *addr, spinlock_t *writeq_lock)
 {

but unfortunately it did not fix the kernel panic issue.

I have been trying to track down any possible bugs in the kernel source code, but the second last function call, pci_bus_read_config_byte() does not seem to have a definition anywhere in the source. The function that actually causes the error, pci_generic_config_read(), could be having an issue with dereferencing pointers though. I am going to try to get the full dmesg output from the failed boot.

[geerlingguy]

@joshuaboud - That sounds eerily similar to some of the problems we've been encountering with AMD's GPU drivers in #4 — I wonder if the mpt3sas driver is missing some of the megaraid sas changes that were made for better ARM compatibility in general?

[joshuaboud]

LSI 9405W-16i Now Working on Custom 64-bit Kernel

The Fix

This is very strange. Removing quiet from /boot/cmdline.txt fixed the kernel panic issue with the Storinator JR custom 64-bit kernel.

Reason

We have no idea. Maybe there is a timing issue in the PCI driver that is solved by the slow down from printing console messages.

Use

lspci -vvv:

01:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3616 Fusion-MPT Tri-Mode I/O Controller Chip (IOC) (rev 02)
        Subsystem: LSI Logic / Symbios Logic SAS3616 Fusion-MPT Tri-Mode I/O Controller Chip (IOC)
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 49
        Region 0: Memory at 600200000 (64-bit, prefetchable) [size=1M]
        Region 2: Memory at 600300000 (64-bit, prefetchable) [size=1M]
        Region 4: Memory at 600000000 (32-bit, non-prefetchable) [size=1M]
        Region 5: I/O ports at <unassigned> [disabled]
        [virtual] Expansion ROM at 600100000 [disabled] [size=1M]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [70] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 1024 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [b0] MSI-X: Enable+ Count=128 Masked-
                Vector table: BAR=0 offset=00002000
                PBA: BAR=0 offset=00003000
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
        Capabilities: [148 v1] Power Budgeting <?>
        Capabilities: [158 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 0
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [168 v1] #19
        Capabilities: [264 v1] #16
        Capabilities: [294 v1] Vendor Specific Information: ID=0002 Rev=2 Len=100 <?>
        Capabilities: [394 v1] Vendor Specific Information: ID=0001 Rev=1 Len=038 <?>
        Capabilities: [3cc v0] Virtual Channel
                Caps:   LPEVC=1 RefClk=100ns PATEntryBits=1
                Arb:    Fixed- WRR32- WRR64- WRR128- ??6+
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32+ WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable- ID=0 ArbSelect=Fixed TC/VC=00
                        Status: NegoPending- InProgress-
        Kernel driver in use: mpt3sas

lsblk:

NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda            8:0    0 232.9G  0 disk 
sdb            8:16   0 232.9G  0 disk 
sdc            8:32   0 232.9G  0 disk 
sdd            8:48   0 232.9G  0 disk 
sde            8:64   0 232.9G  0 disk 
sdf            8:80   0 232.9G  0 disk 
sdg            8:96   0 232.9G  0 disk 
sdh            8:112  0 232.9G  0 disk 
sdi            8:128  0 232.9G  0 disk 
sdj            8:144  0 232.9G  0 disk 
mmcblk0      179:0    0  29.1G  0 disk 
├─mmcblk0p1  179:1    0   256M  0 part /boot
└─mmcblk0p2  179:2    0  28.9G  0 part /
mmcblk0boot0 179:32   0     4M  1 disk 
mmcblk0boot1 179:64   0     4M  1 disk

[geerlingguy]

Reason

We have no idea.

🤣 love it

[wallentx]

Oooooo this is about to get interesting..
I have the following:

• SAS 9300-8e
• SAS 9305-16e
• SAS 9440-8i
• CM4 8GB RAM - 32GB eMMC arriving tomorrow
• Waveshare CM4 IO Board (Type B) w/PoE in transit
• A variety of PCIe 1x to 16x risers
• The following volumes stored across 2 JBODs that can be accessed with 2 SFF-8644 cables:

FILESYSTEM TYPE     (=) USED      FREE (-)  %USED AVAILABLE  TOTAL MOUNTED ON
/dev/sda1  ext4     [||||||||||||||||----]  77.4%     56.8G 251.0G /
/dev/sdb   xfs      [||||||||||||--------]  56.3%      1.4T   3.2T /mnt/ssdarray1
/dev/md0   xfs      [|-------------------]   0.7%     10.8T  10.9T /mnt/ssdarray2
/dev/sdo   ext4     [||||||||||||||||||||]  98.5%     60.7G   4.0T /mnt/plots
/dev/sdh   ext4     [||||||||||||||||||||]  98.0%     80.5G   4.0T /mnt/plots2
/dev/sdi   ext4     [||||||||||||||||||||]  98.0%     80.4G   4.0T /mnt/plots3
/dev/sdj   ext4     [||||||||||||||||||||]  98.0%     80.3G   4.0T /mnt/plots4
/dev/sdk   ext4     [||||||||||||||||||||]  98.0%     80.1G   4.0T /mnt/plots5
/dev/sdl   ext4     [|||||||||||||||||||-]  94.2%    232.4G   4.0T /mnt/plots6
/dev/sdm   ext4     [||||||||||||||||||||]  98.0%     80.3G   4.0T /mnt/plots7
/dev/sdn   ext4     [||||||||||||||||||||]  98.0%     80.3G   4.0T /mnt/plots8
/dev/sdg   ext4     [||||||||||||||||||||]  98.0%     80.2G   4.0T /mnt/plots9
/dev/sdp   ext4     [||||||||||||||||||||]  98.2%     73.8G   4.0T /mnt/plots10
/dev/sdq   ext4     [||||||||||||||||||||]  98.3%     67.5G   4.0T /mnt/plots11
/dev/sdr   ext4     [|||||||||||||||||||-]  92.7%    291.2G   4.0T /mnt/plots12
/dev/sdt   ext4     [||||||||||||||||||||]  98.7%     40.4G   3.0T /mnt/plots14
/dev/sdu   ext4     [||||||||||||||||||||]  98.0%     80.5G   4.0T /mnt/plots15
/dev/sdv   ext4     [||||||||||||||||||||]  98.0%     80.0G   4.0T /mnt/plots16
/dev/sdw   ext4     [||||||||||||||||||||]  98.0%     80.4G   4.0T /mnt/plots17
/dev/sdx   ext4     [||||||||||||||||||||]  98.0%     80.1G   4.0T /mnt/plots18
/dev/sdad  ext4     [||||||||||||||||||||]  98.0%     39.5G   2.0T /mnt/plots19
/dev/sdy   ext4     [||||||||||||||||||||]  99.1%     92.5G  10.0T /mnt/plots20
/dev/sdz   ext4     [||||||||||||||||||||]  99.1%     93.5G  10.0T /mnt/plots21
/dev/sdaa  ext4     [||||||||||||||||||||]  99.1%     92.4G  10.0T /mnt/plots22
/dev/sdab  ext4     [||||||||||||||||||||]  99.8%     24.1G  12.0T /mnt/plots23
/dev/sdah  ext4     [||||||||||||||||||||]  99.3%     52.8G   8.0T /mnt/plots24
/dev/sdag  ext4     [||||||||||||||||||||]  99.8%     19.9G   8.0T /mnt/plots25
/dev/sdam  ext4     [||||||||||||||||||||]  99.3%     54.0G   8.0T /mnt/plots26
/dev/sdac1 btrfs    [||||||||||||||||||||]  99.3%     69.7G  10.0T /mnt/plots27
/dev/sdal1 btrfs    [||||||||||||||||||||]  99.3%     66.0G  10.0T /mnt/plots28
/dev/sdak1 btrfs    [||||||||||||||||||||]  99.3%     68.7G  10.0T /mnt/plots29
/dev/sdai  ext4     [||||||||||||||||||||]  99.1%     93.5G  10.0T /mnt/plots30
/dev/sdaj  ext4     [||||||||||||||||||||]  99.1%     93.6G  10.0T /mnt/plots31
/dev/sdaf  ext4     [||||||||||||||||||||]  99.1%     92.4G  10.0T /mnt/plots32
/dev/sdbe  f2fs     [||||||||||||||||||||]  99.7%     27.9G  10.0T /mnt/plots33
/dev/sdbf  ext4     [||||||||||||||||||||]  99.8%     12.4G   6.0T /mnt/plots34
/dev/sdbg  ext4     [||||||||||||||||||||]  98.0%     80.8G   4.0T /mnt/plots35
/dev/sdbh  ext4     [||||||||||||||||||||]  98.0%     80.7G   4.0T /mnt/plots36
/dev/sdbi  f2fs     [||||||||||||||||||||]  99.0%    103.1G  10.0T /mnt/plots37
/dev/sdbj  ext4     [||||||||||||||||||||]  98.0%     80.8G   4.0T /mnt/plots38
/dev/sdbk  ext4     [||||||||||||||||||||]  98.0%     80.7G   4.0T /mnt/plots39
/dev/sdbl  btrfs    [||||||||||||||||||||]  99.3%     69.8G  10.0T /mnt/plots40
/dev/sdbm  ext4     [||||||||||||||||||||]  99.8%     12.9G   6.0T /mnt/plots41
/dev/sdbn  ext4     [||||||||||||||||||||]  98.0%     80.9G   4.0T /mnt/plots42
/dev/sdbo  ext4     [||||||||||||||||||||]  98.0%     81.0G   4.0T /mnt/plots43
/dev/sdbp  f2fs     [||||||||||||||||||||]  99.7%     27.9G  10.0T /mnt/plots44
/dev/sdbq  ext4     [||||||||||||||||||||]  99.8%     13.0G   6.0T /mnt/plots45
/dev/sdbr  ext4     [||||||||||||||||||||]  98.0%     80.6G   4.0T /mnt/plots46
/dev/sdbs  ext4     [||||||||||||||||||||]  98.0%     80.8G   4.0T /mnt/plots47
/dev/sdbt  ext4     [||||||||||||||||||||]  98.0%     80.7G   4.0T /mnt/plots48
/dev/sdbu  f2fs     [||||||||||||||||||||]  99.7%     28.0G  10.0T /mnt/plots49
/dev/sdbv  ext4     [||||||||||||||||||||]  99.8%     12.3G   6.0T /mnt/plots50
/dev/sdbw  ext4     [||||||||||||||||||||]  98.0%     80.8G   4.0T /mnt/plots51
/dev/sdbx  ext4     [||||||||||||||||||||]  98.0%     81.0G   4.0T /mnt/plots52
/dev/sdby  ext4     [||||||||||||||||||||]  98.0%     80.7G   4.0T /mnt/plots53
/dev/sdbz  f2fs     [||||||||||||||||||||]  99.7%     27.9G  10.0T /mnt/plots54
/dev/sdca  ext4     [||||||||||||||||||||]  98.5%     88.3G   6.0T /mnt/plots55
/dev/sdae  f2fs     [||||||||||||||||||||]  99.7%     28.3G  10.0T /mnt/plots56
/dev/sdcb  ext4     [||||||||||||||||||||]  98.0%     80.8G   4.0T /mnt/plots57
/dev/sdcc  f2fs     [||||||||||||||||||||]  99.7%     27.8G  10.0T /mnt/plots58
/dev/sdcd  ext4     [||||||||||||||||||||]  98.0%     80.7G   4.0T /mnt/plots59
/dev/sdce  ext4     [||||||||||||||||||||]  98.0%     80.7G   4.0T /mnt/plots60
/dev/sdcf  ext4     [||||||||||||||||||||]  98.0%     80.7G   4.0T /mnt/plots61
/dev/sdcg  ext4     [||||||||||||||||||||]  98.0%     80.7G   4.0T /mnt/plots62
/dev/sdch  ext4     [||||||||||||||||||||]  98.0%     80.8G   4.0T /mnt/plots63
/dev/sdci  f2fs     [||||||||||||||||||||]  99.7%     27.8G  10.0T /mnt/plots64
/dev/sdck  f2fs     [||||||||||||||||||||]  99.7%     27.8G  10.0T /mnt/plots65
/dev/sdcl  ext4     [||||||||||||||||||||]  98.0%     80.9G   4.0T /mnt/plots66
/dev/sdcm  ext4     [||||||||||||||||||||]  98.0%     80.9G   4.0T /mnt/plots67
/dev/sdco  f2fs     [||||||||||||||||||||]  99.7%     28.2G  10.0T /mnt/plots68
/dev/sdcp  f2fs     [||||||||||||||||||||]  99.0%    103.4G  10.0T /mnt/plots69
/dev/sdcq  ext4     [||||||||||||||||||||]  98.0%     80.9G   4.0T /mnt/plots70
/dev/sdcr  ext4     [||||||||||||||||||||]  98.0%     80.9G   4.0T /mnt/plots71
/dev/sdcs  f2fs     [||||||||||||||||||||]  99.7%     28.0G  10.0T /mnt/plots72
/dev/sdct  ext4     [||||||||||||||||||||]  98.0%     80.7G   4.0T /mnt/plots73
/dev/sdas  ext4     [||||||||||||||||||||]  98.0%     80.9G   4.0T /mnt/plots74
/dev/sdar  ext4     [||||||||||||||||||||]  98.0%     80.6G   4.0T /mnt/plots75
/dev/sdaq  ext4     [||||||||||||||||||||]  98.0%     80.6G   4.0T /mnt/plots76
/dev/sdap  ext4     [||||||||||||||||||||]  98.0%     80.6G   4.0T /mnt/plots77
/dev/sdao  ext4     [||||||||||||||||||||]  98.0%     80.9G   4.0T /mnt/plots78
/dev/sdan  ext4     [||||||||||||||||||||]  98.0%     80.9G   4.0T /mnt/plots79
/dev/sday  ext4     [||||||||||||||||||||]  98.0%     80.8G   4.0T /mnt/plots80
/dev/sdax  ext4     [||||||||||||||||||||]  98.0%     80.8G   4.0T /mnt/plots81
/dev/sdaw  ext4     [||||||||||||||||||||]  98.0%     80.8G   4.0T /mnt/plots82
/dev/sdav  ext4     [||||||||||||||||||||]  98.0%     80.7G   4.0T /mnt/plots83
/dev/sdau  ext4     [||||||||||||||||||||]  98.0%     80.8G   4.0T /mnt/plots84
/dev/sdat  ext4     [||||||||||||||||||||]  98.0%     80.7G   4.0T /mnt/plots85
/dev/sdcu  ext4     [||||||||||||||||||||]  98.0%     80.8G   4.0T /mnt/plots86
/dev/sdcv  ext4     [||||||||||||||||||||]  98.0%     80.5G   4.0T /mnt/plots87
/dev/sdcw  f2fs     [||||||||||||||||||||]  99.7%     28.0G  10.0T /mnt/plots88
/dev/sdcy  ext4     [||||||||||||||||||||]  98.0%     80.9G   4.0T /mnt/plots89
/dev/sdcz  ext4     [||||||||||||||||||||]  98.0%     80.7G   4.0T /mnt/plots90
/dev/sdda  ext4     [||||||||||||||||||||]  98.0%     80.7G   4.0T /mnt/plots91
/dev/sddb  ext4     [||||||||||||||||||||]  99.8%     12.7G   6.0T /mnt/plots92
/dev/sddc  ext4     [||||||||||||||||||||]  98.0%     80.8G   4.0T /mnt/plots93
/dev/sdbd  ext4     [||||||||||||||||||||]  99.1%     93.6G  10.0T /mnt/plots94
/dev/sdcx  ext4     [||||||||||||||||||||]  99.1%     93.9G  10.0T /mnt/plots95
/dev/sdcj  ext4     [||||||||||||||||||||]  99.1%     94.0G  10.0T /mnt/plots96
/dev/sdcn  ext4     [||||||||||||||||||||]  99.1%     93.6G  10.0T /mnt/plots97
SUM:                [||||||||||||||||||||]  96.7%     19.3T 589.2T

Using the delicious breadcrumbs left by you wonderful folks in this GH issue, I'll be trying to get one of these working. Is there any interest in seeing my results shared somewhere? (AKA - where can I ask for help when I fail)

[geerlingguy]

@wallentx - According to the Broadcom engineer I spoke with, none of the 93xx series cards will work on the Pi because drivers don't support ARM for that generation. Only 94xx/95xx and newer cards should be able to work (so your SAS 9440-8i has a fighting chance!).