Consistent filesystem corruption

[servusdei2018]

Describe the bug
My root filesystem gets constantly corrupted.

To reproduce
Usually, I shutdown with shutdown now, and when I power it up again, I get

[    4.503731]  sda: sda1 sda2
[    4.511049] sd 0:0:0:0: [sda] Attached SCSI disk
[    4.535157] EXT4-fs (sda2): INFO: recovery required on readonly filesystem
[    4.540448] EXT4-fs (sda2): write access will be enabled during recovery
[    4.648339] EXT4-fs (sda2): recovery complete
[    4.654438] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: (null)
[    4.659839] VFS: Mounted root (ext4 filesystem) readonly on device 8:2.

Expected behaviour
I don't expect my filesystem to get corrupted.

Actual behaviour
My filesystem gets corrupted, pretty often, and even when I run fsck from another linux machine it happens again.

System

• Which model of Raspberry Pi? 3B+
• Which OS and version (cat /etc/rpi-issue)? Raspbian Stretch, Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 7e0c786c641ba15990b5662f092c106beed40c9f, stage5
• Which firmware version (vcgencmd version)? Dec 21 2018 15:27:13 Copyright (c) 2012 Broadcom version da468960fe03ecbaa8e3f1ee01c7217c3bd01fa8 (clean) (release)
• Which kernel version (uname -a)? Linux raspberryPi 4.19.57-v7+ #1244 SMP Thu Jul 4 18:45:25 BST 2019 armv7l GNU/Linux

Logs
Here's the applicable part of dmesg:

[    4.503731]  sda: sda1 sda2
[    4.511049] sd 0:0:0:0: [sda] Attached SCSI disk
[    4.535157] EXT4-fs (sda2): INFO: recovery required on readonly filesystem
[    4.540448] EXT4-fs (sda2): write access will be enabled during recovery
[    4.648339] EXT4-fs (sda2): recovery complete
[    4.654438] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: (null)
[    4.659839] VFS: Mounted root (ext4 filesystem) readonly on device 8:2.

Additional context
The root filesystem is mounted on an external SSD, connected via USB. I've been running this setup for almost a year now, without errors like this.

Sometimes, while running the desktop (lxde), the windows just close one after the other, and when I try typing a command in the open lxterminal (bash), I get "Input/Output Error".

Note: looking trough other people's dmesg, it seems that this is not a unique problem.

[servusdei2018]

After extensive testing (booting and shutting it down), I think I've fount a piece of information that may help solve this.

My SATA-to-USB adapter thankfully has a active light. Here's what I found. After a shutdown (shutdown now), the light goes off. But after a couple seconds, it comes back on again, so power has been restored to USB?

Pulling the power plug after the light has come back on again corrupts my filesystem, pulling the power plug before it comes back on again does not.

[timg236]

Yes, USB is powered in the HALT state unless you set POWER_OFF_ON_HALT=1 and WAKE_ON_GPIO=0 in the bootloader which turns off the PMIC
https://www.raspberrypi.org/documentation/hardware/raspberrypi/bcm2711_bootloader_config.md

In future it might be possible to get the bootloader to power off the USB ports after a reset (which IIRC will enable power to the ports) but that won't be very soon.

[servusdei2018]

@timg236 , thanks, how would I do this on a Raspberry Pi 3b+? The documentation seems to be for RPi4.

[JamesH65]

@timg236 Any recollection of stuff like this on the Pi3B+? Presumably the config settings are Pi4 specific?