6.12.41 Rpi 5 Kernel NULL pointer and paging request crash during camera streaming #7102
Description
Describe the bug
Raspberry Pi 5 Kernel crashes during camera streaming tests. The tests include streaming the camera for longer hours, doing a resolution switch test or stress testing the camera with repeated start stop behavior. Upstream Raspberry Pi 5 kernel's imx290.c driver is used for the test.
We see different patterns of crash anywhere between 15m and 15hrs of running the tests. In some cases it's NULL pointer crash (mostly from DRM subsystem) and in some it is paging request. Please see the attached log files for the crash logs. Sample snippet below.
==================
NULL Pointer crash:
[26466.254279] vc4-drm axi:gpu: [drm] ERROR [CRTC:92:crtc-2] cleanup_done timed out
[26466.254304] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000148
[26466.263146] Mem abort info:
[26466.265943] ESR = 0x0000000096000005
[26466.269706] EC = 0x25: DABT (current EL), IL = 32 bits
[26466.275039] SET = 0, FnV = 0
[26466.278281] EA = 0, S1PTW = 0
[26466.282187] FSC = 0x05: level 1 translation fault
[26466.287110] Data abort info:
[26466.289997] ISV = 0, ISS = 0x00000005, ISS2 = 0x00000000
[26466.295510] CM = 0, WnR = 0, TnD = 0, TagAccess = 0
[26466.300581] GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
[26466.305934] user pgtable: 16k pages, 47-bit VAs, pgdp=0000000003398000
[26466.312558] [0000000000000148] pgd=0000000000000000, p4d=0000000000000000, pud=0000000000000000
[26466.321350] Internal error: Oops: 0000000096000005 [#1] PREEMPT SMP
[26466.327642] Modules linked in: rfcomm snd_seq_dummy snd_hrtimer snd_seq snd_seq_device algif_hash algif_skcipher af_alg bnep brcmfmac_wcc vc4 binfmt_misc brcmfmac brcmutil snd_soc_hdmi_codec imx290 drm_display_helper v4l2_cci regmap_i2c cec cfg80211 rpi_hevc_dec drm_dma_helper snd_soc_core v4l2_mem2mem rp1_cfe pisp_be v3d snd_compress v4l2_fwnode v4l2_async videobuf2_dma_contig snd_pcm_dmaengine videobuf2_memops snd_pcm gpu_sched videobuf2_v4l2 snd_timer videodev drm_shmem_helper snd drm_kms_helper rp1_mailbox i2c_designware_platform videobuf2_common rp1_pio mc i2c_designware_core rp1_adc spidev joydev hci_uart btbcm bluetooth aes_ce_blk aes_ce_cipher ghash_ce gf128mul sha2_ce sha256_arm64 sha1_ce ecdh_generic ecc rfkill libaes sha1_generic raspberrypi_hwmon i2c_brcmstb raspberrypi_gpiomem spi_bcm2835 rp1_fw gpio_keys nvmem_rmem uio_pdrv_genirq uio i2c_dev drm ledtrig_pattern drm_panel_orientation_quirks fuse backlight dm_mod ip_tables x_tables ipv6
[26466.412233] CPU: 0 UID: 1000 PID: 952 Comm: labwc Not tainted 6.12.41-v8-16k+ #3
[26466.419656] Hardware name: Raspberry Pi 5 Model B Rev 1.1 (DT)
[26466.425506] pstate: 00400009 (nzcv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[26466.432493] pc : drm_atomic_helper_setup_commit+0x2ec/0x7f8 [drm_kms_helper]
[26466.439586] lr : drm_atomic_helper_setup_commit+0x2c8/0x7f8 [drm_kms_helper]
[26466.446671] sp : ffffc00083d7ba60
[26466.449990] x29: ffffc00083d7ba60 x28: ffff8000034f1800 x27: 0000000000000004
[26466.457152] x26: ffff80000335ce00 x25: 0000000000000028 x24: ffff80000adae000
[26466.464313] x23: 0000000000000000 x22: ffffd06fcf6103e8 x21: ffff800009be9d80
[26466.471475] x20: ffffd06fcda753c0 x19: ffff800201c4c5c0 x18: ffffffffffffffff
[26466.478636] x17: 6d697420656e6f64 x16: 5f70756e61656c63 x15: 205d322d63747263
[26466.485797] x14: 3a32393a43545243 x13: 74756f2064656d69 x12: 7420656e6f645f70
[26466.492958] x11: 756e61656c63205d x10: ffffd06fcfce04e8 x9 : ffffd06fce505410
[26466.500119] x8 : 00000000ffffefff x7 : ffffd06fcfce04e8 x6 : 0000000000000000
[26466.507280] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000004
[26466.514441] x2 : 0000000000000038 x1 : ffff80000325b400 x0 : 0000000000000000
[26466.521602] Call trace:
[26466.524049] drm_atomic_helper_setup_commit+0x2ec/0x7f8 [drm_kms_helper]
[26466.530784] drm_atomic_helper_commit+0x60/0x190 [drm_kms_helper]
[26466.536909] drm_atomic_nonblocking_commit+0x64/0x80 [drm]
[26466.542460] drm_mode_atomic_ioctl+0x6a8/0xcd8 [drm]
[26466.547468] drm_ioctl_kernel+0xc0/0x140 [drm]
[26466.551955] drm_ioctl+0x210/0x4e8 [drm]
[26466.555915] __arm64_sys_ioctl+0xb0/0x100
[26466.559938] invoke_syscall+0x50/0x120
[26466.563696] el0_svc_common.constprop.0+0x48/0xf8
[26466.568414] do_el0_svc+0x28/0x40
[26466.571734] el0_svc+0x30/0x100
[26466.574881] el0t_64_sync_handler+0x13c/0x158
[26466.579248] el0t_64_sync+0x190/0x198
[26466.582918] Code: 52800702 f94012a1 9ba20400 f9400c00 (f940a413)
[26466.589032] ---[ end trace 0000000000000000 ]---
=======================
Paging Request crash:
[ 4452.638694] Unable to handle kernel paging request at virtual address 000905ba00470775
[ 4452.646665] Mem abort info:
[ 4452.649461] ESR = 0x0000000096000021
[ 4452.653217] EC = 0x25: DABT (current EL), IL = 32 bits
[ 4452.658544] SET = 0, FnV = 0
[ 4452.661602] EA = 0, S1PTW = 0
[ 4452.664747] FSC = 0x21: alignment fault
[ 4452.668764] Data abort info:
[ 4452.671647] ISV = 0, ISS = 0x00000021, ISS2 = 0x00000000
[ 4452.677148] CM = 0, WnR = 0, TnD = 0, TagAccess = 0
[ 4452.682212] GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
[ 4452.687539] [000905ba00470775] address between user and kernel address ranges
[ 4452.694700] Internal error: Oops: 0000000096000021 [#1] PREEMPT SMP
[ 4452.700988] Modules linked in: rfcomm snd_seq_dummy snd_hrtimer snd_seq snd_seq_device algif_hash algif_skcipher af_alg bnep binfmt_misc vc4 snd_soc_hdmi_codec drm_display_helper cec imx662 at24 drm_dma_helper brcmfmac_wcc joydev v4l2_cci regmap_i2c spidev brcmfmac hci_uart snd_soc_core aes_ce_blk rpi_hevc_dec btbcm aes_ce_cipher brcmutil ghash_ce gf128mul bluetooth pisp_be sha2_ce sha256_arm64 cfg80211 sha1_ce snd_compress rp1_cfe snd_pcm_dmaengine ecdh_generic v4l2_mem2mem ecc sha1_generic libaes snd_pcm v4l2_fwnode v4l2_async videobuf2_dma_contig v3d rfkill raspberrypi_hwmon snd_timer gpu_sched videobuf2_memops videobuf2_v4l2 drm_shmem_helper spi_bcm2835 snd videodev i2c_brcmstb gpio_keys drm_kms_helper videobuf2_common mc rp1_pio rp1_adc rp1_mailbox rp1_fw raspberrypi_gpiomem i2c_designware_platform i2c_designware_core nvmem_rmem uio_pdrv_genirq uio i2c_dev ledtrig_pattern drm drm_panel_orientation_quirks backlight fuse dm_mod ip_tables x_tables ipv6
[ 4452.785986] CPU: 0 UID: 0 PID: 47 Comm: kworker/u17:1 Not tainted 6.12.41-v8-16k+ #2
[ 4452.793758] Hardware name: Raspberry Pi 5 Model B Rev 1.1 (DT)
[ 4452.799609] Workqueue: events_unbound commit_work [drm_kms_helper]
[ 4452.805839] pstate: 204000c9 (nzCv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 4452.812825] pc : _raw_spin_lock_irq+0x28/0x80
[ 4452.817198] lr : wait_for_completion_timeout+0x40/0x158
[ 4452.822440] sp : ffffc00082273c50
[ 4452.825759] x29: ffffc00082273c50 x28: ffff8000074ef4f0 x27: ffff800003f492c0
[ 4452.832921] x26: ffff800100778080 x25: 0000000000000038 x24: ffff800002ceb100
[ 4452.840083] x23: ffffd06faf2b23c8 x22: 0000000000000038 x21: 000905ba00470775
[ 4452.847244] x20: 000905ba0047076d x19: 00000000000009c4 x18: 0000000000000000
[ 4452.854406] x17: 0000000000000000 x16: ffffd06fcf1a1ee0 x15: 00007ffed8077680
[ 4452.861567] x14: 0000000400000026 x13: 0000000400000025 x12: 0000000100000024
[ 4452.868728] x11: 00000000000000c0 x10: 0000000000001a40 x9 : ffffd06fcf1a1f20
[ 4452.875890] x8 : ffff8000024985a0 x7 : 0000000000000075 x6 : 0000000000000000
[ 4452.883051] x5 : 0000000000000004 x4 : ffff8000028262a8 x3 : ffff800002826200
[ 4452.890212] x2 : 0000000000000001 x1 : 0000000000000000 x0 : 000905ba00470775
[ 4452.897374] Call trace:
[ 4452.899821] _raw_spin_lock_irq+0x28/0x80
[ 4452.903840] drm_atomic_helper_wait_for_flip_done+0x74/0xd8 [drm_kms_helper]
[ 4452.910927] vc4_atomic_commit_tail+0x154/0x928 [vc4]
[ 4452.916005] commit_tail+0xac/0x1a0 [drm_kms_helper]
[ 4452.920998] commit_work+0x1c/0x30 [drm_kms_helper]
[ 4452.925901] process_one_work+0x15c/0x3c0
[ 4452.929921] worker_thread+0x2e4/0x3f0
[ 4452.933679] kthread+0x120/0x130
[ 4452.936912] ret_from_fork+0x10/0x20
[ 4452.940497] Code: b9000841 d503201f 52800001 52800022 (88e17c02)
[ 4452.946610] ---[ end trace 0000000000000000 ]---
[ 4452.951239] note: kworker/u17:1[47] exited with irqs disabled
[ 4452.957097] note: kworker/u17:1[47] exited with preempt_count 1
[ 4465.738891] Unable to handle kernel paging request at virtual address 000905ba00470775
[ 4465.746858] Mem abort info:
[ 4465.749654] ESR = 0x0000000096000021
[ 4465.753410] EC = 0x25: DABT (current EL), IL = 32 bits
[ 4465.758737] SET = 0, FnV = 0
[ 4465.761795] EA = 0, S1PTW = 0
[ 4465.764939] FSC = 0x21: alignment fault
[ 4465.768957] Data abort info:
[ 4465.771839] ISV = 0, ISS = 0x00000021, ISS2 = 0x00000000
[ 4465.777340] CM = 0, WnR = 0, TnD = 0, TagAccess = 0
[ 4465.782406] GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
[ 4465.787733] [000905ba00470775] address between user and kernel address ranges
[ 4465.794894] Internal error: Oops: 0000000096000021 [#2] PREEMPT SMP
[ 4465.801184] Modules linked in: rfcomm snd_seq_dummy snd_hrtimer snd_seq snd_seq_device algif_hash algif_skcipher af_alg bnep binfmt_misc vc4 snd_soc_hdmi_codec drm_display_helper cec imx662 at24 drm_dma_helper brcmfmac_wcc joydev v4l2_cci regmap_i2c spidev brcmfmac hci_uart snd_soc_core aes_ce_blk rpi_hevc_dec btbcm aes_ce_cipher brcmutil ghash_ce gf128mul bluetooth pisp_be sha2_ce sha256_arm64 cfg80211 sha1_ce snd_compress rp1_cfe snd_pcm_dmaengine ecdh_generic v4l2_mem2mem ecc sha1_generic libaes snd_pcm v4l2_fwnode v4l2_async videobuf2_dma_contig v3d rfkill raspberrypi_hwmon snd_timer gpu_sched videobuf2_memops videobuf2_v4l2 drm_shmem_helper spi_bcm2835 snd videodev i2c_brcmstb gpio_keys drm_kms_helper videobuf2_common mc rp1_pio rp1_adc rp1_mailbox rp1_fw raspberrypi_gpiomem i2c_designware_platform i2c_designware_core nvmem_rmem uio_pdrv_genirq uio i2c_dev ledtrig_pattern drm drm_panel_orientation_quirks backlight fuse dm_mod ip_tables x_tables ipv6
[ 4465.886187] CPU: 0 UID: 0 PID: 2203 Comm: kworker/u17:2 Tainted: G D 6.12.41-v8-16k+ #2
[ 4465.895619] Tainted: [D]=DIE
[ 4465.898502] Hardware name: Raspberry Pi 5 Model B Rev 1.1 (DT)
[ 4465.904353] Workqueue: events_unbound commit_work [drm_kms_helper]
[ 4465.910579] pstate: 204000c9 (nzCv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 4465.917568] pc : _raw_spin_lock_irq+0x28/0x80
[ 4465.921941] lr : wait_for_completion_timeout+0x40/0x158
[ 4465.927182] sp : ffffc00082273c50
[ 4465.930501] x29: ffffc00082273c50 x28: ffff800003bc0cf0 x27: ffff800002df2580
[ 4465.937664] x26: ffff800100778080 x25: 0000000000000038 x24: ffff800002e01500
[ 4465.944826] x23: ffffd06faf2b23c8 x22: 0000000000000038 x21: 000905ba00470775
[ 4465.951988] x20: 000905ba0047076d x19: 00000000000009c4 x18: 0000000000000000
[ 4465.959149] x17: 0000000000000000 x16: ffffd06fcf1a1ee0 x15: 0000000000000000
[ 4465.966311] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
[ 4465.973472] x11: 00000000000000c0 x10: 0000000000001a40 x9 : ffffd06fcf1a1f20
[ 4465.980634] x8 : ffff800002f722a0 x7 : ffff8000034dc600 x6 : 0000000000000000
[ 4465.987795] x5 : 0000000000000004 x4 : ffff8000028262a8 x3 : ffff800002826200
[ 4465.994957] x2 : 0000000000000001 x1 : 0000000000000000 x0 : 000905ba00470775
[ 4466.002119] Call trace:
[ 4466.004566] _raw_spin_lock_irq+0x28/0x80
[ 4466.008586] drm_atomic_helper_wait_for_flip_done+0x74/0xd8 [drm_kms_helper]
[ 4466.015675] vc4_atomic_commit_tail+0x154/0x928 [vc4]
[ 4466.020754] commit_tail+0xac/0x1a0 [drm_kms_helper]
[ 4466.025745] commit_work+0x1c/0x30 [drm_kms_helper]
[ 4466.030648] process_one_work+0x15c/0x3c0
[ 4466.034669] worker_thread+0x2e4/0x3f0
[ 4466.038427] kthread+0x120/0x130
[ 4466.041659] ret_from_fork+0x10/0x20
[ 4466.045245] Code: b9000841 d503201f 52800001 52800022 (88e17c02)
[ 4466.051358] ---[ end trace 0000000000000000 ]---
[ 4466.055987] note: kworker/u17:2[2203] exited with irqs disabled
[ 4466.062068] note: kworker/u17:2[2203] exited with preempt_count 1
Steps to reproduce the behaviour
Steps to reproduce the issue:
- Install Raspberry Pi 5 kernel built from the kernel code base : https://github.com/raspberrypi/linux/tree/rpi-6.12.y (commit ac69f09)
- Connect 2 IMX462 cameras to CAM0 and CAM1 ports respectively.
- Set the dtoverlays in config.txt with the value imx462,cam0 and imx462,cam1
- Reboot the kit.
- IMX290 driver from upstream Raspberry Pi 5 kernel should probe the sensor.
- Open Terminal and start streaming using the command: libcamera-vid --mode 1920:1080:10:U --width 1920 --height 1080 -t 0 -f --camera --tuning-file=
- Within 30 minutes to 15 hrs, kernel crashes with a NULL pointer or paging request.
Device (s)
Raspberry Pi 5
System
Kernel: 6.12.41-v8-16k+
libcamera version: 0.5.0+rpt20250429-1
OS: Bookworm
Version in bootsplash logo: Release 5.6 - April 2025
cat /etc/debian_version: 12.10
Logs
dmesg3.txt.txt
dmesg4.txt.txt
dmesg1.txt.txt
dmesg2.txt.txt
Additional context
No response