Frequent crash (SIGABRT) on high frame input

[mariotaku]

• I confirm that this is an issue rather than a question.

Bug report

I have created a video source by sending image frames via flatbuffers server. I noticed that there are frequent crash especially under high framerate.

Starting program: /usr/bin/hyperiond
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
[New Thread 0xb2d69350 (LWP 2491)]
[New Thread 0xb24a8350 (LWP 2492)]
2021-05-19T02:01:25.624 hyperiond DAEMON       : <INFO> CEC handler created
2021-05-19T02:01:25.631 hyperiond EFFECTFILES  : <INFO> 39 effects loaded from directory :/effects/
2021-05-19T02:01:25.634 hyperiond EFFECTFILES  : <INFO> 22 effect schemas loaded from directory :/effects/schema/
2021-05-19T02:01:25.634 hyperiond EFFECTFILES  : <INFO> 0 effects loaded from directory /home/pi/.hyperion/custom-effects
[New Thread 0xb1aff350 (LWP 2493)]
2021-05-19T02:01:25.681 hyperiond DAEMON       : <INFO> set screen capture device to 'dispmanx'
[New Thread 0xb10ff350 (LWP 2494)]
[New Thread 0xb06fe350 (LWP 2495)]
[New Thread 0xafefd350 (LWP 2496)]
[New Thread 0xaf6fc350 (LWP 2497)]
2021-05-19T02:01:25.684 hyperiond DISPMANXGRAB : <INFO> Display opened with resolution: 1920x1080
2021-05-19T02:01:25.685 hyperiond DAEMON       : <INFO> DISPMANX frame grabber created
2021-05-19T02:01:25.686 hyperiond V4L2:AUTO    : <INFO> Signal threshold set to: {12, 12, 12}
2021-05-19T02:01:25.686 hyperiond V4L2:AUTO    : <INFO> CEC detection is now disabled
2021-05-19T02:01:25.686 hyperiond V4L2:AUTO    : <INFO> Signal detection is now disabled
2021-05-19T02:01:25.686 hyperiond V4L2:AUTO    : <INFO> Signal detection area set to: 0.250000,0.250000 x 0.750000,0.750000
2021-05-19T02:01:25.687 hyperiond JSONSERVER   : <INFO> Started on port 19444
[New Thread 0xaeefb350 (LWP 2498)]
[New Thread 0xae6fa350 (LWP 2499)]
2021-05-19T02:01:25.695 hyperiond FLATBUFSERVE : <INFO> Started on port 19400
[New Thread 0xadcff350 (LWP 2500)]
2021-05-19T02:01:25.696 hyperiond PROTOSERVER  : <INFO> Started on port 19445
[New Thread 0xad4fe350 (LWP 2501)]
[New Thread 0xacaff350 (LWP 2502)]
[New Thread 0xac2fe350 (LWP 2503)]
2021-05-19T02:01:25.699 hyperiond FLATBUFCONN  : <INFO> Connecting to Hyperion: 127.0.0.1:19401
[New Thread 0xab8ff350 (LWP 2504)]
[New Thread 0xaaeff350 (LWP 2505)]
2021-05-19T02:01:25.741 hyperiond LEDDEVICE    : <INFO> Start LedDevice 'file'.
2021-05-19T02:01:25.742 hyperiond EFFECTENGINE : <INFO> Run effect "Rainbow swirl fast" on channel 0
[New Thread 0xaa6fe350 (LWP 2506)]
2021-05-19T02:01:25.751 hyperiond HYPERION     : <INFO> Initial foreground effect 'Rainbow swirl fast' started
2021-05-19T02:01:25.753 hyperiond HYPERION     : <INFO> Hyperion instance 'First LED Hardware instance' has been started
[New Thread 0xa9dc4350 (LWP 2507)]
2021-05-19T02:01:26.004 hyperiond WEBSERVER    : <INFO> Started on port 8090 name 'Hyperion Webserver'
[New Thread 0xa95c3350 (LWP 2508)]
2021-05-19T02:01:26.008 hyperiond WEBSERVER    : <INFO> Started on port 8092 name 'Hyperion Webserver'
[New Thread 0xa8dc2350 (LWP 2509)]
[New Thread 0xa83ff350 (LWP 2510)]
[New Thread 0xa7bfe350 (LWP 2511)]
[Thread 0xa83ff350 (LWP 2510) exited]
[New Thread 0xa83ff350 (LWP 2512)]
[Thread 0xa83ff350 (LWP 2512) exited]
[Thread 0xa7bfe350 (LWP 2511) exited]
QJson: Document too large to store in data structure 124 -1378877356 134217727
corrupted double-linked list

Thread 10 "FlatBufferServe" received signal SIGABRT, Aborted.
[Switching to Thread 0xae6fa350 (LWP 2499)]
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50      ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0xb5757230 in __GI_abort () at abort.c:79
#2  0xb57a751c in __libc_message (action=action@entry=do_abort, fmt=<optimized out>)
    at ../sysdeps/posix/libc_fatal.c:181
#3  0xb57ae044 in malloc_printerr (str=<optimized out>) at malloc.c:5341
#4  0xb57ae2fc in malloc_consolidate (av=av@entry=0xadd00010) at malloc.c:4488
#5  0xb57b1694 in _int_malloc (av=av@entry=0xadd00010, bytes=bytes@entry=16401) at malloc.c:3695
#6  0xb57b3318 in __GI___libc_malloc (bytes=16401) at malloc.c:3057
#7  0xb5be2d44 in QArrayData::allocate(unsigned int, unsigned int, unsigned int, QFlags<QArrayData::AllocationOption>) () from /usr/share/hyperion/bin/../lib/libQt5Core.so.5
#8  0xb5e09288 in QEventDispatcherGlib::unregisterSocketNotifier(QSocketNotifier*) ()
   from /usr/share/hyperion/bin/../lib/libQt5Core.so.5
#9  0xae6f99d8 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Steps to reproduce

1. Send image frames via flatbuffers server
2. Crash will occur, especially under high framerate

What is expected?

Hyperion handles high framerate properly without crashing

What is actually happening?

SIGABRT, see log above.

System

Hyperion Server:

• Build: (HEAD detached at 83338dc) (Paulchen Panther-ca50487e/83338dc3-1605733041)
• Build time: Nov 18 2020 21:34:12
• Git Remote: https://github.com/hyperion-project/hyperion.ng
• Version: 2.0.0-alpha.9
• UI Lang: auto (BrowserLang: en-US)
• UI Access: default
• Avail Capt: dispmanx,v4l2,framebuffer,qt
• Database: read/write

Hyperion Server OS:

• Distribution: Raspbian GNU/Linux 10 (buster)
• Architecture: arm
• CPU Model: ARMv7 Processor rev 3 (v7l)
• CPU Type: Raspberry Pi 4 Model B Rev 1.4
• CPU Revision: d03114
• CPU Hardware: BCM2711
• Kernel: linux (5.10.17-v7l+ (WS: 32))
• Qt Version: 5.7.1
• Python Version: 3.5.3
• Browser: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36

[Paulchen-Panther]

What does high framerate mean for you?

[mariotaku]

@Paulchen-Panther I tried to send 60 captured video frames per second, to the server. Also found this might be happened only when preview is on.

[tpmodding]

1080p with 60fps? What are you planning to do? 60fps is vay toooo much.... and a looooot to calculate for hyperion....you will get a big delay on your leds...

[mariotaku]

@tpmodding No, it's 108p, 192x108@60fps.

[Paulchen-Panther]

@mariotaku
When I send 60 fps with the external hyperion-v4l2 program at 640x480 resolution, nothing crashes. Maybe there is a problem with your program!?!
The error message:

QJson: Document too large to store in data structure 124 -1378877356 134217727
corrupted double-linked list

is very strange.

[mariotaku]

@Paulchen-Panther what about to preview (with video on) in the browser? This is indeed very strange.

[Paulchen-Panther]

If the frame size and frame rate are too high, the connection to Hyperion may be interrupted. I am not aware of Hyperion crashing under these circumstances. Maybe you can explain us the use of your external program and we can look for a solution together. 😃

[mariotaku]

This external program, like hyperion-framebuffer, is a custom video source. Another interesting thing is if you don't open preview screen, crash won't occur. I'll do more tests to find the root cause.

[DavidBuchanan314]

I haven't been able to narrow down on the issue much, but I'm pretty sure it's a heap corruption issue in hyperiond (likely UAF, or maybe a buffer overflow).

Amusingly, the crash never seems to happen when running hyperiond under valgrind...

[Lord-Grey]

Hi
as @mariotaku outlines, this happens with preview open.
We know that there is currently an issue with the custom WebSocket implementation under high load feeding the UI.
That is the same reason as for #1130.

Edit: @mariotaku Could you test with alpha10 again, please. We did a minor fix, but it might not cover everything.
If not, we might need to migrate to e.g. QtWebSockets

[mariotaku]

I believe this issue can still be reproduced in alpha10.

[tpmodding]

@mariotaku can you recheck it or can you provide the packages you use? thanks