CIRCUITPY_PYSTACK_SIZE=4000 crashes ESP32S3 boards to safe mode

[RetiredWizard]

CircuitPython version

Adafruit CircuitPython 8.1.0-alpha.2-17-g00a03c323 on 2023-02-23; Adafruit Feather ESP32S3 4MB Flash 2MB PSRAM with ESP32S3
Adafruit CircuitPython 8.1.0-alpha.2-17-g00a03c323 on 2023-02-23; BPI-PicoW-S3 with ESP32S3

Code/REPL

No code.py file is executed.

Behavior

On esp32s3 boards setting CIRCUITPY_PYSTACK_SIZE=4000 causes a crash into safe mode.

Auto-reload is off.
Running in safe mode! Not running saved code.

You are in safe mode because:
CircuitPython core code crashed hard. Whoops!
Fault detected by hardware.
Please file an issue with your program at https://github.com/adafruit/circuitpython/issues.
Press reset to exit safe mode.

Press any key to enter the REPL. Use CTRL-D to reload.

A value of 3500 works fine.

Description

I've used values up to 7000 on a board with the raspberry pi RP2040 micro controller without a similar crash.

This may be a total red herring but building with micropy_stackless=1 prevents the crash, however setting CIRCUITPY_PYSTACK_SIZE > 3500 still doesn't seem to improve the stack limitations (EDIT: I'm not sure about this now, my stack depth test may have been faulty). I remember @bill88t mentioning that on some boards the parameter had an effective upper limit but I can't find the note right now. Perhaps the ESP32S3 was one of those boards.

Additional information

No response

[bill88t]

On S2 I could go up to a megabyte before I added checks. I assumed I hit some interger limit there.

The effective upper limit is 3700 for S2, 7000 for rp2.

Currently I have covid and I am pretty much unable to fix this. However can you please find the exact value it fails on?

The fix would pretty much be defining an upper limit for every mcu. If the value set is higher than that, lower it to the maximum.

[bill88t]

My stack depth test:

def test(no):
    print(no)
    try:
        test(no+1)
    except RuntimeError:
        print("Done")
test(0)

[RetiredWizard]

Rest up and get better, this functionality is awesome and is absolutely functional even with this issue. I'm not around much this weekend but I'll try and nail down the exact setting that crashes when I have a chance. Thanks SO much for this work 😁.

[RetiredWizard]

On the S3 it seems that a setting of 3504 or higher will lead to a safe mode hard crash if the stack depth test is run. 3503 or lower seems stable.

[tannewt]

One interest would be to print out the memory address that are allocated to see what they are near.

[RetiredWizard]

Built on an ESP32-S3-DevKitC-1-N8R2. The crash limit was actually a little higher but I haven't spent any time looking at that yet. I set the pystack size to 3600 and got the following trackeback:

***ERROR*** A stack overflow in task main has been detected.


Backtrace: 0x40378646:0x3fcf2da0 0x4038262d:0x3fcf2dc0 0x40385daa:0x3fcf2de0 0x403843b0:0x3fcf2e60 0x403826dc:0x3fcf2e90 0x403826d2:0x2d797063 |<-CORRUPTED




ELF file SHA256: 199f30faa96abdc6

CPU halted.

I tried running the decode_backtrace function but the board doesn't seem to have the subprocess library baked in. I also don't know if it would be of much value anyway since the traceback is flagged as corrupted.

[RetiredWizard]

One interest would be to print out the memory address that are allocated to see what they are near.

I'm not real fluent in c++ pointer addresses but I used the following to print what I'm hoping is the pystack memory address:
mp_printf(&mp_plat_print, "Pystack address %d",(mp_int_t) pystack); and got the following: Pystack address 1070215376

[tannewt]

I think the crash makes it pretty clear that we're not checking that we're within the C stack. There is stack checking code but its probably not checking the C stack and the pystack.

[bill88t]

I have finished reading the cp source for all stack code (pystack, cstack, heap).
The C stack is allocated before pystack. Starting from main.c:416, function stack_resize calls supervisor/shared/stack.c:75 which in turn calls :46 function allocate_stack.
This function is similar to the other stack allocations, using allocate_memory for a generic supervisor_allocation.
As a reminder allocate_memory is 'safe'. Populated in supervisor/shared/memory.c:233, it uses allocate_memory_node which keeps track of given records (using high/low_head) and should allocate unique, non-overlapping, blocks.
Since it all is supervisor_allocation's, assuming overflow to the C stack would mean, something has gone horribly wrong with the allocations.
Even so, the allocations always happen in the following order: cstack -> pystack -> heap. So if anything, it should be impossible for pystack to 'overflow' to cstack..
I can't use the pointer addresses since I don't have and can't get the board.
I will build and try to reproduce it on C3 and rp2.

[bill88t]

The following build is latest master, built manually cuz I just reinstalled my desktop and was testing it.

[bill88t@KeyFalse | espressif]> time beetle-cleanmake
[ ... ]
real    0m41,789s
user    4m51,394s
sys     1m18,136s

Quite a bit faster than my pi400.

Reproduced on C3, with a pystack of just 2000!!

Adafruit CircuitPython 8.1.0-beta.0-4-g8a1006999 on 2023-03-04; DFRobot Beetle ESP32-C3 with ESP32-C3FN4
>>> def test(no):
...     print(no)
...     try:
...         test(no+1)
...     except:
...         print("Done")
... 
>>> test(0)
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
Done
>>> 
Auto-reload is off.
Running in safe mode! Not running saved code.

You are in safe mode because:
CircuitPython core code crashed hard. Whoops!
Fault detected by hardware.
Please file an issue with your program at https://github.com/adafruit/circuitpython/issues.
Press reset to exit safe mode.

Press any key to enter the REPL. Use CTRL-D to reload.

[bill88t]

It boots and works till you access that memory. Stock pystack size works just fine, so as an emergency workaround I suggest CIRCUITPY_SETTABLE_PYSTACK is disabled for C3 and S3 till this issue is solved.