TinyPico D4 possible memory overrun #7288
Comments
|
I'm using the TinyPICO V2 |
|
Do you have other similar boards you could try this on to see if it is board-specific? |
|
The TinyPICO V2 is currently my only ESP32-D4 board. I don't actually know if there is another D4 board that supports circuitpython. I have run this same code on ESP32, ESP32-S2, ESP32-S3 and ESP32C3 boards running CP without this issue occurring. I believe the QTPY ESP32-Pico uses a similar microprocessor that I haven't tested yet, I'll go ahead and order one of those. If anyone knows of another D4 board, let me know :) |
|
Just as a swipe in the dark I decided to look at the boards/unexpectedmaker_tinypico/sdkconfig file and I set the CONFIG_SPIRAM_SIZE parameter first to 2097152, then 524288 and finally to 2048. Each time when the board started gc.mem_free() returned 4095360. I also tried using the sdkconfig file from the qtpy pico board but that build didn't startup on the tinyPICO. |
|
We retrieve espidf's own idea of the amount of psram present, ultimately via |
|
This is the first things that come through on the Thonny console: ets Jun 8 2016 00:22:57 rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT) Device is busy or does not respond. Your options:
Adafruit CircuitPython 8.0.0-beta.4-58-gc8390a791 on 2022-11-30; TinyPICO with ESP32-PICO-D4 Auto-reload is on. Simply save files over USB to run them or enter REPL to disable. Code done running. Press any key to enter the REPL. Use CTRL-D to reload. |
|
Interesting side note, when I started customizing the sdkconfig file the CP flash filesystem was initialized so I recopied my code down to it. Before getting the boot messages I posted in the previous comment, I restored the Adafruit build from circuitpython.org and the flash filesystem was also restored to the same state it was in prior to my loading the custom images. That probably makes perfect sense to someone that understands how the flash is handled but just in case I thought I'd mention it.... |
|
I don't know if this will help, but I commented out the CONFIG_ESP32_SPIRAM_SUPPORT=y line in sdkconfig and added CIRCUITPY_ESP32_CAMERA = 0 to the mpconfigboard.mk file and gc.mem_free() now reports 106192 but the PyDOS code now seems to run without the variables being clobbered. |
|
Sorry folks, been AWOL with production line issues for the last few weeks :( I'll have another look over my TinyPICO board settings compared to MicroPython. Maybe I missed something, or set something wrong, though I'm pretty sure I checked them before doing the PR. I'll also look into how MP includes the Flash/PSRAM cache issue IDF fixes, as they need to be included to have both Flash and PSRAM working together, and PSRAM to work properly. I'll get onto this as soon as I can @RetiredWizard |
|
My QT Py ESP32 Pico was delivered today. It does not seem to have this same memory issue. |
That's using a different ESP32 chip and different PSRAM. So different settings and it doesn't require any cache fixes for IDF. |
|
If I build without the SPI RAM everything works great (except there's not much RAM 😁). The Micropython build also works fine so I've been comparing the sdkconfig files and one thing I've noticed is that CircuitPython configures the CONFIG_SPIRAM_SPEED_40M=y parameter regardless of whether I set CONFIG_SPIRAM_SPEED_80M=y in the boards/unexpectedmaker_tinypico/sdkconfig or not. The Micropython build looks to me like it's using the 80M setting so I'd like to try a build with that setting but haven't figured out how yet. EDIT: I managed to get the 80M setting but still no luck running the larger python scripts. |
|
I've hacked away at this on and off for a week or so without any breakthroughs. My current theory is that the CircuitPython variable structure (sorry, I know nothing about the internals) is somehow getting misplaced or corrupted (or perhaps a variable pointer) when the SPI RAM is used. Everything on the board seems to work fine with the SPI RAM, but when larger python programs are run, sometime during their execution when CP accesses a variable it actually retrieves a different variable's contents. If I build CP without the SPI RAM the same larger programs will run without any variable corruptions. |
|
I ran the test code from #2687 with SPI RAM enable and disabled and got the same results as described in the issue. It's not entirely clear to me from the open issue if this is really an issue or not but it did highlight the id() function for me and since both the working and not working memory models on this board behave the same way I'm thinking #2687 probably isn't a factor here. I did try using the id() function to get a better feel of what's going on. I ended up chopping my main application up until I got a crash that resulted in a python backtrace rather than freezing and then started inserting print statements. I suspect the freezes occurred when a function object was corrupted and python attempted to transfer program control to garbage memory locations. This code segement: def prDir(dirPath,swBits):
if swBits & (swAllB-int('010110',2)):
print("Illegal switch, Command Format: DIR[/p][/w][/s] [path][file]")
return
savDir = os.getcwd()
print('absPath:',id(absolutePath))
print('slh:',slh,id(slh),'dirPath:',dirPath,id(dirPath),'savDir:',savDir,id(savDir))
#fullPath = absolutePath(dirPath,savDir)
fullPath = None
print("fullPath:",fullPath,id(fullPath),"slh:",slh,id(slh),"dirPath:",dirPath,id(dirPath))prints: and then freezes apparently on the 'fullPath = None` statement. If I modify the print statement by removing the text literals: def prDir(dirPath,swBits):
if swBits & (swAllB-int('010110',2)):
print("Illegal switch, Command Format: DIR[/p][/w][/s] [path][file]")
return
savDir = os.getcwd()
print('absPath:',id(absolutePath))
print(slh,id(slh),dirPath,id(dirPath),savDir,id(savDir))
#fullPath = absolutePath(dirPath,savDir)
fullPath = None
print("fullPath:",fullPath,id(fullPath),"slh:",slh,id(slh),"dirPath:",dirPath,id(dirPath))
fullPath = (savDir+dirPath if dirPath[0] != slh else dirPath)The following output occurs: In both cases it appears to me that the program has reached a point where it can no longer add new variables to the variable structure. In the first example I created new literals in the print statement and after that it seems objects could no longer be properly identified. In the second case the same thing happens after the fullPath variable is created. Since all this weirdness only happens when using the SPI RAM I'm thinking it must have something to do with where the data structures are being located (possibly based on the available RAM???). |
|
With Scott's recent memory work on 9.x, I decided to take another look at this since I believe it's some sort of memory management bug, however the issue still seems to occur using the latest build from main. I was able to build a debug build by disabling WiFi which actually ended up creating a much smaller image (about 500K free flash, rather than 50K free flash for the standard build). That build still showed the issue but it took much more activity (memory use) before the problem occurred. I was able to grab the ESP Debug output immediately after power up as well: I'll test this again once 9.x is released. |
|
@RetiredWizard Curious I just had another look at the 8.2 branch compared to MP and the only thing different that stands out (but that I don't think it the issue) is CP is not setting That setting will try to put smaller allocations in internal SRAM instead of PSRAM - which might cause less fragmentation in PSRAM. You could try building with that added to the TinyPICO settings. Also, MP 1.21 introduced a new way to deal with allocating and tracking free space in PSRAM (maybe memory in general) and I believe the plan is to merge MP 1.21 into 9.x now that 1.20 has been merged. But the fact that you previously tried MP way back beef these new memory management parts were added, and didn't see the issue in MP, only in CP doesn't give me much confidence that the new 1.21 changes will help. It's likely just a missing or different |
|
Thanks @UnexpectedMaker! I threw I also just loaded the latest v1.22 preview Micropython again and it still seems to work fine. |
|
Yeah, I didn't think it would help - but bummer anyway, sorry. |
|
I believe this has been fixed by #8553 |
CircuitPython version
Code/REPL
Behavior
Starting Py-DOS...
Warning wild card length set to: 11
(4071120)/>dir
0 16
<> <>
Traceback (most recent call last):
File "", line 1, in
File "PyDOS.py", line 1208, in
File "PyDOS.py", line 573, in PyDOS
File "PyDOS.py", line 307, in prDir
TypeError: unsupported types for sub: '', 'int'
PyDOS.zip
Description
I believe the Unexpected Maker TinyPico D4 CircuitPython build has some sort of memory overrun problem.
I've only been able to really see the memory error on larger programs 1000+ lines of code. It seems that some of the program objects are being clobbered as the program runs. For example if I set a variable to a text string, later on in the program I get an error indicating "TypeError: can't convert 'dict' object to str implicitly" and if I print out the variable it's no longer the string variable but a dict from some other part of the program. If I set the variable to the string again just before the line indicated by the error the program continues but has a similar error a litter further down. Other large programs I test behave similarly. Moderatly smaller programs 400-500 lines seem to either run or simply hang (or hang the serial port).
All of the programs I'm seeing issues on are exactly the same versions that I've run on 10 or more different boards.
The errors also seem to be related to how many files are stored in the Flash filesystem??? If I put just a few files on the the flash either the errors occur at different locations or the program simply runs.
Since I"m guessing this is a memory error, I've tried a couple simple scripts to gobble up memory and see if I can reproduce the variable clobbering but so far they have only run the board out of memory,
I've spent several hours trying to get something to demonstrate the error that doesn't involve 1000 lines of code and I'll keep working on that front but I've attached a zipped copy of PyDOS.py that demonstrates the issue.
On line 192 at the start of the prDIR function I print out two variables, swBits and swAllB, between line 192 and 305 there is a single function definition only, and then on line 306 I print out swBits and swAllB again but their values have changed.
Additional information
I ran this quickly by @UnexpectedMaker on Discord but without an issue and some code that reproduced the issue he didn't think there was much that could be looked at.
The text was updated successfully, but these errors were encountered: