Tune auto-growing split heap

[RetiredWizard]

I may be jumping the gun on this a bit, but I've had a little time and started playing with the memory allocation values for the Espressif port and thought this would be a good place to follow the progress.

[RetiredWizard]

I've been focusing on running large python scripts that churn through lots of memory. Setting the memory allocation up so that it always grabbed 512 bytes (256 or less didn't appear to work at all), vastly increased the length of time my largest test programs would run. I'm thinking for the test category I'm looking at (which I know is not a common use case), a non increasing small allocation is the best edge case but based on the comments in py/gc.c -> gc_try_add_heap() will lead to fragmentation issues.

I've been printing out the memory map using gc_dump_alloc_table when an allocation fails and it looks to me like the reason the split heap doesn't perform as well as the earlier <9.x memory model is that the memory allocations are separated by chunks of memory that are not available to CircuitPython. Scott mentioned that CP allocations on the heap are intermingled with "IDF" allocations. I'm guessing the not available chunks are the IDF allocations?

Is there any place I can look to learn more about the IDF allocations?

[tannewt]

Is there any place I can look to learn more about the IDF allocations?

The IDF won't allocate with increasing addresses necessarily. Instead, it uses TLSF for the allocation which tries to do a good fit allocation. It is interesting to see the heap layout when the allocation fails and to also know what the IDF says is the largest free section.

Please post complete zips of code for your large project if you want me to look at them.

[RetiredWizard]

Thanks Scott!

I wasn't really thinking that you'd be looking specifically at my use case. I'm running the PyBasic version of Adventure from within PyDOS. There's also an interactive component as once everything loads and appears to run okay I can continue to stress the memory by playing through the game for a bit.

If you really want to do some testing with PyDOS/PyBasic let me know what board you'll be testing on and I'll zip up a set of files for you but I'm also fine with posting any progress I make here or if you'd prefer we can close this and use specific issues like the #8574 as they come up.