ctypes infinite pointer cache

[earonesty]

Bug(ish?) report

The following function has a cache. If you are using a factory to call ctypes.POINTER in a loop, the memory usage is unbounded and unable to be reclaimed.

https://docs.python.org/3/library/ctypes.html#ctypes.POINTER

The documentation should mention that this is unbounded and should not be called in a loop, or the cache should be changed to a configurable, bounded LRU cache.

Example of a variable length type factory used by windows, and a bad func that cannot be called in a loop:

def shitemid_factory(size: int) -> Type[ctypes.Structure]:
    class SHITEMID_Var(ctypes.Structure):
        _fields_ = (
            ("cb", USHORT),
            ("abID", BYTE * size),
        )

    return SHITEMID_Var

def bad_func():
    SHITEMID_Var = shitemid_factory(sz - ctypes.sizeof(USHORT))
    item_var = ctypes.cast(item_ptr, ctypes.POINTER(SHITEMID_Var))

Linked PRs

• gh-100926: Move ctype's pointers cache to StgInfo #131282

[mara004]

We've run into a similar issue with pypdfium2: pypdfium2-team/pypdfium2#346
An unlimited cache for pointer types seems problematic. Also, the docs aren't very clear about the peril of using POINTER() to get the data of variable-size arrays, or what alternatives there are.

[picnixz]

cc @encukou

[encukou]

I'm aware of the issue, can bump it up my list of priorities. Still not sure I'll be able to get a fix in 3.14.
If anyone wants to take a stab at it before I get to it, I can review/discuss.

IMO, the POINTER should be cached in stginfo, rather than a global cache, so when a type goes away so does the pointer to it (there'd probably be a ref cycle for the GC to resolve, but that's better than an unbounded cache).
I've only peeked down this rabbit hole enough to see that it's not straightforward, but should be solvable.

[mara004]

Thanks.
In the meantime, is (type * size).from_address(addressof(first_ptr.contents)) OK to work around this issue and replace our previous use of cast(first_ptr, POINTER(type * size)).contents ?

[sergey-miryanov]

I'm aware of the issue, can bump it up my list of priorities. Still not sure I'll be able to get a fix in 3.14. If anyone wants to take a stab at it before I get to it, I can review/discuss.

I would like to try, if you don't mind.

[sergey-miryanov]

@encukou I have made a PR, please take a look.

[encukou]

Thanks @earonesty for the report, @sergey-miryanov for the PR, @neonene for input & reviews, @mara004 and @picnixz for pings!

@junkmd, you'll want something like this to avoid deprecation warnings:

-        from ctypes import _pointer_type_cache  # type: ignore
-        _pointer_type_cache[self] = p
+        if sys.version_info >= (3, 14):
+            self.__pointer_type__ = p
+        else:
+            from ctypes import _pointer_type_cache  # type: ignore
+            _pointer_type_cache[self] = p