__class_getitem__ Unexpectedly Falls Back to the Metaclass

[ericsnowcurrently]

Bug report

Bug description:

A major point of __class_getitem__ (PEP 560) is to avoid metaclasses. This implies that the metaclass should never be involved in the mechanism. However, currently (and since the feature landed) we actually do fall back to the metaclass:

(example)

class Spam:
    def __class_getitem__(cls, key):
        return f'spam got {key!r}'


class Meta(type):
    def __class_getitem__(cls, key):
        return f'meta got {key!r}'


class Eggs(metaclass=Meta):
    def __class_getitem__(cls, key):
        return f'eggs got {key!r}'


class Ham(metaclass=Meta):
    pass

print(Spam[10])
print(Eggs[10])
print(Ham[10])

Output:

spam got 10
eggs got 10
meta got 10

I was expecting the last one to raise TypeError: type 'object' is not subscriptable, since Ham does not implement __class_getitem__. The actual outcome is surprising because the metaclass can already define __getitem__. Falling back to the metaclass __class_getitem__ doesn't make much sense and can be confusing. The metaclass __class_getitem__ should only be used when subscripting the metaclass, not its instances.

The PEP doesn't really address the question of a metaclass that defines __class_getiem__ ¹ (nor does the documentation as far as I noticed). Overall, it seems like this was simply not noticed nor considered. It's certainly not an obvious case. Regardless, I think we should fix it.

The fix would involve skipping the metaclass part. That would be in the implementation for the subscript syntax (PyObject_GetItem() in Objects/abstract.c). There's a part where it specially handles the case where a class is being subscripted. In that case it looks up __class_getitem__ on the class. (See gh-4732.) However, currently it uses PyObject_GetOptionalAttr(), which involves descriptors and the metaclass (the object's type) and the type's __mro__. Again, the fix is to skip the metaclass part.

FWIW, __init_subclass__ is fairly similar, but it doesn't fall back to the metaclass. Instead, it effectively does getattr(super(cls), '__init_subclass__'). (See type_new_init_subclass() in Objects/typeobject.c.) We should probably do something similar in PyObject_GetItem().

CC @ilevkivskyi @gvanrossum

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Linked PRs

• gh-122634: Deprecate __class_getitem__ on metaclasses #122743

Footnotes

1. The PEP does say Note that this method is used as a fallback, so if a metaclass defines __getitem__, then that will have the priority. but that's specifically about falling back to meta.__getitem__. ↩

[sobolevn]

Quoting PEP:

Note that this method is used as a fallback, so if a metaclass defines __getitem__, then that will have the priority.

Moreover, https://docs.python.org/3/reference/datamodel.html#class-getitem-versus-getitem goes into details about Meta.__getitem__ and Class.__class_getitem__, but it never says that Meta.__class_getitem__ should not be called in this case.

So: nor docs, nor PEP say that Meta.__class_getitem__ must not be called.

Here's the example algorithm:

from inspect import isclass

def subscribe(obj, x):
    """Return the result of the expression 'obj[x]'"""

    class_of_obj = type(obj)

    # If the class of obj defines __getitem__,
    # call class_of_obj.__getitem__(obj, x)
    if hasattr(class_of_obj, '__getitem__'):
        return class_of_obj.__getitem__(obj, x)

    # Else, if obj is a class and defines __class_getitem__,
    # call obj.__class_getitem__(x)
    elif isclass(obj) and hasattr(obj, '__class_getitem__'):
        return obj.__class_getitem__(x)

    # Else, raise an exception
    else:
        raise TypeError(
            f"'{class_of_obj.__name__}' object is not subscriptable"
        )

And your example follows this logic:

>>> class Meta(type):
...     def __class_getitem__(cls, key):
...         return f'meta got {key!r}'
  
>>> class Ham(metaclass=Meta):
...     pass

>>> hasattr(Ham, '__class_getitem__')
True

I am pretty sure that since this is so old, many people rely on this behavior anyway.
So, my opinion is that we just accept and document / test this corner case.

[ilevkivskyi]

Yeah, this part was missed during the original discussion. After thinking about this a bit, I think the __init_subclass__()-like behavior would be more consistent, but as @sobolevn says it is a bit dangerous to change semantics this late.

[ethanfurman]

It feels like a bug to me. We can raise a DeprecationWarning if we need to, but I think we should fix it.

[ericsnowcurrently]

but as @sobolevn says it is a bit dangerous to change semantics this late.

The current behavior is actively problematic but also extremely unlikely to be triggered. Thus I think it's especially unlikely that there would be user-facing issues fixing this.

[AlexWaygood]

Moreover, docs.python.org/3/reference/datamodel.html#class-getitem-versus-getitem goes into details about Meta.__getitem__ and Class.__class_getitem__, but it never says that Meta.__class_getitem__ should not be called in this case.

FWIW, I wrote those docs a couple of years ago (after __class_getitem__ had already been around for a while), and I wrote them with the intention of describing the current behaviour rather than prescribing what the behaviour should be... though that doesn't really matter much now, I suppose. Now that it's been documented as such for a few years, users may be relying on the behaviour as-documented.

The current behavior is actively problematic but also extremely unlikely to be triggered.

I don't know that I've seen sufficient evidence to agree that this is "actively problematic" (how many people are really affected by this? what's the realistic use case where it crops up?)... but I agree it seems pretty surprising and buggy. And I agree that it's certainly very unlikely to be triggered. I guess I vote for fixing the bug!

[sobolevn]

Ok, let's fix it! There are two ways:

1. With a deprecation note when __class_getitem__ is called on a metaclass
2. Without a deprecation

And about backports:

1. We can backport this to 3.12.x and 3.13.1, I don't think that rc phase will allow such change
2. We can not backport it at all, only fix in new versions.

What do you think? Which one is better?

[JelleZijlstra]

No backporting; we shouldn't change core language semantics in a bugfix release. If anyone needs to branch on this behavior, they should be able to write if sys.version_info >= (3, 14), not if sys.version_info >= (3, 13, 1) or (3, 12, 3) <= sys.version_info < (3, 13): or something like that.

I would vote for raising a deprecation first.