Series.diff returns Series['float'] even for nullable integers

[MarcoGorelli]

Describe the bug
A clear and concise description of what the bug is.

To Reproduce

1. Provide a minimal runnable pandas example that is not properly checked by the stubs.
2. Indicate which type checker you are using (mypy or pyright).
3. Show the error message received from that type checker while checking your example.

import pandas as pd


reveal_type(pd.Series([1,2,3], dtype='Int64').diff())
#  note: Revealed type is "pandas.core.series.Series[builtins.float]"

Please complete the following information:

• OS: [e.g. Windows, Linux, MacOS] linux
• OS Version [e.g. 22] 22.04
• python version 3.12.5
• version of type checker mypy 1.15
• version of installed pandas-stubs 2.2.3.241126

Additional context
noticed looking into #1108

[Dr-Irv]

This one could take some work to fix, because we don't differentiate between pd.Series([1,2,3], dtype='Int64') and pd.Series([1,2,3], dtype=int) in the stubs. pd.Series([1,2,3], dtype='Int64').diff() remains a series of integers (with pd.NA as the missing value), whereas pd.Series([1,2,3], dtype=int) becomes a Series[float] because np.nan becomes the missing value.

It might be possible to introduce a class hierarchy of base class of AllInt with subclasses BaseInt and NullableInt and then we replace Series[int] with Series[AllInt] everywhere, modify Series.__new__() to return either Series[BaseInt] or Series[NullableInt] based on dtype and then in Series.diff() have overloads to return different results based on whether you had Series[BaseInt] or Series[NullableInt].

This would be a big change - would have to modify a lot of PYI files and tests.

[cmp0xff]

I am replying here so that the idea can have a broader audience.

It seems to me that we can add a generic variable to Index and Series (and maybe even DataFrame) to indicate the type of value.

• By default, say Index([1]), Series([1]), .value gives np.ndarray. They are not nullable.
  • .array gives NumpyExtensionArray.
• We can also have Index([1], dtype="Int64"), Series([1], dtype="Int64"). .value gives IntegerArray. They are nullable.
  • .array gives IntegerArray.
• We can also have Index([1], dtype="int64[pyarrow]"), Series([1], dtype="int64[pyarrow]"). .value gives ArrowExtensionArray. They are nullable.
  • .array gives ArrowExtensionArray.
• We can also have Index([1], dtype="category"), Series([1], dtype="category"). .value gives Categorical. They are kind of nullable.
  • .array gives Categorical.

What do you think?

[Dr-Irv]

It seems to me that we can add a generic variable to Index and Series (and maybe even DataFrame) to indicate the type of value.

I think you mean the storage backend? i.e., we have numpy, nullable and pyarrow backends. So you want to add that to Index and Series ??

[cmp0xff]

I think you mean the storage backend? i.e., we have numpy, nullable and pyarrow backends. So you want to add that to Index and Series ??

I indeed have the storage backend in my mind, but I am not an expert of Pandas implementation and wanted to avoid the tech term. I believe using the storage backend can help us solving several issues, including

• distinguishing nullable and non-nullable dtypes (Series.diff returns Series['float'] even for nullable integers #1110, Nullable dtypes for Series.__new__ and astype #1395, etc)
• typing categorical Series and Indexes (How should we type Series with a categorical dtype? #1415)

It also preserved the current typing overloads, like arithmetic, because the storage backend is an additional piece of information, and we are not doing something breaking like Series[NullableInt], Series[NonNullableInt].

[Dr-Irv]

It also preserved the current typing overloads, like arithmetic, because the storage backend is an additional piece of information, and we are not doing something breaking like Series[NullableInt], Series[NonNullableInt].

So are you suggesting we'd have something like:

• `Series[int, "numpy"]
• `Series[int, "nullable"]
• `Series[int, "pyarrow"]

in the stubs? If so, then wouldn't we have to change anything that currently takes Series[int] as an argument to Series[int, Any] ?

[cmp0xff]

So are you suggesting we'd have something like:

* `Series[int, "numpy"]

* `Series[int, "nullable"]

* `Series[int, "pyarrow"]

in the stubs?

Yep

If so, then wouldn't we have to change anything that currently takes Series[int] as an argument to Series[int, Any] ?

Yep, those are the ideas. But we need to try and see if it works.

[cmp0xff]

I mean, there will be no Series[int, Any]. The default backend is np.ndarray, so we have Series[int, np.ndarray] as default.

[Dr-Irv]

I mean, there will be no Series[int, Any]. The default backend is np.ndarray, so we have Series[int, np.ndarray] as default.

Sure, but let's say there is a method that currently has Series[int] as an argument. If we are indifferent on the backend, wouldn't you have to change that argument type to be Series[int, Any] so that any backend would work?