A new home for pyarrow-stubs?

[yangdanny97]

The maintainer of pyarrow-stubs (a set of Python type stubs for arrow) has expressed interest in donating the stubs since they no longer have time to maintain them: zen-xu/pyarrow-stubs#186

I'm opening a discussion here to see if there's any interest in merging the stubs into the main arrow project. They may remain as stubs, or if there is interest, they can also be merged into the Python bindings as inline types.

I'm happy to help set up the appropriate CI to support future maintenance and make sure the types are accurate/up-to-date going forward.

I see an existing issue for this topic (#30426), but not a lot of active discussion, so I'm hoping to hear what the maintainers think.

[AlenkaF]

Thank you for opening up this discussion!

There is a general consensus on the need for type annotations in PyArrow. In my opinion, the main challenge to progress is the lack of agreement on the approach to take.

To add some context, here are a few key comments from the ongoing discussion in #32609:

• The latest summary from @jorisvandenbossche: [Python] Type checking support #32609 (comment)
• Insights from @paleolimbot regarding the nanoarrow experience: [Python] Type checking support #32609 (comment)
• Advice from the pyarrow-stubs author: [Python] Type checking support #32609 (comment)

[raulcd]

Thanks for opening the discussion @yangdanny97 and thanks for the links @AlenkaF , really useful to get the history around it!
I've closed:

• [Python] Type stubs #30426

As a duplicate of:

• [Python] Type checking support #32609

And added some links to try and keep things organized on a single place.

[yangdanny97]

Thanks for the context!

If there isn't a strong consensus on inline types, I'd probably recommend the types be kept as stubs (with some tools like stubtest to make sure they're complete and in-sync). The reason is that if you change your mind, it's easier to move from stubs to inline types, whereas removing inline types will cause a lot of churn and merge conflicts.

Hand-written stubs will be much better quality than completely auto-generated stubs, and to the best of my knowledge all existing stub-generation tools do not produce stubs that are good enough to be used out-of-the-box without manual tweaks. Almost all the stubs in typeshed are maintained by hand, with a few exceptions like protobuf which has its own stub generation script.

I can see generated stubs being used as a temporary supplement to fill any gaps that pyarrow-stubs has (according to the author it's relatively complete), but after being generated the stubs will need to be fixed and maintained by hand.