DayTransform issues with pyarrow timestamp[ms]

[rongxinyu]

I have a table with timestamp[ms] and created a partition on this column with DayTransform. The pyarrow_transform raises due to epoch is timestamp[us]

[sungwy]

Hi @rongxinyu - thanks for reporting this issue!

I think this is the limitation of the current implementation that makes use of the pyarrow.compute.days_between function.

pyarrow.lib.ArrowNotImplementedError: Function 'days_between' has no kernel matching input types (timestamp[ms], timestamp[us])

Thankfully, there is an open PR that will use in the pyiceberg-core transforms that will handle this case which is waiting on the 0.5.0 pyiceberg-core release: #1592

[rongxinyu]

Got it, thank you!

[kevinjqliu]

And we're actively working to get 0.5.0 pyiceberg-core out :)
https://github.com/apache/iceberg-rust/milestone/4

almost there

[Fokko]

Since #1833 has been merged, I'll close this issue. If you want to give it a try already, you can give the nightly built a try: https://py.iceberg.apache.org/nightly-build/