Support generating Substrait plans

[wjones127]

Description

From @roeap:

Generate query plans for operations requiring current table data as substrait (https://substrait.io) plans to integrate with different query backends

• (py)arrow has initial support for substrait - https://github.com/apache/arrow/blob/master/python/pyarrow/substrait.py
• Datafusion has experimental substrait support - https://github.com/datafusion-contrib/datafusion-substrait
• DuckDB seems to be working on it - https://github.com/duckdblabs/duckdb-substrait-demo

Use Case

Related Issue(s)

[wjones127]

I'm very interested in pursuing this as well, but I haven't looked into it enough to have a clear idea of what API we should offer.

One idea I had was a function DeltaTable.expand_substrait(read_rel: ReadRelation) -> SubstraitPlan, where you would pass a read relation with the projection and filter, and we would give the expanded plans with all the Delta prototol details embedded and files already pruned. But perhaps there's a more straight forward way.

[wjones127]

Thinking about this a little more, one goal might be to implement Substrait support that will allow us to write an ADBC driver for Delta Lake. That will allow us to support more engines / databases without having to do a bunch of one-off implementations. I'll try to draft a rough design of what we need for ADBC, and then use that to determine what we might need for Substrait integration. I also need to see what the state of these are in the Rust ecosystem.

[houqp]

I think both approaches (read relation & ADBC) are worth pursuing with their own tradeoffs and use-cases.