Merge snapshots into 1 under transaction of multiple operations

[jim-ngoo]

Question

For my use case, I have a daily cron job that batch process and append data but I only want a single snapshot record after the whole process. I tried to do them under a single transaction but still there are multiple snapshots created. Can pyiceberg provide the options such that only 1 snapshot will be generated under a transaction?

[ForeverAngry]

Question

For my use case, I have a daily cron job that batch process and append data but I only want a single snapshot record after the whole process. I tried to do them under a single transaction but still there are multiple snapshots created. Can pyiceberg provide the options such that only 1 snapshot will be generated under a transaction?

Can you share what function you are using to do the append?

[jim-ngoo]

Question

For my use case, I have a daily cron job that batch process and append data but I only want a single snapshot record after the whole process. I tried to do them under a single transaction but still there are multiple snapshots created. Can pyiceberg provide the options such that only 1 snapshot will be generated under a transaction?

Can you share what function you are using to do the append?

So here's a sample code:

from pyiceberg.catalog.memory import InMemoryCatalog
from pyiceberg.schema import Schema
from pyiceberg.types import LongType, NestedField
import pyarrow as pa

if __name__ == "__main__":
    catalog = InMemoryCatalog(name="test_catalog")
    catalog.create_namespace_if_not_exists("test_ns")
    table = catalog.create_table_if_not_exists(
        "test_ns.test_table",
        schema=Schema(NestedField(field_id=1, name="a", field_type=LongType())),
    )
    with table.transaction() as transaction:
        transaction.append(pa.Table.from_pylist([{"a": 1}]))
        transaction.append(pa.Table.from_pylist([{"a": 2}]))

    for entry in table.metadata.snapshot_log:
        print(entry)

And two snapshots are created - is there a way I can do batches of append, but there will be only one snapshot created at the end?

[ForeverAngry]

Hi! There is! However, some of the approaches come to mind might require some changes to how you are looking to add data. So i can better help, do you plan only only ever writing from a dataframe to your iceberg table?

[jim-ngoo]

Hi! There is! However, some of the approaches come to mind might require some changes to how you are looking to add data. So i can better help, do you plan only only ever writing from a dataframe to your iceberg table?

yes for my use case we manipulate data using polars dataframe/ lazyframe, and then append into the iceberg table.

[jayceslesar]

#2152 related