[C++] Footer dumps for IPC Feather/ Parquet

[jayspomodoro]

I have observed in many discussions the inability to read feather/parquet files unless they are complete. Or to be more precise when the file closes, no further writing is possible. This is due to the fact that the reader requires the footer containing metadata to decipher the contents.

Some context - my own requirement for this feature was to continuously store the new data from some process (say a time-series data) in a memory efficient plus reader friendly manner, which can be simultaneously read while being written. Row-based storing (like csv and avro) do not really help a lot - I am talking about huge files here. Column-based storage has the potential for micro-batch processing (not exactly streaming) but has the problem of not being able to read at the same time.

My solution - at least a part of it - has been to make changes in exposed API to enable dumping of footer in a different file. Wanted to be a part of the ongoing development by sharing this knowledge. Let me know if this will be useful to anyone!

[amoeba]

Hi @jayspomodoro, that sounds like an interesting approach. Do you have a demo you could share here? It would help make it more clear what you're end-to-end workflow is and might save others time evaluating.

[SChakravorti21]

I would also be curious about the end to end workflow! It seems difficult to ensure that readers aren't affected by "dirty writes". I suppose if the filesystem allows swapping the footer file atomically, it would be possible for old readers to be unaffected while new readers pick up new data. But then how do old readers find out that there is new data? Or do they not care? Interesting problem!

Edit: come to think of it, this seems a lot like the problem Iceberg solves 🤔

[westonpace]

I agree some kind of atomic swap would be required. This would require the footer to be in a separate location as @jayspomodoro mentioned.

I suppose one way to think of it is that there is no reason an IPC file couldn't have multiple footers (all stored in various locations) to support reading the file at various points in its creation.

Edit: come to think of it, this seems a lot like the problem Iceberg solves 🤔

Yes, this does sound like creating a mini-table format, although one based on appends instead of multiple files. Note that S3 express and Azure blob storage now both support appends so there could even be some cloud storage solutions where this is useful.

Databases that do a lot of ingestion often use some kind of append based WAL. I think they are often built with something like Avro today however. I agree with @amoeba that having a specific use case in mind might be helpful.

[jayspomodoro]

The end to end workflow that I am using currently:
Think of three files -
update_file - always readable
main_file - file in which recordbatches get appended
footer_file - contains the footer data
At any moment I have to dump new data (recordbatch), I write it in the main_file as usual. I dump the new footer in the footer_file, overwriting the entire contents. I maintain the offset of the main_file before this update. I seek to this offset in the update_file, overwrite the original footer metadata with the new recordbatch just by copying it, and then copy the new footer on top of it. This maintains the structure of the feather/parquet at the end of these steps. Of course I had to maintain flocks for synchronous read and writes.

[jayspomodoro]

But I wanted to expand more on the reader side of things as well. Since before completion the files only get appended by new recordbatches, giving access to footer at any moment will give us power to read the file till a certain point (works even if the footer file is stale and also at the same moment when the file is being written on). So I was thinking in terms of giving reader two files to read - the main_file and the footer_file - through which it will be able to decipher the contents upto the mentioned recordbatches in the footer. Synchronous update of footer can simply be done by atomic rename operation.

[kou]

Can we use the Apache Arrow IPC Streaming Format https://arrow.apache.org/docs/format/Columnar.html#ipc-streaming-format instead of Apache Arrow IPC File Format https://arrow.apache.org/docs/format/Columnar.html#ipc-file-format / Apache Parquet for the use case?

[jayspomodoro]

This works great in many cases but the reader has to stream the recordbatches from beginning every time - the random access of recordbatches was critical in my use case otherwise it slows things down.

[felipecrv]

Start scanning from the byte right after the last byte you've read and you will be able to read the newly added record batch.