Issues using WarpStream/lz4 - Frame encoded lz4 is not supported by default

[anglinb]

Hey! Was trying to get this library up and running against WarpStream w/ lz4 compression and I found that it would fail to decompress b/c the lz4 was frame encoded. I got this error:

!!! error Error: the offset to copy is not contained in the decompressed buffer
    at Object.decompressSync (/Users/brian/superwall/platformic-kafka/src/protocol/compression.ts:141:14)
    at readRecordsBatch (/Users/brian/superwall/platformic-kafka/src/protocol/records.ts:279:30)
    at <anonymous> (/Users/brian/superwall/platformic-kafka/src/apis/consumer/fetch-v15.ts:186:33)
    at Reader.readNullableArray (/Users/brian/superwall/platformic-kafka/src/protocol/reader.ts:319:18)

I found this issue in lz4-napi which explained the issue was due to a frame encoding instead of a "normal" encoding. After some hacking was able to get it working in my fork. But unfortunately, the LZ4 library does not have sync variants of compressFrame and decompressFrame, so I had to use deasync to validate that this was actually the issue.

I have a PR open for sync variants in lz4-napi so hopefully that lands and then this could be enabled w/o my hack.🤞

In the short term, would love to hear about any ideas ya'll might have to fix this, thanks!

[ShogunPanda]

Hi!
Thanks for the thorough report.
My guts here says that we should probably add support to provide custom (de)compressSync functions so that you can eventually hook up anything you need.
This way, when lz4-napi is released with your changes you can opt-in and use them for your needs. How does it looks like?

[anglinb]

Hey @ShogunPanda! I was able to land the sync variants over in the lz4-napi repo, but I'm still waiting for a release. Once that's ready, should be a pretty easy fix to incorporate that here!