[JS] Decoding Dictionary<Uint32, Utf8> incorrectly.

[metalmatze]

Hey everyone! 👋

We have a problem with decoding a Dictionary<Uint32, Utf8> in arrow-js.

We want this array:

[
	"gke-europe-west3-0-preemptible-t2d-st-ec27d3db-pwwz"
	"gke-europe-west3-0-preemptible-t2d-st-add19435-w74v"
	"gke-europe-west3-0-preemptible-t2d-st-717888db-nrfr"
]

but we get this:

[
	"gke-europe-west3-0-preemptible-t2d-st-ec27d3db-pwwz", 
	"gke-europe-west3-0-preemptible-t2d-st-ec27d3db-pwwz",
	null
]

It seems like we're reading the same string twice, and then in the second batch, we read NULL. Something along those lines.
Something with either decoding the dictionaries is broken, or reading the dictionary back into JS.

Reproducing

For us, there's a stream with incoming Arrow Flight Data chunks.
We have validated that the backend Go code is returning the correct payload.

The Flight Data is transformed into chunks using similar code to this:
https://github.com/lancedb/flight-sql-js-client/blob/decae7347da105554957edac5dc53a7f192c06dc/src/arrow_util.ts#L28-L53

The chunks are base64 encoded for reproducibility in the unit test.

import {tableFromIPC,StructRow} from "apache-arrow";

describe('Arrow Reader', () => {
    const chunksDistinct = [
        '3AAAABAAAAAAAAoADAAKAAkABAAKAAAAEAAAAAABBAAIAAgAAAAEAAgAAAAEAAAAAQAAABQAAAAQABQAEAAOAA8ABAAAAAgAEAAAABgAAAAMAAAAAAABDXAAAAABAAAAGAAAALD///8QABgAFAAOAA8ABAAQAAgAEAAAADwAAAAwAAAAAAABBRAAAAAwAAAACAAKAAAABAAIAAAADAAAAAAABgAIAAQABgAAACAAAAAAAAAABAAEAAQAAAAEAAAAbm9kZQAAAAATAAAAYXR0cmlidXRlc19yZXNvdXJjZQA=',
        'qAAAABAAAAAMABgAFgAVAAQACAAMAAAAHAAAAMAAAAAAAAAAAAAAAAACBAAIAAoAAAAEAAgAAAAQAAAAAAAKABgADAAIAAQACgAAACwAAAAQAAAAAQAAAAAAAAAAAAAAAQAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAwAAAAAAAAAAAAAAAQAAAAAAAABAAAAAAAAAAAgAAAAAAAAAgAAAAAAAAAAzAAAAAAAAAP8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAZ2tlLWV1cm9wZS13ZXN0My0wLXByZWVtcHRpYmxlLXQyZC1zdC1lYzI3ZDNkYi1wd3d6AAAAAAAAAAAAAAAAAA==',
        'qAAAABAAAAAMABoAGAAXAAQACAAMAAAAIAAAAMAAAAAAAAAAAAAAAAAAAAMEAAoAGAAMAAgABAAKAAAAPAAAABAAAAABAAAAAAAAAAAAAAACAAAAAQAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAwAAAAAAAAAAAAAAAQAAAAAAAABAAAAAAAAAAAEAAAAAAAAAgAAAAAAAAAAEAAAAAAAAAP8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==',
        'qAAAABAAAAAMABgAFgAVAAQACAAMAAAAHAAAAAABAAAAAAAAAAAAAAACBAAIAAoAAAAEAAgAAAAQAAAAAAAKABgADAAIAAQACgAAACwAAAAQAAAAAgAAAAAAAAAAAAAAAQAAAAIAAAAAAAAAAAAAAAAAAAAAAAAAAwAAAAAAAAAAAAAAAQAAAAAAAABAAAAAAAAAAAwAAAAAAAAAgAAAAAAAAABmAAAAAAAAAP8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMwAAAGYAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAZ2tlLWV1cm9wZS13ZXN0My0wLXByZWVtcHRpYmxlLXQyZC1zdC1hZGQxOTQzNS13NzR2Z2tlLWV1cm9wZS13ZXN0My0wLXByZWVtcHRpYmxlLXQyZC1zdC03MTc4ODhkYi1ucmZyAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=',
        'qAAAABAAAAAMABoAGAAXAAQACAAMAAAAIAAAAMAAAAAAAAAAAAAAAAAAAAMEAAoAGAAMAAgABAAKAAAAPAAAABAAAAACAAAAAAAAAAAAAAACAAAAAgAAAAAAAAAAAAAAAAAAAAIAAAAAAAAAAAAAAAAAAAAAAAAAAwAAAAAAAAAAAAAAAQAAAAAAAABAAAAAAAAAAAEAAAAAAAAAgAAAAAAAAAAIAAAAAAAAAP8AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAD/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA==',
    ];

    describe('decode SELECT DISTINCT', () => {
        const chunks: Buffer[] = [];
        const chunksString: string[] = [];
        chunksDistinct.forEach((b64) => {
            const decoded = Buffer.from(b64, 'base64');
            chunks.push(decoded);
            // for debugging
            chunksString.push(new TextDecoder().decode(decoded));
        });

        console.log(chunksString);

        // Using RecordBatchReader makes no difference
        // const reader = RecordBatchReader.from(chunks)
        // for (const batch of reader) {
        //     console.log(batch.length);
        //
        // }


        const table = tableFromIPC(chunks);

        it('should have the correct data', () => {
            expect(table.numRows).toBe(3);
            expect(table.numCols).toBe(1);
            expect(table.schema.fields[0].toString()).toBe('attributes_resource: Struct<{node:Dictionary<Uint32, Utf8>}>');

            const nodes = [];
            table.toArray().forEach((row: StructRow) => {
                nodes.push(row['attributes_resource']['node']);
            })

            expect(nodes).toContain("gke-europe-west3-0-preemptible-t2d-st-ec27d3db-pwwz",);
            expect(nodes).toContain("gke-europe-west3-0-preemptible-t2d-st-add19435-w74v",);
            expect(nodes).toContain("gke-europe-west3-0-preemptible-t2d-st-717888db-nrfr",);
        });
    });
});

Using my debugger, I can introspect these chunks.

The expected strings are inside these chunks.

Just a thought: Maybe we should ignore chunks at index 2 and 4?

The unit test fails:

Error: expect(received).toContain(expected) // indexOf

Expected value: "gke-europe-west3-0-preemptible-t2d-st-add19435-w74v"
Received array: ["gke-europe-west3-0-preemptible-t2d-st-ec27d3db-pwwz", "gke-europe-west3-0-preemptible-t2d-st-ec27d3db-pwwz", null]

    at Object.<anonymous> (arrow.test.ts:45:27)

I hope the code above is able to run on your machine.
Please let me know if you have follow-up questions!

Does arrow-js have a Slack or Discord? I wasn't able to find anything.

Thank you!

[metalmatze]

It turns out we're using DictionaryHandling::Resend in our backend.
Using DictionaryHandling::Hydrate, things are starting to get decoded just fine.

https://arrow.apache.org/rust/arrow_flight/encode/enum.DictionaryHandling.html#variants

[assignUser]

Ah I'm just the build guy, so I'm totally not in the arrow JS ecosystem but you tagged the right ones already 👍

And you can of course always open an issue/discussion/ml thread!

[trxcllnt]

Does arrow-js have a Slack or Discord? I wasn't able to find anything.

There is an arrow-js channel in the-asf.slack.com.

The rust flight doc says

The current implementation does not support “delta” dictionaries so a new dictionary batch will be sent each time the encoder sees a dictionary which is not pointer-equal to the previously observed dictionary for a given dict_id.

The JS implementation supports either replacing or appending (i.e. "delta") dictionaries. Perhaps the Rust implementation is replacing dictionaries its already sent with ones that have fewer elements, and that's why you're seeing a null value?

[westonpace]

Hmm, if I run the unit test as-is it passes for me. Did you perhaps update the test with the Resend version? If not, maybe it is an arrow-js version issue? I tested with 18.1.0.

I'm thinking about opening an issue asking for general Flight support (it seems like at least two companies are already working on a solution themselves)

That would definitely be nice to have. We're happy to donate anything in https://github.com/lancedb/flight-sql-js-client/tree/decae7347da105554957edac5dc53a7f192c06dc (or even the entire repo). I'm not sure I have developer availability at the moment to extend much (though happy to debug / help occasionally). The intent was to donate it eventually, as we filled out functionality (based on customer demand) but that could be a while. There is a lot missing (it pretty much only handles the query path) though that's all flight-sql and there is maybe not too much missing on the flight side.

[metalmatze]

Hey everybody,
Thank you for the lively discussion!
Upon reading "I run the unit test as-is, it passes for me," I double checked our dependencies, and it turns out we were running v12. 😑 😅

Updating to [email protected] has solved the above issues, and the tests are indeed passing for us now, too.

We would happily work with you, @westonpace, to get an upstream version of the Flight client. 👍

Thank you all for the help! Much appreciated!

[westonpace]

I'll watch #45943 (reply in thread)

Once things split we can take the discussion there (I can maybe try and make a PR to the repo)