large data joins using collapse?

[SebKrantz]

Hi, I just wanted to ask why you took collapse out of the 50Gb join benchmark, given that it had initially successfully completed it? collapse has a single-threaded implementation of joins which is very memory efficient. It should be able to complete the task.

[Tmonster]

Oops, sorry about that. I will test it again

[SebKrantz]

Please use v2.1.2 - there was some missing garbage collection in earlier versions which may have led to memory overconsumption.

[Tmonster]

Latest results will use v2.1.2 You can see here.
https://github.com/duckdblabs/db-benchmark/pull/126/files

Also running the large join on collapse, hopefully can have the results up later today

[Tmonster]

Hi @SebKrantz,

So I've run the benchmark for collapse a couple of times on the large machine large dataset. Collapse @v2.1.2 seems to fail each time during ingestion. I know the benchmark page says "not tested" but I will update that soon. I have other priorities now, so can't spend too much time debugging the issue, but feel free to request another run or open a PR if you have more information as to what might be the problem