Running into `Error: doXHR failed (bug)!`

[sabberworm]

Amazing project! I had to try it out with a 15 GiB database of my own (sorry, I couldn’t resist testing the limits).

I started with simple queries and gradually made them more complex. This worked fine. But when I tried running a complex query immediately without having any other data loaded first, I would run into the error doXHR failed (bug)! (after a minute or so of loading gradually increasing chunks up to the configured max chunk size of 5 MiB).

I have built a sample Observable Notebook that exhibits the problem: https://observablehq.com/@sabberworm/imdb.

The “Aggregate all years” query always runs into this error (though I have seen it running fine when starting from a simpler query, see above).

I assume the other error that pops up (“database disk image is malformed”) is a consecutive error stemming from the missing data blocks.

[phiresky]

sorry, I couldn’t resist testing the limits

No need to apologize, it's always great to find out what the limits are!

I think i found the issue, try 0.8.9 and let me know, looks like I can't access the database if I fork your notebook.

Also, for such large queries we'd probably really need to add cache eviction to prevent the whole database from being in RAM at some point.

[phiresky]

Your queries also seem to be doing a lot of requests, it just keeps going... I can't see which queries are causing which reads since it looks like they all start at the same time. Is it just the last one?

I've added a bit of general information about reducing data fetching to the readme. Let me know if I can help you more specifically.

[sabberworm]

Thanks for the fix, 0.8.9 seems to work.

Don’t worry, I don’t plan to use do complete table scan queries in the future, I just wanted to see how far I could push it. Cache eviction would be cool but it’s not a must-have for me personally.

Also, I loosened the CORS restrictions on my server, you should now be able to fork the notebook and still have access.

[GibreelAbdullah]

On the latest code, still getting this error.
DB size is around 500 mb, it is divided into 49 chunks. When I queried the main unpartitioned DB is worked fine.

[GibreelAbdullah]

Added the database in github
ConfigUrl - https://raw.githubusercontent.com/GibreelAbdullah/hadith-search-api/master/db/config.json
Repo with DB - https://github.com/GibreelAbdullah/hadith-search-api/tree/master/db

index.ts file -

import { createDbWorker } from "sql.js-httpvfs";

const workerUrl = new URL(
  "sql.js-httpvfs/dist/sqlite.worker.js",
  import.meta.url
);
const wasmUrl = new URL("sql.js-httpvfs/dist/sql-wasm.wasm", import.meta.url);

async function load() {
  const worker = await createDbWorker(
    [
      {
        from: "jsonconfig",
        configUrl: "https://raw.githubusercontent.com/GibreelAbdullah/hadith-search-api/master/db/config.json"
      }
    ],
    workerUrl.toString(),
    wasmUrl.toString()
  );

  const result = await worker.db.query(`select * from hadith where shortname = 'abudawud' and hadithnumber = 1 and language = 'eng'`);

  document.body.textContent = JSON.stringify(result);
}

load();

[GibreelAbdullah]

Noticed that it fails if it has to make a lot of requests (if I don't use index), but works if the number of requests are less.