Memory handling with the query (bigquery datasource)

[dmitry-mightydevops]

So I have the following query against bigquery DS:

SELECT
  event_name,
  event_timestamp,
  (
    SELECT value.string_value
    FROM UNNEST(event_params)
    WHERE key = "clone_id"
  ) AS clone_id,
  (
    SELECT value.string_value
    FROM UNNEST(event_params)
    WHERE key = "sync_to"
  ) AS sync_to,
  (
    SELECT value.string_value
    FROM UNNEST(event_params)
    WHERE key = "error_class"
  ) AS error_class,
  (
    SELECT value.string_value
    FROM UNNEST(event_params)
    WHERE key = "ga_session_id"
  ) AS ga_session_id
FROM
  `prod.analytics_151274232.events_*`
WHERE
  user_id IS NOT NULL
  AND _TABLE_SUFFIX >= '20250701'

And I'm getting OOMs events due in {pod="redash-adhocworker-6bc46f8db5-h9h6q"}
So the memory usage of the POD grew from 750M to 4G pretty quickly

How query results are processed and stored inside dedicated redash postgres DB?

• Are they stored in memory until the query is fully complete?
  • And only after that they will be inserted in the redash postgres DB?

redashprod=> select id, query, query_hash from queries where id=3;
 id |                        query                        |            query_hash
----+-----------------------------------------------------+----------------------------------
  3 | SELECT                                             +| 31091ec7165a0409f1b1bd13ac4b1581
    |   event_name,                                      +|
    |   event_timestamp,                                 +|
    |   (                                                +|
    |     SELECT value.string_value                      +|
    |     FROM UNNEST(event_params)                      +|
    |     WHERE key = "clone_id"                         +|
    |   ) AS clone_id,                                   +|
    |   (                                                +|
    |     SELECT value.string_value                      +|
    |     FROM UNNEST(event_params)                      +|
    |     WHERE key = "sync_to"                          +|
    |   ) AS sync_to,                                    +|
    |   (                                                +|
    |     SELECT value.string_value                      +|
    |     FROM UNNEST(event_params)                      +|
    |     WHERE key = "error_class"                      +|
    |   ) AS error_class,                                +|
    |   (                                                +|
    |     SELECT value.string_value                      +|
    |     FROM UNNEST(event_params)                      +|
    |     WHERE key = "ga_session_id"                    +|
    |   ) AS ga_session_id                               +|
    | FROM                                               +|
    |   `prod.analytics_151274232.events_*`+|
    | WHERE                                              +|
    |   user_id IS NOT NULL                              +|
    |   AND _TABLE_SUFFIX >= '20250701'                   |
(1 row)

redashprod=> \d query_results;
                                         Table "public.query_results"
     Column     |           Type           | Collation | Nullable |                  Default
----------------+--------------------------+-----------+----------+-------------------------------------------
 id             | integer                  |           | not null | nextval('query_results_id_seq'::regclass)
 org_id         | integer                  |           | not null |
 data_source_id | integer                  |           | not null |
 query_hash     | character varying(32)    |           | not null |
 query          | text                     |           | not null |
 data           | text                     |           | not null |
 runtime        | double precision         |           | not null |
 retrieved_at   | timestamp with time zone |           | not null |
Indexes:
    "query_results_pkey" PRIMARY KEY, btree (id)
    "ix_query_results_query_hash" btree (query_hash)
Foreign-key constraints:
    "query_results_data_source_id_fkey" FOREIGN KEY (data_source_id) REFERENCES data_sources(id)
    "query_results_org_id_fkey" FOREIGN KEY (org_id) REFERENCES organizations(id)
Referenced by:
    TABLE "queries" CONSTRAINT "queries_latest_query_data_id_fkey" FOREIGN KEY (latest_query_data_id) REFERENCES query_results(id)

redashprod=> SELECT pg_column_size(data) AS data_size_bytes
FROM query_results
WHERE query_hash = '31091ec7165a0409f1b1bd13ac4b1581';
 data_size_bytes
-----------------
          305228
          305032
          305657
(3 rows)

redashprod=> SELECT pg_size_pretty(pg_database_size(current_database())) AS size;
  size
--------
 116 MB
(1 row)