Description
Fork is a foundational building block to support multiple-subqueries, RRF, and much more.
What is FORK?
Conceptually, fork is:
- a bifurcation of the stream, with all data going to each fork branch, followed by
- a merge of the branches, enhanced with a discriminator column
The name, fork, is somewhat inspired by unix fork, and other streamy frameworks, since the concept of forked execution is quite familiar. Other names considered and discounted are: union, merge, combine, tee, tpipe. While conceptually similar, the aforementioned names would likely lead to confusion with similar (but different) concepts in other languages, e.g. SQL union.
Example:
FROM test
| FORK
( WHERE content:"fox" )
( WHERE content:"dog" )
| SORT _fork
| KEEP _fork, id, content
Conceptual data flow:
Actual execution flow:
The planner and execution is free to reorganise things as long as it adheres to the conceptual flow of data.
Building upon the previous example, now with a common pre-filter:
FROM test
| WHERE id > 1 // common pre-filter
| FORK
( WHERE content:"fox" )
( WHERE content:"dog" )
| SORT _fork
| KEEP _fork, id, content
Where the FORK is “pushable”, then the common pre-filter and the WHERE of each fork branch is pushed down to be an effective subquery.
Where the FORK is not pushable, e.g. after a STATS, then the fork implementation will “fan-out” and merge within the compute engine. That is, the implementation will be more like the initial conceptual diagram above.
### Initial Restrictions
A number of initial restrictions have been put in place in order to make progress and unblock other development efforts dependent on Fork, e.g. RRF.
The restrictions are:
- First level data retrieval only - not yet general purpose bifurcation of the stream. This allows us to support multiple different subqueries. For bifurcation of the stream, then the planner will have to determine that the fork is actually being performed in second stage retrieval. This is a pragmatic limitation that we can lift later.
- All branches of the fork must return the same data scheme (same columns). This is a pragmatic limitation that we can lift later. For this reason, only WHERE, SORT, and LIMIT, are supported within fork subqueries.
- No fork within a fork. This is a pragmatic limitation that we can lift later.
- Lucene queries are independent - no point-in-time. We can add this later
- Fork branches are automatically named. We can provide the ability to name the branches later.
Development outline and evolution
We will lift all the restrictions as outlined above, but not all at once and not necessarily in the outlined order.
Since FORK is a significant feature, its development will be broken down over several other smaller PRs and issues. This section is intended to capture the current state and future plans as we progress towards a complete implementation. As such, consider this section "live", as new PRs and issues are filed they can be linked here.
- Initial FORK with restrictions #121950
- FORK: allow more commands in FORK branches #126553
- use union types to resolve schema conflicts between fork branches when possible
- Nested Fork commands
- Support FORK after FORK
Bugs:
These are all the known bugs with FORK:
- ES|QL: FORK memory management #130072
- FORK can add extra an extra warning header when implicit date nanos conversion fails for some rows (when the values indicate a pre 1970 time). This will likely be fixed when we improve the field caps resolution. This caused the following test failures:
- queries that are using the implicit date nanos conversion might return extra columns. Note this only happens when using union types with date nanos/datetime. The date nanos/datetime conversion is under snapshot, which makes this less urgent to fix. (see [ESQL] Ensure date/date_nanos implicit casting rule behind snapshot #130026)
The following query returns an extra$$nanos$converted_to$date_nanoscolumn:
FROM date_nanos, date_nanos_union_types
| EVAL a = nanos::date_nanos::datetime
| WHERE millis < "2023-10-23T13:00:00"
| LIMIT 1
| FORK (where true) (where true)
Follow ups from #121950:
- improve field resolution ES|QL: tests for FORK's evaluation of field names used in field_caps resolve calls #127208
- FORK and unmapped fields with INSIST (INSIST is still WIP so this is not urgent for now)
- fix CCS support ES|QL: Skip FORK tests for CCS for now #127309 ES|QL: Test FORK with multi cluster #128310
- Guides/more examples on when to use FORK