Skip to content

[ES|QL] Dense vector embedding function proof of concept #130383

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 14 commits into
base: main
Choose a base branch
from

Conversation

afoucret
Copy link
Contributor

@afoucret afoucret commented Jul 1, 2025

This is a POC implementation for a function that allow to embed text as dense vector in ES|QL

TODO list:

  • Implement the InferenceOperator
  • Implement InferenceFunction
    • Basic function implementation (signature and parsing)
    • Add dimensions resolution for the Logical plam
  • Implement logical and physical plan
    • Logical plan
    • Physical plan
    • Logical to physical mapping
    • Logical plan analysis
    • Logical plan optimization (Already implemented for all InferencePlan instances)
    • Physical planning
  • Implement dense vector embedding function rewrite to a logical plan
  • Implement foldable evaluation
  • Implement inference resolution for function

Also missing (not in the scope of the POC):

  • Tests
    • Unit
      • Analyzer, Logical plan optimization
    • Java REST and CSV tests
  • Good documentation

Gaps:

  • DENSE_VECTOR datatype does not support multivalued fields.

@afoucret afoucret force-pushed the esql-dense-vector-embedding branch from f4be9cf to 3d075fd Compare July 1, 2025 11:52
Copy link
Contributor

github-actions bot commented Jul 1, 2025

🔍 Preview links for changed docs:

🔔 The preview site may take up to 3 minutes to finish building. These links will become live once it completes.

Copy link
Contributor

github-actions bot commented Jul 2, 2025

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants