|
| 1 | +[[semantic-search]] |
| 2 | +== Semantic search |
| 3 | + |
| 4 | +Semantic search is a search method that helps you find data based on the intent |
| 5 | +and contextual meaning of a search query, instead of a match on query terms |
| 6 | +(lexical search). |
| 7 | + |
| 8 | +{es} provides semantic search capabilities using {ml-docs}/ml-nlp.html[natural |
| 9 | +language processing (NLP)] and vector search. Deploying an NLP model to {es} |
| 10 | +enables it to extract text embeddings out of text. Embeddings are vectors that |
| 11 | +provide a numeric representation of a text. Pieces of content with similar |
| 12 | +meaning have similar representations. |
| 13 | + |
| 14 | +[[semantic-search-diagram]] |
| 15 | +.A simplified representation of encoding textual concepts as vectors |
| 16 | +image::images/search/vector-search-oversimplification.png[A simplified representation of encoding textual concepts as vectors,align="center"] |
| 17 | + |
| 18 | +At query time, {es} can use the same NLP model to convert a query into |
| 19 | +embeddings, enabling you to find documents with similar text embeddings. |
| 20 | + |
| 21 | +This guide shows you how to implement semantic search with {es}, from selecting |
| 22 | +an NLP model, to writing queries. |
| 23 | + |
| 24 | +[discrete] |
| 25 | +[[semantic-search-select-nlp-model]] |
| 26 | +=== Select an NLP model |
| 27 | + |
| 28 | +{es} offers the usage of a |
| 29 | +{ml-docs}/ml-nlp-model-ref.html#ml-nlp-model-ref-text-embedding[wide range of NLP models], |
| 30 | +including both dense and sparse vector models. Your choice of the language model |
| 31 | +is critical for implementing semantic search successfully. |
| 32 | + |
| 33 | +While it is possible to bring your own text embedding model, achieving good |
| 34 | +search results through model tuning is challenging. Selecting an appropriate |
| 35 | +model from our third-party model list is the first step. Training the model on |
| 36 | +your own data is essential to ensure better search results than using only BM25. |
| 37 | +However, the model training process requires a team of data scientists and ML |
| 38 | +experts, making it expensive and time-consuming. |
| 39 | + |
| 40 | +To address this issue, Elastic provides a pre-trained representational model |
| 41 | +called {ml-docs}/ml-nlp-elser.html[Elastic Learned Sparse EncodeR (ELSER)]. |
| 42 | +ELSER, currently available only for English, is an out-of-domain sparse vector |
| 43 | +model that does not require fine-tuning. This adaptability makes it suitable for |
| 44 | +various NLP use cases out of the box. Unless you have a team of ML specialists, |
| 45 | +it is highly recommended to use the ELSER model. |
| 46 | + |
| 47 | +In the case of sparse vector representation, the vectors mostly consist of zero |
| 48 | +values, with only a small subset containing non-zero values. This representation |
| 49 | +is commonly used for textual data. In the case of ELSER, each document in an |
| 50 | +index and the query text itself are represented by high-dimensional sparse |
| 51 | +vectors. Each non-zero element of the vector corresponds to a term in the model |
| 52 | +vocabulary. The ELSER vocabulary contains around 30000 terms, so the sparse |
| 53 | +vectors created by ELSER contain about 30000 values, the majority of which are |
| 54 | +zero. Effectively the ELSER model is replacing the terms in the original query |
| 55 | +with other terms that have been learnt to exist in the documents that best match |
| 56 | +the original search terms in a training dataset, and weights to control how |
| 57 | +important each is. |
| 58 | + |
| 59 | + |
| 60 | +[discrete] |
| 61 | +[[semantic-search-deploy-nlp-model]] |
| 62 | +=== Deploy the model |
| 63 | + |
| 64 | +After you decide which model you want to use for implementing semantic search, |
| 65 | +you need to deploy the model in {es}. |
| 66 | + |
| 67 | +include::{es-repo-dir}/tab-widgets/semantic-search/deploy-nlp-model-widget.asciidoc[] |
| 68 | + |
| 69 | +[discrete] |
| 70 | +[[semantic-search-field-mappings]] |
| 71 | +=== Map a field for the text embeddings |
| 72 | + |
| 73 | +Before you start using the deployed model to generate embeddings based on your |
| 74 | +input text, you need to prepare your index mapping first. The mapping of the |
| 75 | +index depends on the type of model. |
| 76 | + |
| 77 | +include::{es-repo-dir}/tab-widgets/semantic-search/field-mappings-widget.asciidoc[] |
| 78 | + |
| 79 | +[discrete] |
| 80 | +[[semantic-search-generate-embeddings]] |
| 81 | +=== Generate text embeddings |
| 82 | + |
| 83 | +Once you have created the mappings for the index, you can generate text |
| 84 | +embeddings from your input text. This can be done by using an |
| 85 | +<<ingest,ingest pipeline>> with an <<inference-processor,inference processor>>. |
| 86 | +The ingest pipeline processes the input data and indexes it into the destination |
| 87 | +index. At index time, the inference ingest processor uses the trained model to |
| 88 | +infer against the data ingested through the pipeline. After you created the |
| 89 | +ingest pipeline with the inference processor, you can ingest your data through |
| 90 | +it to generate the model output. |
| 91 | + |
| 92 | +include::{es-repo-dir}/tab-widgets/semantic-search/generate-embeddings-widget.asciidoc[] |
| 93 | + |
| 94 | +Now it is time to perform semantic search! |
| 95 | + |
| 96 | +[discrete] |
| 97 | +[[semantic-search-search]] |
| 98 | +=== Search the data |
| 99 | + |
| 100 | +Depending on the type of model you have deployed, you can query rank features |
| 101 | +with a text expansion query, or dense vectors with a kNN search. |
| 102 | + |
| 103 | +include::{es-repo-dir}/tab-widgets/semantic-search/search-widget.asciidoc[] |
| 104 | + |
| 105 | +[discrete] |
| 106 | +[[semantic-search-hybrid-search]] |
| 107 | +=== Beyond semantic search with hybrid search |
| 108 | + |
| 109 | +In some situations, lexical search may perform better than semantic search. For |
| 110 | +example, when searching for single words or IDs, like product numbers. |
| 111 | + |
| 112 | +Combining semantic and lexical search into one hybrid search request using |
| 113 | +<<rrf,reciprocal rank fusion>> provides the best of both worlds. Not only that, |
| 114 | +but hybrid search using reciprocal rank fusion {blog-ref}improving-information-retrieval-elastic-stack-hybrid[has been shown to perform better |
| 115 | +in general]. |
| 116 | + |
| 117 | +include::{es-repo-dir}/tab-widgets/semantic-search/hybrid-search-widget.asciidoc[] |
| 118 | + |
| 119 | +[discrete] |
| 120 | +[[semantic-search-read-more]] |
| 121 | +=== Read more |
| 122 | + |
| 123 | +* Tutorials: |
| 124 | +** <<semantic-search-elser,Semantic search with ELSER>> |
| 125 | +** {ml-docs}/ml-nlp-text-emb-vector-search-example.html[Semantic search with the msmarco-MiniLM-L-12-v3 sentence-transformer model] |
| 126 | +* Blogs: |
| 127 | +** {blog-ref}may-2023-launch-sparse-encoder-ai-model[Introducing Elastic Learned Sparse Encoder: Elastic's AI model for semantic search] |
| 128 | +** {blog-ref}lexical-ai-powered-search-elastic-vector-database[How to get the best of lexical and AI-powered search with Elastic's vector database] |
| 129 | +** Information retrieval blog series: |
| 130 | +*** {blog-ref}improving-information-retrieval-elastic-stack-search-relevance[Part 1: Steps to improve search relevance] |
| 131 | +*** {blog-ref}improving-information-retrieval-elastic-stack-benchmarking-passage-retrieval[Part 2: Benchmarking passage retrieval] |
| 132 | +*** {blog-ref}may-2023-launch-information-retrieval-elasticsearch-ai-model[Part 3: Introducing Elastic Learned Sparse Encoder, our new retrieval model] |
| 133 | +*** {blog-ref}improving-information-retrieval-elastic-stack-hybrid[Part 4: Hybrid retrieval] |
| 134 | + |
| 135 | +include::semantic-search-elser.asciidoc[] |
0 commit comments