🚀 Feature: Enhancing Milvus VectorDB Instrumentation for Improved search Monitoring

[divyapathak24]

Which component is this feature for?

Milvus Instrumentation

🔖 Feature description

Description:

We propose enhancing the existing Milvus VectorDB search method instrumentation by introducing additional attributes for better monitoring and analysis. These changes aim to provide deeper insights into search performance, retrieved results, distances, the ranking of retrieved results, and other relevant key metrics that are crucial for end to end observability of a RAG agent.

Current State:

The current Milvus instrumentation for search does not include the following essential attributes:

• Output Fields: Tracks the specific fields returned in search results.
• Metric Type: Example: L2, Euclidean, Cosine.
• Radius
• Input Query Vector Dimension: Provides the dimension of the input query vector.
• Search Status: Indicates the success or failure of the search operation
• Search Duration: Monitors the time taken for each search operation (in milliseconds).
• Result Count: Reports the total number of results returned, this number can be less than or equal to the limit specified to the search.
• Result Distances: The distances returned by the search.
• Min, Max, Avg Distances: Min, max and average of the returned result distances.
• Result IDs: Lists the returned IDs of the retrieved results.
• Retrieved Results: Displays the IDs, Output Fields, Distances returned by the search.
• Support for Multi-vector searches: Tracks per-query attributes for multiple queries in a single search operation.

Why do we need these attributes:

• Adding these attributes are important as it provides insights into how the search results vary with different search metrics, radius, and other configurations. Additionally, attributes such as IDs of the returned results, Distances allow developers and end-users to understand how and why certain results are returned, ensuring that any issues or inconsistencies can be addressed efficiently.
• The pymilvus search method does support multi-vector searches, but the current instrumentation implementation does not account for tracking each query's individual results.

Before and After Comparison of Search Instrumentation:

Note: Highlighted boxes show the new proposed attributes which are not present currently:

Single-query Search:

Multi-Vector Search:

Track per-query attributes for multiple query vectors in a single search operation.

🎤 Why is this feature needed ?

This feature is needed to enhance the end-to-end observability of Milvus search operations of a RAG agent. By adding this feature, we aim to provide deeper insights into search performance and monitoring.

✌️ How do you aim to achieve this?

1. Track the attributes: https://github.com/traceloop/openllmetry/blob/main/packages/opentelemetry-instrumentation-milvus/opentelemetry/instrumentation/milvus/wrapper.py

• Add new attributes such as Output Fields, Metric Type, Radius, Input Query Vector Dimension to the existing function:
  

• A new function should be added to handle processing of search results and introduce the required events and attributes (such as Result Count, Result Distances, Retrieved Results, Multi-vector searches)
  

2. Incorporate new attributes: packages/opentelemetry-semantic-conventions-ai/opentelemetry/semconv_ai/init.py

3. Testing and Validation: Verifying that the new attributes function properly with various parameter configurations in the search operation, and ensuring the attributes are correctly added to the spans through unit tests.

🔄️ Additional Information

Our team at IBM would like to contribute to the creation of this feature.

👀 Have you spent some time to check if this feature request has been raised before?

• I checked and didn't find similar issue

Are you willing to submit PR?

Yes I am willing to submit a PR!

[nirga]

@divyapathak24 that would be amazing! Let me know if we can help