-
Notifications
You must be signed in to change notification settings - Fork 933
Comparing changes
Open a pull request
base repository: explodinggradients/ragas
base: v0.2.13
head repository: explodinggradients/ragas
compare: v0.2.14
- 19 commits
- 41 files changed
- 11 contributors
Commits on Feb 5, 2025
-
feat: add http request-response logging with env flag control (#1903)
Add raw request/response logging with RAGAS_ENABLE_HTTP_LOG env flag
Configuration menu - View commit details
-
Copy full SHA for a336eb0 - Browse repository at this point
Copy the full SHA a336eb0View commit details
Commits on Feb 8, 2025
-
Configuration menu - View commit details
-
Copy full SHA for 7c4997f - Browse repository at this point
Copy the full SHA 7c4997fView commit details
Commits on Feb 10, 2025
-
Configuration menu - View commit details
-
Copy full SHA for cb63a82 - Browse repository at this point
Copy the full SHA cb63a82View commit details
Commits on Feb 14, 2025
-
Fixed simple criteria metric (#1909)
The issue arises because `self.single_turn_prompt.instruction` and `self.multi_turn_prompt.instruction` were not being properly assigned during the initialization of `SimpleCriteriaScore()`.
Configuration menu - View commit details
-
Copy full SHA for 146d2e2 - Browse repository at this point
Copy the full SHA 146d2e2View commit details -
Configuration menu - View commit details
-
Copy full SHA for d96e167 - Browse repository at this point
Copy the full SHA d96e167View commit details -
fix SingleHopQuerySynthesizer::prepare_combinations() (#1921)
This bug probably raise a KeyError since `personas` is not set. This could be a bug due to an oversight. Related to #1917
Configuration menu - View commit details
-
Copy full SHA for 6ef4f9a - Browse repository at this point
Copy the full SHA 6ef4f9aView commit details -
docs: fix semantic similarity description (cross-encoder -> bi-encode…
…r) (#1910) This PR updates the documentation to correctly describe the Semantic similarity. ### Issue The documentation previously stated that a **cross-encoder** was used for computing the semantic similarity score. However, after reviewing the implementation, it is clear that the current approach follows a **bi-encoder** strategy: - The ground truth and response are encoded independently - Their embeddings are then compared using cosine similarity A cross-encoder would typically process both texts together in a single forward pass (e.g., concatenating them before encoding), which is not the case in the current implementation. ### Current Implementation For example, in the current implementation: ```python embedding_1 = np.array(await self.embeddings.embed_text(ground_truth)) embedding_2 = np.array(await self.embeddings.embed_text(answer)) # Normalization factors of the above embeddings norms_1 = np.linalg.norm(embedding_1, keepdims=True) norms_2 = np.linalg.norm(embedding_2, keepdims=True) embedding_1_normalized = embedding_1 / norms_1 embedding_2_normalized = embedding_2 / norms_2 similarity = embedding_1_normalized @ embedding_2_normalized.T score = similarity.flatten() ``` This code shows that the ground truth and response are encoded separately, and their similarity is computed using cosine similarity, which is characteristic of a **bi-encoder** approach. ### Fix The term "cross-encoder" has been corrected to "bi-encoder" in the documentation to ensure consistency with the actual implementation.
Configuration menu - View commit details
-
Copy full SHA for dcfd58b - Browse repository at this point
Copy the full SHA dcfd58bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 620c6b0 - Browse repository at this point
Copy the full SHA 620c6b0View commit details
Commits on Feb 18, 2025
-
nvidia end-2-end accuracy, relevance and groundedness metrics (#1913)
first iteration of Nvidia accuracy metric Added a simple implementation for single turn accuracy, context_relevance and answer_groundedness metrics --------- Co-authored-by: jjmachan <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for c9305dd - Browse repository at this point
Copy the full SHA c9305ddView commit details -
Enable Runtime Checks for ModeMetric Protocol (#1929)
Add @t.runtime_checkable to the ModeMetric protocol to allow runtime type checking using isinstance() and issubclass().
Configuration menu - View commit details
-
Copy full SHA for a94feb2 - Browse repository at this point
Copy the full SHA a94feb2View commit details
Commits on Feb 20, 2025
-
- r2r integration - fixed spelling mistake in swarm tutorial - added @t.runtime_checkable to metric mode --------- Co-authored-by: Jithin James <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 76e14b0 - Browse repository at this point
Copy the full SHA 76e14b0View commit details -
Haystack llm and embedding wrapper (#1901)
Co-authored-by: Jithin James <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 2bc29a2 - Browse repository at this point
Copy the full SHA 2bc29a2View commit details
Commits on Feb 24, 2025
-
Easy fix, Groundedness metric, 5 retries early break. (#1935)
Easy fix Groundedness metric, 5 retries early break. Added logger to the 3 nv_metrics retries. Fixed input context max lenght to 7k to avoid 8k break.
Configuration menu - View commit details
-
Copy full SHA for 48b82ab - Browse repository at this point
Copy the full SHA 48b82abView commit details
Commits on Mar 3, 2025
-
Configuration menu - View commit details
-
Copy full SHA for 0a32987 - Browse repository at this point
Copy the full SHA 0a32987View commit details -
unnecessary Noise Sensitivity name update (#1943)
- the current code was logging a name and mode of noise sensitivity metric instead of just the name
Configuration menu - View commit details
-
Copy full SHA for 88e5fd3 - Browse repository at this point
Copy the full SHA 88e5fd3View commit details -
When saving kg only save the node id in the relationships (#1926)
When saving the knowledge graph, all the fields of each node is no longer stored in the relationships. This can save a very large amount of hard disk space.
Configuration menu - View commit details
-
Copy full SHA for f5ea3d5 - Browse repository at this point
Copy the full SHA f5ea3d5View commit details -
fix: return valid JSON output schema (#1933)
Use json.dumps() on the dict output of Pydantics's BaseModel.model_json_schema() to ensure the string representation is valid JSON using double quotes instead of single quotes.
Configuration menu - View commit details
-
Copy full SHA for 776afaa - Browse repository at this point
Copy the full SHA 776afaaView commit details -
Update default.py - Missing
cosine_similarity
transform for docs wi……th Token Count 101–500 (#1934) The `default_ transforms` function defined at `src/ragas/testset/transforms/default.py` has a problem with handling transforms for documents with 101-500 tokens. The code divides the `transforms` configurations based on the document's token count. Several transforms are instantiated when the ["101-500" token count bins the first quartile (Q1](https://github.com/explodinggradients/ragas/blob/2bc29a2b8358ddb6b167fdf7ab0518ad9371463c/src/ragas/testset/transforms/default.py#L128), among them the `cosine_sim_builder`. While `cosine_sim_builder` is correctly **instantiated** (line 139), it's then **not included** in the list of transforms that are actually returned (line 153). It appears that `cosine_sim_builder` was likely unintentionally omitted from the returned transforms list. The intended behavior should probably mirror how `ner_overlap_sim` is handled (line 120), where `cosine_sim_builder` is instantiated and added to the returned list. The current code effectively instantiates `cosine_sim_builder` but then discards it. This omission might impact the number of relationships created in the knowledge graph.
Configuration menu - View commit details
-
Copy full SHA for 1d4b2ff - Browse repository at this point
Copy the full SHA 1d4b2ffView commit details
Commits on Mar 4, 2025
-
Ndarray dtype fix, improve error wording (#1924)
1. I have noticed that if `response.statements` is an empty list, then the output of `verify_claims` function is `array([], dtype=float64)`, which raises a type error when `~` operation is applied to `reference_response` variable. This occurs in a rare case when hypothesis_list is empty (when there are no claims in the response). 2. A small change is the choice of words in `LLMDidNotFinishException` --------- Co-authored-by: Mikhail Zybin <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 414a518 - Browse repository at this point
Copy the full SHA 414a518View commit details
This comparison is taking too long to generate.
Unfortunately it looks like we can’t render this comparison for you right now. It might be too big, or there might be something weird with your repository.
You can try running this command locally to see the comparison on your machine:
git diff v0.2.13...v0.2.14