Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: explodinggradients/ragas
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v0.2.13
Choose a base ref
...
head repository: explodinggradients/ragas
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: v0.2.14
Choose a head ref
  • 19 commits
  • 41 files changed
  • 11 contributors

Commits on Feb 5, 2025

  1. feat: add http request-response logging with env flag control (#1903)

    Add raw request/response logging with RAGAS_ENABLE_HTTP_LOG env flag
    ganeshrvel authored Feb 5, 2025
    Configuration menu
    Copy the full SHA
    a336eb0 View commit details
    Browse the repository at this point in the history

Commits on Feb 8, 2025

  1. Configuration menu
    Copy the full SHA
    7c4997f View commit details
    Browse the repository at this point in the history

Commits on Feb 10, 2025

  1. Configuration menu
    Copy the full SHA
    cb63a82 View commit details
    Browse the repository at this point in the history

Commits on Feb 14, 2025

  1. Fixed simple criteria metric (#1909)

    The issue arises because `self.single_turn_prompt.instruction` and
    `self.multi_turn_prompt.instruction` were not being properly assigned
    during the initialization of `SimpleCriteriaScore()`.
    sahusiddharth authored Feb 14, 2025
    Configuration menu
    Copy the full SHA
    146d2e2 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    d96e167 View commit details
    Browse the repository at this point in the history
  3. fix SingleHopQuerySynthesizer::prepare_combinations() (#1921)

    This bug probably raise a KeyError since `personas` is not set.
    
    This could be a bug due to an oversight.
    
    Related to #1917
    lryan599 authored Feb 14, 2025
    Configuration menu
    Copy the full SHA
    6ef4f9a View commit details
    Browse the repository at this point in the history
  4. docs: fix semantic similarity description (cross-encoder -> bi-encode…

    …r) (#1910)
    
    This PR updates the documentation to correctly describe the Semantic
    similarity.
    
    ### Issue
    The documentation previously stated that a **cross-encoder** was used
    for computing the semantic similarity score. However, after reviewing
    the implementation, it is clear that the current approach follows a
    **bi-encoder** strategy:
    
    - The ground truth and response are encoded independently
    - Their embeddings are then compared using cosine similarity
    
    A cross-encoder would typically process both texts together in a single
    forward pass (e.g., concatenating them before encoding), which is not
    the case in the current implementation.
    
    ### Current Implementation
    For example, in the current implementation:
    
    ```python
    embedding_1 = np.array(await self.embeddings.embed_text(ground_truth))
    embedding_2 = np.array(await self.embeddings.embed_text(answer))
    # Normalization factors of the above embeddings
    norms_1 = np.linalg.norm(embedding_1, keepdims=True)
    norms_2 = np.linalg.norm(embedding_2, keepdims=True)
    embedding_1_normalized = embedding_1 / norms_1
    embedding_2_normalized = embedding_2 / norms_2
    similarity = embedding_1_normalized @ embedding_2_normalized.T
    score = similarity.flatten()
    ```
    
    This code shows that the ground truth and response are encoded
    separately, and their similarity is computed using cosine similarity,
    which is characteristic of a **bi-encoder** approach.
    
    ### Fix
    The term "cross-encoder" has been corrected to "bi-encoder" in the
    documentation to ensure consistency with the actual implementation.
    Ayaka-mogumogu authored Feb 14, 2025
    Configuration menu
    Copy the full SHA
    dcfd58b View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    620c6b0 View commit details
    Browse the repository at this point in the history

Commits on Feb 18, 2025

  1. nvidia end-2-end accuracy, relevance and groundedness metrics (#1913)

    first iteration of Nvidia accuracy metric
    Added a simple implementation for single turn accuracy,
    context_relevance and answer_groundedness metrics
    
    ---------
    
    Co-authored-by: jjmachan <[email protected]>
    titericz and jjmachan authored Feb 18, 2025
    Configuration menu
    Copy the full SHA
    c9305dd View commit details
    Browse the repository at this point in the history
  2. Enable Runtime Checks for ModeMetric Protocol (#1929)

    Add @t.runtime_checkable to the ModeMetric protocol to allow runtime
    type checking using isinstance() and issubclass().
    sahusiddharth authored Feb 18, 2025
    Configuration menu
    Copy the full SHA
    a94feb2 View commit details
    Browse the repository at this point in the history

Commits on Feb 20, 2025

  1. r2r integration (#1918)

    - r2r integration
    - fixed spelling mistake in swarm tutorial
    - added @t.runtime_checkable to metric mode
    
    ---------
    
    Co-authored-by: Jithin James <[email protected]>
    sahusiddharth and jjmachan authored Feb 20, 2025
    Configuration menu
    Copy the full SHA
    76e14b0 View commit details
    Browse the repository at this point in the history
  2. Haystack llm and embedding wrapper (#1901)

    Co-authored-by: Jithin James <[email protected]>
    sahusiddharth and jjmachan authored Feb 20, 2025
    Configuration menu
    Copy the full SHA
    2bc29a2 View commit details
    Browse the repository at this point in the history

Commits on Feb 24, 2025

  1. Easy fix, Groundedness metric, 5 retries early break. (#1935)

    Easy fix Groundedness metric, 5 retries early break.
    Added logger to the 3 nv_metrics retries.
    Fixed input context max lenght to 7k to avoid 8k break.
    titericz authored Feb 24, 2025
    Configuration menu
    Copy the full SHA
    48b82ab View commit details
    Browse the repository at this point in the history

Commits on Mar 3, 2025

  1. nvidia docs (#1940)

    sahusiddharth authored Mar 3, 2025
    Configuration menu
    Copy the full SHA
    0a32987 View commit details
    Browse the repository at this point in the history
  2. unnecessary Noise Sensitivity name update (#1943)

    - the current code was logging a name and mode of noise sensitivity
    metric instead of just the name
    sahusiddharth authored Mar 3, 2025
    Configuration menu
    Copy the full SHA
    88e5fd3 View commit details
    Browse the repository at this point in the history
  3. When saving kg only save the node id in the relationships (#1926)

    When saving the knowledge graph, all the fields of each node is no
    longer stored in the relationships.
    
    This can save a very large amount of hard disk space.
    lryan599 authored Mar 3, 2025
    Configuration menu
    Copy the full SHA
    f5ea3d5 View commit details
    Browse the repository at this point in the history
  4. fix: return valid JSON output schema (#1933)

    Use json.dumps() on the dict output of Pydantics's
    BaseModel.model_json_schema() to ensure the string representation is
    valid JSON using double quotes instead of single quotes.
    alasdairsmith authored Mar 3, 2025
    Configuration menu
    Copy the full SHA
    776afaa View commit details
    Browse the repository at this point in the history
  5. Update default.py - Missing cosine_similarity transform for docs wi…

    …th Token Count 101–500 (#1934)
    
    The `default_ transforms` function defined at
    `src/ragas/testset/transforms/default.py` has a problem with handling
    transforms for documents with 101-500 tokens.
    
    The code divides the `transforms` configurations based on the document's
    token count. Several transforms are instantiated when the ["101-500"
    token count bins the first quartile
    (Q1](https://github.com/explodinggradients/ragas/blob/2bc29a2b8358ddb6b167fdf7ab0518ad9371463c/src/ragas/testset/transforms/default.py#L128),
    among them the `cosine_sim_builder`. While `cosine_sim_builder` is
    correctly **instantiated** (line 139), it's then **not included** in the
    list of transforms that are actually returned (line 153).
    
    It appears that `cosine_sim_builder` was likely unintentionally omitted
    from the returned transforms list. The intended behavior should probably
    mirror how `ner_overlap_sim` is handled (line 120), where
    `cosine_sim_builder` is instantiated and added to the returned list. The
    current code effectively instantiates `cosine_sim_builder` but then
    discards it. This omission might impact the number of relationships
    created in the knowledge graph.
    rgrizzo-linksmt authored Mar 3, 2025
    Configuration menu
    Copy the full SHA
    1d4b2ff View commit details
    Browse the repository at this point in the history

Commits on Mar 4, 2025

  1. Ndarray dtype fix, improve error wording (#1924)

    1. I have noticed that if `response.statements` is an empty list, then
    the output of `verify_claims` function is `array([], dtype=float64)`,
    which raises a type error when `~` operation is applied to
    `reference_response` variable. This occurs in a rare case when
    hypothesis_list is empty (when there are no claims in the response).
    2. A small change is the choice of words in `LLMDidNotFinishException`
    
    ---------
    
    Co-authored-by: Mikhail Zybin <[email protected]>
    zybinmikhail and Mikhail Zybin authored Mar 4, 2025
    Configuration menu
    Copy the full SHA
    414a518 View commit details
    Browse the repository at this point in the history
Loading