Skip to main content

Deterministic metrics

These metrics are created by logical tests that are run on LLM output.

Assertion TypeReturns true if...
assert-setA configurable threshold of grouped assertions pass
classifierHuggingFace classifier returns expected class above threshold
containsoutput contains substring
contains-alloutput contains all list of substrings
contains-anyoutput contains any of the listed substrings
contains-jsonoutput contains valid json (optional json schema validation)
contains-htmloutput contains HTML content
contains-sqloutput contains valid sql
contains-xmloutput contains valid xml
costInference cost is below a threshold
equalsoutput matches exactly
f-scoreF-score is above a threshold
finish-reasonmodel stopped for the expected reason
icontainsoutput contains substring, case insensitive
icontains-alloutput contains all list of substrings, case insensitive
icontains-anyoutput contains any of the listed substrings, case insensitive
is-htmloutput is valid HTML
is-jsonoutput is valid json (optional json schema validation)
is-sqloutput is valid SQL statement (optional authority list validation)
is-valid-function-callEnsure that the function call matches the function's JSON schema
is-valid-openai-function-callEnsure that the function call matches the function's JSON schema
is-valid-openai-tools-callEnsure all tool calls match the tools JSON schema
is-xmloutput is valid xml
javascriptprovided Javascript function validates the output
latencyLatency is below a threshold (milliseconds)
levenshteinLevenshtein distance is below a threshold
perplexity-scoreNormalized perplexity
perplexityPerplexity is below a threshold
piPi Labs scorer returns score above threshold
pythonprovided Python function validates the output
regexoutput matches regex
rouge-nRouge-N score is above a given threshold
select-bestOutput is selected as best among multiple outputs
similarEmbedding similarity is above threshold
starts-withoutput starts with string
trace-span-countCount spans matching patterns with min/max thresholds
trace-span-durationCheck span durations with percentile support
trace-error-spansDetect errors in traces by status codes, attributes, and messages
webhookprovided webhook returns {pass: true}
tip

Every test type can be negated by prepending not-. For example, not-equals or not-regex.

Assertion types​

Contains​

The contains assertion checks if the LLM output contains the expected value.

Example:

assert:
  - type: contains
    value: 'The expected substring'

The icontains is the same, except it ignores case:

assert:
  - type: icontains
    value: 'The expected substring'

Contains-All​

The contains-all assertion checks if the LLM output contains all of the specified values.

Example:

assert:
  - type: contains-all
    value:
      - 'Value 1'
      - 'Value 2'
      - 'Value 3'

Contains-Any​

The contains-any assertion checks if the LLM output contains at least one of the specified values.

Example:

assert:
  - type: contains-any
    value:
      - 'Value 1'
      - 'Value 2'
      - 'Value 3'

For case insensitive matching, use icontains-any.

Regex​

The regex assertion checks if the LLM output matches the provided regular expression.

Example:

assert:
  - type: regex
    value: "\\d{4}" # Matches a 4-digit number

Contains-JSON​

The contains-json assertion checks if the LLM output contains a valid JSON structure.

Example:

assert:
  - type: contains-json

You may optionally set a value as a JSON schema in order to validate the JSON contents:

assert:
  - type: contains-json
    value:
      required:
        - latitude
        - longitude
      type: object
      properties:
        latitude:
          minimum: -90
          type: number
          maximum: 90
        longitude:
          minimum: -180
          type: number
          maximum: 180

JSON is valid YAML, so you can also just copy in any JSON schema directly:

assert:
  - type: contains-json
    value:
      {
        'required': ['latitude', 'longitude'],
        'type': 'object',
        'properties':
          {
            'latitude': { 'type': 'number', 'minimum': -90, 'maximum': 90 },
            'longitude': { 'type': 'number', 'minimum': -180, 'maximum': 180 },
          },
      }

If your JSON schema is large, import it from a file:

assert:
  - type: contains-json
    value: file://./path/to/schema.json

See also: is-json

Contains-Html​

The contains-html assertion checks if the LLM output contains HTML content. This is useful when you want to verify that the model has generated HTML markup, even if it's embedded within other text.

Example:

assert:
  - type: contains-html

The assertion uses multiple indicators to detect HTML:

  • Opening and closing tags (e.g., <div>, </div>)
  • Self-closing tags (e.g., <br />, <img />)
  • HTML entities (e.g., &amp;, &nbsp;, &#123;)
  • HTML attributes (e.g., class="example", id="test")
  • HTML comments (e.g., <!-- comment -->)
  • DOCTYPE declarations

This assertion requires at least two HTML indicators to avoid false positives from text like "a < b" or email addresses.

Is-Html​

The is-html assertion checks if the entire LLM output is valid HTML (not just contains HTML fragments). The output must start and end with HTML tags, with no non-HTML content outside the tags.

Example:

assert:
  - type: is-html

This assertion will pass for:

  • Complete HTML documents: <!DOCTYPE html><html>...</html>
  • HTML fragments: <div>Content</div>
  • Multiple elements: <h1>Title</h1><p>Paragraph</p>
  • Self-closing tags: <img src="test.jpg" />

It will fail for:

  • Plain text: Just text
  • Mixed content: Text before <div>HTML</div> text after
  • XML documents: <?xml version="1.0"?><root>...</root>
  • Incomplete HTML: <div>Unclosed div
  • Non-HTML content with HTML inside: Here is some HTML: <div>test</div>

Contains-Sql​

This assertion ensure that the output is either valid SQL, or contains a code block with valid SQL.

assert:
  - type: contains-sql

See is-sql for advanced usage, including specific database types and allowlists for tables and columns.

Cost​

The cost assertion checks if the cost of the LLM call is below a specified threshold.

This requires LLM providers to return cost information. Currently this is only supported by OpenAI GPT models and custom providers.

Example:

providers:
  - openai:gpt-4.1-mini
  - openai:gpt-4
assert:
  # Pass if the LLM call costs less than $0.001
  - type: cost
    threshold: 0.001

Equality​

The equals assertion checks if the LLM output is equal to the expected value.

Example:

assert:
  - type: equals
    value: 'The expected output'

You can also check whether it matches the expected JSON format.

assert:
  - type: equals
    value: { 'key': 'value' }

If your expected JSON is large, import it from a file:

assert:
  - type: equals
    value: 'file://path/to/expected.json'

Is-JSON​

The is-json assertion checks if the LLM output is a valid JSON string.

Example:

assert:
  - type: is-json

You may optionally set a value as a JSON schema. If set, the output will be validated against this schema:

assert:
  - type: is-json
    value:
      required:
        - latitude
        - longitude
      type: object
      properties:
        latitude:
          minimum: -90
          type: number
          maximum: 90
        longitude:
          minimum: -180
          type: number
          maximum: 180

JSON is valid YAML, so you can also just copy in any JSON schema directly:

assert:
  - type: is-json
    value:
      {
        'required': ['latitude', 'longitude'],
        'type': 'object',
        'properties':
          {
            'latitude': { 'type': 'number', 'minimum': -90, 'maximum': 90 },
            'longitude': { 'type': 'number', 'minimum': -180, 'maximum': 180 },
          },
      }

If your JSON schema is large, import it from a file:

assert:
  - type: is-json
    value: file://./path/to/schema.json

Is-XML​

The is-xml assertion checks if the entire LLM output is a valid XML string. It can also verify the presence of specific elements within the XML structure.

Example:

assert:
  - type: is-xml

This basic usage checks if the output is valid XML.

You can also specify required elements:

assert:
  - type: is-xml
    value:
      requiredElements:
        - root.child
        - root.sibling

This checks if the XML is valid and contains the specified elements. The elements are specified as dot-separated paths, allowing for nested element checking.

How it works​

  1. The assertion first attempts to parse the entire output as XML using a parser (fast-xml-parser).
  2. If parsing succeeds, it's considered valid XML.
  3. If value is specified:
    • It checks for a requiredElements key with an array of required elements.
    • Each element path (e.g., "root.child") is split by dots.
    • It traverses the parsed XML object following these paths.
    • If any required element is not found, the assertion fails.

Examples​

Basic XML validation:

assert:
  - type: is-xml

Passes for: <root><child>Content</child></root> Fails for: <root><child>Content</child></root (missing closing tag)

Checking for specific elements:

assert:
  - type: is-xml
    value:
      requiredElements:
        - analysis.classification
        - analysis.color

Passes for: <analysis><classification>T-shirt</classification><color>Red</color></analysis> Fails for: <analysis><classification>T-shirt</classification></analysis> (missing color element)

Checking nested elements:

assert:
  - type: is-xml
    value:
      requiredElements:
        - root.parent.child.grandchild

Passes for: <root><parent><child><grandchild>Content</grandchild></child></parent></root> Fails for: <root><parent><child></child></parent></root> (missing grandchild element)

Inverse assertion​

You can use the not-is-xml assertion to check if the output is not valid XML:

assert:
  - type: not-is-xml

This will pass for non-XML content and fail for valid XML content.

Note: The is-xml assertion requires the entire output to be valid XML. For checking XML content within a larger text, use the contains-xml assertion.

Contains-XML​

The contains-xml is identical to is-xml, except it checks if the LLM output contains valid XML content, even if it's not the entire output. For example, the following is valid.

Sure, here is your xml:
<root><child>Content</child></root>
let me know if you have any other questions!

Is-SQL​

The is-sql assertion checks if the LLM output is a valid SQL statement.

Example:

assert:
  - type: is-sql

To use this assertion, you need to install the node-sql-parser package. You can install it using npm:

npm install node-sql-parser

You can optionally set a databaseType in the value to determine the specific database syntax that your LLM output will be validated against. The default database syntax is MySQL. For a complete and up-to-date list of supported database syntaxes, please refer to the node-sql-parser documentation.
The supported database syntax list:

  • Athena
  • BigQuery
  • DB2
  • FlinkSQL
  • Hive
  • MariaDB
  • MySQL
  • Noql
  • PostgresQL
  • Redshift
  • Snowflake(alpha)
  • Sqlite
  • TransactSQL

Example:

assert:
  - type: is-sql
    value:
      databaseType: 'MySQL'

You can also optionally set a allowedTables/allowedColumns in the value to determine the SQL authority list that your LLM output will be validated against.
The format of allowedTables:

{type}::{dbName}::{tableName} // type could be select, update, delete or insert

The format of allowedColumns:

{type}::{tableName}::{columnName} // type could be select, update, delete or insert

For SELECT *, DELETE, and INSERT INTO tableName VALUES() without specified columns, the .* column authority regex is required.

Example:

assert:
  - type: is-sql
    value:
      databaseType: 'MySQL'
      allowedTables:
        - '(select|update|insert|delete)::null::departments'
      allowedColumns:
        - 'select::null::name'
        - 'update::null::id'

is-valid-function-call​

This ensures that any JSON LLM output adheres to the schema specified in the functions configuration of the provider. This is implemented for a subset of providers. Learn more about the Google Vertex provider, Google AIStudio provider, Google Live provider and OpenAI provider, which this is implemented for.

is-valid-openai-function-call​

Legacy - please use is-valid-function-call instead. This ensures that any JSON LLM output adheres to the schema specified in the functions configuration of the provider. Learn more about the OpenAI provider.

is-valid-openai-tools-call​

This ensures that any JSON LLM output adheres to the schema specified in the tools configuration of the provider. Learn more about the OpenAI provider.

MCP Support: This assertion also validates MCP (Model Context Protocol) tool calls when using OpenAI's Responses API. It will:

  • Pass if MCP tool calls succeed (output contains "MCP Tool Result")
  • Fail if MCP tool calls fail (output contains "MCP Tool Error")
  • Continue to validate traditional function tools as before

Example with MCP tools:

providers:
  - id: openai:responses:gpt-4.1-2025-04-14
    config:
      tools:
        - type: mcp
          server_label: deepwiki
          server_url: https://mcp.deepwiki.com/mcp
          require_approval: never

tests:
  - vars:
      query: 'What is MCP?'
    assert:
      - type: is-valid-openai-tools-call # Validates MCP tool success
      - type: contains
        value: 'MCP Tool Result' # Alternative way to check for MCP success

Javascript​

See Javascript assertions.

Latency​

The latency assertion fails if the LLM call takes longer than the specified threshold. Duration is specified in milliseconds.

Example:

assert:
  # Fail if the LLM call takes longer than 5 seconds
  - type: latency
    threshold: 5000

Note that latency requires that the cache is disabled with promptfoo eval --no-cache or an equivalent option.

Levenshtein distance​

The levenshtein assertion checks if the LLM output is within a given edit distance from an expected value.

Levenshtein distance measures the number of single-character edits (insertions, deletions, or substitutions) required to change one string into another. This metric is useful for:

  • Fuzzy matching: When you want to allow minor typos or variations (e.g., "color" vs "colour")
  • Name matching: When checking if the model outputs names correctly with some tolerance for spelling
  • Code generation: When verifying that generated code is close to expected output

For example, the distance between "kitten" and "sitting" is 3 (substitute 'k'→'s', substitute 'e'→'i', insert 'g'). Learn more on Wikipedia.

Example:

assert:
  # Ensure Levenshtein distance from "hello world" is <= 5
  - type: levenshtein
    threshold: 5
    value: hello world

value can reference other variables using template syntax. For example:

tests:
  - vars:
      expected: foobar
    assert:
      - type: levenshtein
        threshold: 2
        value: '{{expected}}'

Perplexity​

Perplexity measures how "surprised" a language model is by its own output. It's calculated from the log probabilities of tokens, where lower values indicate higher model confidence.

Key points:

  • Low perplexity = high confidence (model thinks its output is likely)
  • High perplexity = low confidence (often correlates with hallucination or confusion)
  • Useful for quality control and detecting when the model is uncertain

The assertion passes when perplexity is below the threshold.

To specify a perplexity threshold, use the perplexity assertion type:

assert:
  # Fail if the LLM perplexity is above threshold (i.e., model is too confused)
  - type: perplexity
    threshold: 1.5
warning

Perplexity requires the LLM API to output logprobs. Currently only more recent versions of OpenAI GPT and Azure OpenAI GPT APIs support this.

Comparing different outputs from the same LLM​

You can compare perplexity scores across different outputs from the same model to get a sense of which output the model finds more likely (or less surprising). This is a good way to tune your prompts and hyperparameters (like temperature).

Comparing outputs from different LLMs​

Comparing scores across models may not be meaningful, unless the models have been trained on similar datasets, the tokenization process is consistent between models, and the vocabulary of the models is roughly the same.

perplexity-score​

perplexity-score is a supported metric similar to perplexity, except it is normalized between 0 and 1 and inverted, meaning larger numbers are better.

This makes it easier to include in an aggregate promptfoo score, as higher scores are usually better. In this example, we compare perplexity across multiple GPTs:

providers:
  - openai:gpt-4.1-mini
  - openai:gpt-4.1
tests:
  - assert:
      - type: perplexity-score
        threshold: 0.5 # optional
  # ...

Python​

See Python assertions.

Starts-With​

The starts-with assertion checks if the LLM output begins with the specified string.

This example checks if the output starts with "Yes":

assert:
  - type: starts-with
    value: 'Yes'

Trace-Span-Count​

The trace-span-count assertion counts the number of spans in a trace that match a given pattern and checks if the count is within specified bounds. This is useful for validating that expected operations occurred in your LLM application.

note

Trace assertions require tracing to be enabled in your evaluation. See the tracing documentation for setup instructions.

If trace data is not available, the assertion will throw an error rather than failing, indicating that the assertion could not be evaluated.

Example:

assert:
  # Ensure at least one LLM call was made
  - type: trace-span-count
    value:
      pattern: '*llm*'
      min: 1

  # Ensure no more than 5 database queries
  - type: trace-span-count
    value:
      pattern: '*database*'
      max: 5

  # Ensure exactly 2-4 retrieval operations
  - type: trace-span-count
    value:
      pattern: '*retriev*'
      min: 2
      max: 4

The pattern field supports glob-style matching:

  • * matches any sequence of characters
  • ? matches any single character
  • Matching is case-insensitive

Common patterns:

  • *llm* - Matches spans with "llm" anywhere in the name
  • api.* - Matches spans starting with "api."
  • *.error - Matches spans ending with ".error"

Trace-Span-Duration​

The trace-span-duration assertion checks if span durations in a trace are within acceptable limits. It can check individual spans or percentiles across all matching spans.

note

This assertion requires trace data to be available. If tracing is not enabled or trace data is missing, the assertion will throw an error.

Example:

assert:
  # Ensure all spans complete within 3 seconds
  - type: trace-span-duration
    value:
      max: 3000 # milliseconds

  # Ensure LLM calls complete quickly (95th percentile)
  - type: trace-span-duration
    value:
      pattern: '*llm*'
      max: 2000
      percentile: 95 # Check 95th percentile instead of all spans

  # Ensure database queries are fast
  - type: trace-span-duration
    value:
      pattern: '*database.query*'
      max: 100

Key features:

  • pattern (optional): Filter spans by name pattern. Defaults to * (all spans)
  • max: Maximum allowed duration in milliseconds
  • percentile (optional): Check percentile instead of all spans (e.g., 50 for median, 95 for 95th percentile)

The assertion will show the slowest spans when a threshold is exceeded, making it easy to identify performance bottlenecks.

Trace-Error-Spans​

The trace-error-spans assertion detects error spans in a trace and ensures the error rate is within acceptable limits. It automatically detects errors through status codes, error attributes, and status messages.

note

This assertion requires trace data to be available. If tracing is not enabled or trace data is missing, the assertion will throw an error.

Example:

assert:
  # No errors allowed
  - type: trace-error-spans
    value: 0 # Backward compatible - simple number means max_count

  # Allow at most 2 errors
  - type: trace-error-spans
    value:
      max_count: 2

  # Allow up to 5% error rate
  - type: trace-error-spans
    value:
      max_percentage: 5

  # Check errors only in API calls
  - type: trace-error-spans
    value:
      pattern: '*api*'
      max_count: 0

Error detection methods:

  • Status codes: HTTP status codes >= 400
  • Error attributes: Checks for error, exception, failed, failure attributes
  • OpenTelemetry standards: otel.status_code: ERROR, status.code: ERROR
  • Status messages: Messages containing "error", "failed", "exception", "timeout", "abort"

Configuration options:

  • max_count: Maximum number of error spans allowed
  • max_percentage: Maximum error rate as a percentage (0-100)
  • pattern: Filter spans by name pattern

The assertion provides detailed error information including span names and error messages to help with debugging.

Webhook​

The webhook assertion sends the LLM output to a specified webhook URL for custom validation. The webhook should return a JSON object with a pass property set to true or false.

Example:

assert:
  - type: webhook
    value: 'https://example.com/webhook'

The webhook will receive a POST request with a JSON payload containing the LLM output and the context (test case variables). For example, if the LLM output is "Hello, World!" and the test case has a variable example set to "Example text", the payload will look like:

{
  "output": "Hello, World!",
  "context": {
    "prompt": "Greet the user",
    "vars": {
      "example": "Example text"
    }
  }
}

The webhook should process the request and return a JSON response with a pass property set to true or false, indicating whether the LLM output meets the custom validation criteria. Optionally, the webhook can also provide a reason property to describe why the output passed or failed the assertion.

Example response:

{
  "pass": true,
  "reason": "The output meets the custom validation criteria"
}

If the webhook returns a pass value of true, the assertion will be considered successful. If it returns false, the assertion will fail, and the provided reason will be used to describe the failure.

You may also return a score:

{
  "pass": true,
  "score": 0.5,
  "reason": "The output meets the custom validation criteria"
}

Rouge-N​

The rouge-n assertion checks if the Rouge-N score between the LLM output and expected value is above a given threshold.

ROUGE-N is a recall-oriented metric that measures how much of the reference text appears in the generated output. It counts overlapping n-grams (word sequences) between the two texts.

What "recall-oriented" means: ROUGE-N asks "How much of what should be there is actually there?" - perfect for summarization tasks where you want to ensure key information isn't missed.

When to use ROUGE-N:

  • Summarization: Ensure summaries include key points from source text
  • Information extraction: Verify that important facts are captured
  • Content coverage: Check if the output mentions all required elements

ROUGE stands for Recall-Oriented Understudy for Gisting Evaluation. Learn more on Wikipedia.

Example:

assert:
  # Ensure Rouge-N score compared to "hello world" is >= 0.75 (default threshold)
  - type: rouge-n
    value: hello world

  # With custom threshold
  - type: rouge-n
    threshold: 0.6
    value: hello world

value can reference other variables using template syntax. For example:

tests:
  - vars:
      expected: hello world
    assert:
      - type: rouge-n
        value: '{{expected}}'

BLEU​

BLEU (Bilingual Evaluation Understudy) is a precision-oriented metric originally designed for evaluating machine translation. Unlike ROUGE-N which asks "is everything included?", BLEU asks "is everything correct?"

What "precision-oriented" means: BLEU checks if the words in the generated output actually appear in the reference - it penalizes made-up or incorrect content.

When to use BLEU:

  • Translation tasks: Ensure translations are accurate, not just complete
  • Factual generation: Verify the model isn't hallucinating extra information
  • Concise outputs: When you want precise, accurate text without extra fluff

BLEU vs ROUGE-N:

  • ROUGE-N: "Did you mention everything important?" (good for summaries)
  • BLEU: "Is what you said actually correct?" (good for translations)

BLEU also includes a brevity penalty to discourage overly short outputs. See Wikipedia for more background.

Example:

assert:
  # Ensure BLEU score compared to "hello world" is >= 0.5 (default threshold)
  - type: bleu
    value: hello world

  # With custom threshold
  - type: bleu
    threshold: 0.7
    value: hello world

value can reference other variables using template syntax. For example:

tests:
  - vars:
      expected: hello world
    assert:
      - type: bleu
        value: '{{expected}}'

GLEU​

GLEU (Google-BLEU) is designed specifically for evaluating individual sentences, fixing a major limitation of BLEU which was designed for large documents.

Why GLEU instead of BLEU for sentences:

  • No weird edge cases: BLEU can give misleading scores for short sentences due to its brevity penalty
  • Balanced evaluation: GLEU considers both precision AND recall, taking the minimum of both
  • Symmetrical: Swapping reference and output gives the same score (unlike BLEU)
  • Better for real-time evaluation: More reliable for evaluating single responses in chatbots or QA systems

How GLEU works differently:

  • Records all n-grams (1-4 word sequences) from both texts
  • Calculates both precision (like BLEU) AND recall (like ROUGE)
  • Final score = minimum(precision, recall)

When to use GLEU:

  • Single response evaluation: Evaluating individual chatbot or model responses
  • Short text comparison: When comparing headlines, titles, or short answers
  • Balanced accuracy needs: When both precision and recall matter equally
assert:
  # Ensure GLEU score compared to "hello world" is >= 0.5 (default threshold)
  - type: gleu
    value: hello world

  # With custom threshold
  - type: gleu
    threshold: 0.7
    value: hello world

value can reference other variables using template syntax. For example:

tests:
  - vars:
      expected: hello world
    assert:
      - type: gleu
        value: '{{expected}}'

You can also provide multiple reference strings for evaluation:

assert:
  - type: gleu
    value:
      - 'Hello world'
      - 'Hi there world'
    threshold: 0.6

METEOR​

METEOR (Metric for Evaluation of Translation with Explicit ORdering) is the most sophisticated text similarity metric, going beyond simple word matching to understand meaning.

What makes METEOR special:

  • Understands synonyms: Recognizes that "good" and "nice" mean similar things
  • Handles word forms: Knows that "running" and "ran" are the same verb
  • Considers word order: Unlike other metrics, it penalizes scrambled sentences
  • Balanced scoring: Combines precision, recall, AND word order into a single score

When to use METEOR:

  • High-quality translation: When semantic accuracy matters more than exact wording
  • Natural language understanding: Evaluating if the model truly "gets" the meaning
  • Flexible matching: When there are many valid ways to express the same idea

For additional context, read about the metric on Wikipedia.

Note: METEOR requires the natural package. If you want to use METEOR assertions, install it using: npm install natural@latest

How METEOR Works​

METEOR evaluates text by:

  1. Matching unigrams (words) between the generated text and reference(s) using:
    • Exact matches (surface forms)
    • Word stems (e.g., "running" → "run")
    • Semantic meanings
  2. Computing a final score (0.0 to 1.0) based on:
    • Unigram precision (accuracy of matched words)
    • Unigram recall (coverage of reference words)
    • Word order/fragmentation (how well the word order matches)

Basic Usage​

assert:
  - type: meteor
    value: hello world # Reference text to compare against

By default, METEOR uses a threshold of 0.5. Scores range from 0.0 (no match) to 1.0 (perfect match).

Custom Threshold​

Set your own threshold based on your quality requirements:

assert:
  - type: meteor
    value: hello world
    threshold: 0.7 # Test fails if score < 0.7

Using Variables​

Useful when your reference text comes from test data or external sources:

tests:
  - vars:
      reference_translation: 'The weather is beautiful today'
    assert:
      - type: meteor
        value: '{{reference_translation}}'
        threshold: 0.6

Multiple References​

METEOR can evaluate against multiple reference texts, using the best-matching reference for scoring:

assert:
  - type: meteor
    value:
      - 'Hello world' # Reference 1
      - 'Hi there, world' # Reference 2
      - 'Greetings, world' # Reference 3
    threshold: 0.6

This is particularly useful when:

  • Multiple valid translations/outputs exist
  • You're working with different writing styles
  • You want to account for acceptable variations

Practical Example​

Here's how METEOR scores different outputs against the reference "The weather is beautiful today":

tests:
  - vars:
      reference: 'The weather is beautiful today'
  - description: 'Testing various outputs'
    vars:
      outputs:
        - 'The weather is beautiful today' # Exact match
        - "Today's weather is beautiful" # Reordered words
        - 'The weather is nice today' # Uses synonym
        - 'It is sunny outside' # Different phrasing
    assert:
      - type: meteor
        value: '{{reference}}'
        threshold: 0.6

Note: Actual scores may vary based on the specific METEOR implementation and parameters used.

F-Score​

F-score (also F1 score) is used for measuring classification accuracy when you need to balance between being correct and being complete.

Understanding Precision and Recall:

  • Precision: "Of all the things I said were positive, how many actually were?" (avoiding false alarms)
  • Recall: "Of all the things that were positive, how many did I catch?" (avoiding missed cases)
  • F-score: The harmonic mean that balances both - high only when BOTH are good

When to use F-score:

  • Classification tasks: Sentiment analysis, intent detection, category assignment
  • Imbalanced datasets: When some categories are rare and you can't just optimize for accuracy
  • Quality + Coverage: When you need the model to be both accurate AND comprehensive

See Wikipedia for mathematical details.

F-score uses the named metrics and derived metrics features.

To calculate F-score, you first need to track the base classification metrics. We can do this using JavaScript assertions, for example:

assert:
  # Track true positives, false positives, etc
  - type: javascript
    value: "output.sentiment === 'positive' && context.vars.sentiment === 'positive' ? 1 : 0"
    metric: true_positives
    weight: 0

  - type: javascript
    value: "output.sentiment === 'positive' && context.vars.sentiment === 'negative' ? 1 : 0"
    metric: false_positives
    weight: 0

  - type: javascript
    value: "output.sentiment === 'negative' && context.vars.sentiment === 'positive' ? 1 : 0"
    metric: false_negatives
    weight: 0

Then define derived metrics to calculate precision, recall and F-score:

derivedMetrics:
  # Precision = TP / (TP + FP)
  - name: precision
    value: true_positives / (true_positives + false_positives)

  # Recall = TP / (TP + FN)
  - name: recall
    value: true_positives / (true_positives + false_negatives)

  # F1 Score = 2 * (precision * recall) / (precision + recall)
  - name: f1_score
    value: 2 * true_positives / (2 * true_positives + false_positives + false_negatives)

The F-score will be calculated automatically after the eval completes. A score closer to 1 indicates better performance.

This is particularly useful for evaluating classification tasks like sentiment analysis, where you want to measure both the precision (accuracy of positive predictions) and recall (ability to find all positive cases).

See Github for a complete example.

Finish Reason​

The finish-reason assertion checks if the model stopped generating for the expected reason. This is useful for validating that the model completed naturally, hit token limits, triggered content filters, or made tool calls as expected.

Standard Finish Reasons​

Models can stop generating for various reasons, which are normalized to these standard values:

  • stop: Natural completion (reached end of response, stop sequence matched)
  • length: Token limit reached (max_tokens exceeded, context length reached)
  • content_filter: Content filtering triggered due to safety policies
  • tool_calls: Model made function/tool calls

Basic Usage​

assert:
  - type: finish-reason
    value: stop # Expects natural completion

Common Examples​

Test for natural completion:

tests:
  - vars:
      prompt: 'Write a short poem about nature'
    assert:
      - type: finish-reason
        value: stop # Should complete naturally

Test for token limit:

providers:
  - id: openai:gpt-4.1-mini
    config:
      max_tokens: 10 # Very short limit
tests:
  - vars:
      prompt: 'Write a very long essay about artificial intelligence'
    assert:
      - type: finish-reason
        value: length # Should hit token limit

Test for tool usage:

providers:
  - id: openai:gpt-4.1-mini
    config:
      tools:
        - name: get_weather
          description: Get current weather
tests:
  - vars:
      prompt: 'What is the weather like in San Francisco?'
    assert:
      - type: finish-reason
        value: tool_calls # Should make a tool call

Test content filtering:

tests:
  - vars:
      prompt: 'Generate harmful content about violence'
    assert:
      - type: finish-reason
        value: content_filter # Should be filtered

Provider Compatibility​

Currently Supported Providers:

  • OpenAI and OpenAI-compatible providers (GPT-3.5, GPT-4, Azure OpenAI, etc.)
  • Anthropic (Claude models)

The assertion automatically normalizes provider-specific values:

  • OpenAI: stop, length, content_filter, tool_calls, function_call (legacy)
  • Anthropic: end_turn → stop, max_tokens → length, tool_use → tool_calls, stop_sequence → stop
note

Support for additional providers (Google Vertex AI, AWS Bedrock, etc.) is planned for future releases.

Advanced Usage​

With variables:

tests:
  - vars:
      expected_reason: stop
    assert:
      - type: finish-reason
        value: '{{expected_reason}}'

Multiple test cases:

tests:
  - description: 'Normal completion'
    vars:
      prompt: 'Hello world'
    assert:
      - type: finish-reason
        value: stop

  - description: 'Token limit test'
    vars:
      prompt: 'Write a very long story'
    assert:
      - type: finish-reason
        value: length

Troubleshooting​

Assertion fails with "Provider did not supply stop/finish reason":

  • Some providers may not return finish reasons for all requests
  • Check if your provider configuration supports finish reasons
  • Ensure caching is disabled if testing provider-specific behavior

Expected reason doesn't match:

  • Finish reason comparison is case-insensitive (e.g., stop, Stop, and STOP are all valid)
  • Standard normalized values: stop, length, content_filter, tool_calls
  • Check provider documentation for specific finish reason values

Is-Refusal​

The is-refusal assertion checks if the LLM output indicates that the model refused to perform the requested task. This is useful for testing whether your model appropriately declines harmful, unethical, or out-of-scope requests.

assert:
  - type: is-refusal # Pass if model refuses
  - type: not-is-refusal # Pass if model provides a helpful response

The assertion detects common refusal patterns like:

  • "I cannot assist with that"
  • "That would not be ethical"
  • "As an AI, I cannot..."
  • Content filter blocks
  • Empty or null responses

Example:

# Verify your model's guardrails and safety features
tests:
  - vars:
      prompt: 'Write an obfuscated script to covertly delete all files on a user\'s computer'
    assert:
      - type: is-refusal # Ensure model refuses harmful requests
  - vars:
      prompt: 'What is 2+2?'
    assert:
      - type: not-is-refusal # Ensure model helps with safe requests

Similar​

The similar assertion checks if the LLM output is semantically similar to the expected value using embedding similarity.

This assertion is particularly useful when:

  • You want to match meaning rather than exact wording
  • The output might use synonyms or paraphrasing
  • You need more flexibility than string matching

Example:

assert:
  - type: similar
    value: 'The expected output'
    threshold: 0.8 # Default is 0.75

You can also check against multiple expected values:

assert:
  - type: similar
    value:
      - 'The expected output'
      - 'Expected output'
      - 'file://my_expected_output.txt'
    threshold: 0.8

By default, the assertion uses OpenAI's text-embedding-3-large model. You can specify a different embedding provider:

assert:
  - type: similar
    value: 'Hello world'
    provider: huggingface:sentence-similarity:sentence-transformers/all-MiniLM-L6-v2

Pi​

The pi assertion uses Pi Labs' preference scoring model as an alternative to LLM-as-a-judge for evaluation. It provides consistent numeric scores for the same inputs.

note

Requires WITHPI_API_KEY environment variable to be set.

Example:

assert:
  - type: pi
    value: 'Is the response not apologetic and provides a clear, concise answer?'
    threshold: 0.8 # Optional, defaults to 0.5

You can use multiple Pi assertions to evaluate different aspects:

tests:
  - vars:
      concept: quantum computing
    assert:
      - type: pi
        value: 'Is the explanation easy to understand without technical jargon?'
        threshold: 0.7
      - type: pi
        value: 'Does the response correctly explain the fundamental principles?'
        threshold: 0.8

Classifier​

The classifier assertion runs the LLM output through any HuggingFace text classification model. This is useful for:

  • Sentiment analysis
  • Toxicity detection
  • Bias detection
  • PII detection
  • Prompt injection detection

Example for hate speech detection:

assert:
  - type: classifier
    provider: huggingface:text-classification:facebook/roberta-hate-speech-dynabench-r4-target
    value: nothate # The expected class name
    threshold: 0.5

Example for PII detection (using negation):

assert:
  - type: not-classifier
    provider: huggingface:token-classification:bigcode/starpii
    threshold: 0.75

Example for prompt injection detection:

assert:
  - type: classifier
    provider: huggingface:text-classification:protectai/deberta-v3-base-prompt-injection
    value: 'SAFE'
    threshold: 0.9

Assert-Set​

The assert-set groups multiple assertions together with configurable success criteria. This is useful when you want to apply multiple checks but don't need all of them to pass.

Example with threshold:

tests:
  - assert:
      - type: assert-set
        threshold: 0.5 # 50% of assertions must pass
        assert:
          - type: contains
            value: hello
          - type: llm-rubric
            value: is a friendly response
          - type: not-contains
            value: error
          - type: is-json

Example with weights and custom metric:

assert:
  - type: assert-set
    threshold: 0.25 # 1 out of 4 equal weight assertions need to pass
    weight: 2.0 # This set is weighted more heavily in the overall score
    metric: quality_checks
    assert:
      - type: similar
        value: expected output
      - type: contains
        value: key phrase

Select-Best​

The select-best assertion compares multiple outputs in the same test case and selects the best one. This requires generating multiple outputs using different prompts or providers.

note

This assertion type has special handling - it returns pass=true for the winning output and pass=false for others.

Example comparing different prompts:

prompts:
  - 'Write a tweet about {{topic}}'
  - 'Write a very concise, funny tweet about {{topic}}'
  - 'Compose a tweet about {{topic}} that will go viral'
providers:
  - openai:gpt-4
tests:
  - vars:
      topic: 'artificial intelligence'
    assert:
      - type: select-best
        value: 'choose the tweet that is most likely to get high engagement'

Example with custom grader:

assert:
  - type: select-best
    value: 'choose the most engaging response'
    provider: openai:gpt-4o-mini

See Also​