Generator: Add support for Google Gemini models #1306

dchiitmalla · 2025-07-22T22:26:31Z

Faced issues trying to run gemini models using Litellm. This generator natively supports google models using Google’s official google.generativeai library—more reliable than LiteLLM. It supports multiple models (2.5 Pro, Flash, etc.) with error handling and model-specific config options. #443

jmartin-tech

Working with GCP based services has shown that auth via api_key is not always as straight forward as providing a single environment variable value. Can you provide details on how the key used would be scoped or generated.

Also consider enabling the google library to attempt auth even when no key is provided.

garak/generators/gemini.py

jmartin-tech · 2025-07-23T13:30:08Z

garak/generators/gemini.py

+        responses = []
+        import logging
+
+        for _ in range(generations_this_call):


Iterating generations like this is will not combine with backoff correctly. As written any raised backoff exception will throw away all completed generations and start over.

Looking at gemini docs multiple generations can be obtained in a single call using GenerateContentConfig by setting candidateCount= generations_this_call and passing generate_content() a config named parameter.

If calling for more than one generation please validate how the response object will be formatted.

Addressed - The implementation now uses candidate_count=generations_this_call in GenerateContentConfig and makes a single API call via _generate_content_with_backoff(). This ensures backoff retries don't discard completed generations. The response format has been validated and properly handles multiple candidates from the API response.

1. Update Gemini generator to handle test model names gracefully 2. Add missing documentation file for Gemini generator 3. Add Gemini generator to documentation toctree 4. Add google-generativeai dependency to pyproject.toml

remove api validation Co-authored-by: Jeffrey Martin <[email protected]> Signed-off-by: Divya Chitimalla <[email protected]>

- Update _call_model to use Gemini's native candidateCount parameter for multiple generations - Process response candidates correctly to extract text from each generation - Remove generation config from model initialization and set it per request - Fix backoff handling to properly retry the entire batch of generations - Ensure consistent number of responses are returned

jmartin-tech

General comments, the auth concern may be reasonable to defer to require environment variable based API key for initial acceptance.

Another concern that may be important to address:
This PR uses the package google-generativeai however current docs recommend using google-genai which has the same imported package name but a different interface.

tests/generators/test_gemini.py

jmartin-tech · 2025-07-23T18:06:24Z

tests/generators/test_gemini.py

+
+    # Create the generator with a native audio model
+    generator = GeminiGenerator(name="gemini-2.5-flash-native-audio")
+    output = generator._call_model("Transcribe this text.")


This does not make sense the input modality suggested by this test is audio not text.

The current generator also inherits the default modality, {"in": {"text"}, "out": {"text"}} unless overridden this generator should only accept text prompts.

garak/generators/gemini.py

Co-authored-by: Jeffrey Martin <[email protected]> Signed-off-by: Divya Chitimalla <[email protected]>

Fixed Failing Tests

Migrating it to the new google-genai , Support both API Keys and Vertex AI (ADC) and Improving the backoff logic

github-actions · 2025-08-08T16:02:09Z

DCO Assistant Lite bot All contributors have signed the DCO ✍️ ✅

Abhiraj-GetGarak · 2025-08-08T16:20:24Z

I have read the DCO Document and I hereby sign the DCO

Abhiraj-GetGarak · 2025-08-08T16:20:53Z

recheck

Abhiraj-GetGarak · 2025-08-08T16:36:51Z

Multiple generations backoff issue - Fixed using new google-genai library with candidate_count in
single API call.
Authentication flexibility - Enhanced with both API key and Vertex AI (ADC) support using new client
architecture.
Package migration completed - Successfully migrated from google-generativeai>=0.8.5 to
google-genai>=1.0.0

jmartin-tech

Code only review, full functional testing will be performed after comments and are addressed.

jmartin-tech · 2025-09-05T20:05:28Z

garak/generators/gemini.py

+        """Clear the model to avoid pickling issues."""
+        self.model = None
+
+    def __init__(self, name="", config_root=_config, api_key=None):


api_key can be injected via the config_root parameter, please use signature consistent with base.Generator:

Suggested change

def __init__(self, name="", config_root=_config, api_key=None):

def __init__(self, name="", config_root=_config):

jmartin-tech · 2025-09-05T20:08:11Z

garak/generators/gemini.py

+        self.temperature = self.DEFAULT_PARAMS["temperature"]
+        self.top_p = self.DEFAULT_PARAMS["top_p"]
+        self.top_k = self.DEFAULT_PARAMS["top_k"]
+        self.max_output_tokens = self.DEFAULT_PARAMS["max_output_tokens"]
+        self.api_key = api_key


These values are populated by the Configurable._load_config() call made in super().__init__() defer to common code:

Suggested change

self.temperature = self.DEFAULT_PARAMS["temperature"]

self.top_p = self.DEFAULT_PARAMS["top_p"]

self.top_k = self.DEFAULT_PARAMS["top_k"]

self.max_output_tokens = self.DEFAULT_PARAMS["max_output_tokens"]

self.api_key = api_key

jmartin-tech · 2025-09-05T20:13:19Z

garak/generators/gemini.py

+        # Defer model loading until actually needed
+        self.client = None
+        self.model = None


I would suggest, early validation that all required configuration is available during construction outweighs benefits of deferred loading here.

Suggested change

# Defer model loading until actually needed

self.client = None

self.model = None

self._load_model()

This becomes more clearly required when considering the fact that _load_model() currently impacts self.modality for the generator which need to be viable to evaluate on the instantiating thread before any request for inference has been performed.

jmartin-tech · 2025-09-05T20:21:36Z

garak/generators/gemini.py

+            logging.error(f"All retries failed for {self.name}: {e}")
+            return [None] * generations_this_call
+
+    @backoff.on_exception(backoff.expo, APIError, max_tries=5)


Note this suggestion mirrors how other generators currently implement backoff, garak is tenacious and currently is not expected to abort or skip a generation unless the error indicates that there is no possibility of recovery.

Suggested change

@backoff.on_exception(backoff.expo, APIError, max_tries=5)

@backoff.on_exception(backoff.fibo, APIError, max_value=70)

jmartin-tech · 2025-09-05T20:24:01Z

garak/generators/gemini.py

+                config=generation_config
+            )
+            return response
+            raise e  # This will trigger backoff retry


This would not trigger backoff, e does not exist and return ends execution of the block.

Suggested change

raise e # This will trigger backoff retry

jmartin-tech · 2025-09-05T20:44:37Z

garak/generators/gemini.py

+        try:
+            response = self.client.models.generate_content(
+                model=self.name,
+                contents=prompt,
+                config=generation_config
+            )
+            return response
+            raise e  # This will trigger backoff retry
+        except Exception as e:
+            logging.error(f"Unexpected error when calling {self.name}: {e}")
+            raise e  # Re-raise to be handled by caller


What value add is there in this try block?

Suggested change

try:

response = self.client.models.generate_content(

model=self.name,

contents=prompt,

config=generation_config

)

return response

raise e # This will trigger backoff retry

except Exception as e:

logging.error(f"Unexpected error when calling {self.name}: {e}")

raise e # Re-raise to be handled by caller

response = self.client.models.generate_content(

model=self.name,

contents=prompt,

config=generation_config

)

return response

If the exception is of type APIError then @backoff.on_exception will trigger otherwise it will bubble up for the caller to handle.

jmartin-tech · 2025-09-05T20:54:48Z

garak/generators/gemini.py

+            responses = [None] * generations_this_call
+
+        # Ensure we return the expected number of responses
+        while len(responses) < generations_this_call:
+            responses.append(None)
+
+        return responses[:generations_this_call]


This logic seems odd, I would expect the response format either to have all generations required or none (Maybe that is an invalid assumption on my part). I suspect this method can be a bit less opinionated and simply extract results from the response. The caller can be responsible for deciding what to do if the number of found responses do not total generations_this_call.

To that end generations_this_call does not need to be passed into this method.

Suggested change

responses = [None] * generations_this_call

# Ensure we return the expected number of responses

while len(responses) < generations_this_call:

responses.append(None)

return responses[:generations_this_call]

responses = [None]

return responses

jmartin-tech · 2025-09-08T15:24:31Z

garak/generators/gemini.py

+    ]
+
+    DEFAULT_PARAMS = Generator.DEFAULT_PARAMS | {
+        "name": "gemini-2.5-pro",


Do we want a default model here, the churn as model names change could be tedious to maintain.

Suggested change

"name": "gemini-2.5-pro",

"name": "gemini-2.5-pro",

jmartin-tech · 2025-09-08T15:32:31Z

garak/generators/gemini.py

+    SUPPORTED_MODELS = [
+        "gemini-2.5-pro",
+        "gemini-2.5-flash",
+        "gemini-2.5-flash-lite-preview",
+        "gemini-2.5-flash-native-audio",
+        "gemini-2.5-flash-preview-text-to-speech",
+        "gemini-2.5-pro-preview-text-to-speech",
+        "gemini-2.0-flash"
+    ]


Most of these models have very short TTL, retirement dates suggests that models listed are Auto-updated aliases and are versioned further. To denote possible need for clarification of results it might be helpful to have some method added to extract and document the Stable version reference to the model tested by garak during a run.

jmartin-tech · 2025-09-08T15:37:01Z

garak/generators/gemini.py

+    Supported models:
+    - gemini-2.5-pro: Gemini 2.5 Pro model (default)
+    - gemini-2.5-flash: Gemini 2.5 Flash model
+    - gemini-2.5-flash-lite-preview: Gemini 2.5 Flash Lite Preview model
+    - gemini-2.5-flash-native-audio: Gemini 2.5 Flash Native Audio model
+    - gemini-2.5-flash-preview-text-to-speech: Gemini 2.5 Flash Preview Text-to-Speech model
+    - gemini-2.5-pro-preview-text-to-speech: Gemini 2.5 Pro Preview Text-to-Speech model
+    - gemini-2.0-flash: Gemini 2.0 Flash model


I like having this level of detail however I think the lifecycle of models will cause the list to become stale quickly. Consider removing from the static docs.

dchiitmalla added 3 commits July 22, 2025 15:13

add gemini generator

435bcac

gemini test

e66394e

fix test

5a34e77

dchiitmalla mentioned this pull request Jul 22, 2025

generators: gemini #443

Closed

update req

0ff360b

jmartin-tech reviewed Jul 23, 2025

View reviewed changes

dchiitmalla and others added 3 commits July 23, 2025 10:56

Fix Gemini generator tests and documentation

c8a790d

1. Update Gemini generator to handle test model names gracefully 2. Add missing documentation file for Gemini generator 3. Add Gemini generator to documentation toctree 4. Add google-generativeai dependency to pyproject.toml

Update garak/generators/gemini.py

7e653be

remove api validation Co-authored-by: Jeffrey Martin <[email protected]> Signed-off-by: Divya Chitimalla <[email protected]>

jmartin-tech requested changes Jul 31, 2025

View reviewed changes

dchiitmalla and others added 15 commits August 6, 2025 09:07

Update tests/generators/test_gemini.py

43ad484

Co-authored-by: Jeffrey Martin <[email protected]> Signed-off-by: Divya Chitimalla <[email protected]>

Update tests/generators/test_gemini.py

492fed0

Co-authored-by: Jeffrey Martin <[email protected]> Signed-off-by: Divya Chitimalla <[email protected]>

Update garak/generators/gemini.py

aea4e3a

Co-authored-by: Jeffrey Martin <[email protected]> Signed-off-by: Divya Chitimalla <[email protected]>

Update garak/generators/gemini.py

d08dc1b

Co-authored-by: Jeffrey Martin <[email protected]> Signed-off-by: Divya Chitimalla <[email protected]>

fixed test

cb4319e

Merge pull request #2 from Abhiraj-GetGarak/fix/gemini_tests

e279a46

Fixed Failing Tests

Improve backoff handling for multi-generation requests

f6f64ff

migrate Gemini generator to the new google-genai library

225b8e7

Enhance authentication to support both API Key and Vertex AI (ADC)

9febc99

fixed tests

8c798dc

fixed bug

d64cb1e

fix gemini test

8fb2d48

fix minor issues

20f393a

lazy loading

3715abe

Merge pull request #3 from Abhiraj-GetGarak/gemini

ff45220

Migrating it to the new google-genai , Support both API Keys and Vertex AI (ADC) and Improving the backoff logic

github-actions bot added a commit that referenced this pull request Aug 8, 2025

@Abhiraj-GetGarak has signed the CLA in #1306

2774eb5

jmartin-tech requested changes Sep 8, 2025

View reviewed changes

	def __init__(self, name="", config_root=_config, api_key=None):
	def __init__(self, name="", config_root=_config):

	@backoff.on_exception(backoff.expo, APIError, max_tries=5)
	@backoff.on_exception(backoff.fibo, APIError, max_value=70)

Generator: Add support for Google Gemini models #1306

Are you sure you want to change the base?

Generator: Add support for Google Gemini models #1306

Uh oh!

Conversation

dchiitmalla commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jmartin-tech left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Abhiraj-GetGarak Aug 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jmartin-tech left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Aug 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Abhiraj-GetGarak commented Aug 8, 2025

Uh oh!

Abhiraj-GetGarak commented Aug 8, 2025

Uh oh!

Abhiraj-GetGarak commented Aug 8, 2025

Uh oh!

jmartin-tech left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dchiitmalla commented Jul 22, 2025 •

edited

Loading

Abhiraj-GetGarak Aug 8, 2025 •

edited

Loading

jmartin-tech left a comment •

edited

Loading

github-actions bot commented Aug 8, 2025 •

edited

Loading