Gemini googleSearch grounding tool is dropped when running prompts via invoke / OpenAI-compat proxy

[justin-rwazi]

Summary

A Braintrust prompt configured with Gemini's search-grounding tool (tools: [{ "googleSearch": {} }]) does not perform grounding when the prompt is run through Braintrust (prompt invoke, experiments, or the /v1/proxy/chat/completions endpoint). The tool is silently dropped — the logged LLM span shows tools: null — so the model answers from training data only, and groundingMetadata is never produced.

This looks like the request-side root cause of #1700 (grounding metadata not captured): grounding never actually runs via the OpenAI-compatible proxy path, so there is no metadata to capture.

Environment

• braintrust JS SDK: 3.15.0
• Model: gemini-2.5-pro / gemini-2.5-flash
• Prompt prompt_data.tools: "[{\"googleSearch\":{}}]"

Reproduction

1. Create a prompt for gemini-2.5-pro whose tools array is the Gemini grounding tool [{ "googleSearch": {} }].
2. Run it via the prompt invoke API (or as an experiment task via invoke()).
3. Inspect the logged LLM span.

Observed: span request metadata has tools: null; the model responds without searching; fields that require fresh web data come back empty. No groundingMetadata.

Direct proxy evidence

The OpenAI-compatible proxy rejects/drops the Gemini-native grounding tool:

# (A) googleSearch on the OpenAI-compat endpoint -> gateway error
curl https://api.braintrust.dev/v1/proxy/chat/completions \
  -H "Authorization: Bearer $BT" -H "Content-Type: application/json" \
  -d '{"model":"gemini-2.5-flash","messages":[{"role":"user","content":"latest news?"}],"tools":[{"googleSearch":{}}]}'
# -> "Braintrust gateway error: Unable to detect request source format"

# (B) {"type":"web_search"} -> same gateway error
# (C) a normal function tool (e.g. google_web_search) -> returned as finish_reason=tool_calls,
#     never executed, so no grounding actually happens

The same grounding tool works through the native Google passthrough endpoint — but only on braintrustproxy.com, not api.braintrust.dev:

curl https://braintrustproxy.com/v1/proxy/google/models/gemini-2.5-flash:generateContent \
  -H "Authorization: Bearer $BT" -H "Content-Type: application/json" \
  -d '{"contents":[{"role":"user","parts":[{"text":"latest news?"}]}],"tools":[{"googleSearch":{}}]}'
# -> candidates[0].groundingMetadata present (webSearchQueries + groundingChunks)

For comparison, the same path on api.braintrust.dev returns HTTP 405 "Method POST not supported".

Expected

Running a prompt with the Gemini googleSearch grounding tool through Braintrust (invoke / experiments / playground) should forward it to Gemini so grounding actually runs — e.g. by routing gemini + googleSearch to the native generateContent request — instead of dropping the tool on the OpenAI-compatible path.

Workaround

Bypass invoke and call the native passthrough (https://braintrustproxy.com/v1/proxy/google/models/{model}:generateContent) with the native tools: [{ "googleSearch": {} }] body. Grounding runs and uses the Braintrust-configured Google key.

Related

• Google GenAI groundingMetadata not captured in streaming aggregation or span metadata #1700 — Google GenAI grounding metadata not captured (this issue is the request-side cause).

[lforst]

Hi, thanks for the report! We'll take a look asap.