[safe-output-health] Safe Output Health Report - 2026-04-23

[github-actions[bot]]

Executive Summary

• Period: 2026-04-23, last 24 hours
• Runs Analyzed: 30 total (10 with full artifacts in summary, 20 workflow-logs-only)
• Workflows Active: 10 distinct workflows (in summary)
• Safe Output Jobs Executed: 0 — no safe output jobs ran in any run
• Agent Runs (agent actually executed): 2 (Smoke OpenCode, Smoke Gemini)
• Error Clusters Identified: 2

Overall Status: The safe output infrastructure conditional logic is working correctly. The absence of safe output executions is explained entirely by agent-level failures or agent skips — not by any fault in the safe output jobs themselves.

---

Safe Output Job Statistics

Job Type	Total Executions	Failures	Success Rate
All safe output types	0	0	N/A — no jobs ran

Why no safe output jobs ran: The safe_outputs job condition requires (needs.agent.result != 'skipped') && (needs.detection.result == 'success'). Detection only runs when the agent produces output_types or has_patch. In all runs this period, either the agent was skipped (20 workflow-logs-only runs) or the agent ran but produced no output (Smoke OpenCode, Smoke Gemini), so detection evaluated to false and safe output jobs were correctly gated off.

---

Error Clusters

Cluster 1: Docker Container Health Failure — awf-api-proxy

• Count: 1 occurrence
• Affected Workflow: Smoke Gemini (§24837114224)
• Engine: Google Gemini CLI
• Sample Error:

  Container awf-api-proxy  Error
  dependency failed to start: container awf-api-proxy is unhealthy
  [ERROR] Failed to start containers: Error: Command failed with exit code 1: docker compose up -d --pull never
  

• Root Cause: awf-api-proxy container failed its health check after ~10 seconds. awf-squid was healthy. The container sandbox for the Gemini agent could not start, preventing any agent execution.
• Impact: Agent never ran; detection evaluated false; no safe output triggered.

Cluster 2: Missing API Key — Google Generative AI

• Count: 1 occurrence
• Affected Workflow: Smoke OpenCode (§24837114187)
• Engine: OpenCode (internally uses Google Gemini models)
• Sample Error:

  AI_LoadAPIKeyError: Google Generative AI API key is missing.
  Pass it using the 'apiKey' parameter or the GOOGLE_GENERATIVE_AI_API_KEY environment variable.
  

• Root Cause: GOOGLE_GENERATIVE_AI_API_KEY environment variable not available to the OpenCode agent. Process exited with code 0 (silent failure).
• Impact: Agent produced no output; detection evaluated false; no safe output triggered.

---

Root Cause Analysis

Infrastructure Issues

awf-api-proxy container health check failure (Smoke Gemini): The awf-api-proxy container, which proxies API traffic for the agent sandbox, failed its Docker health check. awf-squid (proxy) was healthy in 7 seconds; awf-api-proxy errored at the 10-second mark. This appears to be a transient container startup issue, possibly a flaky health check or a race condition. This is a PR-triggered run on branch copilot/modify-pre-agent-steps-injection (PR #28082).

Smoke Crush hang (run-24837114237): Run was still in_progress at audit time with limited log data. The agent job was in_progress but no agent logs were downloaded. May be a slow-running test or a timeout situation. Also triggered by PR #28082.

Configuration Issues

Missing Google API key for OpenCode: The Smoke OpenCode workflow runs OpenCode CLI which uses Google Gemini as its LLM backend but does not have the GOOGLE_GENERATIVE_AI_API_KEY set. This causes a silent failure (exit code 0) with no agent output.

Expected Behavior: Label-Filtered Skips

20 workflow-logs-only runs were triggered by a PR labeled event (label: smoke, PR #28082) but each individual workflow requires its own specific activation label (e.g., water, metal, smoke-opencode, etc.). All correctly evaluated pre_activation.if to false and skipped. This is working as designed.

---

Recommendations

Critical Issues (Immediate Action Required)

1. Investigate awf-api-proxy container health failure

   • Priority: High
   • Root Cause: Container becomes unhealthy during Gemini smoke test; docker compose startup fails
   • Recommended Action: Check awf-api-proxy image health check configuration; investigate if the PR branch copilot/modify-pre-agent-steps-injection changes affect container startup behavior; add retry logic or extend health check timeout
   • Affected: Smoke Gemini workflow, PR Run pre-agent-steps before MCP gateway startup #28082

2. Configure GOOGLE_GENERATIVE_AI_API_KEY for Smoke OpenCode

   • Priority: High
   • Root Cause: OpenCode uses Google Gemini internally but the API key is not present in the workflow's environment
   • Recommended Action: Add GOOGLE_GENERATIVE_AI_API_KEY secret to the Smoke OpenCode workflow configuration; verify OpenCode is expected to use Google Gemini (not a local/different backend)
   • Affected: Smoke OpenCode workflow

3. Investigate Smoke Crush run hang

   • Priority: Medium
   • Root Cause: Run remained in_progress at audit time; no agent log artifacts were available
   • Recommended Action: Check the current status of run §24837114237; determine if there's a timeout configuration issue for the Crush engine

---

Work Item Plans

Work Item 1: Diagnose and Fix awf-api-proxy Container Startup Failure

• Type: Bug Fix / Investigation
• Priority: High
• Description: The awf-api-proxy container becomes unhealthy when the Smoke Gemini workflow runs on the copilot/modify-pre-agent-steps-injection PR branch. This prevents any agent execution and is blocking smoke test validation of the PR.
• Acceptance Criteria:
  • Root cause identified (health check config, PR branch changes, or transient runner issue)
  • Smoke Gemini smoke test passes on re-run or fix is applied
  • awf-api-proxy container starts healthy within expected timeout
• Technical Approach: Compare awf-api-proxy Docker image configuration between the PR branch and main; check if any recent changes to proxy startup or health check logic are involved; consider adding retry logic for container startup in awf binary
• Estimated Effort: Small–Medium
• Dependencies: Access to PR Run pre-agent-steps before MCP gateway startup #28082 branch changes; Docker image definitions for awf-api-proxy

Work Item 2: Fix OpenCode Google API Key Configuration

• Type: Configuration Fix
• Priority: High
• Description: OpenCode smoke test silently fails because GOOGLE_GENERATIVE_AI_API_KEY is not configured, causing OpenCode's internal Gemini API calls to fail with AI_LoadAPIKeyError. The process exits with code 0 masking the failure.
• Acceptance Criteria:
  • GOOGLE_GENERATIVE_AI_API_KEY is available in Smoke OpenCode workflow environment
  • OpenCode agent executes successfully and produces output
  • Smoke test creates expected GitHub issue and PR comment
• Technical Approach: Add GOOGLE_GENERATIVE_AI_API_KEY as a repository or organization secret and wire it into the Smoke OpenCode workflow's environment; alternatively verify if OpenCode should be configured to use a different LLM backend
• Estimated Effort: Small
• Dependencies: Google API key provisioning; Smoke OpenCode workflow YAML access

---

Historical Context

This is the first audit for this repository. No prior safe output health data in cache memory.

Metrics and KPIs

• Overall Safe Output Success Rate: N/A (0 executions)
• Agent-level failure rate this period: 2/10 analyzed runs (20%)
• Most common failure mode: Infrastructure/configuration failures preventing agent execution
• Safe output infrastructure health: ✅ Conditional logic working correctly

Next Steps

• Investigate and fix awf-api-proxy container health failure (PR Run pre-agent-steps before MCP gateway startup #28082, Smoke Gemini)
• Configure GOOGLE_GENERATIVE_AI_API_KEY for Smoke OpenCode workflow
• Check current status of Smoke Crush run §24837114237
• Re-run affected smoke tests on PR Run pre-agent-steps before MCP gateway startup #28082 after fixes

References:

• §24837114224 — Smoke Gemini (docker failure)
• §24837114187 — Smoke OpenCode (missing API key)
• §24837114237 — Smoke Crush (in_progress / hang)

Generated by Safe Output Health Monitor · ● 470.4K · ◷

• expires on Apr 24, 2026, 1:19 PM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Safe Output Health Monitor.

A newer discussion is available at Discussion #28271.