Add models check step for engine API access verification

[Copilot]

Summary

Adds a pre-agent step that requests the list of models from the agentic engine provider before the agent step is launched, verifying API access and surfacing secret validity issues early.

Changes

New interface: ModelsProvider

Defines an optional ModelsProvider interface in the agentic engine architecture:

type ModelsProvider interface {
    GetModelsRoute(workflowData *WorkflowData) *ModelsRoute
}

Along with a ModelsRoute struct that captures the URL, auth header, auth scheme, secret env var, secret expression, extra headers, and base URL env var configuration.

Engine implementations

• ClaudeEngine: Implements ModelsProvider using https://api.anthropic.com/v1/models with x-api-key auth and anthropic-version: 2023-06-01 header. Respects ANTHROPIC_BASE_URL from engine.env for custom API endpoints. Skipped when custom command or top-level environment: is configured.
• CodexEngine: Implements ModelsProvider using https://api.openai.com/v1/models with Authorization: Bearer auth. Respects OPENAI_BASE_URL from engine.env for custom API endpoints. Skipped when custom command or top-level environment: is configured.

Generated step: "Verify engine API access"

When an engine implements ModelsProvider, a bash step is emitted before the agent execution step that:

1. Resolves the models URL at runtime: uses ${ANTHROPIC_BASE_URL%/}/models (or ${OPENAI_BASE_URL%/}/models) when the base URL env var is set, falling back to the default endpoint otherwise
2. Calls the /models endpoint using curl with the engine's secret
3. On success (2xx): reports ✅ Engine API access verified and lists available models in GITHUB_STEP_SUMMARY
4. On failure (non-2xx): sets models_check_failed=true in step output, reports ❌ Models request failed to GITHUB_STEP_SUMMARY, and exits 1 (failing the job)

Environment variable parity with the agent step

The models check step builds its env: block using the same layered approach as the agent execution step:

1. Route defaults — sets the secret expression (e.g. ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }})
2. Base URL fallback — always maps ${{ vars.ANTHROPIC_BASE_URL || '' }} so GitHub Actions variables are visible to the bash script (they are not automatically available as process env vars)
3. engine.env merge — any user-configured override takes precedence, overwriting the defaults above. This enables the GitHub Apps pattern where a token minted in a pre-step is passed as ANTHROPIC_API_KEY: ${{ steps.get-app-token.outputs.token }} in engine.env, and both the models check step and the agent step use the same dynamic credential
4. agentConfig.Env merge — for full parity with the agent execution step

Custom API base URL support

When ANTHROPIC_BASE_URL or OPENAI_BASE_URL is set in engine.env, the models check step hits the same custom endpoint the agent will use. When not set in engine.env, the base URL env var is still mapped from the GitHub Actions variable (${{ vars.ANTHROPIC_BASE_URL || '' }}) so it can be configured at the repository/organisation level without touching the workflow file.

Job output and conclusion job integration

• The agent job exposes models_check_failed as a job output (defaults to 'false')
• The conclusion job receives GH_AW_MODELS_CHECK_FAILED from the agent job output
• handle_agent_failure.cjs reads this env var and adds a **⚠️ Engine API Access Failed** context to the failure issue/comment, indicating the secret may be incorrect or outdated
• The {models_check_failed_context} placeholder is added to both agent_failure_issue.md and agent_failure_comment.md templates (right after {secret_verification_context}, reusing the same issue)

Testing

Added pkg/workflow/models_check_test.go with tests covering:

• Models check step generated for Claude and Codex engines
• models_check_failed job output added when ModelsProvider is implemented
• GH_AW_MODELS_CHECK_FAILED passed to conclusion job
• No models check for Copilot (doesn't implement ModelsProvider)
• Step skipped when custom command is specified
• Step skipped when top-level environment: is configured
• Step appears before the agent execution step
• Custom ANTHROPIC_BASE_URL from engine.env is used in the generated step for Claude
• Custom OPENAI_BASE_URL from engine.env is used in the generated step for Codex
• Default URL used when no custom base URL is configured
• engine.env API key override replaces the default secret expression in the models check step (GitHub Apps pattern)

[pelikhan]

@copilot you must also add support for the api base url

[Copilot]

@copilot you must also add support for the api base url

Done in 177a205. The models check step now reads ANTHROPIC_BASE_URL (Claude) and OPENAI_BASE_URL (Codex) from engine.env when configured. At runtime, the step uses a bash conditional to build the URL: ${BASE_URL_ENV_VAR%/}/models when the env var is set, falling back to the default endpoint otherwise. The env var (which may be a literal URL or a GitHub Actions expression) is passed explicitly to the step's env: block.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• https://api.github.com/graphql
  • Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE node (http block)
  • Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE sh (http block)
  • Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw GOMOD GOMODCACHE node /opt�� run lint:cjs 64/bin/go GOSUMDB GOWORK 64/bin/go sh (http block)
• https://api.github.com/repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b
  • Triggering command: /usr/bin/gh gh api /repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b --jq [.object.sha, .object.type] | @tsv GOSUMDB GOWORK 64/bin/go GOINSECURE GOMOD GOMODCACHE sh -c npx prettier --cGOINSECURE GOPROXY 64/bin/go GOSUMDB GOWORK 64/bin/go sh (http block)
  • Triggering command: /usr/bin/gh gh api /repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b --jq [.object.sha, .object.type] | @tsv "prettier" --cheGOINSECURE GOPROXY $name) { hasDiscussionsEnabled } } GOSUMDB GOWORK 64/bin/go git -c log.showsignaturGOINSECURE log 64/bin/go -n1 --format=format:-C --end-of-options/home/REDACTED/work/gh-aw/gh-aw/.github/workflows go (http block)
  • Triggering command: /usr/bin/gh gh api /repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b --jq [.object.sha, .object.type] | @tsv che/go-build/51/GOINSECURE GOPROXY 64/bin/go GOSUMDB GOWORK 64/bin/go /opt/hostedtoolcGOPROXY -o /tmp/go-build364GOSUMDB -trimpath 64/bin/go -p main -lang=go1.25 go (http block)
• https://api.github.com/repos/github/gh-aw
  • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw --jq .default_branch -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } GOINSECURE GOMOD GOMODCACHE go (http block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

[pelikhan]

@copilot you need to pass the env var to the steps so that it reads it

[Copilot]

@copilot you need to pass the env var to the steps so that it reads it

Fixed in 1aa8b23. The step now always includes the base URL env var in its env: block. When not explicitly set in engine.env, it maps from the GitHub Actions variable — e.g. ANTHROPIC_BASE_URL: ${{ vars.ANTHROPIC_BASE_URL || '' }} — so the bash conditional actually reads it.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• https://api.github.com/graphql
  • Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE node (http block)
  • Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE npx (http block)
  • Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE sh (http block)
• https://api.github.com/repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b
  • Triggering command: /usr/bin/gh gh api /repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b --jq [.object.sha, .object.type] | @tsv GOSUMDB GOWORK 64/bin/go GOINSECURE GOMOD GOMODCACHE npm run lint:cjs GOPROXY 64/bin/go GOSUMDB GOWORK 64/bin/go sh (http block)
  • Triggering command: /usr/bin/gh gh api /repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b --jq [.object.sha, .object.type] | @tsv prettier --check $name) { hasDiscussionsEnabled } } **/*.ts **/*.json --ignore-path go env GOPATH node 64/bin/go tierignore ../../../**/*.js-C 64/bin/go go (http block)
  • Triggering command: /usr/bin/gh gh api /repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b --jq [.object.sha, .object.type] | @tsv prettier --check 64/bin/go **/*.ts **/*.json --ignore-path go list�� -mod=readonly -e 64/bin/go tierignore ../../../**/*.jsapi 64/bin/go go (http block)
• https://api.github.com/repos/github/gh-aw
  • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw --jq .default_branch -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } GOINSECURE GOMOD GOMODCACHE go (http block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

[pelikhan]

@copilot make this step uses the same variables as the agent step. Including support for github apps.

[Copilot]

@copilot make this step uses the same variables as the agent step. Including support for github apps.

Done in 4627eef. The models check step now builds its env block the same way as the agent execution step:

1. Start with the route's default secret expression (e.g. ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }})
2. Set the base URL env var with a ${{ vars.X || '' }} fallback (so GitHub Actions variables are visible)
3. Merge all engine.env vars on top — this is where GitHub Apps support comes in: if the user sets ANTHROPIC_API_KEY: ${{ steps.get-app-token.outputs.token }} in engine.env, it overrides the default secret, and both the models check step and the agent step use the same dynamic credential
4. Merge agentConfig.Env for full parity with the agent step

The now-dead BaseURLEnvExpr field was also removed from ModelsRoute since engine.env is merged directly.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• https://api.github.com/graphql
  • Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE node (http block)
  • Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE npx (http block)
  • Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw GOMOD GOMODCACHE go env -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE sh (http block)
• https://api.github.com/repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b
  • Triggering command: /usr/bin/gh gh api /repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b --jq [.object.sha, .object.type] | @tsv GOSUMDB GOWORK 64/bin/go GOINSECURE GOMOD GOMODCACHE npm run lint:cjs GOPROXY 64/bin/go GOSUMDB GOWORK 64/bin/go sh (http block)
  • Triggering command: /usr/bin/gh gh api /repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b --jq [.object.sha, .object.type] | @tsv prettier --check $name) { hasDiscussionsEnabled } } **/*.ts **/*.json --ignore-path go env GOPATH node 64/bin/go tierignore ../../../**/*.js-C 64/bin/go go (http block)
  • Triggering command: /usr/bin/gh gh api /repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b --jq [.object.sha, .object.type] | @tsv prettier --check 64/bin/go **/*.ts **/*.json --ignore-path go list�� -mod=readonly -e 64/bin/go tierignore ../../../**/*.jsapi 64/bin/go go (http block)
• https://api.github.com/repos/github/gh-aw
  • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw --jq .default_branch -json GO111MODULE 64/bin/go GOINSECURE GOMOD GOMODCACHE go env -json GO111MODULE repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } GOINSECURE GOMOD GOMODCACHE go (http block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)