Skip to content

FEAT: Runtime capability discovery for prompt targets#1699

Open
hannahwestra25 wants to merge 12 commits intomicrosoft:mainfrom
hannahwestra25:hawestra/query_target_capabilities
Open

FEAT: Runtime capability discovery for prompt targets#1699
hannahwestra25 wants to merge 12 commits intomicrosoft:mainfrom
hannahwestra25:hawestra/query_target_capabilities

Conversation

@hannahwestra25
Copy link
Copy Markdown
Contributor

@hannahwestra25 hannahwestra25 commented May 8, 2026

Description

Adds query_target_capabilities.py, which probes a PromptTarget at runtime to determine what the underlying endpoint actually accepts. Useful for custom OpenAI-compatible endpoints, gateways that strip features, or any deployment where declared capabilities may not match real behavior.

New public API (exported from pyrit.prompt_target)

  • query_target_capabilities_async — probes boolean capability flags (SYSTEM_PROMPT, MULTI_MESSAGE_PIECES, MULTI_TURN, JSON_OUTPUT, JSON_SCHEMA).
  • verify_target_modalities_async — probes which input-modality combinations are accepted.
  • verify_target_async — runs both and returns a populated TargetCapabilities.

Each probe is bounded by per_probe_timeout_s (default 30s) and retried once on transient errors. The target's configuration is temporarily replaced with a permissive one so _validate_request doesn't short-circuit. Probe-written memory rows are tagged with prompt_metadata["capability_probe"] == "1".

Caveats (in docstrings)

  • "Supported" means the request was accepted — silent ignores aren't detected.
  • Output modality probing is intentionally not provided.
  • Not safe to call concurrently against the same target instance.

Tests & docs

  • 32 unit tests, 98% coverage of the new module.
  • Notebook: 6_1_target_capabilities.ipynb.
  • New subsection in 0_prompt_targets.md.

)

# Probe a single dimension:
verified_caps = await query_target_capabilities_async(target=target)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be more intuitive IMO if it was target.get_capabilities() or (even better) target.capabilities (and similarly target.input_modalities / target.output_modalities) since these are static after instantiation (right?).

Having to import a function makes it a bit more obscure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants