From 9dbc17efd939b190e70b7f046fb78bfa404f4355 Mon Sep 17 00:00:00 2001 From: JonJagger Date: Tue, 12 May 2026 12:46:40 +0100 Subject: [PATCH] docs: rewrite evaluate_trails_with_opa tutorial to fix false-positive compliance footgun The previous tutorial used `allow if { count(violations) == 0 }` throughout. This pattern silently grants compliance when the violations rule body fails to fire -- for example when the attestation key is wrong. The kosli-public/cli flow names its pull-request attestation "pr", not "pull-request". Under the old policy, those trails were shown as ALLOWED not because PRs had approvers, but because the violations rule never matched and the empty set vacuously passed. The tutorial was demonstrating the exact footgun it should have warned against. Rewrites all policies to drive `allow` through positive assertions (`every`) rather than absence of violations. Parameterises the attestation name via --params so the policy works across orgs with different naming conventions. Explains the three design rules (fail-safe default, positive assertion, violations as diagnostics only) and adds a missing-param fail-safe test. Also fixes a Rego v1 strict-mode compile error: unused iteration variable replaced with `_`. Co-Authored-By: Claude Sonnet 4.6 --- tutorials/evaluate_trails_with_opa.mdx | 226 ++++++++++++++++++------- 1 file changed, 168 insertions(+), 58 deletions(-) diff --git a/tutorials/evaluate_trails_with_opa.mdx b/tutorials/evaluate_trails_with_opa.mdx index cb43b07..79d643e 100644 --- a/tutorials/evaluate_trails_with_opa.mdx +++ b/tutorials/evaluate_trails_with_opa.mdx @@ -1,11 +1,11 @@ --- title: "Evaluate trails with OPA policies" -description: "Learn how to use kosli evaluate trail and kosli evaluate trails to check your Kosli trails against custom OPA/Rego policies. This tutorial walks through writing a policy that verifies pull requests have been approved." +description: "Learn how to write safe OPA/Rego policies for kosli evaluate trail and kosli evaluate trails, including design rules that prevent false-positive compliance results." --- The `kosli evaluate` commands let you evaluate Kosli trails against custom policies written in [Rego](https://www.openpolicyagent.org/docs/latest/policy-language/). This is useful for enforcing rules like "every artifact must have an approved pull request" or "all security scans must pass", and for gating deployments in CI/CD pipelines based on those rules. -In this tutorial, we'll write a policy that checks whether pull requests on a trail have been approved, then evaluate it against real trails in public Kosli orgs. +In this tutorial, you'll write and evaluate policies against real trails in public Kosli orgs. Along the way, you'll learn three design rules that prevent a Rego policy from granting false-positive compliance results. @@ -22,7 +22,7 @@ To follow this tutorial, you need to: ``` -You don't need OPA installed — the Kosli CLI has a built-in Rego evaluator. You just need to write a `.rego` policy file. +You don't need OPA installed -- the Kosli CLI has a built-in Rego evaluator. You just need to write a `.rego` policy file. @@ -36,27 +36,58 @@ package policy import rego.v1 +pr_attestation_name := data.params.pr_attestation_name + default allow = false violations contains msg if { some trail in input.trails - some pr in trail.compliance_status.attestations_statuses["pull-request"].pull_requests + some pr in trail.compliance_status.attestations_statuses[pr_attestation_name].pull_requests count(pr.approvers) == 0 msg := sprintf("trail '%v': pull-request %v has no approvers", [trail.name, pr.url]) } +trail_is_approved(trail) if { + every pr in trail.compliance_status.attestations_statuses[pr_attestation_name].pull_requests { + count(pr.approvers) > 0 + } +} + +allow if { + every trail in input.trails { + trail_is_approved(trail) + } +} +``` + +This policy applies three design rules that every evaluate policy should follow. + +**Rule 1: `default allow = false` -- fail safe** + +Trails are denied unless the policy explicitly allows them. Anything the policy cannot positively verify is treated as non-compliant. This matches Kosli's compliance direction: a false non-compliant blocks a good trail (recoverable); a false compliant passes a bad one (not recoverable). + +The alias `pr_attestation_name := data.params.pr_attestation_name` reads the attestation name from a params file rather than hardcoding it. Different orgs and flows use different names for their pull-request attestation (for example `"pull-request"` or `"pr"`). If the param is absent, `pr_attestation_name` is undefined, the lookup into `attestations_statuses` fails, `trail_is_approved` does not fire, and `allow` stays `false` -- the correct fail-safe. + +**Rule 2: Drive `allow` via a positive assertion, not the absence of violations** + +`allow` fires through `trail_is_approved`, which makes a positive claim: every PR has at least one approver. It is never driven by `count(violations) == 0`. + +The following pattern looks equivalent but is not safe: + +```rego +# unsafe allow if { count(violations) == 0 } ``` -Let's break down what this policy does: +If the `violations` rule body references a field that does not exist -- a typo in `pull_requests`, an unexpected schema change, a missing key -- the rule body silently produces no messages. The violations set is empty, `count(violations) == 0` is true, and `allow` fires even though no PRs were actually checked. The trail receives a false-positive compliant result. + +With the safe pattern, if `pull_requests` is undefined, `every pr in ...` fails to evaluate, `trail_is_approved` does not fire, and `allow` stays `false`. -* **`package policy`** — every evaluate policy must use the `policy` package. -* **`import rego.v1`** — use Rego v1 syntax (the `if`/`contains` keywords). -* **`default allow = false`** — trails are denied unless explicitly allowed. -* **`violations`** — a set of messages describing why the policy failed. The rule iterates over trails, then over pull requests within the `pull-request` attestation, looking for PRs where `approvers` is empty. -* **`allow`** — trails are allowed only when there are no violations. +**Rule 3: Violations provide diagnostics only** + +`violations` explains why a policy was denied -- it does not decide whether it was denied. When a `violations` rule body encounters an undefined reference, it silently produces no message. This is the safe failure mode: you lose a diagnostic, not a compliance check. See the [Rego Policy reference](/policy-reference/rego_policy) for the full policy contract, input data shape, and exit code behaviour. @@ -71,6 +102,7 @@ Let's evaluate several trails from the public `cyber-dojo` org against our polic ```shell kosli evaluate trails \ --policy pr-approved.rego \ + --params '{"pr_attestation_name": "pull-request"}' \ --org cyber-dojo \ --flow dashboard-ci \ 9978a1ca82c273a68afaa85fc37dd60d1e394f84 \ @@ -89,11 +121,12 @@ VIOLATIONS: trail '5abd63aa1d64af7be5b5900af974dc73ae425bd6': pull-request http trail 'cb3ec71f5ce1103779009abaf4e8f8a3ed97d813': pull-request https://github.com/cyber-dojo/dashboard/pull/341 has no approvers ``` -Now try the `kosli-public` org, where PRs do have approvers: +Now try the `kosli-public` org, where PRs do have approvers. This org names the attestation `"pr"`: ```shell kosli evaluate trails \ --policy pr-approved.rego \ + --params '{"pr_attestation_name": "pr"}' \ --org kosli-public \ --flow cli \ 5a0f3c0 \ @@ -109,17 +142,30 @@ RESULT: ALLOWED -The `kosli evaluate trail` (singular) command evaluates facts within a single trail, which is a different use case from comparing across multiple trails. For example, you might check that a snyk container scan found no high-severity vulnerabilities. +The `kosli evaluate trail` (singular) command evaluates facts within a single trail. For example, you might check that a Snyk container scan found no high-severity vulnerabilities. Save this as `snyk-no-high-vulns.rego`: -```rego +```rego snyk-no-high-vulns.rego package policy import rego.v1 default allow = false +artifact_scan_is_clean(artifact) if { + snyk := artifact.attestations_statuses["snyk-container-scan"] + every result in snyk.processed_snyk_results.results { + result.high_count == 0 + } +} + +allow if { + every _, artifact in input.trail.compliance_status.artifacts_statuses { + artifact_scan_is_clean(artifact) + } +} + violations contains msg if { some name, artifact in input.trail.compliance_status.artifacts_statuses snyk := artifact.attestations_statuses["snyk-container-scan"] @@ -127,13 +173,9 @@ violations contains msg if { result.high_count > 0 msg := sprintf("artifact '%v': snyk container scan found %d high severity vulnerabilities", [name, result.high_count]) } - -allow if { - count(violations) == 0 -} ``` -This policy iterates over every artifact in the trail, looks up its `snyk-container-scan` attestation, and checks whether any result has a non-zero `high_count`. +`allow` fires only when `artifact_scan_is_clean` succeeds for every artifact. If `snyk-container-scan` is absent or `processed_snyk_results` is undefined, `artifact_scan_is_clean` fails to fire and `allow` stays `false`. Use `--attestations` to enrich only the snyk data (faster than fetching all attestation details). The value uses the format `artifact-name.attestation-type`. Here, `dashboard` is the artifact name and `snyk-container-scan` is the attestation name: @@ -161,81 +203,150 @@ The `input.trail` / `input.trails` distinction and the full input data shape are -Policies sometimes need configurable values — for example, a threshold that varies between environments. Instead of hardcoding these, use the `--params` flag to pass data into the policy as `data.params`. +Policies often need thresholds that vary by environment -- stricter in production than in staging. Use the `--params` flag to pass these values as `data.params` rather than hardcoding them in the policy. -Save this as `check-threshold.rego`: +Save this as `snyk-severity-threshold.rego`: -```rego check-threshold.rego +```rego snyk-severity-threshold.rego package policy import rego.v1 -default allow := false +max_high := data.params.max_high +max_medium := data.params.max_medium + +default allow = false + +artifact_within_threshold(artifact) if { + snyk := artifact.attestations_statuses["snyk-container-scan"] + every result in snyk.processed_snyk_results.results { + result.high_count <= max_high + result.medium_count <= max_medium + } +} + +allow if { + every _, artifact in input.trail.compliance_status.artifacts_statuses { + artifact_within_threshold(artifact) + } +} + +violations contains msg if { + some name, artifact in input.trail.compliance_status.artifacts_statuses + snyk := artifact.attestations_statuses["snyk-container-scan"] + some result in snyk.processed_snyk_results.results + result.high_count > max_high + msg := sprintf("artifact '%v': %d high-severity vulnerabilities exceed limit of %d", [name, result.high_count, max_high]) +} + +violations contains msg if { + some name, artifact in input.trail.compliance_status.artifacts_statuses + snyk := artifact.attestations_statuses["snyk-container-scan"] + some result in snyk.processed_snyk_results.results + result.medium_count > max_medium + msg := sprintf("artifact '%v': %d medium-severity vulnerabilities exceed limit of %d", [name, result.medium_count, max_medium]) +} +``` -default threshold := 10 +**Why the aliases at the top matter** -threshold := data.params.threshold if { data.params.threshold } +`max_high := data.params.max_high` is not just shorthand. In the compliance path, `result.high_count <= max_high` is a positive bound check. If `max_high` is absent from the params file, this condition is undefined, `artifact_within_threshold` fails to fire, and `allow` stays `false`. That is the correct fail-safe behaviour. -allow if { input.score >= threshold } +Compare this to a policy that drives `allow` through the absence of violations: +```rego +# unsafe violations contains msg if { - input.score < threshold - msg := sprintf("score %d is below threshold %d", [input.score, threshold]) + result.high_count > data.params.max_high # fails silently if max_high is absent + msg := ... } + +allow if { count(violations) == 0 } ``` -This policy: +If `data.params.max_high` is absent, the `violations` rule body fails silently, the set stays empty, and `allow` fires. A misconfigured params file grants compliance rather than denying it. + +**Testing with `kosli evaluate input`** -* Defines a **default threshold** of `10` — used when no `--params` are provided. -* Overrides the threshold with `data.params.threshold` when present. -* Allows the input only if `input.score` meets the threshold. +You can test a policy locally without a real trail using `kosli evaluate input`. Create a minimal input file: + +```shell +cat > scan-input.json << 'EOF' +{ + "trail": { + "compliance_status": { + "artifacts_statuses": { + "dashboard": { + "attestations_statuses": { + "snyk-container-scan": { + "processed_snyk_results": { + "results": [{"high_count": 2, "medium_count": 5}] + } + } + } + } + } + } + } +} +EOF +``` -You can test this locally with `kosli evaluate input`. First, create an input file: +Evaluate with permissive staging thresholds: ```shell -echo '{"score": 5}' > score-input.json +kosli evaluate input \ + --input-file scan-input.json \ + --policy snyk-severity-threshold.rego \ + --params '{"max_high": 5, "max_medium": 10}' +``` + +```plaintext +RESULT: ALLOWED ``` -Evaluate without params (uses the default threshold of `10`): +Apply stricter production thresholds: ```shell kosli evaluate input \ - --input-file score-input.json \ - --policy check-threshold.rego + --input-file scan-input.json \ + --policy snyk-severity-threshold.rego \ + --params '{"max_high": 0, "max_medium": 3}' ``` ```plaintext RESULT: DENIED -VIOLATIONS: score 5 is below threshold 10 +VIOLATIONS: artifact 'dashboard': 2 high-severity vulnerabilities exceed limit of 0 + artifact 'dashboard': 5 medium-severity vulnerabilities exceed limit of 3 ``` -Now pass a lower threshold via `--params`: +Now verify the fail-safe: omit `max_high` from params entirely: ```shell kosli evaluate input \ - --input-file score-input.json \ - --policy check-threshold.rego \ - --params '{"threshold": 3}' + --input-file scan-input.json \ + --policy snyk-severity-threshold.rego \ + --params '{"max_medium": 10}' ``` ```plaintext -RESULT: ALLOWED +RESULT: DENIED ``` -The score of `5` is now above the threshold of `3`, so the policy allows it. +`allow` is `false` even though no violation message was produced. The missing param causes the compliance check to fail, not to vacuously pass. Always verify this explicitly when writing a policy that relies on params. You can also load parameters from a file using the `@` prefix: ```shell -echo '{"threshold": 3}' > params.json +echo '{"max_high": 0, "max_medium": 3}' > params-prod.json kosli evaluate input \ - --input-file score-input.json \ - --policy check-threshold.rego \ - --params @params.json + --input-file scan-input.json \ + --policy snyk-severity-threshold.rego \ + --params @params-prod.json ``` -The `--params` flag works the same way on `kosli evaluate trail` and `kosli evaluate trails` — parameters are always available as `data.params` in the policy. +The `--params` flag works the same way on `kosli evaluate trail` and `kosli evaluate trails` -- parameters are always available as `data.params` in the policy. @@ -286,9 +397,7 @@ Use the `--attestations` flag to limit which attestations are enriched with full -The `kosli evaluate` commands exit with `0` on allow and `1` on deny or error — making them straightforward to use as pipeline gates. See the [Rego Policy reference](/policy-reference/rego_policy#exit-codes) for details on distinguishing denial from command failure. - - +The `kosli evaluate` commands exit with `0` on allow and `1` on deny or error -- making them straightforward to use as pipeline gates. See the [Rego Policy reference](/policy-reference/rego_policy#exit-codes) for details on distinguishing denial from command failure. ```shell # Example: gate a deployment on policy evaluation @@ -297,10 +406,10 @@ if kosli evaluate trail \ --org "$KOSLI_ORG" \ --flow "$FLOW_NAME" \ "$GIT_COMMIT"; then - echo "Policy passed — proceeding with deployment" + echo "Policy passed -- proceeding with deployment" # ... deploy commands ... else - echo "Policy denied — blocking deployment" + echo "Policy denied -- blocking deployment" exit 1 fi ``` @@ -313,7 +422,7 @@ This pattern lets you enforce custom compliance rules as part of your delivery p After evaluating a trail, you can record the result as an attestation. This creates an audit record in Kosli that captures the policy, the full evaluation report, and any violations. -This step requires write access to your Kosli org. The examples below use variables you'd set in your CI/CD pipeline. In your own pipeline you'd use your own policy file — here we use `my-policy.rego` as a placeholder: +This step requires write access to your Kosli org. The examples below use variables you'd set in your CI/CD pipeline. In your own pipeline you'd use your own policy file -- here we use `my-policy.rego` as a placeholder: ```shell # Run the evaluation and save the full JSON report to a file @@ -326,7 +435,7 @@ kosli evaluate trail "$TRAIL_NAME" \ --output json > eval-report.json 2>/dev/null || true # Read the allow/deny result from the report -is_compliant=$(jq -r '.allow' eval-report.json) +is_compliant=$(jq --raw-output '.allow' eval-report.json) # Extract violations as structured user-data jq '{violations: .violations}' eval-report.json > eval-violations.json @@ -344,12 +453,12 @@ kosli attest generic \ This creates a generic attestation on the trail with: -* **`--compliant`** set based on whether the policy allowed or denied — read directly from the JSON report rather than relying on the exit code, which avoids issues with `set -e` in CI environments like GitHub Actions +* **`--compliant`** set based on whether the policy allowed or denied -- read directly from the JSON report rather than relying on the exit code, which avoids issues with `set -e` in CI environments like GitHub Actions * **`--attachments`** containing the Rego policy (for reproducibility) and the full JSON evaluation report (including the input data the policy evaluated) * **`--user-data`** containing the violations, which appear in the Kosli UI as structured metadata on the attestation -Use `--compliant=value` (with `=`) not `--compliant value` (with a space). Boolean flags in Kosli CLI require the `=` syntax when passing `false` — otherwise `false` is interpreted as a positional argument. +Use `--compliant=value` (with `=`) not `--compliant value` (with a space). Boolean flags in Kosli CLI require the `=` syntax when passing `false` -- otherwise `false` is interpreted as a positional argument. @@ -358,9 +467,10 @@ Use `--compliant=value` (with `=`) not `--compliant value` (with a space). Boole ## What you've accomplished -You have written OPA/Rego policies and evaluated Kosli trails against them, both across multiple trails and within a single trail. You've also recorded evaluation results as attestations, creating a tamper-proof audit record of every policy decision linked to a specific trail. +You have written OPA/Rego policies using the three design rules that prevent false-positive compliance results: fail-safe default, compliance via positive assertion, and violations as diagnostics only. You've evaluated Kosli trails against those policies, tested safety properties locally with `kosli evaluate input`, and recorded evaluation results as attestations. From here you can: * Explore evaluated trails in the [Kosli app](https://app.kosli.com) * Gate deployments in CI/CD pipelines using `kosli evaluate trail` exit codes +* Use environment-specific params files to enforce different thresholds per environment * Extend your policies to check other attestation types. See [`kosli evaluate trail`](/client_reference/kosli_evaluate_trail) and [`kosli evaluate trails`](/client_reference/kosli_evaluate_trails) for the full flag reference