Skip to content

Allow inline eval cases to carry explicit origin metadata.#2104

Merged
max-braintrust merged 2 commits into
mainfrom
max/use-inline-row-origin
Jun 10, 2026
Merged

Allow inline eval cases to carry explicit origin metadata.#2104
max-braintrust merged 2 commits into
mainfrom
max/use-inline-row-origin

Conversation

@max-braintrust

Copy link
Copy Markdown
Contributor

Previously, eval span origins were inferred only when running directly from a Dataset, using the dataset row id/xact metadata on each datum. Callers that resolve or transform rows before invoking Eval as inline data had no generic way to preserve the original source-row association.

This change lets an EvalCase provide an explicit origin, and keeps the existing dataset-derived origin inference as a fallback. That enables batched inline eval flows to preserve per-row provenance without changing behavior for normal dataset-backed evals.

Comment thread js/src/framework.ts
? evaluator.data
: undefined;
const origin =
datum.origin ??

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one question I have here:
is there some way we could do something like only take the object_type and object_id from the base row rather than the whole origin - since technically the origin id should always be the row id? That way we don't need to change the interface

Something like that could convert dataset_id on the base row -> object_type: dataset, object_id: dataset_id

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we'd still have to change the API surface to do something like this - right now we are passing these rows inline, so there is no dataset_id in the eval input:

z.object({ data: z.array(z.unknown()) }),

I don't think there is any reference to the base dataset in the existing shape that we could use - I could add that instead of the origin change, but then I think we are still basically implicitly changing the contract.

@max-braintrust max-braintrust merged commit 35b142d into main Jun 10, 2026
48 checks passed
@max-braintrust max-braintrust deleted the max/use-inline-row-origin branch June 10, 2026 22:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants