Allow inline eval cases to carry explicit origin metadata.#2104
Conversation
| ? evaluator.data | ||
| : undefined; | ||
| const origin = | ||
| datum.origin ?? |
There was a problem hiding this comment.
one question I have here:
is there some way we could do something like only take the object_type and object_id from the base row rather than the whole origin - since technically the origin id should always be the row id? That way we don't need to change the interface
Something like that could convert dataset_id on the base row -> object_type: dataset, object_id: dataset_id
There was a problem hiding this comment.
I think we'd still have to change the API surface to do something like this - right now we are passing these rows inline, so there is no dataset_id in the eval input:
I don't think there is any reference to the base dataset in the existing shape that we could use - I could add that instead of the origin change, but then I think we are still basically implicitly changing the contract.
Previously, eval span origins were inferred only when running directly from a
Dataset, using the dataset row id/xact metadata on each datum. Callers that resolve or transform rows before invokingEvalas inline data had no generic way to preserve the original source-row association.This change lets an
EvalCaseprovide an explicitorigin, and keeps the existing dataset-derived origin inference as a fallback. That enables batched inline eval flows to preserve per-row provenance without changing behavior for normal dataset-backed evals.