Score saved outputs and define judges

Current beta console·member+ applies; owner/admin defines native judges·Verified 2026-11-13 · Isolated local QA (code + Boolean model rubrics)Verification details

Add supplementary assessments to a completed evaluation without sending its prompts to the target again. Use code criteria for deterministic checks or a versioned model judge for an explicitly defined rubric. The original canonical exact-match results remain intact.

Before you start

Use a Completed run in a writable project in the current beta console. Members, admins and owners can apply scorers. Creating or versioning a custom judge requires owner or admin access.

Model judging additionally must be enabled in your organization’s environment and requires your explicit acknowledgement of provider charges. A disabled criterion labelled Requires operator setup cannot be enabled by acknowledging cost. Ask your administrator about availability. Reading or editing a judge definition does not itself invoke the model.

1. Apply a code criterion

1. Open Evals → Runs and select a completed run.
2. Click Score saved outputs.
3. Leave Suite version at Ad hoc criteria.
4. Select the code criteria you need: Exact match, Contains expected text, Valid JSON or Response length (characters).
5. Click Apply scorers once and wait for the saved-output job to finish.

Contains expected text, Valid JSON and Response length score retained answers. Model options require operator setup in this QA environment. Open full size ↗

Only the saved outputs are scored. Valid JSON is a syntax check, not your business schema. Contains expected text is a substring check, not semantic correctness. Response length counts Unicode characters. Choose criteria that actually measure the property you care about.

Inspect a result’s Assessments after the job completes. The teaching response READY contains its expected text and has five characters, while failing JSON syntax. Its original exact-match result still passes. The recorded native persistence state tells you whether the additional feedback has been retained.

READY contains the expected text and has five characters, but is not valid JSON. All three code feedback entries show MLflow persisted; the original exact-match result still passes. Open full size ↗

2. Add built-in model opinions

1. Open Score saved outputs on the completed run and leave Suite version at Ad hoc criteria.
2. Clear criteria you do not want, then select Correctness judge and Safety judge when they are available.
3. Read and select I acknowledge that model judging can incur provider charges.
4. Click Apply scorers once. Reopen Score saved outputs to follow the job, then wait for its terminal state.
5. Select the corresponding Assessment job and inspect a result’s Assessments. Read the value, explanation and Model opinion persistence state together.

The real local QA environment exposes both model judges. Only correctness and safety are selected, and the owner has acknowledged provider charges before admission. Open full size ↗

The local teaching example produced six values across three saved responses, with zero judgment errors. Correctness and safety were both rated true for these benign examples. Assessment identity retained the model/judge revision; the original exact-match outcomes stayed unchanged. These ratings describe this small sample, not the target’s general security.

The saved READY result has true safety and correctness opinions, explanations and persisted MLflow feedback. Assessment identity records the Bedrock GLM-5 judge revision. One pending feedback delivery was reconciled without requesting another judgment. Open full size ↗

If a saved value and explanation exist but their MLflow state is pending, use Sync saved feedback to reconcile that assessment’s delivery. Wait, refresh and reselect the original job. This does not request another model judgment. One assessment in the recorded example needed this action before all six native feedback records were retained.

3. Define a custom model rubric

1. As an owner/admin, open Evals → Judges and click New judge.
2. Enter a stable Judge name, such as answer_quality, and a useful Description.
3. Write Instructions describing what to judge from the saved input/output and expected value. For example:

Text

Assess whether {{ outputs }} answers {{ inputs }} with the required factual
content from {{ expectations }}. Return true only when the required answer is
present and the response does not contradict it. Explain the decision using
the saved text; do not assume external actions or unrecorded facts.

Copy

4. Choose Output type: boolean, numeric, categorical or text. For numeric output, set Minimum and Maximum; for categorical output, enter one category per line in Category options.
5. Click Save version 1. Reopen the judge and confirm the saved immutable version.

Owner enters the docs_answer_present Boolean rubric before Save version 1. Saving the definition invokes no judging model. Open full size ↗

A judge name begins with a lowercase letter and uses lowercase letters, digits or underscores. Keep the rubric specific and review it against examples before relying on its ratings. Saving a rubric does not establish judge accuracy.

4. Apply a particular judge version

1. Return to the completed run and open Score saved outputs.
2. Under Ad hoc criteria, select the custom judge. Use the version selector beside it to choose the intended version, including a historical version if needed.
3. If you choose Policy compliance judge, fill Policy guidelines.
4. Read and select I acknowledge that model judging can incur provider charges.
5. Click Apply scorers and wait for the job’s terminal state.
6. Select that Assessment job, inspect a result’s Assessments, and check its typed value, rationale and persistence state. Refresh and verify the same job and feedback remain recorded.

Create and apply v2

For the pictured differences, use the completed comparison-variant run with the saved outputs ready, hello and 4.. Your own responses can receive identical opinions when both rubrics accept them.

1. Open Evals → Judges, find the same judge, and choose Create next version.
2. Edit Description and Instructions. In this example, v2 requires the expected letter case and punctuation; v1 accepts the required answer despite those differences.
3. Click Save version 2. Reopen the judge to read Version history and confirm that both definitions remain available. Saving v2 does not apply it to a run.

The immutable judge name remains disabled. Owner prepares a stricter Boolean v2 rubric requiring expected case and punctuation, then saves it as a new version. Open full size ↗

The real Version history panel retains both Boolean definitions, their original instructions and capture-time creation dates. Saving v2 did not replace v1. Open full size ↗

4. Return to the same completed run and open Score saved outputs. Under Ad hoc criteria, select the custom judge and choose v2 in its adjacent version selector.
5. Read and select I acknowledge that model judging can incur provider charges. Then click Apply scorers once.
6. Wait for the new job to finish, select its Assessment job, and inspect the retained Assessments. Refresh and reselect that job to verify persistence.

V2 rejects the extra period while the earlier v1 accepts the required answer. Both native feedback records, rationales, jobs and definition digests remain visible. The identical engine-v2 source suffix is separate from each rubric version. Open full size ↗

The recorded v1 and v2 jobs produced three true and three false opinions on the same saved responses. Those ratings reflect different instructions. The target’s two canonical passes and one failure stayed unchanged; existing jobs retained their original rubric version.

Use a historical version

The catalogue checkbox label names the latest version. Use the adjacent version selector to pin an older one; the historical v1 remains selectable after v2 exists.

To score another completed run with that older rubric, repeat the application steps at the start of this section, selecting v1 before cost acknowledgement and Apply scorers. To inspect an existing v1 result, reselect its saved Assessment job.

The checkbox label identifies the latest catalogue version; the adjacent selector pins v1. This actual selection was admitted against baseline EV-1001 after v2 existed and retained three native v1 assessments. Open full size ↗

For a retained version record, select the relevant Assessment job, use Results → Export JSON, and inspect assessments[].provenance["martian.native_scorer_version"]. Assessment identity shows the job, definition digest and runtime source. A source suffix such as martian-eval-judge-v2 is the judging engine’s revision; it is separate from your rubric’s version number.

Interpret and recover

Boolean results can form pass rates; numeric, category and text outputs retain their own types. Coverage and errors matter alongside scores. A model opinion is separate from a verified security finding or independently observed business outcome.

Ad hoc judgments do not automatically define a passing gate. Unconfigured means no threshold policy was supplied, even if every Boolean rating is true. For built-in criteria, a scoring suite can supply a threshold and minimum coverage. Registered custom rubrics cannot be attached to suite versions in the current console; their jobs have no suite-threshold control. Review their typed values and coverage directly.

A scoring request supports one to seven unique criteria. One active scoring job is allowed per organization. Completed runs are required. If submission is unresolved, use Retry saved scoring request to inspect that same operation. Stop controls are cooperative and may allow the in-flight judgment to finish; see stopping and recovery.

If feedback delivery remains uncertain, retain the run and job identifiers for administrator review rather than rerunning paid work. Applying a different job or rubric does not overwrite the first job’s provenance.

Next, apply a reusable suite or collect human reviews.