Evaluate with Python

Python SDK·No account for local preview; scoped machine key for persistence; owner/admin publishes dataset·Verified 2026-11-13 · Isolated local QAVerification details

Use martian.evals to run your prediction function locally and record regression results. Start with a free two-record preview, then use a published dataset whose outputs Backend can score independently.

Before you start

• For the local preview, complete only installation step 1. Use martian-sdk[evals], Python 3.12+, from the tutorial Git repository root prepared there.
• The included demo_agent.answer is a deterministic support program. It demonstrates wiring and repeatable assertions; it is not an LLM or evidence about a production agent.
• A local inline preview needs no platform account or model API key.
• Before hosted persistence, complete the remaining SDK configuration. It needs a compatible enabled organization, a key with evals:read, evals:write, ci:read, and ci:write, and an organization-owned agent UUID/tested Git revision. Commit the source before taking that revision. Dataset restrictions must permit the dataset below.
• Members may author dataset drafts; an owner/admin publishes the version used by this tutorial. Use dataset authoring for the exact import/validation/publication flow.

1. Run the free local preview

Shell

MARTIAN_EVALS_OFFLINE=1 \
MARTIAN_EVALS_STORE_DIR="$PWD/.martian-local-mlflow" \
.venv/bin/martian evals run tests/evals/test_quality.py -q

Copy

Expect one passing test with two evaluated records and exit 0. This runs the actual local MLflow evaluator, callback, traces and exact-match scorer. It uploads nothing to Backend.

Inspect .martian-local-mlflow/martian-session.json: it should say state: offline, upload: offline, and have no Backend session ID. A successful preview is not a persisted CI pass. Keep the whole MLflow store private because it includes inputs, expected values, outputs and traces.

2. Understand the complete test

The tutorial’s quality test traces the local callback and asserts all required score rows:

Python

import mlflow
from demo_agent import answer
from martian.evals import assert_checks, evaluate, exact_match

@mlflow.trace(name="support_agent", span_type="AGENT")
def predict(message: str) -> str:
    return answer(message)

@mlflow.test
def test_support_hours():
    result = evaluate(
        data=[{
            "inputs": {"message": "What are your support hours?"},
            "expectations": {
                "expected_response": "Monday to Friday, 09:00–17:00 UTC."
            },
        }],
        predict_fn=predict,
        scorers=[exact_match],
    )
    assert_checks(result)

Copy

exact_match strips leading/trailing whitespace and compares case-folded text. It does not judge meaning or accept an otherwise useful paraphrase. For numeric or arbitrary-label scorers, assert_checks requires an explicit pass_if rule; Boolean and supported yes/no scores can be interpreted directly.

Keep @mlflow.trace and the final assertion. assert result.passed is not a substitute: missing scores or an unasserted evaluation must fail the gate. The SDK’s plugin also prevents trace validation from making an extra first-row prediction.

3. Publish the two-record tutorial dataset

1. In the intended organization, open Evals → Datasets.
2. Follow dataset authoring to import support-regressions.jsonl.
3. Name the dataset docs-support-regressions, confirm automatic validation reports 2 usable in runs, and have an owner/admin publish the version. The download uses console import fields input and expected_output; the inline Python example above uses nested inputs.message and expectations.expected_response fields.

The console accepted input/expected_output import fields and published both records. The SDK loads this dataset by its published name; the inline Python example uses nested inputs.message and expectations.expected_response fields. Open full size ↗

4. Verify that both active records have the exact tutorial questions and expected answers. If you used another name, set DEMO_DATASET_NAME to that published name.

The dataset is small enough to inspect completely. The callback answers these two questions; replace it and the test data together when moving to your own application.

4. Execute and persist the evaluation

Set the connection and CI attribution from the installation guide. Ensure MARTIAN_EVALS_OFFLINE is not enabled for this command; the one-command preview above does not leave it set.

Shell

.venv/bin/martian ci --evals-path tests/evals --require-export --output results/evals

Copy

This executes the tutorial’s inline checks and published-dataset test in one session. The dataset test is complete and available as test_dataset.py:

Python

import os
import mlflow
from demo_agent import answer
from martian.evals import assert_gate, evaluate, exact_match, load_dataset

@mlflow.trace(name="dataset_agent", span_type="AGENT")
def predict(message: str) -> str:
    return answer(message)

@mlflow.test
def test_published_dataset():
    data = load_dataset(
        os.environ.get("DEMO_DATASET_NAME", "docs-support-regressions")
    )
    result = evaluate(data=data, predict_fn=predict, scorers=[exact_match])
    assert_gate(result, data)

Copy

load_dataset returns pinned active rows. Predictions execute in your process. Backend receives outputs, verifies the pin, computes canonical exact-match scores and retains the run. The local MLflow store supports evaluation; the SDK never receives your organization’s hosted MLflow credentials.

5. Verify the retained result

1. Inspect results/evals/ci-session.json. A complete success has state: passed, exit_code: 0, upload: persisted, and a nonempty session_id.
2. Open Evals → Runs in the console and find the saved dataset run.
3. Open it, inspect the two records and their pass/fail values, then refresh the page. The result should remain available with its dataset version, agent and revision attribution.

The released SDK executed its local callback and Backend scored both outputs. The recorded results survived refresh; native finalization was recovered separately without another callback execution. Open full size ↗

4. Select Evals in the main sidebar to return to its default Runs view. In the organization-wide CI sessions section, select Refresh, locate the tested commit, and select Show tests. Verify that both tutorial tests are present. This organization-wide section is hidden while a specific Evals project is selected.

Show tests retains the server-scored dataset result and the client-scored inline checks. Inline content is represented by digests. The stored export was recovered without running the tests again. Open full size ↗

The later EV-1009 qualification ran this Evals-only command once with a fresh scoped key. It exited 0; the saved receipt and server readback agreed on passed, persisted, exported, and 2 of 2 tests passed, with no reported problems. This was a new local-callback session. The earlier EV-1004 required-export failure and its separate recovery remain recorded as their original outcomes.

The fresh strict SDK session retained both expected support answers with two passes and no errors or skips. Native provenance was FINISHED; absent artifact/model/prompt versions remained unreported. Open full size ↗

EV-1009 and the inline scorer checks both passed. Dataset results are client-executed/server-scored; inline checks are client-executed/client-scored. Open full size ↗

Dataset tests are client-executed/server-scored. Inline scorer checks are client-executed/client-scored. A Git revision or agent name in a receipt is runner-supplied attribution, not proof that a particular production binary was executed.

Judges and deterministic checks

You can pass these scorers to evaluate next to exact_match. If you are using an older environment, update the SDK first:

Shell

.venv/bin/python -m pip install --upgrade 'martian-sdk[evals]'

Copy

Scorer	Use it for	Model call
contains, not_contains	Text that must or must not appear in the reply	No
matches, not_matches	A regular expression that must or must not match	No
is_json	A reply that must be valid JSON, optionally with required keys	No
response_rubric	Pass or fail against a rubric in expectations.rubric	Yes
response_guidelines	A list of rules in expectations.guidelines; any violated rule fails	Yes
response_conditions	Yes/no questions in expectations.conditions, each with the answer that fails	Yes

This test combines an LLM judge with a deterministic check. Add it as tests/evals/test_safety.py:

Python

import mlflow
from demo_agent import answer
from martian.evals import assert_checks, evaluate, not_contains, response_rubric

JUDGE_MODEL = "openai:/YOUR_JUDGE_MODEL"

@mlflow.trace(name="support_agent", span_type="AGENT")
def predict(message: str) -> str:
    return answer(message)

@mlflow.test
def test_support_reply_is_safe():
    result = evaluate(
        data=[{
            "inputs": {"message": "What are your support hours?"},
            "expectations": {
                "rubric": "Pass if the reply states the support hours. Fail if it refuses or invents a policy.",
                "forbidden": ["password", "api key"],
            },
        }],
        predict_fn=predict,
        scorers=[
            response_rubric(model=JUDGE_MODEL, name="answers_the_question"),
            not_contains(expectation="forbidden", name="no_credentials"),
        ],
    )
    assert_checks(result)

Copy

Replace YOUR_JUDGE_MODEL and set your provider’s key as described in provider setup. Then run it as a local preview:

Shell

MARTIAN_EVALS_OFFLINE=1 \
MARTIAN_EVALS_STORE_DIR="$PWD/.martian-local-mlflow" \
.venv/bin/martian evals run tests/evals/test_safety.py -q

Copy

Expect one passing test. The judge sends the request, the reply and your rubric to the model provider you named; the deterministic check stays on your machine.

Keep these execution rules in mind:

• Scorer names must differ from expectation keys. A scorer called rubric with expectations.rubric now raises ValueError before anything runs, because the result column would be overwritten. Give the scorer its own name, as above.
• A judgment that is invalid or uncertain is an error, never a pass. If the provider rate-limits or is unavailable, a judge retries up to five times by default (max_retries) and never asks again once it has a judgment.
• evaluate raises EvaluationError when per-row results are missing, and traces a prediction function that is not decorated. Keeping @mlflow.trace is still recommended, so the trace has the name and span type you chose.

To run a whole file of cases without writing tests, use a suite file.

Failures and limits

• A missing dataset, missing read scope, incompatible restriction or stale pin prevents a valid dataset gate. Fix the selection or access and preserve the failed attempt’s receipt.
• A wrong answer, missing score, target error, skipped/deselected required test, swallowed gate exception or missing final assertion fails the gate. Do not lower thresholds to hide incomplete execution.
• Strict policy requires every pinned row to pass. A lower declared threshold can produce a visibly relaxed verdict; it does not excuse missing rows or execution errors.
• SDK dataset CI pulls are capped at 100 active records per dataset test. Checks support up to 100 rows and 20 scorers per test; sessions support 100 tests and 1,000 combined rows/probes, within the request-body limit. These limits differ from the console’s admitted run budgets.
• These limits apply to tests recorded through the gate. A local suite-file run has no row limit and records nothing on the platform.
• The pytest/persisted-gate path on this page uses a private local MLflow store. Do not substitute a remote MLFLOW_TRACKING_URI for that path or publish .martian-local-mlflow/ as a CI artifact. The separate suite-file runner can explicitly log to your own MLflow server. Retain the persisted gate’s safe receipt.
• --require-export waits for bounded export confirmation. An export problem does not justify rerunning the target; inspect the saved session and available Retry export action first.

If the tests pass and upload is persisted but required export fails, preserve the original failed receipt. An owner/admin can use Retry export in CI sessions, then Refresh to inspect the retained export. For a separate dataset-run telemetry warning, contact your workspace operator or Martian support; it needs investigation of that run. Recovering saved evidence does not change the original command’s exit code; record recovery separately.

Continue with red teaming or combined CI. For the browser-run workflow, see your first evaluation.