Results & Artifacts

Everything a job produces lands under this block's artifacts/ directory. This page maps out what is written and where.

Per-job outputs

artifacts/jobs/<job>/
├── result.json                              # aggregate job stats (rewards, errors)
├── config.yaml                              # snapshot of the active config at launch
├── analysis/                                # post-eval job analysis (see below)
└── <task>/
    ├── agent/litellm-trajectory.jsonl       # one raw trajectory per task
    ├── result.json                          # trial status, reward, and exception
    └── verifier/                            # per-task verdict, scoring, test logs

• litellm-trajectory.jsonl is the replayable log of one trial, written by the LiteLLM logger.
• verifier/ holds the per-task grading output — verdict, scoring, and test logs that determine the reward.
• result.json at the job root aggregates rewards and error stats across all trials.

This directory is the block's eval_results_dir output (declared in config.yaml → runtime_info.output). eval is a terminal block, so there is no downstream consumer.

Analysis outputs

When job analysis runs (automatically after each eval, or manually via scripts/analyze_job.sh), it writes the files the dashboard reads:

artifacts/jobs/<job>/analysis/
├── report_failed.json / report_resolved.json   # primary/axis failure & resolve distributions
├── report_task_analysis.json                    # task difficulty tiers, domain/bug-type breakdowns
├── traj_analysis/score_comparison.json          # resolved vs unresolved metrics
├── instance_analysis/{summary,correlations}.json
├── instances.jsonl
└── analysis_config.yaml                         # self-contained config snapshot

See Job Analysis for the pipeline and its gold-dataset dependency.

Run archives

After each run, a snapshot is archived under:

artifacts/archives/run_NNN/
├── metadata.yaml   # run id, timestamps, status, job dir, and repo commit ids
├── config.yaml     # config as it was at run time
└── scripts/        # copy of executed scripts

and one entry is appended to artifacts/index.yaml when the run exits. Use it for completed/failed/interrupted archive history; live progress remains under the active job directory (see Status).

Cleaning up

scripts/clean.sh removes disposable logs, generated LiteLLM state, and other unrecognized entries. It preserves env//envs/, index.yaml, archives/, jobs/, datasets/, and runtime/. Pass -n/--dry-run to preview the deletions. It does not touch repos/; remove sensitive job/archive data explicitly before packaging a worktree.