Run Jobs

An eval job rolls a coding agent out across a registry-resolved benchmark inside Harbor containers, capturing one trajectory and per-task evaluation artifacts per task. This section covers the full run path: selecting a benchmark, fronting your model with a LiteLLM proxy, and inspecting the outputs.

The whole flow is driven from a single config.yaml and executed by scripts/start.sh:

bash scripts/update_repos.sh   # clone/update Harbor at the pinned commit (first time)
bash scripts/dryrun.sh         # validate config, repo, envs, and registry entry
bash scripts/probe_llm_completion.sh  # validate the live upstream
bash scripts/start.sh          # generate proxy config, start proxy, run Harbor job, analyze

start.sh repeats the dry-run and completion gates, generates a per-job LiteLLM config, starts the proxy on the configured port, builds the Harbor command (--dataset <name>@<version> --registry-path repos/harbor/registry.json, plus --exclude-task-name flags from HARBOR_EXCLUDE_TASKS), runs the job under artifacts/jobs/, and then runs post-eval job analysis (unless disabled).

Select Benchmark — the registry-driven benchmark catalog and how to switch
LiteLLM Proxy — the per-job proxy in front of your model API
Results & Artifacts — where trajectories, results, analysis, and archives land

Proxy cleanup

start.sh stops its per-job LiteLLM process group in its EXIT trap. After a forced interruption, verify cleanup and stop only the process owned by this job; never kill unrelated LiteLLM or vLLM processes.