How to run Harbor evaluations in parallel cloud sandboxes

Harbor is a framework for evaluating agents and models on containerized tasks, from the creators of Terminal-Bench. It is also the official harness for Terminal-Bench 2.0. You give it a task or a dataset and an agent, and it runs trials and scores each one.

By default, Harbor runs each trial in a Docker container on your own machine. That's fine for one task. A full benchmark is different: many tasks, several attempts per task, and an agent that can spend minutes on each one. This guide shows how to run Harbor evaluations in the cloud instead, with every trial in its own Superserve sandbox, so you can run trials in parallel without a local Docker daemon.

Why run Harbor evaluations in the cloud

A local Docker daemon puts three limits on an evaluation:

• Your machine sets the concurrency. Every trial's container shares the host's CPUs and memory. Raise -n past what the host can hold, and trials slow each other down.
• Containers share a kernel. An evaluation is only as good as its isolation. If two trials share a kernel, a filesystem, or a network, one agent's mess can leak into another agent's score.
• You need Docker where you run Harbor. CI runners and dev laptops don't always have a Docker daemon, and the ones that do weren't sized for a benchmark.

Harbor's superserve environment removes all three. Each trial gets a Firecracker microVM of its own. Harbor sizes it from the task's cpus, memory_mb, and storage_mb, so the values in task.toml are both what the trial gets and the most it can use. When the trial ends, Harbor deletes the sandbox. Harbor itself only needs Python and network access.

Prerequisites

Step 1: Install Harbor with the Superserve environment

Install Harbor with the superserve extra, and export your Superserve API key:

bashCopy
uv tool install --python 3.13 "harbor[superserve]>=0.25.1.dev202611040613"
export SUPERSERVE_API_KEY=ss_live_...

Until the next stable Harbor release, the superserve environment ships in Harbor's nightly build, which is why the version starts at a nightly. Because the requirement names a nightly, uv allows a prerelease for Harbor only, and installs stable releases of everything else. --python 3.13 pins a Python version that all of Harbor's dependencies support.

Step 2: Check the setup with the oracle agent

Before you spend model tokens, run a task's reference solution with the oracle agent:

bashCopy
harbor run -t hello-world/hello-world -e superserve -a oracle

-e superserve is the only change from a local run. -t runs one task from the Harbor registry. The oracle doesn't call a model, so if this run fails, the problem is the setup, not the agent. When it passes, the summary shows one trial, no exceptions, and a mean reward of 1.000.

Step 3: Run a real agent

Now point Harbor at an agent and a model:

bashCopy
export ANTHROPIC_API_KEY=sk-ant-...
harbor run -t hello-world/hello-world -e superserve \
  -a claude-code -m anthropic/claude-haiku-4-5

Harbor installs the agent in the trial's sandbox, gives it the task's instructions, and runs the task's tests when the agent finishes. The trial's reward comes from those tests.

Step 4: Run trials in parallel

Two flags control the size of a run:

• -n sets how many trials run at the same time. The default is 4.
• -k sets how many attempts each task gets. The default is 1. More attempts give you a pass@k score and a steadier mean.

bashCopy
harbor run -t hello-world/hello-world -e superserve -a oracle -n 3 -k 3

To evaluate a whole benchmark, swap -t for -d <dataset>@<version>. For example, to run Terminal-Bench 2.0 on a sample of two tasks:

bashCopy
harbor run -d terminal-bench@2.0 -e superserve \
  -a claude-code -m anthropic/claude-haiku-4-5 -l 2 -n 2

-l caps the number of tasks, and -i and -x include or exclude tasks by name, with glob patterns. Start with a small sample to check cost and timing, then drop -l and raise -n for the full run. Use -p <path> to run your own task or dataset directory instead of one from the registry.

Since each trial runs in its own sandbox, your machine's CPUs and memory no longer limit -n. The limits that apply are your Superserve team's:

• Concurrent sandboxes. If Superserve refuses a sandbox because the team is at its limit, Harbor keeps retrying for up to 30 minutes, and the trial starts when another one finishes and frees a slot.
• Resources per sandbox. Each task's cpus, memory_mb, and storage_mb must fit your plan's resource limits. GPUs aren't supported.
• Template builds. One Harbor process builds at most four templates at a time. Change that with --ek max_concurrent_builds=..., and keep it at or below your team's build limit.

Step 5: Read the results

When a run ends, Harbor prints the number of trials, the number of exceptions, the mean reward, and how many trials got each reward. It also writes everything to a job directory under jobs/, named after the start time. Pass -o to use a different directory.

• jobs/<job>/result.json has the job's totals: completed and errored trials, the mean, pass@k when you ran more than one attempt, and which trials got which reward.
• jobs/<job>/<trial>/ holds one trial: its result.json, trial.log, the agent's logs in agent/, and the verifier's output in verifier/, including reward.txt and the test output in test-stdout.txt.

To browse trials and agent trajectories in a web UI, run:

bashCopy
harbor view jobs

If a run stops before it finishes, resume it from its job directory with harbor job resume -p jobs/<job>.

How task images become sandboxes

A Harbor task defines its environment with a container image, a Dockerfile, or a Compose file. Getting that environment ready is work you only want to do once per image, not once per trial.

If the task sets docker_image, Superserve builds a template from the image and boots each trial from it. The template is named harbor/<task>-<hash>, and later trials with the same image and resources reuse it. The first trial waits for the build, and later trials start from the cached template while it exists.

If the task has a Dockerfile or docker-compose.yaml, the trial boots from a Docker host template and builds the task's image or Compose project inside the sandbox. Multi-container Compose tasks run this way too, and tasks with the same resources share the Docker host template.

Some images can't boot as a sandbox, for example a distroless base. For those, Harbor runs the image with Docker inside the sandbox instead.

Enforce the task's network policy

Plenty of tasks only mean something if the agent can't look up the answer. Harbor tasks declare a network policy, and the superserve environment turns it into Superserve network rules:

• public adds no rules, so the sandbox can reach the public internet.
• no-network denies everything.
• allowlist allows the hosts you list, as exact hostnames, wildcards, IPv4 addresses, or CIDRs, and denies the rest. When the list has a hostname, Harbor also allows the sandbox's DNS resolvers.

Hostname entries cover HTTP and HTTPS on ports 80 and 443. If a task changes its policy partway through a trial, the change applies to new connections.

Keep model keys out of the sandbox

The agent you're evaluating needs a model key, and it also reads a lot of task content that someone else wrote. Normally the key sits in an environment variable inside the sandbox, where anything the agent runs can read it.

With Superserve Secrets, it doesn't have to. Create a team secret, then bind it to the trial's sandboxes with --ek:

bashCopy
harbor run -t hello-world/hello-world -e superserve \
  -a claude-code -m anthropic/claude-haiku-4-5 \
  --ek 'secrets={"ANTHROPIC_API_KEY": "anthropic-eval"}'

The agent sees a stand-in value. Superserve attaches the real key only to requests that go to the secret's allowed hosts, so the key never enters the sandbox.

Debug a failed trial

When a trial fails, you usually want to see what the agent left behind. Pass --no-delete, and Harbor pauses the sandbox instead of deleting it. A paused sandbox keeps its filesystem and isn't billed for compute. Superserve deletes it after seven days by default, and you can change that with --ek auto_delete_seconds=....

Troubleshooting and FAQ

Harbor says SUPERSERVE_API_KEY isn't set

Harbor checks for the key before it starts any trial. Export SUPERSERVE_API_KEY in the shell that runs harbor run.

harbor run fails with ModuleNotFoundError for a postgrest module

The install pulled prerelease builds of Harbor's dependencies, which happens when you install with --prerelease allow. Run uv tool uninstall harbor, then install again with the command in Step 1, which allows a prerelease for Harbor only.

Why is the first trial of a task slower than the rest?

It waits for the task's template to build. Later trials with the same image and resources reuse the template, so they skip the build.

Trials wait before they start

The team is at its concurrent sandbox limit, so Harbor waits for running trials to finish before it creates more sandboxes. Lower -n to stay under the limit.

Harbor deleted a harbor/ template

When the team reaches its template limit, Harbor deletes the oldest harbor/ template that no running trial uses, and logs a warning. The next run that needs it builds it again. Harbor keeps its templates between runs, so delete the ones you no longer need. The guide shows how.

Do I need Docker installed?

No. Harbor talks to the Superserve API, and every image build and container runs in Superserve. Tasks with a Dockerfile or Compose file use Docker inside their own sandbox.

Does the image's ENTRYPOINT run?

No. A sandbox booted from a template doesn't run the image's ENTRYPOINT or CMD. Commands run as the image's USER, or root if the image doesn't set one, unless Harbor asks for a specific user.

Next steps

The Harbor guide in the Superserve docs covers every environment option, how resources and network policies map, and how to clean up harbor/ templates. The environment's source is in the Harbor repo. If something doesn't work the way the guide says, open an issue.