A nowcasting RL task built in verifiers 0.4.0 has to close four routes to its own answer

Two January payroll numbers for one verifiers task

On 2 February 2024 the first release of US total nonfarm payrolls put January's gain at 353,000 jobs. The vintage of the same series that ALFRED served on 14 October 2026 puts it at 175,000.¹ ALFRED, the St. Louis Fed's vintage archive, keeps every version of a series. In the FRED API's terms, "the real-time period marks when facts were true or when information was known until it changed."² An environment that trains a language-model agent to nowcast payrolls, estimating a month's figure before its release, has to pick one of the two before it computes a reward.

verifiers is Prime Intellect's Python package for building such environments. Release 0.4.0 reached PyPI on 12 October 2026 under the MIT license, with the whole interface in one subpackage, imported as import verifiers.v1 as vf.³ An environment is an installable package. It can be published on the Environments Hub, a community registry,⁴ and trained with prime-rl, Prime's open-source asynchronous RL trainer.⁵ Hosted Training runs prime-rl on Prime's GPUs.⁶

The listings come from nowcast_alfred, twelve tasks built on ALFRED payroll vintages for 2024, and the eval config that ran it end to end against verifiers 0.4.0 on the subprocess runtime with a canned model that always answers 200. The training config was not run, and nothing was trained.

Four verifiers classes make up the nowcasting taskset

vf.TaskData is a frozen pydantic row holding the prompt, a network allowlist, timeouts, resources and the environment's own fields. vf.Task wraps one row and carries the hooks: an async def setup(self, trace, runtime), the reward and metric methods, and a toolsets classmethod that returns tool servers for each rollout. vf.Taskset has one abstract method, load(self) -> Iterable[TaskT], and vf.Toolset registers every @vf.tool method on an MCP server. The loader accepts an installed package whose __all__ exports exactly one Taskset subclass.³

class NowcastData(vf.TaskData):
    series_id: str
    as_of: str
    reference_month: str
    realized: float  # first release: hidden from the agent, read only by scoring
    realized_latest: float  # a later vintage, recorded as a metric for contrast
    baseline: float  # naive forecast: last change known on as_of
    scale: float  # std of the last 24 known monthly changes


class NowcastTaskset(vf.Taskset[NowcastTask, NowcastConfig]):
    def load(self) -> list[NowcastTask]:
        rows = json.loads(FIXTURE.read_text())["tasks"]
        return [
            NowcastTask(
                NowcastData(
                    id=f"{row['target_series']}@{row['as_of']}",
                    prompt=PROMPT.format(as_of=row["as_of"], month=row["reference_month"][:7]),
                    network_allow=[] if self.config.closed_egress else ["*"],
                    series_id=row["target_series"],
                    as_of=row["as_of"],
                    reference_month=row["reference_month"],
                    realized=row["realized_first"],
                    realized_latest=row["realized_latest"],
                    baseline=row["baseline"],
                    scale=row["scale"],
                ),
                self.config.task,
            )
            for row in rows
        ]

Twelve tasks are a fixture, not a training set. Prime defines group size as "Rollouts generated per task" and sets floors of 8 for group size and 64 for batch size.⁷ Batch size counts rollouts. The training page's knob table says tasks, but Prime's Wordle run counts 32 tasks of 16 rollouts as a batch of 512.⁸ At the floors one step draws 8 tasks, and a 100-step run draws 800, so each of twelve months would be seen about 67 times. A policy trained on one task learns its realized number. A method is learned only when it has to serve many series and as-of dates.

The nowcast reward scores the first release

Reward and metric hooks are decorated methods on the Task, and in 0.4.0 they must be coroutines. verifiers runs them through asyncio.gather, so a plain def returning a float raises TypeError: An asyncio.Future, a coroutine or an awaitable is required. Parameters are matched by name: task is the TaskData row, not the Task, trace is the rollout record and runtime is the agent's box. The Task's three type parameters name the row, the per-rollout State and the task config. The trace's reward is the sum of each score times its weight.³

FORECAST = re.compile(r"FORECAST:\s*(-?\d+(?:\.\d+)?)", re.IGNORECASE)


def parse(trace: vf.Trace) -> float | None:
    match = FORECAST.search(trace.last_reply)
    return float(match.group(1)) if match else None


class NowcastTask(vf.Task[NowcastData, CatalogState, NowcastTaskConfig]):
    @vf.reward(weight=1.0)
    async def beats_baseline(self, task: NowcastData, trace: vf.Trace) -> float:
        forecast = parse(trace)
        if forecast is None:
            return -1.0
        gain = abs(task.baseline - task.realized) - abs(forecast - task.realized)
        return max(-1.0, min(1.0, gain / task.scale))

    @vf.metric
    async def abs_error_latest(self, task: NowcastData, trace: vf.Trace) -> float:
        forecast = parse(trace)
        return float("nan") if forecast is None else abs(forecast - task.realized_latest)

    @vf.metric
    async def lookups(self, trace: vf.Trace) -> float:
        return float(trace.state.lookups)

The January task shows the arithmetic. The naive forecast of 216 missed the first release of 353 by 137, and the stub's 200 missed it by 153. The gain is minus 16, and divided by that task's scale of 163.3 it gives the reward of -0.098 the run recorded.

Whether a quantity belongs in the reward turns on one test: should the policy be pushed toward it? Error against the latest vintage depends on revisions nobody could know on the as-of date, so it is a metric. The lookup count describes how the agent worked, so it is a metric too. Metrics land in trace.metrics and never enter the reward.³

Over 2024, revisions moved the monthly change by 97.7 thousand jobs on average between first release and the October 2026 vintage. The naive forecast missed the first release by 104.3 thousand on average.¹ Scoring against the latest vintage would move the target about as far as the baseline misses it. The first release is a standard cut: the FRED API returns it with output type 4, "Observations, Initial Release Only".⁹ The figures cover one series over twelve months.

One forecast window is also a noisy reward. In October 2024 the first release showed a gain of 12 thousand against a naive forecast of 254 thousand.¹ Every rollout in a group faces that one draw, so the group's ranking says which forecast was closest that month and nothing about other months. A larger group sharpens the ranking on the draw. Only more tasks average over draws. Prime's group floor exists because when a group's rollouts all succeed or all fail, "the within-group advantage collapses to zero and the trainer learns nothing from that task".⁷ The floor applies to group-relative algorithms such as the default, GRPO.¹⁰ The stub run hit that case: its fixed forecast gave identical rewards within each group of two. The reward is a clipped, scale-normalized gain because a hit-or-miss threshold would tie more groups, a design judgment Prime's docs do not state.

What in verifiers 0.4.0 closes each route to the answer

A snapshot taken on 1 February 2024 ends at December 2023, at 157,232 thousand jobs. The release on 2 February added January at 157,700 and revised December to 157,347, so January's change of 353 sits in any catalog dated 2 February or later.¹ A task dated one day late is scored against a number its own catalog contains. The nowcast_alfred build dates each snapshot at month end and asserts that no snapshot contains its own reference month. The documented FRED API builds such a snapshot from realtime_start and realtime_end set to the as-of date, or from vintage_dates.⁹

    @classmethod
    def toolsets(cls, config: NowcastTaskConfig) -> list[vf.Toolset]:
        return [CatalogToolset(config.tools)]

    async def setup(self, trace: vf.Trace, runtime: vf.Runtime) -> None:
        # Stage the as-of snapshot only. The realized value never enters the box.
        row = next(
            t
            for t in json.loads(FIXTURE.read_text())["tasks"]
            if t["reference_month"] == self.data.reference_month
        )
        for sid, obs in row["snapshot"].items():
            lines = "date,value\n" + "".join(f"{d},{v}\n" for d, v in sorted(obs.items()))
            await runtime.write(f"catalog/{sid}.csv", lines.encode())

network_allow=[] does not mean no network. verifiers turns an empty allowlist into a policy that blocks everything except framework routes, and those include the model endpoint and every MCP tool URL. The policy is applied after task setup, so Task.setup runs with open egress and can fetch a snapshot itself. On the subprocess runtime the rollout fails with "task 0 requires a network policy, but the subprocess runtime does not support framework-aware policies". vf-eval recorded that as an error on the episode, with empty rewards, and still exited 0.³

class CatalogState(vf.State):
    lookups: int = 0


class CatalogToolset(vf.Toolset[vf.ToolsetConfig, CatalogState]):
    TOOL_PREFIX = "catalog"

    async def setup(self) -> None:
        self.fixture = json.loads(FIXTURE.read_text())
        self.snapshot: dict[str, dict[str, float]] = {}

    async def setup_task(self, task) -> None:
        # Serve only the vintage this task is dated at, never the realized value.
        row = next(
            t for t in self.fixture["tasks"] if t["reference_month"] == task.reference_month
        )
        self.snapshot = row["snapshot"]

    @vf.tool
    async def observations(self, series_id: str) -> dict[str, float]:
        """Return every observation of one series known on the task's as-of date."""
        self.state.lookups += 1
        return self.snapshot.get(series_id, {})


if __name__ == "__main__":
    CatalogToolset.run()

The tool server is the route the egress policy leaves open on purpose. setup runs once per server process, setup_task once per task, and self.state is the rollout's typed state, synced with trace.state on each call. By default a toolset is not colocated and runs as a subprocess on the env-worker host, with that host's network and files.³ A tool that queried FRED live would hand the agent today's vintage through that open route.

The answer file is the route the subprocess run actually opened. nowcast_alfred ships fixture.json, realized values included, inside its package. On subprocess the agent's box is the host, and a command run there printed "realized_first": 353.0 from the installed fixture. With colocated=True, verifiers installs the toolset's whole package, fixture and all, into the agent's runtime.³

The answer is hidden from the prompt and not from Prime. The realized value rides on every trace as part of the task data, and vf-eval uploads runs to Prime unless given --no-push or push = false.³¹¹

Pricing a verifiers rollout, then measuring one with vf-eval

A rollout under the default prime runtime gets a box with 1 CPU, 2 GB of memory and 5 GB of disk on python:3.11-slim, with a 3,600-second idle timeout. The first use of an image triggers an auto-build of about ten minutes.³ Prime's sandbox rates price exactly that box at $0.046 an hour, billed while it runs.¹² Prime publishes no per-rollout, token or tool-call cost, and its Hosted Training GPU-hour rate appears only in prime train models.¹³ Traces cost $1 per GiB ingested and $0.10 per GiB-month stored.¹⁴

A box that lives ten minutes costs $0.0077. A run the size of Prime's Wordle example, 100 steps at 512 rollouts a step, is 51,200 rollouts.⁸ At ten minutes a box, that is about $393 of sandbox time before any GPU, token or trace charge. Ten minutes is an assumption. The check that replaces it is a baseline vf-eval run of a few rollouts with the real harness and runtime. Each trace records its setup and agent stage times, and box-minutes times $0.046 / 60 is the sandbox cost per rollout.

model = "stub/model"
num_rollouts = 2
push = false

[select]
limit = 3

[client]
base_url = "http://127.0.0.1:8765/v1"
api_key_var = "STUB_KEY"

[env.taskset]
id = "nowcast-alfred"
closed_egress = false

[env.taskset.task.rewards]
beats_baseline = { weight = 1.0 }

[env.agent.harness]
id = "null"

[env.agent.runtime]
type = "subprocess"

num_rollouts is the group size, and [env.taskset] takes the taskset's own fields. The null harness serves MCP tools but cannot execute code. The default harness, bash, can, and an agent that fits its own model needs it.³ The bash harness was not run here.

A baseline eval decides the supervised warm start

Prime's end-to-end training guide says "a good training environment should have a baseline reward between roughly 10–80%".¹⁵ This reward runs from -1 to 1, and a reply without a FORECAST line scores -1. A base model that never writes the line puts every group at -1, where the advantage is zero and RL learns nothing.

Prime's examples reach for supervised fine-tuning (SFT) first when the base model fails on format. In the Wordle example the base model averaged a reward of about 0.2 with no wins, so the README says "Let's do some SFT warmup to get the model to learn the format of the environment."⁸ Reverse-text also starts with SFT, and alphabet-sort runs RL without it. DeepSeek-AI fine-tuned R1 on long chain-of-thought data "to prevent the early unstable cold start phase of RL training from the base model". R1-Zero skipped SFT and still learned to reason, with poor readability.¹⁶

The judgment for nowcast_alfred follows from the reward. Run the baseline eval with the bash harness and the base model. If most traces lack a FORECAST line, fine-tune on the output format with prime-rl's SFT entrypoint, part of what Prime's docs call an "SFT → RL → SFT" workflow.⁷ If the rewards already vary within groups, start with RL.

[ckpt]

[orchestrator]
batch_size = 64

[orchestrator.train]
group_size = 8

[[orchestrator.train.source]]

[orchestrator.train.source.env.taskset]
id = "nowcast-alfred"
closed_egress = true

[orchestrator.train.source.env.agent.harness]
id = "bash"

[orchestrator.train.source.env.agent.runtime]
type = "prime"

A training source's env field is the same verifiers env config vf-eval reads from eval.toml's [env] tables, so one taskset config serves the baseline and the run.³⁸ Training turns closed_egress back on and moves to the prime runtime, where the empty allowlist holds.

verifiers 0.4.0 has moved past Prime's own docs

Five final releases, 0.2.0 to 0.4.0, came between 10 July and 12 October 2026.¹⁷ The repository took 447 commits from 35 author names in the three months to 14 October.¹¹ The legacy v0 stack was removed on 31 August 2026, and 0.4.0 is the first release without it.¹⁸ Two days after 0.4.0, a pull request deleted vf-init, the scaffolding command 0.4.0 still ships. The 0.4.1.dev18 wheel already on PyPI lacks it.¹⁹²⁰

vf-eval itself prints "vf-eval will be deprecated - move to prime-rl's uv run eval" on every run.³

The package is maintained well enough to build on at a pinned version.

Training on this reward makes sense only after every snapshot predates its release, an isolating runtime enforces the empty allowlist, the box holds no answer file, the tools serve only the snapshot, and a baseline eval on the bash harness shows rewards that differ within groups.