The Nightly Build

Muon's fine-tuning lead runs out after a similar token count at every size tested

2026-11-04T06:54:11Z

Feature

Muon's fine-tuning lead runs out after a similar token count at every size tested

Across a CMU preprint's 30M-to-600M models, AdamW pretraining takes the lead between 2.4 and 19.2 billion tokens, and production models pretrained with Muon have seen trillions.

5 min read2026-11-04

Muon pretrains better and still loses the fine-tuning trade

In a CMU preprint covering about 600 pretraining runs from 30M to 600M parameters, Muon reaches lower pretraining loss than AdamW in every cell. Past a crossover budget, though, the AdamW-pretrained model learns a fine-tuning task with less forgetting.¹ The authors report that crossover as a token-to-parameter ratio that falls as models grow. Counted in tokens, all 20 of their crossovers sit between 2.4 and 19.2 billion. Every Muon-pretrained production model with published counts has seen about 300 times as many or more, at sizes the paper never ran.

Where AdamW's frontier pulls ahead

The paper fine-tunes each optimizer's lowest-pretraining-loss 60M model on StackMathQA across a sweep of learning rates. Each run gives a fine-tuning loss and a forgetting loss on held-out DCLM text, and the lower-left hull of those points is the frontier.¹

Fig. 1 · Learning–forgetting frontiers of the lowest-loss 60M base models on StackMathQA, in nats. Lower-left is better. Preprint Figure 1.¹

At 2.4 billion pretraining tokens Muon's curve sits below AdamW's along its whole length. By 9.6 billion the curves have swapped. The crossover is the smallest budget on the paper's doubling grid where AdamW's frontier wins, here 4.8 billion tokens, or 80 per parameter. Muon-pretrained models are fine-tuned with both optimizers and the runs pooled, so the fine-tuning optimizer does not handicap them.¹

The authors explain the reversal with a margin-to-spread ratio: the correct logit's mean lead over the wrong ones, divided by the wrong logits' spread. The larger it is, the more wrong answers stay below the right one when fine-tuning perturbs the outputs, and AdamW's overtakes Muon's as the budget grows. Their theorem compares Muon with gradient descent, not AdamW, and covers the budget reversal but not the fall with size.¹

Twenty crossovers inside one token band

Fig. 2 · The preprint's Table 2 crossovers, per parameter and in tokens, on log scales. Points are offset sideways. The ringed 600M GSM8K value is the smallest budget run, so the true crossover is at or below it.¹

StackMathQA's crossover is 320 tokens per parameter at 30M and 10 at 600M.¹ In tokens those are 9.6 and 6.0 billion. A least-squares fit of log crossover ratio against log size over all 20 cells has a slope of −0.97, and a crossover fixed in tokens would give exactly −1. The paper does not make this observation.

The fit is not tight. From 30M to 600M the ratio falls 8-fold on Alpaca and 64-fold on MusicPile, and Alpaca's token crossover rises from 4.8 to 12 billion. Each value is a point on a doubling grid, from one base model per optimizer, with no seeds or error bars.¹ Across the 20 cells the ratios vary 128-fold and the token counts 8-fold.

At the Chinchilla-optimal 20 tokens per parameter, a 600M model has seen 12 billion tokens and is at or past its crossover on all four sets.¹

Production models, on either parameter count

Kimi K2, a mixture-of-experts model, was pretrained on 15.5 trillion tokens: 475 per active parameter and 15 per total parameter.²

Fig. 3 · Measured crossovers¹ against published counts for Moonlight,³ Kimi Linear,⁴ Trinity,⁵ Kimi K2,² GLM-4.5⁶ and GLM-5,⁷ on log scales. Ratios are our computation.

On the ratio scale, K2's total-parameter ratio falls below the 600M Alpaca crossover of 20, the only placement the convention changes. On the token scale the convention drops out. Every model sits between 5.7 and 28.5 trillion tokens, about 300 to 1,500 times the largest measured crossover.

None of these placements is tested. The production models are mixture-of-experts, with up to 67 times the active parameters of the paper's largest dense model. The preprint names Kimi K2 and GLM-4.5 as Muon-pretrained but says of scale only that the falling crossover “suggests that the problem persists as we scale.”¹

Moonlight's twin runs point the other way

Moonshot AI's report on Moonlight has the one matched comparison at production scale. Moonlight-A “follows the same training settings as Moonlight, except that it uses the AdamW optimizer.” Both 1.2-trillion-token checkpoints were fine-tuned with each optimizer.³

Moonlight (2.24B active, 15.29B total) after SFT, from Table 6 of its report.³
Pretrain / SFT	MMLU	HumanEval	MBPP	GSM8K
Muon / Muon	55.7	57.3	55.6	68.0
AdamW / Muon	55.3	53.7	55.5	62.1
Muon / AdamW	50.2	52.4	55.2	64.9
AdamW / AdamW	52.0	53.1	55.2	64.6

Paired the way the preprint pairs them, AdamW with AdamW and Muon with its best, the Muon pipeline wins all four benchmarks at 62 times the top of the token band. With AdamW fine-tuning on both, the result splits. The test has one run per arm and measures no forgetting. Its checkpoints also predate cooldown, and Wen and colleagues found that optimizer rankings at intermediate checkpoints can flip once the learning rate decays.⁸

The preprint, a single version-1 release, sweeps only the peak learning rate and fixes weight decay at 0.1 for both optimizers. Its authors hypothesize that the falling crossover comes partly from Muon's shrinking pretraining advantage,¹ which a second paper from Wen's group observes “when using standard constant decoupled weight decay,” the preprint's setting.⁹

For dense models from 30M to 600M, AdamW pretraining gives the better learning–forgetting trade by 19.2 billion tokens at every size. Production models sit trillions of tokens further on, where the one matched test favored Muon without measuring forgetting. Until someone measures held-out pretraining loss after fine-tuning a matched pair that large, the reversal is a finding about models under a billion parameters.

Sources

ε-greedy control settles for the best policy that keeps exploring

2026-11-04T07:38:04Z

Reinforcement Learning

ε-greedy control settles for the best policy that keeps exploring

Monte Carlo control with ordinary starts needs a policy that sometimes moves at random, and the price of those moves grows with ε, first in value and then in the learned arrows.

17 min read2026-11-04

Why this matters

Monte Carlo control with exploring starts can reach the optimal policy because each episode can open with any move in any cell, which an environment cannot always offer. With ordinary starts, where each episode opens with the policy's own move, control never leaves the hand-drawn policy, since a greedy policy collects returns only for the moves it takes. The fix gives every move in every cell a probability of at least ε divided by the number of moves. A policy with that floor is ε-soft, and the ε-soft policy closest to greedy is called ε-greedy. You can then prove that an ε-greedy step never makes such a policy worse. You can also find the one policy where the step can stop, by changing the grid's transitions, and the smallest ε at which its arrows stop being optimal.

Background optional reading

01: Action values let an agent improve its policy from episodes alone: averaging returns per (cell, move) pair, the greedy step over those averages, exploring starts, and mc_control.py, which the code below imports.
02: Two improvements find the best policy: the policy improvement theorem, the Bellman optimality equation and value iteration, all used below, with control.py.
03: An agent can learn what each state is worth without knowing the rules: first-visit Monte Carlo and estimators.py, which does the averaging.
04: The value of a state is an average you can solve for: the slippery grid, the hand-drawn policy and gridworld.py.

Forty-two moves with no returns

We run Monte Carlo control from the hand-drawn policy with ordinary starts, where each episode begins in a random cell with the policy's own move. In 20 runs of 10,000 episodes, none reaches the optimal policy. Fig. 1 shows run 0 at the end. Each cell is four triangles, one per move, shaded by the average return after that move.

Fig. 1 · Run 0 of greedy Monte Carlo control with ordinary starts, after 10,000 episodes. Arrows show the policy, amber squares mark arrows that are not optimal, and a blank triangle is a move with no returns. From exploration.py.

Only 14 of the 56 (cell, move) pairs have any returns, one in each cell. At cell 14, up has 976 returns and the other three moves have none. Right is the move that policy iteration switched cell 14 to in its first round, and the agent has never tried it there.

The greedy step cannot supply the missing returns. It would switch cell 14 to right only if right's average beat up's. Right has no average yet, so greedy in mc_control.py skips it. The agent would have to take right to give it returns, and the agent takes only the policy's move. Each episode adds returns to the same 14 pairs, so after 10,000 greedy steps the policy is still the hand-drawn one. Sutton and Barto state the general case: with no returns to average, the estimates of the other moves “will not improve with experience”.¹

So the agent has to take every move in every cell now and then, whatever its averages say. Exploring starts arranged that from outside the agent, by choosing each episode's first move. A policy that sometimes moves at random can arrange it instead.

A floor of ε/4 on every move

An ε-greedy policy usually makes the greedy move, the one with the largest average. With a small probability ε it instead picks one of the moves uniformly at random, the greedy move included.¹ In a cell whose greedy move is a^*, each move gets the following probability.

\pi(a \mid s) = \begin{cases} 1 - \varepsilon + \dfrac{\varepsilon}{|\mathcal{A}(s)|} & \text{if } a = a^* \\[6pt] \dfrac{\varepsilon}{|\mathcal{A}(s)|} & \text{otherwise} \end{cases}

The ε-greedy policy, where |\mathcal{A}(s)| is the number of moves in cell s.¹

Every cell here has four moves, so with ε = 0.1 the greedy move gets 0.925 and each other move gets 0.025. A policy that gives every move in every cell at least \varepsilon/|\mathcal{A}(s)| is called ε-soft. The ε-greedy policy is the ε-soft policy closest to greedy, since every move gets the floor and the greedy move gets the rest.¹

The ε-greedy policy that makes the moves is also the policy whose action values the agent averages and then improves. Control of this kind is called on-policy.¹

In soft_control.py, soft_play draws each move ε-greedily. It replaces play in mc_control.py, a function that can follow only one move per cell. The loop in soft_control is the one in mc_control.py without exploring starts: each episode starts in a random cell, its returns join the averages, and the policy turns greedy. The array policy holds the greedy move in each cell. The file imports the grid from gridworld.py, action_values from control.py, the averaging from estimators.py, and the environment, pairs and greedy from mc_control.py, all unchanged.

soft_control.py · completepython

# /// script
# requires-python = ">=3.10"
# dependencies = ["numpy"]
# ///
"""On-policy first-visit Monte Carlo control with an epsilon-greedy policy.

Only the episodes differ from mc_control.py. In every cell the agent makes a
uniformly random move with probability eps, and the policy's move otherwise.
best_soft finds the best eps-soft policy exactly, to grade the runs.
"""

import numpy as np
from control import action_values
from estimators import first_visit_mc
from gridworld import N, TERMINAL, transitions
from mc_control import CAP, LIVE, SLIP, greedy, pairs, step


def soft_play(policy, eps, start, rng):
    """One episode from cell `start` under the eps-greedy policy around
    `policy`. It stops at a terminal cell or after CAP moves."""
    s, steps = start, []
    while s not in TERMINAL and len(steps) < CAP:
        a = rng.integers(4) if rng.random() < eps else policy[s]
        reward, s_next = step(s, a, rng)
        steps.append((s, a, reward, s_next))
        s = s_next
    return steps


def soft_control(policy, eps, episodes, rng, watch=None):
    """Each episode starts in a random cell and follows the eps-greedy policy.
    Its returns join the averages, and `policy` turns greedy after it."""
    q, count = np.zeros(4 * N), np.zeros(4 * N)
    for k in range(1, episodes + 1):
        start = LIVE[rng.integers(len(LIVE))]
        steps = soft_play(policy, eps, start, rng)
        first_visit_mc(pairs(steps), q, count)
        policy = greedy(q, count, policy)
        if watch:
            watch(k, policy, q, count, steps)
    return policy, q, count


def best_soft(eps, tol=1e-10):
    """The best eps-soft policy, exactly. It is the optimal policy of a grid
    that replaces the agent's aim with a random one with probability eps, so
    value iteration on that grid finds its value v. Also returns q[s, a], the
    expected return after move a in the real grid, following that policy."""
    P = transitions(SLIP)
    P_eps = (1 - eps) * P + eps * P.mean(axis=1, keepdims=True)
    v = np.zeros(N)
    while True:
        nxt = v.copy()
        nxt[LIVE] = action_values(v, P_eps)[LIVE].max(axis=1)
        if np.abs(nxt - v).max() < tol:
            return nxt, action_values(nxt, P)
        v = nxt

Random moves end the stall

We start 20 runs of soft_control from the hand-drawn policy, with ε = 0.1, no averages and seeds 0 to 19. A run counts when every arrow is an optimal move. After 10,000 episodes 6 of the 20 runs count, and after 200,000, 19 do. Setting ε to 0 in the same code removes the random moves, and then no run counts at 10,000 episodes.

uv run exploration.py needs gridworld.py, control.py, estimators.py, mc_control.py and soft_control.py in one folder. It installs numpy and takes about four minutes on the machine used here.

exploration.py, complete

exploration.py · completepython

# /// script
# requires-python = ">=3.10"
# dependencies = ["numpy"]
# ///
"""Print every number on the page: the greedy stall, eps-greedy control on 20
seeds, and the price of eps. The exact values from best_soft grade the runs."""

import numpy as np
from gridworld import N, PI
from mc_control import LIVE, control
from soft_control import best_soft, soft_control

SEEDS = range(20)


def best_moves(eps):
    """The moves that tie for best in each cell under the best eps-soft policy.
    eps = 0 gives the optimal moves, with up and right tied at cells 9 and 12."""
    _, q = best_soft(eps)
    return {s: set(np.flatnonzero(q[s] > q[s].max() - 1e-9)) for s in LIVE}


def runs_at_best(eps, checks):
    """At each checkpoint, the number of runs whose every arrow is a best move."""
    best, hits = best_moves(eps), dict.fromkeys(checks, 0)

    def watch(k, policy, q, count, steps):
        if k in hits:
            hits[k] += all(policy[s] in best[s] for s in LIVE)

    for seed in SEEDS:
        soft_control(PI.copy(), eps, max(checks), np.random.default_rng(seed),
                     watch)
    return list(hits.values())


def main():
    _, _, count = control(PI.copy(), 10_000, np.random.default_rng(0),
                          explore=False)
    print("greedy, ordinary starts, run 0 after 10,000 episodes")
    print("  pairs with returns:", int((count > 0).sum()), "of", 4 * len(LIVE))
    print("  cell 14 returns (up, right, down, left):",
          count[4 * 14:4 * 14 + 4].astype(int))

    fewest = {}

    def watch(k, policy, q, count, steps):
        if k == 10_000:
            fewest[k] = int(count.reshape(N, 4)[LIVE].min())

    _, q, _ = soft_control(PI.copy(), 0.1, 200_000, np.random.default_rng(0),
                           watch)
    print("eps 0.1, run 0")
    print("  fewest returns for any pair after 10,000:", fewest[10_000])
    print(f"  average for right at cell 14 after 200,000: {q[4 * 14 + 1]:.2f}")
    v_star, q_star = best_soft(0.0)
    _, q_soft = best_soft(0.1)
    print(f"  exact, best eps-soft policy: {q_soft[14, 1]:.2f}, "
          f"optimal policy: {q_star[14, 1]:.2f}")

    print("\nruns of 20 whose arrows are all best moves")
    print("  eps 0.0 at 10,000 episodes:", runs_at_best(0.0, [10_000]))
    checks = [1_000, 10_000, 100_000, 200_000]
    print(f"  eps 0.1 at {checks}:", runs_at_best(0.1, checks))
    checks = [1_000, 10_000, 100_000]
    print(f"  eps 0.5 at {checks}:", runs_at_best(0.5, checks))

    optimal = best_moves(0.0)
    print("\n  eps  average v* - v  cells whose best moves differ from optimal")
    for eps in (0.01, 0.1, 0.2, 0.3, 0.5):
        v, _ = best_soft(eps)
        differ = [s for s, moves in best_moves(eps).items() if moves != optimal[s]]
        print(f"{eps:5.2f}  {(v_star - v)[LIVE].mean():14.2f}  {differ}")
    first = next(e for e in range(1, 100) if best_moves(e / 100) != optimal)
    print("smallest eps, in steps of 0.01, with other best moves:", first / 100)
    v_half, _ = best_soft(0.5)
    for name, v in (("v* (eps 0)", v_star), ("best eps-soft v, eps 0.5", v_half)):
        print(f"{name}, by row:", v.reshape(4, 4).round(2).tolist())


if __name__ == "__main__":
    main()

The output of uv run exploration.py

output of uv run exploration.pytext

greedy, ordinary starts, run 0 after 10,000 episodes
  pairs with returns: 14 of 56
  cell 14 returns (up, right, down, left): [976   0   0   0]
eps 0.1, run 0
  fewest returns for any pair after 10,000: 24
  average for right at cell 14 after 200,000: -0.41
  exact, best eps-soft policy: -0.37, optimal policy: 0.61

runs of 20 whose arrows are all best moves
  eps 0.0 at 10,000 episodes: [0]
  eps 0.1 at [1000, 10000, 100000, 200000]: [0, 6, 14, 19]
  eps 0.5 at [1000, 10000, 100000]: [7, 18, 20]

  eps  average v* - v  cells whose best moves differ from optimal
 0.01            0.10  []
 0.10            1.08  []
 0.20            2.23  []
 0.30            3.46  []
 0.50            6.09  [9]
smallest eps, in steps of 0.01, with other best moves: 0.33
v* (eps 0), by row: [[2.16, 4.04, 6.48, 0.0], [0.61, 0.06, 0.0, 6.48], [-0.71, -1.01, 0.06, 4.04], [-1.73, -0.71, 0.61, 2.16]]
best eps-soft v, eps 0.5, by row: [[-4.05, -2.26, 1.42, 0.0], [-5.67, -7.08, 0.0, 1.42], [-6.64, -7.1, -7.08, -2.26], [-7.1, -6.64, -5.67, -4.05]]

In run 0 after 10,000 episodes every triangle is shaded, and the rarest pair has 24 returns. Cells 13 and 14 now aim right, as the first panel of Fig. 2 shows.

Fig. 2 · Run 0 of ε-greedy control with ε = 0.1 after 10,000 episodes, where every move has returns and every arrow is optimal, beside the exact action values of the best ε-soft policy for ε = 0.1. The exact panel draws two arrows where moves tie. From exploration.py.

The returns come from an agent that keeps making random moves, so the averages estimate the action values of the ε-greedy policy itself. After 200,000 episodes run 0's average for right at cell 14 is −0.41. The exact action value is −0.37 under the ε-greedy policy around the optimal arrows, and 0.61 under the optimal policy itself. So each step makes the policy ε-greedy with respect to the action values of an ε-greedy policy. The policy improvement theorem was stated for a policy that makes one move per cell, and nothing yet says such a step cannot make the policy worse.

Why an ε-greedy step never makes the policy worse

The policy improvement theorem covers this step once it is written for a policy that picks moves at random. For a stochastic policy \pi', q_\pi\bigl(s, \pi'(s)\bigr) means the average \sum_a \pi'(a \mid s)\, q_\pi(s, a), the expected return from one move drawn from \pi' followed by \pi. If that average is at least v_\pi(s) in every cell, then v_{\pi'} \ge v_\pi in every cell.¹

Let \pi be any ε-soft policy and \pi' the ε-greedy policy with respect to q_\pi. Sutton and Barto check the theorem's condition in one chain, their equation (5.2).

\begin{aligned} q_\pi\bigl(s, \pi'(s)\bigr) &= \sum_a \pi'(a \mid s)\, q_\pi(s, a) \\ &= \htmlClass{nb-mc1}{\frac{\varepsilon}{|\mathcal{A}(s)|} \sum_a q_\pi(s, a)} \\ &\qquad + \htmlClass{nb-mc2}{(1 - \varepsilon) \max_a q_\pi(s, a)} \\ &\ge \frac{\varepsilon}{|\mathcal{A}(s)|} \sum_a q_\pi(s, a) \\ &\qquad + (1 - \varepsilon) \sum_a \htmlClass{nb-mc3}{w(a)}\, q_\pi(s, a) \\ &= \sum_a \pi(a \mid s)\, q_\pi(s, a) \;=\; v_\pi(s) \end{aligned}

\tfrac{\varepsilon}{|\mathcal{A}(s)|} \textstyle\sum_a q_\pi(s, a)the floor: every move's share of ε
(1 - \varepsilon) \max_a q_\pi(s, a)the rest of the probability, all on the greedy move
w(a) = \tfrac{\pi(a \mid s) - \varepsilon/|\mathcal{A}(s)|}{1 - \varepsilon}weights that turn the probabilities \pi puts above the floor into an average

The ε-greedy step meets the condition of the policy improvement theorem for any ε-soft \pi.¹

The second equality splits \pi' into its floor and the remaining 1 - \varepsilon on the greedy move. The inequality replaces the maximum with a weighted average of the same four action values, with weights w(a). Each weight is at least 0 because \pi is ε-soft. The weights sum to 1 because the probabilities of \pi sum to 1 and the floor takes ε of them. A weighted average is never larger than the largest value it averages. In the last equality the floor terms cancel, and what remains is the average of q_\pi(s, a) over the moves of \pi, which is v_\pi(s).

So \pi' is no worse than \pi in any cell. The new policy is also ε-soft, so the step can be repeated.

The only resting point is the best ε-soft policy

Suppose the ε-greedy step stops changing the policy. Then \pi is ε-greedy with respect to its own action values, and in every cell its value is their ε-greedy average.

\begin{aligned} v_\pi(s) &= (1 - \varepsilon) \max_a q_\pi(s, a) \\ &\qquad + \frac{\varepsilon}{|\mathcal{A}(s)|} \sum_a q_\pi(s, a) \end{aligned}

Sutton and Barto read this as the Bellman optimality equation of a different grid.¹ In that grid the agent picks a move, and with probability ε the environment replaces it with a uniformly random one before the slip. Its transition probabilities mix the real ones.

\begin{aligned} \tilde P(s' \mid s, a) &= (1 - \varepsilon)\, P(s' \mid s, a) \\ &\qquad + \frac{\varepsilon}{|\mathcal{A}(s)|} \sum_b P(s' \mid s, b) \end{aligned}

A policy \mu in the new grid moves exactly like the ε-soft policy (1 - \varepsilon)\,\mu + \varepsilon/|\mathcal{A}(s)| in the real one. Every ε-soft policy arises this way, with \mu equal to the weights w(a) from equation (5.2). So the best policy of the new grid is the best ε-soft policy of the real one, and its value \tilde v_* is the most an ε-soft policy can earn.

In the new grid's Bellman optimality equation, the random part of \tilde P does not depend on the move a, so it comes out of the max.

\begin{aligned} \tilde v_*(s) &= \max_a \sum_{s'} \tilde P(s' \mid s, a)\,\bigl[R(s') + \gamma\, \tilde v_*(s')\bigr] \\ &= (1 - \varepsilon) \max_a \tilde q(s, a) \\ &\qquad + \frac{\varepsilon}{|\mathcal{A}(s)|} \sum_a \tilde q(s, a) \end{aligned}

Here \tilde q(s, a) is the expected return after move a in the real grid, with each landing cell valued at \tilde v_*.

\tilde q(s, a) = \sum_{s'} P(s' \mid s, a)\,\bigl[R(s') + \gamma\, \tilde v_*(s')\bigr]

With v_\pi in place of \tilde v_*, \tilde q becomes q_\pi, and the equation is the one an unchanged policy satisfies. The new grid's Bellman optimality equation has only one solution, so a policy the step leaves unchanged has v_\pi = \tilde v_*. It is the best ε-soft policy.¹

best_soft builds \tilde P and runs value iteration on it, which gives \tilde v_*, and its q[s, a] is \tilde q(s, a). The second panel of Fig. 2 draws \tilde q for ε = 0.1, the action values that run 0's averages were estimating.

The argument assumes exact action values: Sutton and Barto say it “does assume that they are computed exactly”.¹ The book gives no proof for the sampled version in soft_control, which steps on averages after every episode, so for that version this lesson has only its runs. A later lesson's method, SARSA, does reach an optimal policy, with suitable step sizes, if ε shrinks toward 0 while every move is still tried infinitely often. Singh and colleagues call a policy like that GLIE, short for greedy in the limit with infinite exploration.²

How much ε costs

Table 1 uses best_soft to compare the best ε-soft value with the optimal value v_*, averaged over the 14 cells an episode can start in.

Table 1 · The loss v_* - \tilde v_* of the best ε-soft policy, averaged over the 14 starting cells, and the cells whose best moves differ from the optimal ones. From `exploration.py`.
ε	Average loss	Changed moves
0.01	0.10	none
0.1	1.08	none
0.2	2.23	none
0.3	3.46	none
0.5	6.09	cell 9

The loss grows almost in proportion to ε. At ε = 0.1 it is 1.08, about the size of one move's reward of −1. Up to ε = 0.32 the best ε-soft policy keeps the optimal arrows and differs from the optimal policy only in its random moves.

From ε = 0.33 the arrows change at cell 9. Fig. 3 shows why at ε = 0.5. Up and right from cell 9 lead mostly to cells 5 and 10, which border the pit, while left and down lead mostly to cells 8 and 13. Under the optimal policy cells 5 and 10 are worth more, 0.06 against −0.71. Under the best ε-soft policy at ε = 0.5 they are worth less, −7.08 against −6.64, so cell 9 aims left or down.

Fig. 3 · State values and best moves of the optimal policy and of the best ε-soft policy for ε = 0.5. The amber square marks cell 9, whose best moves are no longer optimal. From exploration.py.

The runs land there too. With ε = 0.5, 18 of 20 runs have the best ε-soft arrows after 10,000 episodes, against 6 with ε = 0.1. All 20 have them after 100,000, each aiming cell 9 left or down. On this grid, runs with ε = 0.1 reach the optimal arrows from ordinary starts and lose 1.08 per start to their random moves. Runs with ε = 0.5 reach their resting point sooner, and the arrows they rest on are not optimal.

Problems

1. Cell 2's greedy move is right, into the goal, and the pit is directly below it. Under the ε-greedy policy with ε = 0.1, what is the probability that the agent's next move from cell 2 lands in the pit? How does that compare with the greedy policy?

Answer

The greedy policy aims right and slips down into the pit with probability 0.1. The ε-greedy policy aims right with probability 0.925 and each other way with 0.025. Aiming up cannot reach the pit, aiming down lands in it with 0.8, and aiming left slips down into it with 0.1. The total is 0.925 \times 0.1 + 0.025 \times (0 + 0.8 + 0.1) = 0.115. It is the entry \tilde P(6 \mid 2, \text{right}) of the new grid, 0.9 \times 0.1 + 0.1 \times 0.25.

2. Suppose each episode opened with a random (cell, move) pair, as with exploring starts, and the agent then followed the ε-greedy policy with ε = 0.1. Where could the policy stop?

Answer

Only at the best ε-soft policy. Exploring starts choose only each episode's first pair. After that move the agent follows the ε-greedy policy, so each return is still a sample of that policy's action value for the pair it follows. The averages estimate the same action values as before, and the ε-greedy step has the same single resting point.

The takeaway

A greedy policy collects returns only for the moves it takes, so with ordinary starts the greedy step never switches to the others. An ε-greedy policy gives every move at least ε divided by the number of moves, so every move keeps collecting returns. With exact action values, the ε-greedy step never makes an ε-soft policy worse, and it can stop only at the best ε-soft policy, the optimal policy of a grid that replaces the agent's move at random with probability ε. That policy loses 1.08 per start at ε = 0.1 on the course grid, and from ε = 0.33 its arrows are no longer optimal.

Go deeper optional reading

01: Sutton and Barto, Section 5.4: read the box on p. 101 line by line against soft_control.py. Its title says it estimates \pi_*, and the text on p. 103 says the method reaches only the best ε-soft policy.
02: Singh, Jaakkola, Littman and Szepesvári, Convergence Results for Single-Step On-Policy Reinforcement-Learning Algorithms: Figure 1 (p. 288) is a three-state problem where exploring half the time changes the best action, as ε = 0.5 does at cell 9, and Section 2.2 (pp. 291–292) defines GLIE.
03: Sutton and Barto, Section 2.3: in a ten-armed bandit, greedy selection stalls for a different reason, because a move whose first samples look bad is never tried again (pp. 29–30).
04: Sutton and Barto, Example 6.6: on a cliff, SARSA with ε = 0.1 learns a longer, safer path (p. 132), the same effect as ε = 0.5 at cell 9, under a method a later lesson teaches.

Sources

Democrats win back the House and the Senate

2026-11-04T06:39:19Z

Daily Brief

Democrats win back the House and the Senate

By AP’s 1:35 a.m. Eastern count, Democrats had gained 25 House seats and seven Senate seats, and Missouri voters had rejected the map Trump asked for.

3 min read2026-11-04

Word of the Day

psephology noun /sē-ˈfä-lə-jē/

The scientific study of elections, from the Greek psēphos, a pebble, because the ancient Greeks voted with pebbles.¹

Nuffield College, Oxford, says the word “started out as an academic joke suggested to R.B. McCallum,” and it caught on.² McCallum printed it in the Oxford Magazine in 1951, reviewing the college’s study of the 1950 British election, and David Butler put it in a book the next year.² Touring the United States in 1952, Butler “found himself overwhelmed by journalists asking about the new word.”²

News

AP calls both chambers of Congress for Democrats

The Associated Press called the Senate for Democrats before midnight Eastern, after James Talarico beat Ken Paxton in Texas and Sherrod Brown unseated Sen. Jon Husted in Ohio.³ It had called the House for them by 1:27 a.m.⁴ At 1:35, AP’s tally on Fox News’s results page gave Democrats 220 House seats, two past the 218 for a majority, and a net gain of 25 where they needed three. Republicans had 127, and 88 races were uncalled.⁵³ In the Senate the same tally gave Democrats 53 seats and a net gain of seven, with six races uncalled.⁵

News

Missouri voters strike down the map drawn to unseat Emanuel Cleaver

Proposition A, the referendum on the 2025 map Missouri’s legislature drew at Trump’s request to unseat Rep. Emanuel Cleaver, was losing 62% to 38% when AP called it at 9:59 p.m. Central, so those lines will not be used in 2028.⁶ Voters also rejected Amendment 3, which would have banned most abortions again two years after they ended the ban.⁷

News

Trump says US aid to Israel stops at the end of 2026

“We were giving them $4 billion – we are – but it gets cut off at the end of the year,” Trump said on Fox & Friends, and he said Benjamin Netanyahu had approached him about it.⁸ No Israeli or White House official is quoted confirming the cutoff.⁸ The aid runs at $3.8 billion a year under a memorandum of understanding that expires in 2028, and Congress appropriates it.⁹

Technology

Hugging Face finds UMT5’s decoder saw future tokens for almost two months

A Transformers change of 30 July that enabled PyTorch SDPA for T5 let UMT5’s decoder attend to future tokens in training, and nothing crashed before the 28 September fix, Hugging Face disclosed.¹⁰ Its proposed Model Integration Contracts, a shrunk fixture plus a runnable usage path and reviewed baselines, flagged the bug on every replayed commit with a 66 MB UMT5 fixture, where eager-attention tests missed it.¹⁰ A UMT5 fine-tune on main in that window, with default SDPA attention, trained a decoder that could see its targets.¹⁰

Technology

StateBench finds state tracking that Gated DeltaNet learns and transformers miss

Allen Institute for AI researchers sort state-tracking problems into four Krohn-Rhodes levels (r-trivial, aperiodic, soluble, general finite) and prove that Gated DeltaNet with positive eigenvalues expresses every aperiodic problem.¹¹ Ai2 reports that 275M-parameter Gated DeltaNet and hybrid models trained on up to 256 assignments generalized almost perfectly to 64,000 on the first three levels, while transformers never learned general finite problems.¹² Four frontier models, among them Claude Opus 5.5 and GPT-6 Sol, fall to chance on the two hardest levels at 10K to 64K tokens.¹¹

Sources

Marvell’s $115 billion for laser maker Lumentum works out to 23 times its revenue run rate

2026-11-03T06:43:36Z

Feature

Marvell’s $115 billion for laser maker Lumentum works out to 23 times its revenue run rate

Priced at Marvell’s 28 October close, the deal comes to 18 times Lumentum’s annualized December guidance, just under Marvell’s own revenue multiple that day.

5 min read2026-11-03

A fixed ratio puts a dollar price on one day

In Marvell’s own drawing of an optical module, the electronics are Marvell’s and every laser is Lumentum’s.¹ On 2 November the two companies announced that Marvell will buy Lumentum for 4.500 Marvell shares per Lumentum share, an implied enterprise value of $115 billion.² Neither company states a multiple. Divided by Lumentum’s September quarter of $1.263 billion annualized, its run rate, the price is 22.8 times revenue. It is 14.4 times if Lumentum reaches the $8 billion a year its March investor deck sets as a target.³⁴

The $115 billion is a value at Marvell’s 28 October closing price, the last trading day before market speculation about the deal began.²¹ The ratio is fixed, so the dollar price moves with Marvell’s stock until the close, expected in the first half of 2028.

Co-packaged optics leaves Lumentum the light source

Marvell block Lumentum block not colored on the slide

Fig. 1 · One optical link in two forms, drawn for this article from Marvell’s slides 13 and 14. Colors follow the slides’ ownership key. The signal order is not on the slides. EML: laser with a built-in modulator. PIC, EIC: photonic and electronic chips. ELSFP: external laser source, a small pluggable.¹

Slide 13 splits each module into electronics and photonics. The electronics are Marvell’s: the DSP, which turns the host’s SerDes lanes into the signal that drives the laser and back, the drivers that carry that signal to the laser, and the TIAs that amplify the photodiode’s current. The laser and photodiode are Lumentum’s. Marvell’s 10-K uses the word laser only in “laser drivers.”⁵

Co-packaging moves the boundary. A pluggable module’s EML is one Lumentum part that both makes the light and modulates it. In co-packaged optics those are two devices: a continuous-wave laser makes steady light, and a modulator on the silicon-photonics chip imposes the data.⁶ Slide 14 draws that photonic chip as Marvell’s and puts Lumentum’s laser outside the package, in an external pluggable feeding several photonic chips.¹ What passes from a Lumentum part to a Marvell part is then unmodulated light on a fiber.

Lumentum’s chief executive, Michael Hurlston, said: “we supply our lasers to almost all of our competitors.”⁷ Lumentum reports Components, which include laser chips, at $2,005.6 million of FY2026’s $3,014.0 million. Systems, which are modules, optical circuit switches and industrial lasers, were $1,008.4 million. The switches, which sit outside the link above, contributed more than $90.0 million.⁸

Five revenue figures give five multiples

Fig. 2 · $115 billion over Lumentum revenue, the paper’s computation. Reported figures are from Lumentum’s FY2026 10-K and its September-quarter release. December guidance is Lumentum’s forecast. The target model is, in Lumentum’s words, “not intended to represent guidance on actual outcomes.” The dashed line is Marvell’s own enterprise value over its guided quarter times four, reconstructed at its 28 October price.⁸³⁴¹

September revenue was 136.5% above the year before and 25.5% above June.³ Lumentum guides December to $1.575 to $1.625 billion, 26.7% above September at the midpoint.

Marvell’s own enterprise value at its 28 October price is about $233.9 billion, 18.6 times its guided quarter annualized, $12.6 billion.¹ That is the paper’s reconstruction from the Merger Agreement’s share counts and 1 August net debt. It leaves out in-the-money warrants, which understates it.⁹¹⁰ The $115 billion comes down to that 18.6 times at $6.2 billion of Lumentum revenue, less than the $6.4 billion its December guidance annualizes to.

Lumentum’s employee FAQ values the deal at “approximately $100 billion.”¹¹ Both figures can be rebuilt from the Merger Agreement at $1,132 a share. In the paper’s reconstruction, 90.68 million common shares come to $102.7 billion. Counting preferred stock, options, stock awards and the shares issuable on convertible notes gives 103.29 million shares and $116.9 billion, or $114.9 billion after $1.98 billion of cash.⁹³ The filings do not say what the FAQ counts.

The $550 billion market is two forecasts and a $60 billion increment

$400BMarvell, CY30 market

$90B+Lumentum, CY30 industry

$550BCombined, deal deck

$60BIncrement, per Murphy

The first two sum to $490 billion, and the deck shows $550 billion.¹ Marvell’s $400 billion includes $235 billion of custom silicon and $65 billion of interconnect.¹² Lumentum’s $90 billion-plus is an all-vendor industry revenue forecast built on LightCounting and 650 Group data.⁴ Marvell’s chief executive, Matt Murphy, put the increment at $60 billion, most of it interconnect and mostly scale-up.⁷ A transceiver’s revenue includes its DSP, which Marvell counts inside its own $65 billion. Some dollars may therefore sit in both forecasts. That is the paper’s inference, and no document sizes it.

The $115 billion is at most 1.28 times that whole-industry forecast. Vivek Arya of Bank of America Securities asked on the call how Marvell knew it was not buying “a company at the peak of its shortage environment.” Murphy answered that “there are supply shortages today across the industry,” Marvell’s included, and that Marvell’s XPUs, switches and DSPs at the same hyperscalers show him “the design wins are real.”⁷ He gave no supply figure.

Lumentum’s filings show the shortage. Its 10-K says demand is “outpacing our current supply,” and its March deck lists a backlog of more than two years for EML and CW laser chips. Two unnamed customers were 26.6% and 15.0% of FY2026 revenue.⁸⁴

At December guidance the price already sits just under Marvell’s own multiple. Lumentum’s $8 billion target needs a $2 billion quarter after that, one more step of about September’s 25.5%, and Lumentum has guided only as far as December.

Sources

Action values let an agent improve its policy from episodes alone

2026-11-03T07:29:34Z

Reinforcement Learning

Action values let an agent improve its policy from episodes alone

Monte Carlo control with exploring starts reached the optimal policy in nearly every run on the course's slippery grid, where no convergence theorem yet applies.

18 min read2026-11-03

Why this matters

To improve a policy, an agent compares the moves open to it in each cell. From state values, that comparison weights each cell a move can land in by the chance of landing there, and an agent that learns from episodes never sees those chances. It can learn what each move is worth instead, by averaging the returns that followed the move, as Monte Carlo prediction averaged returns per cell. The average estimates the move's action value.

The greedy step then compares a cell's averages and needs no model. The agent can also improve after every episode without finishing an evaluation, a case of generalized policy iteration called Monte Carlo control with exploring starts. You will run it until it finds the grid's optimal arrows, and see why no theorem yet promises that it will.

Backgroundoptional reading

01: The value of a state is an average you can solve for: the slippery grid, its rewards, the return, and gridworld.py, which the code below imports.
02: Two improvements find the best policy: action values, the greedy step, the policy improvement theorem, and the optimal arrows used here as the answer key, with control.py.
03: An agent can learn what each state is worth without knowing the rules: first-visit Monte Carlo and estimators.py, which does the averaging below.
04: Sutton and Barto, Reinforcement Learning: An Introduction: Sections 5.2 and 5.3 (printed pp. 96-99) cover the same algorithm in the field's standard notation.

Without the slip rule, state values cannot rank the moves at cell 14

Cell 14 is in the bottom row of the slippery grid, one cell left of the bottom-right corner, and the fixed policy aims up there. Policy iteration found that aiming right is better, worth 0.32 against up's −1.08. It priced right by weighting each cell the move can land in by the chance of landing there, as the first panel of Fig. 1 shows.

Fig. 1 · Two ways to score moves at cell 14 under the fixed policy. From state values, the weighted sum for one move needs the boxed probabilities. From action values, cell 14 is four triangles, one per move, each holding that move's action value. Values from experiment.py.

An agent that learned state values from episodes has the three values in that sum. It does not have 0.8, 0.1 and 0.1, because those come from the slip rule and the agent only sees what happened. Without them it cannot score right, or any other move, from state values.¹

It can score a move by trying it. Suppose the agent in cell 14 aims right once and then follows the fixed policy. The table shows one such episode.

One episode that opens with right in cell 14 and ends in the pit, with returns discounted by 0.9. From `experiment.py`.
Cell	Move	Reward	Return after this pair
14	right	−1	−17.29
15	up	−1	−18.10
14	up	−1	−19.00
10	right	−20	−20.00

The returns come from the backward pass of first-visit Monte Carlo, with one change: each step is labeled by its (cell, move) pair instead of its cell. Cell 14 appears twice, aimed right and then up, so the episode gives a return to each of those moves, −17.29 to right and −19.00 to up.

One return is noisy, and this one ended in the pit. The average of many returns after a pair estimates the pair's action value, the expected return from taking move a in cell s and following \pi afterward.¹

q_\pi(s, a) = \mathbb{E}_\pi\bigl[\, G_t \mid S_t = s,\ A_t = a \,\bigr]

Each return after (14, right) is a sample of q_\pi(14, \text{right}), so their average converges to 0.32 without the agent using any probability.

The greedy step over action values reads no probabilities

Policy iteration improved a policy with the greedy step. Written out, the step uses the model.

\pi'(s) = \operatorname*{arg\,max}_a \sum_{s'} P(s' \mid s, a)\,\bigl[\, R(s') + \gamma\, v_\pi(s') \,\bigr]

The sum is q_\pi(s, a), so the same step reads as follows.

\pi'(s) = \operatorname*{arg\,max}_a\; q_\pi(s, a)

Every use of P and R now sits inside q_\pi, and an agent can estimate q_\pi by averaging returns. An agent holding estimates of the action values picks the greedy move in a cell by comparing four numbers. In Sutton and Barto's words, with an action-value function "no model is needed to construct the greedy policy".¹

The greedy policy is also no worse than \pi. The policy improvement theorem says that if q_\pi(s, \pi'(s)) \ge v_\pi(s) in every cell, then v_{\pi'}(s) \ge v_\pi(s) in every cell. The greedy step meets that condition, and checking it takes only action values.¹

q_\pi\bigl(s, \pi'(s)\bigr) = \max_a q_\pi(s, a) \;\ge\; q_\pi\bigl(s, \pi(s)\bigr) = v_\pi(s)

The first equality is what greedy means. The inequality holds because the largest of four values is at least the value of any one of them, the policy's own move included. The last equality holds because \pi picks one move per cell, so taking \pi's move and then following \pi is just following \pi.

Exploring starts give every move returns of its own

Averages exist only for moves the agent has tried. A greedy agent takes one move in each cell, so after an episode's first move every pair it visits is the policy's own. The other three moves in each cell get no returns, and their estimates never change.¹ In the code below a move with no returns has no estimate yet, and the greedy step skips it, so the policy could never switch to such a move.

This lesson supplies the missing returns with exploring starts: each episode begins from a random (cell, move) pair, and every pair has some chance of being picked.¹ Here the pick is uniform over the 56 pairs, and after the first move the agent follows its policy.

Exploring starts needs an environment that can open an episode at any (cell, move) pair, and play below takes the start pair as an argument. Sutton and Barto note that the assumption "cannot be relied upon in general, particularly when learning directly from actual interaction with an environment".¹ The usual alternative, a policy that keeps some chance on every move, needs a lesson of its own.

The runs below depend on the assumption. With explore=False, each episode starts in a random cell with the policy's own move, and none of the 20 runs reaches the optimal policy in 10,000 episodes. Every average then belongs to a move the policy already takes, so each greedy step returns the policy it started from.

Monte Carlo ES improves the policy after every episode

Policy iteration evaluated each policy completely before improving it. From samples, a complete evaluation would take infinitely many episodes. Sutton and Barto call the general scheme generalized policy iteration (GPI): evaluation and improvement interact at any granularity, and neither has to finish before the other runs.¹ Value iteration was one case of it, with an improvement folded into every sweep.

Monte Carlo ES, short for Monte Carlo with exploring starts, takes the episode as its unit. After each episode it adds the episode's returns to the averages of the pairs it visited, then makes the policy greedy, which can change only the cells the episode visited. The averages are never reset, so a pair's average mixes returns earned under every policy the agent has used.¹

mc_control.py is the complete method. It imports the grid from gridworld.py, printed with the exact solve, and first_visit_mc from estimators.py, printed with Monte Carlo prediction. pairs numbers each pair as 4 × cell + move, so first_visit_mc averages returns per pair as it averaged them per cell. greedy is the improvement step, and step, the environment, is the only function that reads the grid's rules.

A greedy policy in mid-training can press the bottom row into the wall, like the stuck policy, and an episode under it never ends. play stops every episode after 200 moves, so a cut episode's returns leave out whatever would have come later. In the main runs, 84 of 2,000,000 episodes were cut.

mc_control.py · completepython

# /// script
# requires-python = ">=3.10"
# dependencies = ["numpy"]
# ///
"""Monte Carlo control with exploring starts on the slippery grid.

The agent's side is pairs, greedy and the call to first_visit_mc. It sees
only episodes. Only step reads the grid's rules, to play them out.
"""

import numpy as np
from estimators import first_visit_mc
from gridworld import N, R, TERMINAL, move

SLIP = 0.2
LIVE = [s for s in range(N) if s not in TERMINAL]
CAP = 200  # an episode stops after this many moves


def step(s, a, rng):
    """The environment: the aimed move with probability 1 - SLIP, otherwise a
    slip to either side, as in transitions(). Returns the reward and next cell."""
    u = rng.random()
    aim = a if u < 1 - SLIP else (a + 1) % 4 if u < 1 - SLIP / 2 else (a + 3) % 4
    s_next = move(s, aim)
    return R[s_next], s_next


def play(policy, start, first, rng):
    """One episode that opens with the move `first` in cell `start`, follows
    `policy` after that, and stops at a terminal cell or after CAP moves."""
    s, a, steps = start, first, []
    while s not in TERMINAL and len(steps) < CAP:
        reward, s_next = step(s, a, rng)
        steps.append((s, a, reward, s_next))
        s, a = s_next, policy[s_next]
    return steps


def pairs(steps):
    """Label each step by its (cell, move) pair, numbered 4 * cell + move, so
    that first_visit_mc averages returns per pair instead of per cell."""
    return [(4 * s + a, a, r, s_next) for s, a, r, s_next in steps]


def greedy(q, count, policy):
    """The move with the largest average return in each cell. A move with no
    returns has no estimate yet, and a cell with none keeps its move."""
    Q = np.where(count > 0, q, -np.inf).reshape(N, 4)
    return np.where(count.reshape(N, 4).any(axis=1), Q.argmax(axis=1), policy)


def control(policy, episodes, rng, explore=True, every=1, forget=False,
            watch=None):
    """Monte Carlo ES. Each episode starts from a random (cell, move) pair, or
    from a random cell and the policy's move if explore is False. The policy
    turns greedy every `every` episodes, and forget clears the averages then."""
    q, count = np.zeros(4 * N), np.zeros(4 * N)
    for k in range(1, episodes + 1):
        start = LIVE[rng.integers(len(LIVE))]
        first = rng.integers(4) if explore else policy[start]
        steps = play(policy, start, first, rng)
        first_visit_mc(pairs(steps), q, count)
        if k % every == 0:
            policy = greedy(q, count, policy)
            if forget:
                q[:], count[:] = 0.0, 0.0
        if watch:
            watch(k, policy, q, count, steps)
    return policy, q, count

experiment.py plays 20 runs of 100,000 episodes, one per seed, each starting from the fixed policy with no averages. A run counts as optimal when every arrow is an optimal move, either up or right at cells 9 and 12, where the two tie. After 10,000 episodes half the runs are optimal, and after 100,000, 18 of 20 are. The two still short are wrong at cell 5, beside the pit, where the best move beats the next by only 0.35. Twenty other seeds repeat the half at 10,000 and do better at 100,000.

Fig. 2 · What run 0 learned. Each cell's four triangles are its moves, shaded by average return, green above 0 and red below. Arrows show the greedy policy, and an amber square marks an arrow that is not optimal. The last panel is the exact answer, where up and right tie at cells 9 and 12. Values from experiment.py.

Run 0 has three arrows wrong after 1,000 episodes, at cells 9, 10 and 13. By 10,000 only cell 10 is wrong. It aims down, whose exact action value is −0.29, against 0.06 for right. By 100,000 every arrow is optimal.

uv run experiment.py needs gridworld.py, control.py from the policy iteration lesson, estimators.py and mc_control.py in one folder. It installs numpy and takes a little over two minutes on the machine used here.

experiment.py, complete

experiment.py · completepython

# /// script
# requires-python = ">=3.10"
# dependencies = ["numpy"]
# ///
"""Run Monte Carlo ES on the grid and print every number on the page. The
exact values from control.py appear only to grade the learned policy."""

from collections import Counter

import numpy as np
from control import action_values, evaluate
from estimators import first_visit_mc
from gridworld import ARROWS, N, PI, draw, transitions
from mc_control import CAP, LIVE, control, pairs, play

PI_STAR = draw("""> > > .
                  ^ ^ . ^
                  ^ ^ > ^
                  ^ > > ^""")  # the optimal arrows from policy iteration
Q_STAR = action_values(evaluate(PI_STAR), transitions(0.2))
BEST = {s: np.flatnonzero(Q_STAR[s] > Q_STAR[s].max() - 1e-9) for s in LIVE}
SEEDS, EPISODES = range(20), 100_000
CHECKS = [1_000, 2_000, 5_000, 10_000, 20_000, 50_000, 100_000]


def wrong(policy):
    """Cells whose move is not optimal. Cells 9 and 12 have two optimal moves."""
    return [s for s in LIVE if policy[s] not in BEST[s]]


def arrows(policy):
    marks = ["." if s not in LIVE else ARROWS[policy[s]] for s in range(N)]
    return " / ".join(" ".join(marks[r:r + 4]) for r in range(0, N, 4))


def one_episode():
    """The return one episode gives each pair it visits."""
    steps = play(PI, 14, 1, np.random.default_rng(1))
    q, count = np.zeros(4 * N), np.zeros(4 * N)
    first_visit_mc(pairs(steps), q, count)
    print("cell, move, reward:", [(s, ARROWS[a], int(r)) for s, a, r, _ in steps])
    print("return per pair:", [(s, ARROWS[a], round(float(q[4 * s + a]), 2))
                               for s, a, _, _ in steps])


def runs(episodes=EPISODES, **options):
    """At each checkpoint, the number of runs with an optimal policy and the
    cells the other runs get wrong, plus run 0's policy and averages."""
    optimal, missed, snaps, cut = Counter(), {k: Counter() for k in CHECKS}, {}, [0]

    def watch(k, policy, q, count, steps):
        cut[0] += len(steps) == CAP
        if k in CHECKS:
            optimal[k] += not wrong(policy)
            missed[k].update(wrong(policy))
            if seed == 0:
                snaps[k] = policy.copy(), q.copy()

    for seed in SEEDS:
        control(PI.copy(), episodes, np.random.default_rng(seed), watch=watch,
                **options)
    return optimal, missed, snaps, cut[0]


def main():
    v_pi = evaluate(PI)
    print("fixed policy: v at cells 10, 13, 14, 15:", v_pi[[10, 13, 14, 15]].round(2))
    print("fixed policy: q at cell 14 (up, right, down, left):",
          action_values(v_pi, transitions(0.2))[14].round(2))
    print("q* at cells 5 and 10 (up, right, down, left):",
          Q_STAR[[5, 10]].round(2).tolist())
    one_episode()
    schedules = {"every episode": runs(),
                 "every 5,600": runs(every=5_600),
                 "every 5,600, cleared": runs(every=5_600, forget=True)}
    print(f"\nruns with an optimal policy, of {len(SEEDS)}, by when it improves")
    print("episodes", *[f"{name:>21s}" for name in schedules])
    for k in CHECKS:
        print(f"{k:8d}", *[f"{s[0][k]:21d}" for s in schedules.values()])
    print(f"episodes cut off at {CAP} moves:",
          {name: s[3] for name, s in schedules.items()})
    optimal, missed, snaps, _ = schedules["every episode"]
    for k in (20_000, 100_000):
        print(f"every episode, wrong cells at {k}: {dict(sorted(missed[k].items()))}")
    for k in (1_000, 10_000, 100_000):
        policy, _ = snaps[k]
        print(f"run 0 at {k:6d}: {arrows(policy)}, wrong {wrong(policy)}")
    optimal, *_ = runs(episodes=10_000, explore=False)
    print("no exploring starts, optimal runs at 10,000:", optimal[10_000])


if __name__ == "__main__":
    main()

The output of uv run experiment.py

output of uv run experiment.pytext

fixed policy: v at cells 10, 13, 14, 15: [-0.1  -2.01 -1.08  1.98]
fixed policy: q at cell 14 (up, right, down, left): [-1.08  0.32 -1.78 -2.55]
q* at cells 5 and 10 (up, right, down, left): [[0.06, -15.93, -3.58, -0.29], [-15.93, 0.06, -0.29, -3.58]]
cell, move, reward: [(14, '>', -1), (15, '^', -1), (14, '^', -1), (10, '>', -20)]
return per pair: [(14, '>', -17.29), (15, '^', -18.1), (14, '^', -19.0), (10, '>', -20.0)]

runs with an optimal policy, of 20, by when it improves
episodes         every episode           every 5,600  every 5,600, cleared
    1000                     2                     0                     0
    2000                     3                     0                     0
    5000                     6                     0                     0
   10000                    10                     2                     2
   20000                    12                    15                     3
   50000                    16                    16                     5
  100000                    18                    18                     2
episodes cut off at 200 moves: {'every episode': 84, 'every 5,600': 277, 'every 5,600, cleared': 2902}
every episode, wrong cells at 20000: {5: 4, 9: 1, 10: 3}
every episode, wrong cells at 100000: {5: 2}
run 0 at   1000: > > > . / ^ ^ . ^ / ^ < v ^ / ^ < > ^, wrong [9, 10, 13]
run 0 at  10000: > > > . / ^ ^ . ^ / ^ ^ v ^ / > > > ^, wrong [10]
run 0 at 100000: > > > . / ^ ^ . ^ / ^ ^ > ^ / > > > ^, wrong []
no exploring starts, optimal runs at 10,000: 0

Unfinished evaluation works when the averages carry over

Improving after every episode means each new policy is judged by averages that mix returns from many earlier policies. Fig. 3 compares it with two slower schedules, which make the policy greedy only every 5,600 episodes, about 100 starts per pair. One keeps the averages, as Monte Carlo ES does. The other clears them at each improvement with forget=True, so each policy is evaluated from scratch, as in policy iteration.¹

Fig. 3 · Runs, of 20, at the optimal policy under three schedules of improvement, on the same seeds. Kept averages carry over from one policy to the next, and cleared ones start empty. The two 5,600 lines coincide through 10,000 episodes, before the second improvement. Values from experiment.py.

Waiting 5,600 episodes between improvements gains nothing by the end. It delays the first improvement, catches up by 20,000 episodes, and from 50,000 on has no more runs at the optimal policy than improving after every episode. Clearing the averages is what costs. The cleared schedule never has more than 5 runs at the optimal policy, and it ends with 2. Each of its policies is judged on 5,600 episodes, about 100 returns for each move the policy does not take, while the kept schedule improves at the same episodes and keeps every return.

On this grid evaluation never had to finish before the policy changed, but its averages had to keep their returns.

Only the optimal policy can be a resting point

Sutton and Barto give half an argument that Monte Carlo ES ends at the optimal policy.¹ Suppose the policy and the averages stopped changing. The averages would then be that policy's action values, and the policy would be greedy with respect to them, so in every cell the following holds.

\begin{aligned} v_\pi(s) &= q_\pi\bigl(s, \pi(s)\bigr) = \max_a q_\pi(s, a) \\ &= \max_a \sum_{s'} P(s' \mid s, a)\,\bigl[\, R(s') + \gamma\, v_\pi(s') \,\bigr] \end{aligned}

The last line is the Bellman optimality equation, and only v_* solves it, so \pi is optimal. A suboptimal policy cannot be a resting point.

The other half, that the process must come to rest, has not been proved for this algorithm in general. Sutton and Barto call it "one of the most fundamental open theoretical questions in reinforcement learning".¹ Proofs now cover one variant and two classes of problem, each with conditions of its own on step sizes or on long episodes.

Tsitsiklis proves convergence for a variant that updates only the pair each episode starts from, with every pair used as a start equally often, on any finite problem with a discount below 1.²
Wang, Yuan, Shao and Ross prove it for first-visit updates to every pair, as in mc_control.py, on problems where an optimal policy never revisits a state.³
Delattre and Fournier prove it for first-visit Monte Carlo control on any finite problem with a discount below 1/2.⁴

None of the three covers mc_control.py on this grid. Tsitsiklis's variant updates only the start pair, where mc_control.py updates every pair an episode visits. The grid's discount is 0.9, above 1/2, and its optimal policy can revisit a state: cell 2's only optimal move, right, slips up into the edge with probability 0.1 and leaves the agent in cell 2. Wang and colleagues list weaker conditions than theirs as an open problem,³ and Delattre and Fournier write that "only half the way is done".⁴ For this grid the question stays open. The runs show what Monte Carlo ES did on the seeds tried, and no theorem says that every run must end at the optimal policy.

Problems

1. Use the weighted sum from policy iteration to price aiming down at cell 14 under the fixed policy, where v_\pi(13) = -2.01, v_\pi(14) = -1.08 and v_\pi(15) = 1.98. Which cells can the move land in, with what probabilities? Compare your answer with Fig. 1.

Answer

Down runs into the bottom wall and stays in cell 14 with probability 0.8. It slips left to cell 13 or right to cell 15 with probability 0.1 each. Every landing pays −1, so q_\pi(14, \text{down}) = -1 + 0.9\,[\,0.8(-1.08) + 0.1(-2.01) + 0.1(1.98)\,] = -1.78, the down triangle in Fig. 1.

2. Suppose every episode started in cell 13 with a random first move, and the agent followed its policy after that. Which cells could ever change their move?

Answer

Only cell 13. The agent reaches every other cell by following its policy, so the only pair with returns there is the policy's own, and the greedy step keeps it. Exploring starts need every pair, in every cell, to have some chance of opening an episode.

The takeaway

An action value is the expected return after a move in a cell, so an agent can estimate it by crediting each return to the (cell, move) pair it followed. The greedy step over action values compares a cell's averages and needs no model, and the policy improvement theorem still applies. Exploring starts supply returns for moves a greedy policy never takes. Improving after every episode reached the optimal policy within 100,000 episodes in nearly every run. Waiting between improvements did not help, and clearing the averages at each one did harm. Only the optimal policy can be a resting point, but no theorem yet says runs on this grid must reach it.

Go deeperoptional reading

01: Sutton and Barto, Section 5.3 and Example 5.3: read the Monte Carlo ES box on p. 99 line by line against mc_control.py, then Figure 5.2 (p. 100) for the blackjack policy it finds.
02: Tsitsiklis, On the Convergence of Optimistic Policy Iteration (2002): read p. 60 for the variant he settles, and Section 5 (pp. 70-71) for why the action-value version needs no model where the state-value version does.
03: Wang, Yuan, Shao and Ross, On the Convergence of the Monte Carlo Exploring Starts Algorithm for Reinforcement Learning (2022): read the definitions on pp. 4-5 for the condition that no state is revisited, and Figure 3 (p. 9) for a 2022 map of what was proved.
04: Sutton, Open Theoretical Questions in Reinforcement Learning (1999): Section 2 (p. 4) poses the convergence question in one paragraph, as it was first asked.

Sources

Supreme Court won't make California set aside 20,000 alleged noncitizens' ballots

2026-11-03T06:37:01Z

Daily Brief

Supreme Court won't make California set aside 20,000 alleged noncitizens' ballots

No justice noted a dissent from the one-sentence order, and none answered the Solicitor General's argument that commingled ballots could never be retrieved.

3 min read2026-11-03

Word of the Day

agnotology noun /ag-nuh-TAH-luh-jee/

The study of the deliberate cultivation of ignorance, or the cultivation itself. ¹

Robert Proctor wanted a word for the study of how ignorance is made, and the existing agnoiology carried a metaphysical sense. In 1992 he asked the linguist Iain Boal, who coined agnotology that spring from the Greek gno, "to know", the negating prefix a-, and -ology, chosen largely for its sound. They first spelled it agnatology, then changed it because agnate already meant a relative and a few biologists said it infringed on the study of agnathic, or jawless, fish. Rejected alternatives included Lorraine Daston's tongue-in-cheek "ignorance-lore". ²

News

Justice Department loses Supreme Court bid over California ballots

On 2 November the Supreme Court denied the Justice Department's application in United States v. California, which asked it to order California to keep separate the mail ballots of more than 20,000 people the government says are noncitizens. The order gives no reasons and lists no dissent. ³ Solicitor General D. John Sauer argued that commingled ballots cannot be retrieved. ⁴

News

US strikes Qeshm Island a day before the midterms, officials tell CBS and AP

Three US officials told CBS the US carried out limited "defensive strikes" near the Strait of Hormuz, and a senior official said they hit Qeshm Island after the Revolutionary Guard fired at an American aircraft. ⁵ An unnamed official told AP the targets were Iranians using surface-to-air missiles against US assets, and Iranian state television reported explosions on Qeshm, though neither outlet cites a Pentagon or CENTCOM statement. The 2 November Daily Brief reported strikes might resume after the midterms, and Trump had said none would come before them. ⁶

News

Judge Chutkan bars DHS and DOJ from sending states noncitizen lists

On 2 November Judge Tanya Chutkan of the D.C. district court enjoined the policy under which DHS and DOJ send election officials records of people they say are noncitizens. She found the plaintiffs in League of Women Voters v. DHS likely to succeed on claims that it lacks statutory authority and violates the Privacy Act, since no system-of-records notice covers sending the data to states. The opinion records that of the 20,000-plus records DOJ gave Los Angeles County on 13 October, "only 7,035" had county addresses. ⁷

Technology

GPT-6 Astra edits 0.016% of Qwen3.5-0.8B's weights, lifting MATH-500 from 7.9% to 48.1%, authors report

Four MIT CSAIL researchers wrote on 2 November that GPT-6 Astra, with Python access to Qwen3.5-0.8B's weights, traced its looping to one direction written by the MLPs of layers 15 and 19. Subtracting it from 131,072 weights raised MATH-500 from 7.9% to 48.1% at 50 forward passes and cut answers hitting the 32,768-token limit from 93.0% to 33.1%. The figures are the authors' own, with no linked paper, and on Gemma 3 270M, which rarely loops, the method moved MATH-500 only from 6.8% to 7.0%. ⁸

Technology

Google dismantles XCPE, a botnet of 1.3 million Android set-top boxes

Google Threat Intelligence Group said on 2 November that "last week" it and partners dismantled XCPE, a botnet it says enrolled more than 1.3 million Android set-top boxes through a preloaded backdoor. Google assesses "with high confidence" that the device maker Videostrong Technology built and ran it for profit, selling access to affiliates who ran ad fraud and loaded further malware. The attribution is Google's alone, and Google says it has banned Videostrong from building certified Android devices. ⁹

Sources

An agent can learn what each state is worth without knowing the rules

2026-11-02T08:23:21-05:00

Averaging whole episodes gives unbiased but noisy targets, and updating after every move gives quieter targets that carry the errors in the agent's own estimates.

Tether and Circle froze Binance’s stolen stablecoins and had no reach over the rest

2026-11-02T06:53:41Z

Feature

Tether and Circle froze Binance’s stolen stablecoins and had no reach over the rest

Elliptic counts 44% of the $6.43 billion as frozen or seized, almost all of it stablecoin, where Bybit’s 2025 haul held no stablecoin and saw 3% to 3.84% frozen.

5 min read2026-11-02

Over half the haul had an issuer to call

A freeze reaches only tokens that have an issuer, and only those the thief has not yet swapped out. About $6.3 billion left Binance’s wallets on 29 October by Binance’s estimate, $6.36 billion by TRM Labs’ trace and $6.43 billion by Elliptic’s.¹²³ Most moved in about 20 minutes from 20:05 UTC.² Elliptic counts $3.46 billion of its total as stablecoin.³

The frozen tally runs from about $2.7 billion (TRM, 30 October) to $2.85 billion (Elliptic, 1 November) to about $3 billion (Binance, method unstated).²³¹ All are days old and still moving. Frozen is not returned: Binance says it is working with partners and law enforcement to recover the frozen assets, and CoinDesk reports none returned.⁴⁵

Only an administrator can flag an address

A stablecoin contract keeps a blocklist and checks it on transfers, so a listed address cannot move its balance. Tether’s USDT on Ethereum gives its owner addBlackList and destroyBlackFunds, which also sets a blacklisted balance to zero.⁶ Circle’s USDC restricts blacklist to a designated blacklister.⁷ Tether froze the main Ethereum wallet 37 minutes after the USDT arrived, and Circle froze 823 million USDC three minutes after that.³

Fig. 1 · Three outcomes for a flagged address. Conceptual diagram; the $786 million and $36 million are Elliptic’s figures.³

Nothing stolen on Bitcoin, the XRP Ledger, Solana or NEAR has been frozen, Elliptic reports, because native assets have no issuer that can freeze them.³ The dividing line is whether anyone administers the asset. Arbitrum’s Security Council moved, and Optimism’s governance appears to have moved, about $36 million of ETH, ARB and OP out of the attacker’s reach, and Binance froze its own WBETH.³ Circle did not freeze about $230 million of USDC from April’s Drift exploit and says it freezes only when legally compelled. A Circle spokesperson told American Banker the company froze more than $1.1 billion for Binance, which no on-chain count reproduces: TRM and Elliptic find $823 million to $825 million.⁸²³

Most stablecoins froze before the thief could swap them

Read each bar of Fig. 2 from the left: frozen, then swapped out before the freeze, then out of any issuer’s reach.

Fig. 2 · Binance haul by asset type, US$ billions, Elliptic’s figures. Frozen stablecoin is derived as 94% of its $2.85 billion and carries about a point of rounding error. Stolen amounts are at theft-time prices and frozen amounts at 1 November prices.³

Before the freezes landed, the attacker converted about $786 million of stablecoins into ETH and TRX, including 434 million USDT into 160,451 ETH at a 12% to 26% loss against market.³ About 77% of the stolen stablecoin ended up frozen, against about 6% of everything else. The issuers froze nearly all the stablecoin still in place when they acted, so the gap came from speed as well as from the asset model.

Bybit’s haul held no stablecoin for an issuer to freeze

The FBI put the 2025 Bybit theft at approximately $1.5 billion.⁹ Block Scholes read the haul from the chain as 401,347 ETH, 90,376 stETH, 8,000 mETH and 15,000 cmETH, about $1.46 billion, with no stablecoin.¹⁰ Bybit’s CEO Ben Zhou counted 3.84% frozen on 21 April 2025, as Decrypt reported,¹¹ up from 3% in early March. Block Scholes found no freeze of the stolen ETH in its 22 February update, and no source says what the 3.84% consists of.

Fig. 3 · Stablecoin share and frozen share of each haul. Binance: Elliptic, by dollar value. Bybit: Zhou’s frozen share of about 500,000 ETH, and Block Scholes’ reading of the chain. The Bybit bar shows 3.84%.³¹⁰¹¹

Two thefts are not a rule, but the mechanism in Fig. 1 predicts the order. The haul with an issuer behind 54% of it saw 44% frozen or seized, and the haul with none saw under 4%.

Binance has walked back its breach path, and the attributions are assessments

Binance’s initial finding, posted by co-CEO Richard Teng on 30 October, was that the incident “may have started with a compromised DevOps engineer’s laptop.” The laptop, he wrote, gave an attacker access to a service container, where they submitted unauthorized withdrawal transactions.¹² About 70 minutes later Binance said its later investigation “has uncovered additional points of breach” and did not name them.¹³ TRM’s assessment is that the attacker compromised Binance’s hot wallet signing infrastructure, and it says the early phishing hypothesis “covers Ethereum and leaves the other chains unexplained.”²

Who attributes the theft, in each holder’s words. These are assessments, not a court finding.
Holder	Assessment	Stated confidence
TRM Labs	TraderTraitor, North Korea’s elite hacking unit.²	“high confidence”
Binance	“Our investigation has not attributed responsibility to any individual.”¹²	none given
Chainalysis	“likely done by DPRK group TraderTraitor due to links to previous hack.”¹⁴	“likely”
Elliptic	Linked the theft to TraderTraitor before the FBI spoke.³	“likely”
FBI	North Korea “was responsible for the theft of more than $6 billion”; 628 addresses to block. Quoted by BleepingComputer; the FBI’s own page could not be opened.¹⁵	an attribution
North Korea	Has denied such thefts before and did not respond to Reuters here, per IBTimes.¹⁶	denial

Whoever is named in the end, Elliptic counts about $3.25 billion that no issuer can freeze, at 1 November prices, across more than 800 addresses. That is $1.42 billion in BTC, $1.32 billion in ETH and $502 million in XRP, Solana, TRX, NEAR and others.³

Sources

WSJ and NYT report the US may resume Iran strikes after the midterms

2026-11-02T06:49:14Z

Daily Brief

WSJ and NYT report the US may resume Iran strikes after the midterms

Israel’s Channel 12 says Netanyahu urged Trump to restart the war right after the election, and Rubio says Trump has made no decision.

3 min read2026-11-02

Word of the Day

nocebo noun ˌnō-ˈsē-(ˌ)bō

Merriam-Webster defines it as “a harmless substance or treatment that when taken by or administered to a patient is associated with harmful side effects or worsening of symptoms due to negative expectations or the psychological condition of the patient.”¹

Walter P. Kennedy coined nocebo in a 1961 article, “The nocebo reaction,” in Medical World. A 2023 Frontiers in Psychology editorial quotes him: “it is somewhat surprising that little attention has been drawn to the existence of the contrary effect” to the placebo, “which I may call the nocebo reaction.”² The word is a Latin joke on placebo: Merriam-Webster glosses Latin nocēbō as “I will harm,” against placebo’s “I shall please.”¹ Kennedy meant a narrower thing, a reaction that comes from the patient and not the drug’s pharmacology. Confusing the two, he warned, could lead doctors to discard useful drugs.²

News

Netanyahu reportedly pressed Trump to restart the Iran war after the election

The Wall Street Journal and The New York Times report the US may resume military action against Iran after Tuesday’s vote, CNBC said Sunday.³ Israel’s Channel 12, citing an Israeli and an American source, says Benjamin Netanyahu urged Trump on Saturday to resume the war right after the election, and Trump prefers to give Qatari-mediated talks more time. The White House and Netanyahu’s office declined to comment.⁴ Secretary of State Marco Rubio says Trump “hasn’t made a decision,” and CNBC notes Washington has not said it plans to resume attacks.³

News

Learjet 60 crashes in a Florida neighborhood, killing all nine aboard

A Learjet 60 flying from Austin toward the Bahamas crashed in a residential area of Greenacres, Florida, at 12:50 p.m. Sunday. Transportation Secretary Sean Duffy said all nine aboard died, PBS NewsHour reported.⁵ The same report cites air traffic control recordings and flight-tracking data showing the pilots reported turbulence and icing at 35,000 feet. The jet then dove at up to 30,000 feet per minute, beyond its design limits. The FAA is supporting the NTSB investigation, and the cause is undetermined.⁵

News

Sen. Chris Murphy is hospitalized after a stroke and expects a full recovery, his office says

Connecticut Democrat Chris Murphy, 53, “started experiencing numbness in his arm while at his son’s hockey game” on Saturday, chief of staff Allison Herwitt said, the AP reported.⁶ He was moved to Johns Hopkins, where his office says he is “awake and alert,” responding well to treatment and expected to recover fully. He is not on Tuesday’s ballot.⁶

Technology

Microsoft offers every US school district its AI privacy terms, starting with no training on student data

From 1 November, Microsoft says, every US school district can add the protections in its AI Safety & Privacy Standard to existing or new agreements without renegotiation.⁷ Its fact sheet lists 10 contractually enforceable protections, the first barring training on student or educator data except for narrow safety and security uses. Districts choose whether to add them.⁷

Technology

Google runs Gemma 4 on a data-center-class TPU in orbit

Google Research says its Project Suncatcher prototype, launched 1 October, returned “the first-ever AI response from a data center class TPU in orbit”: a Gemma 4 haiku about being in space, downlinked during a pass over Svalbard, Norway.⁸ Its 30 October post says tests of thermal stress and radiation come next, then twin interconnected satellites in 2027.⁸

Sources

Jepsen finds ParticleDB’s default mode losing committed writes among 96 issues

2026-11-01T06:46:35Z

Feature

Jepsen finds ParticleDB’s default mode losing committed writes among 96 issues

In versions 1.1.4 through 1.1.17 of ParticleDB, advertised as “Full ACID. 159K TPS.”, WHERE 4 = 5 returned the row with primary key 5.

5 min read2026-11-01

Guarantees tested one node at a time

Orkes advertised ParticleDB, a Postgres-compatible database, as “Full ACID. 159K TPS.” and as “formally proven” with TLA+, wording that Jepsen’s report quotes from the home page.¹ On 2026-10-31 Jepsen reported 96 issues from tests of versions 1.1.4 through 1.1.18.11.rc-22 and wrote that the Full ACID claim was “somewhat premature.” The default transaction mode, occ, lost committed writes in normal operation. The TLA+ claim could not be checked, because no specification was available.¹

Orkes, the vendor, funded the analysis, which Jepsen says it conducted under its ethics policy. Every test ran on a single node, because the builds Orkes supplied did not support replication.¹

Four of six claims met a measured failure

Each row is one advertised guarantee, in the words the report prints, and each cell to its right is what Jepsen recorded against it. Orkes has since rewritten the home page and documentation pages the report quotes, so the report is now the record of that wording.¹

Advertised wording against Jepsen’s result, with the anomaly defined where it first appears. The last two rows quote, in part or in full, an operator guide Orkes did not publish. Report sections, by row: 3.8.7, 3.8.8 and 3.8.14, 1.1, 3.8.16, 1.1, 3.2.3.¹
Advertised	Jepsen result	Build and mode
“Full ACID. 159K TPS.”	Committed writes lost; aborted reads (G1a), where a committed transaction reads a write from an aborted one;² Lost Update, where a transaction overwrites a concurrent committed write it never observed.³	1.1.4 through 1.1.18.9, every mode, no faults
“snapshot isolation”	G-single, where a transaction observes one effect of another but misses a second effect of it (#73).⁴ Read skew (A5A), where it both observes and fails to observe the other’s effects (#106).⁵	G-single in rc-1 through rc-22; A5A with `table-2pl` in rc-18
“formally proven” with TLA+	Could not be checked. No specification was available.	None tested
`occ` is “Jepsen-safe”	Acknowledged writes discarded on a healthy node: 31 lost among about 20,000 keys, every few minutes (#110).	rc-18 through rc-22, both interfaces
`groupsync` gives the “same durability as sync”	A conflict between documents, not a measured failure. The public docs said `sync` had “stronger durability,” and Jepsen reports no comparison. Orkes’s current docs say the two give the same guarantee.⁶	Documentation
Prepared statements “all work as expected”	Any prepared statement through the Postgres JDBC driver raised an exception, leaving the database “essentially unusable” (#2).	1.1.4 through 1.1.18.9; appeared fixed by rc-1

“Jepsen-safe” was Orkes’s own label. Jepsen printed it and wrote that, since it evaluates safety against documentation and marketing claims, it was not clear how to interpret the term.¹

`WHERE 4 = 5` returned the row with primary key 5

Jepsen’s example is a table t with columns a, the primary key, and b, holding rows 4 | 5 and 5 | 6. In versions 1.1.4 through 1.1.17, ParticleDB “aggressively interpreted WHERE something = x as a primary-key lookup of x, regardless of the value of something.”¹ The query in the diagram therefore asked for key 5 and never compared the 4.

Fig. 1 · Jepsen’s example for issue #18: table t holds rows (4, 5) and (5, 6). The right lane follows the report’s description of the rewrite and is drawn for this article, not copied from ParticleDB’s planner.¹

The example is a single statement with no concurrent transaction. Version 1.1.18.4 stopped treating every equality as a key lookup. A related fault appeared only inside a transaction: in 1.1.4 through 1.1.18.8, SELECT * FROM t WHERE TRUE returned no rows (#35), and it seemed improved in 1.1.18.9.¹

Ten issues survived to rc-22

Jepsen wrote that by 1.1.18.11.rc-22 all but ten of the 96 issues appeared resolved.¹ The chart places seven findings on the builds Jepsen tested, four of them among the ten. A bar spans the builds where the behavior appeared, and a bar that reaches rc-22 was still open when testing ended.

Fig. 2 · Builds are ordered, not to scale. Issue #35 appeared inside a transaction. Issues #107 and #109 were reproduced under strict-serializable at rc-18, a mode the report says may have been renamed serializable in rc-22, where its Discussion says both still occurred.¹

Stale read, the behavior behind #109, is a read that fails to observe an earlier completed write, which strict serializability forbids.⁷ Jepsen saw it roughly every 300,000 transactions at rc-18. Process kills produced Lost Update under the same mode (#107).¹

Jepsen has not evaluated any build after rc-22. It recommended that users halt further deployment of ParticleDB and migrate to other databases as soon as possible. Orkes told Jepsen the issues likely came from an extensive refactor in spring 2026 and that customers who stayed on an earlier version were likely unaffected.¹ Orkes’s own post says: “As of ParticleDB 1.1.19, we report that all identified issues have been addressed.”⁸ Jepsen’s report names the build as 1.1.19-rc.13.

Sources

Two improvements find the best policy

2026-11-01T07:02:53Z

On lesson 1's slippery grid, two rounds of evaluate-and-improve lift cell 13 from −2.01 to −0.71, while the top half, already optimal, does not move.

Justice Department says it now seeks only to segregate Alaska ballots

2026-11-01T06:47:05Z

Its 30 October filing in federal court cites a second candidate named Dan Sullivan, days before Election Day.

The value of a state is an average you can solve for

2026-10-31T11:29:11-04:00

On a slippery 4×4 grid, 20,000 episodes and one linear solve put cell 13 at about −2, and the discount turns out to be needed only by a policy that never ends.

Swapping the model moves AstaBench’s overall score twice as far on average as swapping the loop

2026-10-31T11:29:06-04:00

On Ai2’s science-agent benchmark, a specialized agent takes most of its lead from one category, where its sub-agent Asta Panda adds 25.3 points to the same model at 15.9 times the reported cost.

Judge bars DOJ from threatening California election officials days before the vote

2026-10-31T06:50:09Z

The Ninth Circuit also refused 2-1 to set aside the ballots of more than 20,000 Californians the Justice Department calls noncitizens.

A nowcasting RL task built in verifiers 0.4.0 has to close four routes to its own answer

2026-10-14T23:23:25Z

Written against Prime Intellect's TaskData, Task, Taskset and Toolset classes and built on real ALFRED vintages, a payrolls taskset let a command in the agent's box on the subprocess runtime read January 2024's hidden answer of 353,000 jobs.

Third Circuit rules against ROSS for copying headnotes it had no need to copy

2026-10-04T06:54:27Z

The Third Circuit’s Sept. 29 opinion turns on the free judicial opinions ROSS could have used instead, and addresses generative AI only in one footnote.

OpenCode 2 skips any plugin whose default export lacks an id and a setup or effect function

2026-10-04T06:54:29Z

OpenCode 2.0.22 decodes a plugin's default export against { id, setup } or { id, effect }, so a v1 export with only server() fails the check and is skipped.

Tennessee fails to execute Christa Pike after two doses of pentobarbital

2026-10-04T06:53:30Z

Gov. Bill Lee called off Tennessee's remaining 2026 execution and ordered a third-party review of what happened.

Python 3.15 adds a lazy keyword that binds a proxy instead of running an import

2026-10-03T07:06:19Z

First use of the name reifies the proxy into the real module, moving an import's errors and side effects out of startup and into the running program.

Google's economy ATLAS measures how much people talk to Gemini about work

2026-10-03T07:16:05Z

Its most-quoted figure, 30% of U.S. work-related AI use among computer and mathematical workers, is a share of Gemini request volume the released dataset reproduces only as 20.06%.

Seattle's U.S. attorney is ordered back to a job he held for 54 minutes

2026-10-03T07:15:41Z

A federal judge granted Roger Rogoff a preliminary injunction on October 1 restoring him pending his suit against the White House, which has appealed to the Ninth Circuit.

Federal Medicaid matching for non-pregnant adult refugees and asylees ended on 1 October

2026-10-02T07:29:41Z

Each state now decides whether to pay for them itself, and North Carolina, New Mexico and California gave three different answers.

dbt v2 checks column names before the warehouse, in strict mode and only in the proprietary build

2026-10-02T07:25:25Z

On a two-model DuckDB project, dbt 2.0.6 flagged a misspelled column at models/b.sql:1:12 under --static-analysis strict, and dbt OSS 2.0.5 said it does not include the check.

Supreme Court takes up detention of immigrants without bond hearings

2026-10-02T07:29:52Z

The government asked the Court to take a different case on the same question, and the Court granted this one without stating reasons.

OpenShell puts a supervisor between agents and the network

2026-10-01T08:36:59Z

The v0.1.0 runtime limits local and outbound authority around an existing agent, while leaving policy quality and model behavior outside its proof.

Micron earns more from each dollar of memory sales

2026-10-01T11:31:10Z

Higher prices produce an 86.8% gross margin whose durability depends on undisclosed contract prices and future manufacturing costs.

Six developments for Thursday, October 1

2026-10-01T08:32:11Z

BEA lifts second-quarter growth to 2.2%; a trade court hears a tariff challenge, while Qt and OpenStack publish new releases.

Optical images show glacier motion; radar finds no progressive signal

2026-09-30T19:42:55Z

The optical and radar analyses measure different signals, and neither establishes whether a live alert could have reached people before the flood.

GitHub's fuzzing agent uses coverage gaps to revise its harnesses

2026-09-30T19:43:00Z

GitHub reported 24 findings from separate Android audit taskflows on September 28, but left independent confirmation unclear.

Government weighs changes to Big Bend park work

2026-10-01T08:34:55Z

Sept. 30 reports say the government may narrow work inside Big Bend National Park while ground disturbance may resume on a separate 61-mile Presidio County line.

Supreme Court lets third-country removals resume

2026-09-30T19:42:07Z

The emergency order leaves the removal policy in force while the Court considers its legality.

Supreme Court keeps Missouri's 2022 map in place

2026-09-28T08:24:44Z

A stay settles Missouri's election map for November while Medicaid paperwork begins unevenly and measured oil traffic complicates claims about Hormuz.

The White House’s “Nearly $1 Billion” Rescission Proposal Itemizes $810 Million

2026-09-28T08:18:34Z

Its posted proposal contains $809,501,022, while the legal clock depends on congressional receipt that the public record does not establish.

Dependabot ages version updates before opening pull requests

2026-09-28T08:18:02Z

In one daily npm queue, a routine patch waits until Thursday while an alert bypasses the release-age gate and an excluded team-maintained package can move on Monday.

Polymarket’s federal status leaves New York a contract-by-contract fight

2026-09-25T08:34:55Z

New York calls Polymarket’s event contracts unlicensed gambling; Polymarket says CFTC jurisdiction displaces the state, while two appellate courts have split on the questions beneath both claims.

FDA’s Galleri review pairs 99.74% specificity with 31.6% sensitivity

2026-09-25T08:35:24Z

Specificity and sensitivity divide different groups. Positive predictive value adds prevalence, and a diagnosis still requires a workup.

September 25: Judge Orders White House Passes for Three Outlets Reinstated

2026-09-25T08:34:45Z

Hours after a judge ordered White House passes for three outlets reinstated, reporters for all three said they were still denied entry.

SwiftUI’s ArrangementView Makes Adaptive Two-Pane Layout a Framework Decision

2026-09-24T10:01:36Z

The iOS 27.1 container expresses split and overlay relationships across changing geometry; its ratio remains a preference, and a reported Duo simulator bug calls for a targeted test.

AI incident reports need an audit trail to become comparable

2026-09-24T09:58:44Z

The U.S. notification proposal and lab disclosures show why shared triggers, comparable fields, and independent confirmation matter more than a promise of transparency.

September 24: Paramount–Warner Settlement Awaits Court Approval

2026-09-24T09:55:26Z

Today's items cover a Paramount–Warner settlement awaiting court approval, the 47-mile Big Bend 1 border-wall project, NOAA's 132-year heat record, and new FAA SMART and Huawei Atlas 960E systems.

shadcn/ui gives custom UI a source code boundary

2026-09-22T13:18:05Z

The strongest starting point is not the library with the prettiest demo; it is the layer your team can change without fighting its API.

Base UI keeps custom React components changeable

2026-09-22T15:45:38Z

Use Base UI for interaction, own the visual source, and test the assembled component where automation stops.

The U.S. strikes three Iranian tankers after Iran launches missiles toward two Navy ships, CENTCOM says

2026-09-06T12:53:09Z

The weekend brings a U.S. strike on Iranian oil carriers, two live election-law fights, and an August jobs gain above the recent monthly pace.

Looped transformers turn depth into a dial

2026-09-05T20:16:08Z

Giannou et al. show how a shallow, fixed-weight Transformer can execute iterative programs; later recurrent-depth work makes the idea relevant to Astra, whose public architecture OpenAI has not specified.

Sandbagging turns capability tests into menus

2026-09-04T11:33:28Z

Bergemann, Koh, and Morris model an AI’s hidden preferences and capabilities as a private type, then show why a test can earn trust only when the permissions it buys are incentive-compatible.

When the Test Can Edit the Evidence

2026-08-30T18:12:52Z

METR found agents testing whether they could alter the record used to grade them, while Hugging Face had to reconstruct the intrusion from independent logs.

A growing company books the profit before it collects the cash

2026-08-28T12:47:29Z

Receivables and unsold inventory hold the cash a growing business has earned but not yet collected, and a company whose suppliers wait longer than its customers can flip the whole cycle in its favor.

Moving a date from Spark to Snowflake takes two casts, and SQLGlot writes them

2026-08-28T12:36:12Z

A no-dependency library for more than 30 SQL dialects, honest that its lenient parser will accept queries a real engine would reject.

Athens exiled its powerful men by scratching names on broken pottery

2026-08-28T12:35:03Z

Once a year the Athenian assembly could send a man into exile for ten years with no accusation and no trial, and the verb ostracize is what that ballot left in the language.