Reinforcement Learning
ε-greedy control settles for the best policy that keeps exploring
Monte Carlo control with ordinary starts needs a policy that sometimes moves at random, and the price of those moves grows with ε, first in value and then in the learned arrows.
Action values let an agent improve its policy from episodes alone
Monte Carlo control with exploring starts reached the optimal policy in nearly every run on the course's slippery grid, where no convergence theorem yet applies.
An agent can learn what each state is worth without knowing the rules
Averaging whole episodes gives unbiased but noisy targets, and updating after every move gives quieter targets that carry the errors in the agent's own estimates.
Two improvements find the best policy
On lesson 1's slippery grid, two rounds of evaluate-and-improve lift cell 13 from −2.01 to −0.71, while the top half, already optimal, does not move.
October 2026The value of a state is an average you can solve for
On a slippery 4×4 grid, 20,000 episodes and one linear solve put cell 13 at about −2, and the discount turns out to be needed only by a policy that never ends.