Chapter 3 walked through six hand-picked moves. This capstone lets the same Q-learning update run for real: five full episodes, acting on its own evolving table, starting from nothing and ending at the correct solution to the maze.
Step through all 29 training moves across 5 episodes. Early on, the agent wanders — it bounces off the left wall, backtracks, resets. There's no script telling it what to do anymore; every action comes from whatever the table currently believes, mixed with a fixed rhythm of forced exploration.
Nothing new: every step still applies Chapter 3's exact update,
- — the learned value of taking action from state .
- — the reward received for that step.
- — the state landed in after taking the action.
- — a candidate next action to take from that new state.
- — the discount factor applied to future value.
- — the learning rate: how much of the gap gets corrected on any one update.
- Real episodes instead of a fixed script
Instead of a fixed 6-move script, the agent runs real episodes, resetting to the start each time it reaches the goal (or times out).
- Greedy, except for scheduled exploration
It acts greedily on its own table except for scheduled exploration steps that keep testing the alternative — the same exploit/explore split as the bandit chapter, now driving a full agent.
This is the trained policy — read straight off the final Q-table, no further learning involved. From every state it says exactly one thing: go right. Step through it and watch it solve the maze in the minimum three moves, every single time.
Across all 5 episodes:
- Early episodes: mostly wandering
Episodes 0 through 2 never reach the goal at all within their step limit — forced exploration keeps pulling the agent back toward state 0 before it can complete the corridor.
- Episode 3: the goal, for the first time
jumps to exactly the moment the agent first reaches the goal — identical to Chapter 3's very first successful goal-reaching update, just arrived at through the agent's own exploration instead of a scripted move.
- Episode 4: reinforced, and converged
A second successful run pushes to . By the end, at every one of the three non-terminal states — the greedy policy read off the table is "right" everywhere, the actual optimal solution.
Train until the greedy policy says “right” at every state — the actual optimal solution to this maze.
Every idea from this part shows up somewhere in these 29 steps: exploration versus exploitation from the bandit chapter, the Bellman-style value propagation from the MDP chapter, and the same update rule from Chapter 3, just left running long enough to actually solve something instead of illustrating one update. Nothing here needed a bigger algorithm — it needed more of the same algorithm, applied for longer. That's most of what separates a toy example from a working agent.