Q-learning & temporal-difference updates

Instead of one value per state, Q-learning tracks one value per state-action pair — $Q(s,a)$, "how good is it to take action $a$ from state $s$?" Step through this fixed sequence of moves and watch individual entries in that table update, one at a time, from a starting table of all zeros.

After taking action $a$ from state $s$, landing in $s'$ with reward $r$, the Q-learning update nudges $Q(s,a)$ toward a target built from the best action available next:

• $Q(s,a)$ — the learned value of taking action $a$ from state $s$: how good that state-action pair looks so far.
• $r$ — the reward received for that step.
• $s'$ — the state landed in after taking the action.
• $a'$ — a candidate next action to take from that new state.
• $\gamma$ — the discount factor applied to future value.
• $\alpha$ — the learning rate: how much of the gap gets corrected on any one update.

1. The parenthesized term is a TD error

   It's a temporal-difference error — the gap between what $Q(s,a)$ currently predicts and a slightly better estimate built from one real step of experience plus the table's own best guess about what happens after.

2. Alpha is an ordinary learning rate

   $\alpha$ controls how much of that gap gets corrected on any one update — exactly a learning rate, doing exactly what a learning rate does in every other optimizer in this course.

After all six scripted steps, $Q(2,\text{right})$ already shows $5$ — a real signal that reaching the goal from state 2 is good. Everything one state further back is still negative: the reward hasn't had enough steps yet to propagate that far through the table. Run more steps, and it eventually would.

With $\alpha=0.5$, $\gamma=0.9$, starting from an all-zero table:

1. An update with no signal yet

   From $S_0$, taking "right" lands on $S_1$ with reward $-1$, and $\max_{a'}Q(S_1,a')=0$ (nothing learned there yet). $Q(S_0,\text{right}) \leftarrow 0 + 0.5(-1+0.9(0)-0) = -0.5$.

2. The moment the goal is reached

   Three steps later, from $S_2$, "right" reaches the goal directly: reward $+10$, and the goal has no further actions, so $\max_{a'}Q(S_3,a')=0$. $Q(S_2,\text{right}) \leftarrow 0 + 0.5(10+0.9(0)-0) = 5$ — a single update jumps straight to a strongly positive value.

3. One step back, still catching up

   $Q(S_1,\text{right})$ was updated twice:

   1. First, taking "right" to $S_2$ with reward $-1$ and $\max_{a'}Q(S_2,a')=0$ (nothing learned there yet): $Q(S_1,\text{right}) \leftarrow 0 + 0.5(-1+0.9(0)-0) = -0.5$
   2. Later, the same update runs again, but using $\max_{a'}Q(S_2,a')$ which was still $0$ at that moment (the update at $S_2$ hadn't happened yet): $Q(S_1,\text{right}) \leftarrow -0.5 + 0.5(-1+0.9(0)-(-0.5)) = -0.75$

   Further from the goal's value, not closer, purely because of the order the updates happened in.