Deep Q-Networks (DQN)

Same 4-state gridworld, same 6-move script Q-learning used — but instead of a table entry, $Q(s,a)$ now comes from a network with exactly 6 numbers: one hidden unit feeding two output values, one per action. Step through the same moves and watch those 6 numbers get nudged by gradient descent, using the exact same TD-error signal Q-learning used.

A Deep Q-Network replaces the table with a function $Q(s,a;\theta)$, trained by gradient descent on the squared TD error:

• $\theta$ — the online network's parameters, the ones being trained right now.
• $\theta^-$ — the target network's parameters: a frozen, periodically-refreshed copy of $\theta$.
• $Q(s,a;\theta)$ — the online network's current prediction for state-action pair $(s,a)$.

1. This is Q-learning's TD error, squared into a loss

   The parenthesized term is exactly the same TD error from the Q-learning chapter. DQN just treats it as a regression target and takes a gradient step against it, instead of nudging a table entry directly by $\alpha \times$ error.

2. Why a separate target network at all?

   If $\theta^-=\theta$, the target moves every time $\theta$ does — the network would be chasing a target it just changed, one gradient step ago. Freezing $\theta^-$ for a stretch of steps (only periodically copying $\theta$ into it) gives the online network a stable target to regress toward, instead of an ever-shifting one.

3. Experience replay, in one sentence

   Real DQN also stores past transitions in a buffer and samples random past experience for each update, instead of training only on the transition just seen — this breaks the strong correlation between consecutive steps that plain online updates would otherwise have. The toy script here is too short to need it, but every update it runs has the exact same shape as one replay-buffer sample.

A table needs one slot for every state times every action — double the state count, and it doubles in size. The network's parameter count never moves: the same 6 numbers that handled 4 states could, in principle, be asked to handle 1000. Whether they'd generalize well is a separate question — but they don't run out of room the way a table eventually would.

Starting from $Q_\theta(0,\text{right})=-2$ (computed from the network's initial 6 numbers), with learning rate $0.05$:

1. Step 1 — a clean TD error of exactly 1

   Taking "right" from state 0 lands on state 1 with reward $-1$. Since $\theta^-=\theta$ still at this point, the target network's best action-value at state 1 happens to be exactly $0$ ($\max(Q(1,\text{left}), Q(1,\text{right})) = \max(0,-1) = 0$), so the target is $-1 + 0.9(0) = -1$. TD error $= -1 - (-2) = 1$, loss $=1$. The hidden unit is inactive at state 0 ($h=\max(0, 1\times0+0)=0$), so $Q_\theta(0,\text{right})=w_{\text{right}}\times0+b_{\text{right}}$ depends only on the output bias — the weight's gradient is $-2\times1\times h=0$, but the bias's gradient is $-2\times1\times1=-2$. Gradient descent: $b_{\text{right}} \leftarrow -2 - 0.05\times(-2) = -1.9$ — a small step toward the target, the same shape as a Q-table update but landing on a shared parameter instead of an isolated cell.

2. Step 3 — a 'left' move with zero TD error

   Taking "left" from state 2 happens to produce a target that already exactly matches the current prediction (both $-1$) — TD error $0$, loss $0$, and the left-branch weights don't move at all this step. Zero error means zero gradient, exactly like a reward of $0$ left the softmax policy untouched in the REINFORCE chapter.

3. Step 4 — the target network has just re-synced

   After step 3, the target network copies the online network's weights. Step 4's target computation now uses those freshly-updated numbers instead of the original initialization — the "moving target" problem is kept in check by only letting it move in these periodic jumps, not on every single step.