Actor-Critic architectures (A2C/A3C)

Back to the familiar gridworld, but now two things learn at once: an actor (a softmax policy, one preference per action per state, just like REINFORCE) and a critic (a $V(s)$ table, updated the same TD way as the value-function chapters). Step through a trace and watch both bar charts move together — every single step, off the exact same number.

The critic learns $V(s)$ by ordinary one-step TD, producing a TD error:

The actor then uses that same $\delta$ as its advantage estimate, in place of REINFORCE's raw return $G$:

• $\delta$ — the TD error: how much better or worse the actual outcome was than the critic's own expectation.
• $V(s)$ — the critic's current value estimate for state $s$.
• $\pi_\theta(i)$ — the actor's current probability of action $i$, from the softmax over its preferences.

1. Same update shape as REINFORCE, different scale

   The $(\mathbb{1}[i{=}a] - \pi_\theta(i))$ term is identical to the REINFORCE chapter. The only thing that changed is what multiplies it: a baseline-subtracted advantage instead of a raw return.

2. Why this reduces variance

   $G$ can swing wildly step to step, purely from randomness far in the future. $\delta$ only measures the gap between one real step and the critic's own prediction — a much smaller, steadier number to scale a gradient step by, even though it points in the same useful direction on average.

3. Two learners, one shared signal

   The critic isn't "used" and then discarded — it's trained by its own TD update every step, using the exact same $\delta$ the actor consumes. Neither one needs to fully converge before the other starts learning from it.

Before any updates, $\pi(\cdot \mid S_2)$ is a coin flip between left and right — nothing has been learned yet. After six steps, "right" has pulled ahead, driven entirely by the critic's own value estimates, which shifted right alongside it.

With $\alpha=0.5$ for both actor and critic, $\gamma=0.9$, starting from an all-zero critic and a uniform actor:

1. Step 1 — no signal yet, so the update is uninformative but harmless

   From $S_0$, "right" lands on $S_1$ with reward $-1$; $V(S_1)=0$ still, so $\delta = -1+0.9(0)-0=-1$. $V(S_0) \leftarrow 0.5(-1) = -0.5$. With the policy still uniform ($\pi=0.5$ each way), the actor update $\theta_i \leftarrow \theta_i+0.5(\mathbb{1}[i{=}\text{right}]-\pi_i)\delta$ gives:

   • $\theta(S_0,\text{right}) \leftarrow 0+0.5(1-0.5)(-1) = -0.25$
   • $\theta(S_0,\text{left}) \leftarrow 0+0.5(0-0.5)(-1) = 0.25$

   Discouraging the very action just taken, purely because the critic (still near-zero everywhere) rated the outcome as worse than expected.

2. Step 3 — a real mistake, correctly penalized

   From $S_2$, taking "left" moves away from the goal, landing on $S_1$ (whose value, by now, is already $-0.5$, from step 2's own critic update). $\delta = -1 + 0.9(-0.5) - 0 = -1.45$ — a bigger penalty than step 1's, because this time the critic has real (negative) information about where "left" leads. With $S_2$'s policy still uniform:

   • $\theta(S_2,\text{left}) \leftarrow 0+0.5(1-0.5)(-1.45) = -0.3625$
   • $\theta(S_2,\text{right}) \leftarrow 0+0.5(0-0.5)(-1.45) = 0.3625$

   Already favoring "right" with more than half the probability, after a single update.

3. Step 5 — reaching the goal spikes the advantage

   From $S_2$, "right" reaches the goal directly: reward $+10$, and $V(S_2)=-0.725$ at that point (from step 3's own critic update), so $\delta = 10+0.9(0)-(-0.725) = 10.725$ — far larger than anything seen so far. The critic updates $V(S_2) \leftarrow -0.725+0.5(10.725) = 4.6375$. The actor, starting from step 3's $\theta(S_2,\text{right})=0.3625$ (policy $\pi_{\text{right}}\approx0.674$), updates to $\theta(S_2,\text{right}) \leftarrow 0.3625+0.5(1-0.674)(10.725) \approx 2.11$ — the actor and critic correcting course together, in the same step.