Graph attention networks

Click a node. Every edge in its neighborhood gets its own thickness now — that's an attention weight, not a fixed $\frac{1}{\deg(v)+1}$ share. Node 3's neighbors don't all matter the same amount to it, and the graph now shows exactly how much each one does.

A Graph Attention Network (GAT) layer replaces GCN's fixed mean with a learned, per-neighbor weight. For node $i$ and each member $j$ of its neighborhood (itself included):

• $e_{ij}$ — the raw, unnormalized attention score between node $i$ and neighbor $j$.
• $\alpha_{ij}$ — the normalized attention weight node $i$ assigns to neighbor $j$, after softmax.
• $W$ — a learned weight matrix applied to every node's features before scoring.
• $a$ — a learned attention vector that scores how compatible two transformed features are.
• $h_i$, $h_j$ — node $i$'s and neighbor $j$'s current feature vectors.
• $h_i'$ — node $i$'s updated feature: the attention-weighted sum over itself and its neighbors.
• $\mathcal{N}(i)$ — node $i$'s neighborhood, including itself.

1. W and a score compatibility, like query-key attention

   $W$ is a learned weight matrix and $a$ a learned attention vector — together they score how compatible $i$ and $j$ are, exactly the way a query and key score compatibility in Part IV's attention chapter.

2. This chapter's simplification

   With one scalar feature per node and $W$ fixed to the identity, that score collapses to a straight comparison of the two raw features. This chapter fixes it at $e_{ij} = \text{LeakyReLU}(h_j - h_i)$.

3. LeakyReLU lets asymmetric signal through

   A neighbor with a much larger feature scores far higher, one with a much smaller feature is only mildly suppressed — LeakyReLU's whole point is to leak a little of that signal through instead of zeroing it, the same nonlinearity from Part III.

Every bar is one node's GAT update minus what GCN's plain mean would have produced, on the exact same starting graph. None of them are zero — attention never reduces to a flat average unless every neighbor already agrees. Node 4's bar is the tallest: it's a leaf with exactly one neighbor, so there's no averaging-out effect to soften how hard that neighbor's much larger feature pulls the update.

Node $1$ (feature $2$), attending to itself and its three neighbors — node $0$ (feature $1$), node $2$ (feature $0$), node $3$ (feature $3$):

1. Raw scores

   • $e_{1,1}=\text{LeakyReLU}(2-2)=\text{LeakyReLU}(0)=0$
   • $e_{1,0}=\text{LeakyReLU}(1-2)=\text{LeakyReLU}(-1)=0.2\times(-1)=-0.2$
   • $e_{1,2}=\text{LeakyReLU}(0-2)=\text{LeakyReLU}(-2)=0.2\times(-2)=-0.4$
   • $e_{1,3}=\text{LeakyReLU}(3-2)=\text{LeakyReLU}(1)=1$

   Node $3$'s score is the only one that grows past its raw gap — every negative score gets shrunk to a fifth of its size.

2. Softmax turns those into weights

   Subtracting the max score ($1$) for stability, then exponentiating:

   • $e^{0-1}=e^{-1}\approx0.368$ (self)
   • $e^{-0.2-1}=e^{-1.2}\approx0.301$ (node $0$)
   • $e^{-0.4-1}=e^{-1.4}\approx0.247$ (node $2$)
   • $e^{1-1}=e^{0}=1$ (node $3$)

   Sum $\approx0.368+0.301+0.247+1=1.916$. Normalizing:

   • $\alpha_{1,1}\approx0.368/1.916\approx0.192$
   • $\alpha_{1,0}\approx0.301/1.916\approx0.157$
   • $\alpha_{1,2}\approx0.247/1.916\approx0.129$
   • $\alpha_{1,3}\approx1/1.916\approx0.522$

   Node $3$ alone gets more than half the attention, despite being just one of three neighbors.

3. The weighted sum lands far from a plain average

   $h_1' = 0.192(2) + 0.157(1) + 0.129(0) + 0.522(3) \approx 2.107$. GCN's flat mean of the same four numbers is exactly $1.5$ — attention pulls node $1$'s update a full $0.6$ higher, entirely because node $3$ dominates the weighted sum instead of counting for a plain $\frac{1}{4}$.