Model editing (ROME & MEMIT)

At α = 0 the network still answers "capital of France" with Paris; at α = 1 it answers with Rome instead. Meanwhile "capital of Germany" never moves off Berlin, at either setting — because the edit reused a key vector that Germany's key happens to be orthogonal to.

A fact lives in one weight matrix $W$ as a (key, value) association: $W \cdot k \approx v$. Editing it means finding the change $\Delta W$ that forces $W \cdot k$ onto a new target value $v^*$ exactly:

• $W$ — the original weight matrix, mapping every subject's key to its object's value.
• $k$ — the key vector for the one fact being edited (its subject).
• $v^*$ — the new target value (its new object).
• $\Delta W$ — the rank-1 update: an outer product of the needed residual with the key, scaled by $k^\top k$.

1. It hits the target exactly

   $W'k = Wk + \Delta W k = Wk + (v^*-Wk)\dfrac{k^\top k}{k^\top k} = v^*$.
2. It leaves orthogonal keys alone

   For any other key $k_c$ with $k^\top k_c = 0$: $\Delta W k_c = (v^*-Wk)\dfrac{k^\top k_c}{k^\top k} = 0$, so $W'k_c = Wk_c$ — unchanged.

Drag α from 0 to 1: the France fact's Rome-retrieval score rises in a straight line from 0.25 to exactly 1.00, because the output itself is a linear interpolation between the old and new value. The Germany fact's own retrieval score sits at exactly 1.00 for every value of α — it was never touched.

Editing "capital of France" from Paris to Rome, with $k_{\text{France}}=[1,0]$, $v_{\text{Paris}}=[2,1]$, $v_{\text{Rome}}=[0,4]$, $W = \begin{bmatrix}2 & 0\\1 & 3\end{bmatrix}$, and (for checking that the edit leaves an unrelated fact alone) $k_{\text{Germany}}=[0,1]$, $v_{\text{Berlin}}=[0,3]$:

1. Current output for this key

   $Wk_{\text{France}} = [2(1)+0(0),\ 1(1)+3(0)] = [2,1] = v_{\text{Paris}}$, as expected.

2. Residual

   $v^* - Wk = [0,4]-[2,1] = [-2,3]$. And $k^\top k = 1(1)+0(0) = 1$.
3. Rank-1 update

   $\Delta W = [-2,3] \otimes [1,0] = \begin{bmatrix}-2(1) & -2(0)\\3(1) & 3(0)\end{bmatrix} = \begin{bmatrix}-2 & 0\\3 & 0\end{bmatrix}$.

4. Edited matrix and check

   $W' = W+\Delta W = \begin{bmatrix}2-2 & 0+0\\1+3 & 3+0\end{bmatrix} = \begin{bmatrix}0 & 0\\4 & 3\end{bmatrix}$.

   • $W'k_{\text{France}} = [0(1)+0(0),\ 4(1)+3(0)] = [0,4] = v_{\text{Rome}}$ exactly
   • $W'k_{\text{Germany}} = [0(0)+0(1),\ 4(0)+3(1)] = [0,3] = v_{\text{Berlin}}$, untouched — since $k_{\text{France}}\cdot k_{\text{Germany}} = 1(0)+0(1) = 0$