Inverted residuals & MobileNets

Hook

A ResNet block runs a full dense convolution at every layer — every output channel looking at every input channel, at every position in the kernel. That's exactly the kind of arithmetic a phone's battery can't afford. What if most of those multiplications were spent re-mixing channels that filtering and mixing didn't need to happen together at all?

Intuition

standard 3x3 conv

576

depthwise-separable

136

at 8 channels: separable uses 23.6% of the standard conv's parameters

Same 3x3 kernel, same 4x4 output, same input and output channel count — the only difference is whether one dense convolution does the whole job, or two much smaller ones split it in half. Step through the channel counts and watch the gap between the two bars grow.

Formalize

A standard $K\times K$ convolution mixes space and channels in a single operation. Depthwise separable convolution splits that into two cheaper steps: a depthwise conv that filters each channel on its own (space only), followed by a pointwise $1\times1$ conv that mixes channels (no spatial extent at all):

\text{separable params} = \underbrace{K^2 C}_{\text{depthwise}} + \underbrace{C^2}_{\text{pointwise}} \qquad\text{vs.}\qquad \text{standard params} = K^2 C^2

$K$ — the kernel's spatial size (here, $3$ ).
$C$ — the number of channels (input and output are equal here, so a residual add is possible).

The ratio is exact, and independent of image size
Dividing separable by standard params gives $\frac{K^2C + C^2}{K^2C^2} = \frac{1}{C} + \frac{1}{K^2}$ — a clean identity that holds at any spatial resolution, since the spatial factor cancels out entirely.
Wider layers save more, not less
As $C$ grows, $1/C$ shrinks toward $0$ , so the ratio approaches a floor of $1/K^2 = 1/9$ — roughly a $9\times$ reduction at the widest layers, versus a much smaller saving at narrow ones.
MobileNet inverts the usual bottleneck order
An inverted-residual block puts this depthwise filter inside an expand-then-project bottleneck:
- a $1\times1$ expands $C$ channels to $tC$
- a depthwise $K\times K$ filters the expanded space
- a linear $1\times1$ projects back down to $C$
so the block's output can add back onto its input — the reverse of a ResNet bottleneck, which narrows before it widens.

Play

4 ch

0.361

8 ch

0.236

16 ch

0.174

32 ch

0.142

separable/standard parameter ratio — shrinking as channel count grows, toward a floor of 1/k²

standard FLOPs @ 16ch

36,864

separable FLOPs @ 16ch

6,400

The exact ratio from the formula, plotted across the same channel counts, alongside the raw FLOP counts it implies at 16 channels — over 4x fewer multiply-adds for the identical input and output shape.

Worked example

$K=3$ , a $4\times4$ output feature map, $C=4$ channels, expansion $t=6$ :

Standard vs. separable at 4 channels
- Standard: $K^2C^2 = 9\cdot16=144$ params, $144\times16=2304$ FLOPs
- Separable, depthwise: $K^2C=9\cdot4=36$ params
- Separable, pointwise: $C^2=16$ params
- Separable, total: $36+16=52$ params, $52\times16=832$ FLOPs
About $36\%$ of the standard conv's cost, matching $1/4+1/9\approx0.361$ exactly.
The inverted-residual bottleneck, expanded to 24 channels
- Expand $4\to24$ : $4\times24=96$ params
- Depthwise $3\times3$ on $24$ channels: $9\times24=216$ params
- Project $24\to4$ : $24\times4=96$ params
Total: $96+216+96=408$ params.
What a dense middle conv would have cost instead
Swap the depthwise filter for a dense $3\times3$ over the same expanded $24$ channels: $9\times24\times24=5184$ params, plus the same two $1\times1$ layers, totals $5376$ — over $13\times$ the inverted-residual block's $408$ . The entire saving lives in that one swapped layer.

Checkpoint

Find the channel count where the separable/standard parameter ratio first drops below 0.15.

Pick a channel count to try it

Summary

\frac{\text{separable params}}{\text{standard params}} = \frac{1}{C} + \frac{1}{K^2}

Splitting spatial filtering from channel mixing turns one expensive dense convolution into two cheap ones, saving more as layers get wider. MobileNet's inverted-residual block puts that same depthwise trick inside an expand-then-project bottleneck — the mirror image of a ResNet block's narrow-then-wide shape — so a network can keep the residual connection's training benefits while running on a fraction of the compute. The next chapter picks up a pretrained network like this one and asks how much of it can be reused as-is.