Sequence-to-Sequence models & encoder-decoders

One context vector, decoded out to however many steps you ask for. The encoder that produced it never runs again — everything the decoder knows about the input has to already be packed into that single number.

An encoder RNN reads the entire input sequence and keeps only its final hidden state — the context vector. A separate decoder RNN is initialized from that context and unrolls its own sequence, feeding each output back in as the next step's input, for as many steps as the task needs:

• $c$ — the context vector: the encoder's final hidden state, and the only channel of information passed to the decoder.
• $x_1, \dots, x_n$ — the input sequence, of length $n$.
• $y_1, \dots, y_m$ — the output sequence, of length $m$, produced by the decoder.
• $\text{Encoder}$ — the RNN that reads the entire input sequence and compresses it down to the context vector.
• $\text{Decoder}$ — the RNN that unrolls its own sequence starting from the context vector.

1. Input and output lengths are decoupled

   Nothing ties $n$ and $m$ together — the decoder can unroll for however many steps the task needs, independent of how long the input sequence was.

2. One network compresses, the other generates

   The encoder's job is compression; the decoder's job is generation — and the context vector is the only channel between them.

Three input steps compress down to one number. That same number then drives four decoder steps, each depending only on the context and the decoder's own previous output — the input sequence itself is gone the moment the context vector is formed.

Input sequence $x = (1, -1, 1)$, encoder recurrence $h_t=\tanh(x_t + 0.5\,h_{t-1})$:

1. Encoding compresses three steps into one number

   • $h_1=\tanh(1)\approx 0.7616$
   • $h_2=\tanh(-1+0.5(0.7616))\approx -0.5506$
   • $h_3=\tanh(1+0.5(-0.5506))\approx 0.6198$

   The context vector is $c=h_3\approx 0.6198$ — everything about the 3-token input, folded into one scalar.

2. Decoding starts from the context, not the input

   $y_1=\tanh(c)\approx 0.5510$. From here the decoder only sees its own last output: $y_2=\tanh(0.5\,y_1)\approx 0.2687$.

3. The same context, unrolled further or not at all

   Running the decoder for 4 steps instead of 2, continuing the exact same recurrence from $y_2$:

   • $y_3=\tanh(0.5\,y_2)=\tanh(0.5\times0.2687)\approx 0.1336$
   • $y_4=\tanh(0.5\,y_3)=\tanh(0.5\times0.1336)\approx 0.0667$

   Each output smaller than the last, since nothing new is feeding the recurrence. The output length was never determined by the input length; it's just how many times you choose to run the decoder.