The Power of Powers
Why does the weight matrix decide everything? Because applying it over and over is the same as raising it to a power. Let's unroll the loop.
Start at a₀ and step forward, substituting each result into the next:a₁ = W·a₀a₂ = W·a₁ = W·(W·a₀) = W²·a₀a₃ = W·a₂ = W³·a₀
The pattern is unmistakable:
aᵢ = Wⁱ · a₀
The activity at step i is the weight matrix raised to the i-th power, times the starting activity.
So the circuit's entire future is controlled by what happens to W when you raise it to higher and higher powers. And here's the magic: raising a matrix to a power is easy to understand through its eigenvalues — because along an eigenvector, a matrix is just a number (λ).
That's the bridge. Wⁱ looks scary, but along each eigenvector it becomes plain λⁱ — a single number raised to a power. And you already know how λⁱ behaves: it explodes if λ > 1 and vanishes if λ < 1.