Explode or Decay?
Run the circuit forward and one of two dramatic things happens: the firing rates blow up toward infinity, or they fade away to zero. And it's the weight matrix alone that decides which.
Compare two circuits. Same starting activity, different weight matrices — watch the fates diverge:
Two connected neurons update each step by a(t) = W·a(t−1). Whether the firing rates blow up or fade away is decided entirely by the largest eigenvalue of W: above 1 → explode, below 1 → decay. This is exactly how researchers check whether a recurrent neural network (RNN) is stable.
The Exploding matrix drives the rates up and up; the Decaying matrix pulls them to zero. Notice the readout: whether it explodes or decays lines up perfectly with the largest eigenvalue being above or below 1. That's not a coincidence — it's the whole secret, and the next two lessons show why.
🤖 The RNN training nightmare: exploding activity = exploding gradients; decaying activity = vanishing gradients. These are the classic obstacles to training recurrent networks — and engineers tame them by literally controlling the eigenvalues of the weight matrix.