Expanding attention via a Taylor-series approach yields linear-time transformers with a fixed number of recurrent states
Read original on arXiv cs.LG ↗arXiv:2610.20637v1 Announce Type: new Abstract: We expand the exponential kernel of attention into a truncated Taylor series of $n$ terms and factorize each term into a tensor-power inner product, which collapses...
Original headline: "Expanding Attention: A Taylor-Series Route to Linear-Time Transformers with a Fixed Number of Recurrent States"
Timeline
- October 16, 2026 Expanding Attention: A Taylor-Series Route to Linear-Time Transformers with a Fixed Number of Recurrent States arXiv cs.LG