Format: [Arxiv date] ("*Optional Shorthand Title*") | Title | Final Peer Review

1. Linear Recurrent Sequence Models (LRNNs)
   • 1.1 State-space models (SSM)
     • [Oct 2021;v2.Aug’22] (”S4 paper”) | Efficiently Modeling Long Sequences with Structured State Spaces | ICLR 2022
     • [Dec 2023; v2.May’24] (”o.g. Mamba paper”) | Mamba: Linear-Time Sequence Modeling with Selective State Spaces | COLM 2024
     • [Apr 2024; v3.Mar’25] The Illusion of State in State-Space Models | ICML 2024
     • [May 2024] (”Mamba 2”) | Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality | ICML 2024
     • [Jun 2024] | An Empirical Study of Mamba-based Language Models | Preprint only (as at Oct’10th 2026)
     • [Jul 2024] Longhorn: State Space Models are Amortized Online Learners | ICLR 2025
     • [Mar 2026] (”complex eigenvalues paper”) Mamba-3: Improved Sequence Modeling using State Space Principles | ICLR 2026
     • [Sep 2026] Non-Commutative State Tracking with Input-Dependent Low-Rank Updates in Mamba-3 | Preprint only (as at Oct’10th 2026)
   • 1.2 Linear Attention (LA)
     • [Jun 2020] (”o.g. Linear attention paper”) | Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention | ICML 2020
     • [Feb 2021] (”DeltaNet paper”) | Linear Transformers Are Secretly Fast Weight Programmers | ICML 2021
     • [May 2023] (”RWKV”) | RWKV: Reinventing RNNs for the Transformer Era | EMNLP 2023 Findings
     • [Jul 2023] (”RetNet”) | Retentive Network: A Successor to Transformer for Large Language Models | Preprint only (as at Oct’10th 2026)
     • [Dec 2023] (”Gated Linear Attention”) | Gated Linear Attention Transformers with Hardware-Efficient Training | ICML 2024
     • [May 2024] (”Mambav1 couldn’t do parity paper”) | The Expressive Capacity of State Space Models: A Formal Language Perspective | NeurIPS 2024
     • [Jun 2024] Parallelizing Linear Transformers with the Delta Rule over Sequence Length | NeurIPS 2024
     • [Dec 2024] (”Gated Deltanet paper”) | Gated Delta Networks: Improving Mamba2 with Delta Rule | ICLR 2025
     • [Mar 2025] (”RWKV-7”) | RWKV-7 “Goose” with Expressive Dynamic State Evolution | COLM 2025
     • [Mar 2026] The Expressive Limits of Diagonal SSMs for State-Tracking | ICLR 2026
     • [May 2026] Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention | NeurIPS 2026
     • [Sep 2026] Triadic Linear Attention | Preprint only (as at Oct’10th 2026)
     • [Sep 2026] Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention | Preprint only (as at Oct’19th 2026)
   • 1.3 Overlapping between SSM and LA
     • [Nov 2024] Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues | ICLR 2025
     • [Mar 2026] Why Are Linear RNNs More Parallelizable? | ICML 2026
2. Recurrent Depth (Looped Transformers)
   • [Feb 2025] Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach | NeurIPS 2025
   • [Feb 2025] “Inner Thinking Transformer: Leveraging Dynamic Depth …” | ACL 2025
   • [Apr 2026] Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers | COLM 2026
3. Ordinary Softmax Attention
   • [Jun 2017] (”o.g. Transformer architecture paper”) | Attention is All You Need | NIPS 2017 (back when NeurIPS was NIPS)
   • [Feb 2019] (“GPT-2 paper”) | Language Models are Unsupervised Multitask Learners | OpenAI technical report
   • [May 2025] (“Gated Attention paper”) | Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free ****| NeurIPS 2025
4. Sharing & Compression (of KV Representation)
   • [Nov 2019] (“MQA/Multi-query attention paper”) | Fast transformer decoding: One write-head is all you need | Preprint only (as at Oct’10th 2026)
   • [May 2023] (“GQA/Grouped Query attention paper”) | GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints | EMNLP 2023
   • [May 2024] (”MLA/Multi-head latent attention paper”) | DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model | DeepSeek technical report
   • [Jun 2024] (”MLKV/Multi-layer key-value sharing paper”) | MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decoding | NAACL 2025 Findings
5. Softmax Function alternatives
   • [Apr 2025] Softpick: No Attention Sink, No Massive Activations with Rectified Softmax | ACL 2026 Findings
6. Sparse Attention & Cache Selection
   • [Apr 2025] The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs | ACL 2026 Findings
   • [Jun 2026] VaSE: Value-Aware Stochastic KV Cache Eviction for Reasoning Models | Neurips 2026 (link otw)
   • [July 2026] Octopus: Gated Selective Attention for Memory-Bounded Long-Context Inference in Large Language Models ****| ACL 2026
7. Prediction Objectives
   • [Apr 2024] (”MTP paper”) | Better & Faster Large Language Models via Multi-token Prediction | ICML 2024
   • [Aug 2025] (”TOP Paper”) | Predicting the Order of Upcoming Tokens Improves Language Modeling | ICML 2026
8. Hybrid Models
   • [Mar 2024] Jamba: A Hybrid Transformer-Mamba Language Model | Preprint only (as at Oct’10th 2026)
   • [Mar 2024] (”MAD Paper”) | Mechanistic Design and Scaling of Hybrid Architectures | ICML 2024
   • [Oct 2025] (”Kimi Delta Attention paper”) | Kimi Linear: An Expressive, Efficient Attention Architecture | Preprint only (as at Oct’10th 2026)
9. Recursive Reasoning Models
   • [Jun 2025] (”HRM paper”) | Hierarchical Reasoning Model | Preprint only (as at Oct’10th 2026)
   • [Oct 2025] (”TRM paper”) | Less is More: Recursive Reasoning with Tiny Networks | Preprint only (as at Oct’10th 2026)
10. Positional Embeddings
    • [Apr 2021] (”RoPE”) | RoFormer: Enhanced Transformer with Rotary Position Embedding | Neurocomputing, 2024
    • [Mar 2022] (”NoPE”) | Transformer Language Models without Positional Encodings Still Learn Positional Information | EMNLP’2022 Findings
    • [May 2024] (”Abacus Embeddings paper”) | Transformers Can Do Arithmetic with the Right Embeddings | NeurIPS 2024
11. Mixture of Experts
    • [Apr 2024] Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models | Preprint only (as at Oct’10th 2026)
    • [Sep 2024] Toward Efficient Inference for Mixture of Experts | NeurIPS 2024
    • [June 2025] Can Mixture-of-Experts Surpass Dense LLMs Under Strictly Equal Resources? | ICLR 2026
12. Context Management
    • [Sep 2026] Context Language Models | Preprint only (as at Oct’10th 2026)