跳到主要内容

transformer

注意力、位置编码、架构本体

26 篇,已拆 0 篇。回总览

2025​

  • DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models — paper=deepseek-v3-2@arXiv:2512.02556v1
  • MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention — paper=minimax-m1@arXiv:2506.13585v1

2024​

  • DeepSeek-V3 Technical Report — paper=deepseek-v3-technical-report@arXiv:2412.19437v2
  • What Matters in Transformers? Not All Attention is Needed — paper=what-matters-in-transformers-not-all@arXiv:2406.15786v6

2023​

  • BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models — paper=blip-2@arXiv:2301.12597v3
  • GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints — paper=gqa@arXiv:2305.13245v3
  • Mamba: Linear-Time Sequence Modeling with Selective State Spaces — paper=mamba@arXiv:2312.00752v2
  • Textbooks Are All You Need — paper=textbooks-are-all-you-need@arXiv:2306.11644v2

2022​

  • BERTopic: Neural topic modeling with a class-based TF-IDF procedure — paper=bertopic@arXiv:2203.05794v1
  • Efficient Few-Shot Learning Without Prompts — paper=efficient-few-shot-learning-without-prompts@arXiv:2209.11055v1
  • FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness — paper=flashattention@arXiv:2205.14135v2
  • Training Compute-Optimal Large Language Models — paper=chinchilla@arXiv:2203.15556v1

2021​

  • Decision Transformer: Reinforcement Learning via Sequence Modeling — paper=decision-transformer-reinforcement-learning-via-sequence@arXiv:2106.01345v2
  • GLM: General Language Model Pretraining with Autoregressive Blank Infilling — paper=glm@arXiv:2103.10360v2
  • LoRA: Low-Rank Adaptation of Large Language Models — paper=lora@arXiv:2106.09685v2
  • RoFormer: Enhanced Transformer with Rotary Position Embedding — paper=rope@arXiv:2104.09864v5
  • Show Your Work: Scratchpads for Intermediate Computation with Language Models — paper=show-your-work-scratchpads-for-intermediate@arXiv:2112.00114v1
  • Training Verifiers to Solve Math Word Problems — paper=training-verifiers-to-solve-math-word@arXiv:2110.14168v2

2020​

  • An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale — paper=vit@arXiv:2010.11929v2
  • GLU Variants Improve Transformer — paper=glu-variants-improve-transformer@arXiv:2002.05202v1
  • On Layer Normalization in the Transformer Architecture — paper=on-layer-normalization-in-the-transformer@arXiv:2002.04745v2

2019​

  • Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer — paper=exploring-the-limits-of-transfer-learning@arXiv:1910.10683v4
  • Fast Transformer Decoding: One Write-Head is All You Need — paper=fast-transformer-decoding-one-write-head@arXiv:1911.02150v1

2018​

  • BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — paper=bert@arXiv:1810.04805v2

2017​

  • Attention Is All You Need — paper=attention-is-all-you-need@arXiv:1706.03762v7

2015​

  • Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks — paper=unsupervised-representation-learning-with-deep-convolutional@arXiv:1511.06434v2