Key points
- The Transformer replaces recurrence and convolution with self-attention, allowing models to weigh relationships across an entire sequence in parallel.
- It is the architectural substrate of the large language and diffusion models underlying current generative image, video and motion tools.
- Its scalability enabled the foundation-model era on which the repository's generative-AI entries depend.
Summary
Vaswani et al. (2017) introduced the Transformer, a sequence-to-sequence architecture based entirely on self-attention and feed-forward layers, dispensing with recurrence and convolution. The attention mechanism allows every position in a sequence to attend to every other position in parallel, making both training and inference substantially more scalable than recurrent architectures. The Transformer became the architectural substrate of BERT, GPT, DALL-E, the denoising U-Nets in diffusion models and the motion-generation models used in animation research. Every major generative tool in the repository's collection either is a Transformer or is conditioned by one.
Related items
- High-Resolution Image Synthesis with Latent Diffusion Models
- Denoising Diffusion Probabilistic Models (DDPM)
Source
Source: NeurIPS 2017 ↗ (Research)
Cite this item
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. & Polosukhin, I. (2017). ‘Attention Is All You Need (the Transformer architecture)’, NeurIPS 2017. Available at: https://arxiv.org/abs/1706.03762
Your reference manager can also read this page directly: with the Zotero (or Mendeley) browser connector installed, save it straight to your library. Whole-collection exports: RIS, BibTeX, CSL-JSON.