> For the complete documentation index, see [llms.txt](https://ai.saikatkumardey.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://ai.saikatkumardey.com/llm/the-evolution-of-language-models-from-word2vec-to-gpt-4/4-transformers-attention-is-all-you-need.md).

# \[4] Transformers - Attention Is All You Need

**Title**: Attention Is All You Need

**Authors & Year**: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin, 2017

**Link**: <https://arxiv.org/abs/1706.03762>

**Objective**: Develop a neural architecture for sequence modeling tasks that does not rely on recurrent or convolutional networks, with a focus on machine translation.

**Context**: Recurrent neural networks (RNNs) and convolutional neural networks (CNNs) were the state-of-the-art approaches to sequence modeling tasks, but they suffer from issues like slow training and difficulty in capturing long-range dependencies.

**Key Contributions**:

* Introduced the transformer architecture that uses self-attention mechanisms instead of RNNs or CNNs for sequence modeling.
* Demonstrated the effectiveness of the model on English-to-German and English-to-French machine translation tasks.

**Methodology**:

* The transformer consists of an encoder and a decoder, both composed of multi-head self-attention layers and feedforward layers.
* The self-attention mechanism allows the model to attend to different parts of the input sequence for each output element, avoiding the need for a fixed-length vector representation.
* The transformer is optimized using a novel objective function called "scaled dot-product attention".

**Results**:

* The transformer achieved state-of-the-art performance on several benchmark datasets, including WMT'14 English-to-German and English-to-French machine translation tasks.
* The transformer's parallelizable architecture allowed for faster training than RNNs or CNNs.

**Impact**:

* The transformer introduced a new paradigm for sequence modeling tasks that has been adopted in several NLP applications.
* Inspired further research in NLP, leading to innovations like BERT and GPT-2.

**Takeaways**:

* The transformer is a neural architecture that uses self-attention mechanisms for sequence modeling tasks, eliminating the need for RNNs or CNNs.
* The transformer is faster and easier to parallelize than RNNs or CNNs, leading to faster training times.
* The transformer has significantly influenced the development of NLP models and the direction of NLP research.