Transformers Revisited: Architecture, Attention, and Limitations

Transformers are now the dominant architecture in modern artificial intelligence. Introducedin 2017 by Vaswani et al. in the context of natural language processing, they have sincerevolutionized computer vision, music generation, bioinformatics, and many other fields.This paper presents a progressive and conceptual analysis of their architecture from theattention mechanism to modern positional encodings (RoPE, ALiBi) along with advancedtraining methods (RLHF, instruction tuning) and recent optimizations (FlashAttention, sparseattention). Fundamental limitations and emerging alternatives (state space models, Mamba)are also examined to provide a complete and up-to-date picture of the sequence modelinglandscape.