Few years ago I was learning about transformers and was writing down some notes for myself. Now I come accross those notes and decided to share some part of it here in case any of you find it useful.
Most famous transformers
1. BERT (Bidirectional Encoder Representations from Transformers): BERT is a pre-trained transformer model developed by Google. It has achieved state-of-the-art results in various NLP tasks, such as question answering, sentiment analysis, and text classification.
2. GPT (Generative Pre-trained Transformer): GPT is a series of transformer-based models developed by OpenAI. GPT-3, the most recent version, is a highly influential model known for its impressive language generation capabilities. It has been used in various creative applications, including text completion, language translation, and dialogue generation.
3. Transformer-XL: Transformer-XL is a transformer-based model developed by researchers at Google. It addresses the limitation of standard transformers by incorporating a recurrence mechanism to capture longer-term dependencies in the input sequence. It has been successful in tasks that require modeling long-range context, such as language modeling.
4. T5 (Text-to-Text Transfer Transformer): T5, developed by Google, is a versatile transformer model capable of performing a wide range of NLP tasks. It follows a "text-to-text" framework, where different tasks are cast as text generation problems. T5 has demonstrated strong performance across various benchmarks and has been widely adopted in the NLP community.
5. RoBERTa (Robustly Optimized BERT Pretraining Approach): RoBERTa is a variant of BERT developed by Facebook AI. It addresses some limitations of the original BERT model by tweaking the training setup and introducing additional data. RoBERTa has achieved improved performance on several NLP tasks, including text classification and named entity recognition.
BERT vs RoBERTa vs DistilBERT vs ALBERT
BERT - created by Google, 2018, question answering, summarization, and sequence classification, has 12 Encoders stacked, baseline to others.
RoBERTa - created by Facebook, 2019. literally same architecture as BERT, but improves on BERT by carefully and intelligently optimizing the training hyperparameters for BERT. It's trained on larger data, bigger vocabulary size and longer sentences. It overperforms BERT.
DistilBERT - created by Hugging Face, October 2019. roughly same general architecture as BERT, but smaller, only 6 Encoders. Distilbert is 40% smaller (40% less parameters) than the original BERT-base model, is 60% faster than it, and retains 95+% of its functionality.
ALBERT (A Light BERT) - published/introduced at around the same time as Distilbert. 18x less parameters than BERT, trained 1.7x faster. It doesn't have tradeoff in performance while DistilBERT has it at small extent. This comes from just the core difference in the way the Distilbert and Albert experiments are structured. Distilbert is trained in such a way to use BERT as the teacher for its training/distillation process. Albert, on the other hand, is trained from scratch like BERT. Better yet, Albert outperforms all previous models including BERT, Roberta, Distilbert, and XLNet.
Note: Training speed is not so important to end-users because all those are pre-trained transformer models. Still, in some cases we will need to fine-tune models using our own datasets, which is where speed is important. Also smaller and faster models like DistilBERT and ALBERT can be advantageous when there is not enough memory or computational power.