Data Analytics

If you're interested in learning how to train a large language model from scratch, I recommend checking this out ↓ François Chollet, co-founder of the ARC Prize and creator of Keras, recently suggested a path for learning about LLMs from the ground up. He said that if you're 17 years old or any age, and you want to learn how to create LLMs from scratch, simply read chapters 15 and 16 of his book Deep Learning with Python. I quickly skimmed through them, and these two chapters are definitely worth saving. Chapter 15 starts with the most basic language models and gradually progresses to: Character-level Language Model → Seq2Seq → Attention → QKV → Scaled Dot-Product Attention → Multi-Head Attention → Self-Attention → Transformer There's even a dedicated section explaining why Dot-Product Attention works in the first place. He himself says that this is one of the best explanations of the topic. The author doesn't just stop at formulas like QKᵀ / √d. He starts with Word2Vec and embedding spaces, and then explains how the Transformer layer by layer, through Attention, gradually transforms the relationships between tokens into distance relationships in a vector space. Chapter 16 is even more practical. It directly shows how to train a mini-GPT from scratch. It explains how to take nearly 1 billion tokens from C4, create a SentencePiece vocabulary of 32,000 tokens, train a mini-GPT with 41 million parameters, 8 layers, 8 attention heads, and a hidden state size of 512, and then build a data pipeline, implement weight tying, learning rate warmup, pre-training, and generation with temperature and top-k. From the tokenizer, data pipeline, causal attention, weight tying, and learning rate warmup to pre-training, greedy decoding, temperature, and top-k sampling. In essence, you are guided step-by-step through the entire process of training a GPT model. You don't even need an expensive server for this. The official notes state that the entire example can be run on a free T4 in Google Colab. Training takes about 6 hours. On an A100, it takes just over an hour. Further in the book, Gemma, SFT, RLHF, RAG, and multimodal models are discussed. So, if someone asks me: "I've never trained a large model before. Where do I start?" These two chapters can really be a great starting point. The third edition of Deep Learning with Python is currently available for free online, and all the accompanying notebooks are fully open-source. You can simply download them into Colab and run them. In 2026, it won't be necessary to immediately dive into a hundred research papers to learn about LLMs. If you train a GPT model with 41 million parameters yourself, from data preparation to text generation, many concepts will naturally fall into place. Link: https://deeplearningwithpython.io/