TechNeura

We reproduce the GPT-2 (124M) from scratch. This video covers the whole process: First we build the GPT-2 network, then we optimize its training to be really fast, then we set up the training run following the GPT-2 and GPT-3 paper and their hyperparameters, then we hit run, and come back the next morning to see our results, and enjoy some amusing model generations. Keep in mind that in some places this video builds on the knowledge from earlier videos in the Zero to Hero Playlist (see my channel). You could also see this video as building my nanoGPT repo, which by the end is about 90% similar. Links: - build-nanogpt GitHub repo, with all the changes in this video as individual commits:

- nanoGPT repo:

- llm.c repo:

- my website:

- my twitter:

- our Discord channel:

Supplementary links: - Attention is All You Need paper:

- OpenAI GPT-3 paper:

- OpenAI GPT-2 paper:

The GPU I'm training the model on is from Lambda GPU Cloud, I think the best and easiest way to spin up an on-demand GPU instance in the cloud that you can ssh to:

Chapters: 00:00:00 intro: Let’s reproduce GPT-2 (124M) 00:03:39 exploring the GPT-2 (124M) OpenAI checkpoint 00:13:47 SECTION 1: implementing the GPT-2 nn.Module 00:28:08 loading the huggingface/GPT-2 parameters 00:31:00 implementing the forward pass to get logits 00:33:31 sampling init, prefix tokens, tokenization 00:37:02 sampling loop 00:41:47 sample, auto-detect the device 00:45:50 let’s train: data batches (B,T) → logits (B,T,C) 00:52:53 cross entropy loss 00:56:42 optimization loop: overfit a single batch 01:02:00 data loader lite 01:06:14 parameter sharing wte and lm_head 01:13:47 model initialization: std 0.02, residual init 01:22:18 SECTION 2: Let’s make it fast. GPUs, mixed precision, 1000ms 01:28:14 Tensor Cores, timing the code, TF32 precision, 333ms 01:39:38 float16, gradient scalers, bfloat16, 300ms 01:48:15 torch.compile, Python overhead, kernel fusion, 130ms 02:00:18 flash attention, 96ms 02:06:54 nice/ugly numbers. vocab size 50257 → 50304, 93ms 02:14:55 SECTION 3: hyperpamaters, AdamW, gradient clipping 02:21:06 learning rate scheduler: warmup + cosine decay 02:26:21 batch size schedule, weight decay, FusedAdamW, 90ms 02:34:09 gradient accumulation 02:46:52 distributed data parallel (DDP) 03:10:21 datasets used in GPT-2, GPT-3, FineWeb (EDU) 03:23:10 validation data split, validation loss, sampling revive 03:28:23 evaluation: HellaSwag, starting the run 03:43:05 SECTION 4: results in the morning! GPT-2, GPT-3 repro 03:56:21 shoutout to llm.c, equivalent but faster code in raw C/CUDA 03:59:39 summary, phew, build-nanogpt github repo Corrections: I will post all errata and followups to the build-nanogpt GitHub repo (link above) SuperThanks: I experimentally enabled them on my channel yesterday. Totally optional and only use if rich. All revenue goes to to supporting my work in AI + Education.

Scripts for fine-tuning Meta Llama3 with composable FSDP & PEFT methods to cover single/multi-node GPUs. Supports default & custom datasets for applications such as summarization an...

Self-Supervised Anomaly Detection in Time Series: A Brief Introduction Speaker: Aitor Sanchez Institution: ISG

NCTU CS PhD Seminar Spring 2020 Speaker Sheng-Feng Yu 0786039

This blog is the first in a series in which we consider machine learning from four different viewpoints.

Develop a Monitoring System on Multiple Time Series Sensors

The BrAIn Roads documents the thrilling research journey of a group of researchers of the Computer Vision Centre tasked with developing AI for autonomous driving on rural roads. Funded by the Generalitat de Catalunya, the project aims to offer an economical and sustainable solution to the lack of public transport in rural and isolated areas. The scientific team faces the complex challenge of training the AI on a narrow mountain road with poor visibility, adverse weather conditions, and no mobile connectivity – a proof of concept distinct from existing autonomous driving prototypes, mainly designed for urban areas. Taking us from the laboratories of the CVC to the field tests in the natural park of Alt Pirineu, the documentary explores the intricacies of artificial intelligence research. It provides insight into the successes, setbacks, and technological challenges faced by the team throughout the project. "The BrAIn Roads" offers a captivating perspective on the world of science and innovation, connecting the audience with the inspiring story of a research team dedicated to transforming autonomous driving and paving the way for a more accessible and sustainable mobility of the future. This documentary showcases the outcomes of the Advanced Digital Technologies project focused on autonomous driving in rural areas. The initiative was financially supported by the Generalitat de Catalunya and executed by the CVC, in collaboration with SEAT, Volkswagen, and the i2CAT Foundation. "The BrAIn Roads" is a production by Minifilms TV. More information: www.cvc.uab.es/thebrainroads