Course positioning
CS 336 is one of the most hard-core LLM courses, taught by Percy Liang and Tatsunori Hashimoto. Inspired by 'build an OS from scratch', it walks through the full language-model lifecycle; the assignments need substantial GPU compute, and enrollment requires an application.
What it covers
- Data: collection, cleaning, deduplication, filtering, mixing
- Tokenization: BPE, vocabulary design, multilingual issues
- Model: Transformer architecture, attention variants, positional encoding
- Training: initialization, optimizers, learning rates, mixed precision, distributed training
- Scaling: scaling laws, compute-optimal configurations
- Evaluation: perplexity, benchmarks, contamination, alignment evaluation
- Deployment: inference, quantization, serving
- Safety and ethics: data copyright, bias, misuse
Prerequisites
Strong programming; deep learning background (CS 224N/CS 231N/CS 230 level); application required, GPU resources recommended.
Assessment & workload
Typically about five programming assignments that build the LM step by step, plus a project; compute-heavy. Check the current syllabus for details.
Who it's for
People who want to understand the full LLM stack and later work on pre-training/inference/systems; founders building models.
Study advice
Take CS 224N or CS 231N first. Without GPUs you can run small models, but the assignment experience will be limited.
Related courses
Official course description
Language models serve as the cornerstone of modern natural language processing (NLP) applications and open up a new paradigm of having a single general purpose system address a range of downstream tasks. As the field of artificial intelligence (AI), machine learning (ML), and NLP continues to grow, possessing a deep understanding of language models becomes essential for scientists and engineers alike. This course is designed to provide students with a comprehensive understanding of language models by walking them through the entire process of developing their own. Drawing inspiration from operating systems courses that create an entire operating system from scratch, we will lead students through every aspect of language model creation, including data collection and cleansing for pre-training, transformer model construction, model training, and evaluation before deployment. Application required, apply at http://cs336.stanford.edu/