Stanford AI Course Map 中文 GitHub ↗
← Back to the course map
Stanford · 2026—27 CS 336 LLMs & Agents ★★★★★

Core course · LLMs & Agents

Language Modeling from Scratch

中文:从零构建语言模型

Build a language model from scratch: data cleaning, tokenizer, Transformer, training, scaling, evaluation, and deployment.

Draft translation: official facts are checked; wording may change after native review.

3–5 units Spring 2027 Letter or Credit/No Credit ★★★★★
LLMPre-trainingSystemsHard-core

Course positioning

CS 336 is one of the most hard-core LLM courses, taught by Percy Liang and Tatsunori Hashimoto. Inspired by 'build an OS from scratch', it walks through the full language-model lifecycle; the assignments need substantial GPU compute, and enrollment requires an application.

What it covers

  1. Data: collection, cleaning, deduplication, filtering, mixing
  2. Tokenization: BPE, vocabulary design, multilingual issues
  3. Model: Transformer architecture, attention variants, positional encoding
  4. Training: initialization, optimizers, learning rates, mixed precision, distributed training
  5. Scaling: scaling laws, compute-optimal configurations
  6. Evaluation: perplexity, benchmarks, contamination, alignment evaluation
  7. Deployment: inference, quantization, serving
  8. Safety and ethics: data copyright, bias, misuse

Prerequisites

Strong programming; deep learning background (CS 224N/CS 231N/CS 230 level); application required, GPU resources recommended.

Assessment & workload

Typically about five programming assignments that build the LM step by step, plus a project; compute-heavy. Check the current syllabus for details.

Who it's for

People who want to understand the full LLM stack and later work on pre-training/inference/systems; founders building models.

Study advice

Take CS 224N or CS 231N first. Without GPUs you can run small models, but the assignment experience will be limited.

Related courses

Official course description

Language models serve as the cornerstone of modern natural language processing (NLP) applications and open up a new paradigm of having a single general purpose system address a range of downstream tasks. As the field of artificial intelligence (AI), machine learning (ML), and NLP continues to grow, possessing a deep understanding of language models becomes essential for scientists and engineers alike. This course is designed to provide students with a comprehensive understanding of language models by walking them through the entire process of developing their own. Drawing inspiration from operating systems courses that create an entire operating system from scratch, we will lead students through every aspect of language model creation, including data collection and cleansing for pre-training, transformer model construction, model training, and evaluation before deployment. Application required, apply at http://cs336.stanford.edu/