Stanford AI Course Map 中文 GitHub ↗
← Back to the course map
Stanford · 2026—27 CS 329H LLMs & Agents ★★★★☆

Core course · LLMs & Agents

Machine Learning from Human Preferences

中文:从人类偏好中学习

The theory and practice of RLHF: from choice models and preference learning to assistance games and fairness.

Draft translation: official facts are checked; wording may change after native review.

3 units Autumn 2026 Letter or Credit/No Credit ★★★★☆
RLHFPreferencesAlignmentRecommendation

Course positioning

Taught by Sanmi Koyejo, CS 329H systematically explains how to optimize systems that are used by people. It connects psychometrics, discrete choice, reinforcement learning, information retrieval, and normative theory — a rigorous course for understanding RLHF/DPO and preference modeling in recommender systems.

What it covers

  1. Choice modeling: random utility, Plackett-Luce, axiomatic approaches
  2. Imitation and active learning: behavior cloning, preference queries
  3. RLHF: reward modeling, PPO, DPO, KTO
  4. Assistance games: human-AI collaboration, bandits, recommendation
  5. Behavioral biases and style control: preference heterogeneity, prompt/style
  6. Aggregation and fairness: social choice, aggregating group preferences

Prerequisites

Graduate-level ML (CS 229 level); probability and optimization.

Assessment & workload

Assignments plus a project/paper reproduction, combining theory and experiments. Check the current syllabus for details.

Who it's for

Researchers in alignment, RLHF, recommendation, or HCI; engineers who want to understand why preference data is hard.

Study advice

Understand Bradley-Terry and PPO first. Pair it with the RLHF chapter of CS 224N.

Related courses

Official course description

Machine learning from human preferences (MLHP) provides mechanisms for optimizing systems that are used by humans - with applications in personalization, finetuning, and assistance. MLHP has been important to successes in modern language modeling, but the technical and conceptual underpinnings are spread across fields: psychometrics, discrete choice, reinforcement learning, information retrieval, and normative theory. This course will provide a careful and practical introduction to the main methods and models of MLHP. It will cover: choice modeling and its axiomatic approaches; imitation and static and active learning of choice; reinforcement learning from human feedback; assistance games, bandits, and recommendation; behavioral biases, style control; aggregation and fairness. This is a graduate-level course. By the end of the course, students should be able to build, analyze, and optimize systems that are used by humans.