Course positioning
Taught by Sanmi Koyejo, CS 329H systematically explains how to optimize systems that are used by people. It connects psychometrics, discrete choice, reinforcement learning, information retrieval, and normative theory — a rigorous course for understanding RLHF/DPO and preference modeling in recommender systems.
What it covers
- Choice modeling: random utility, Plackett-Luce, axiomatic approaches
- Imitation and active learning: behavior cloning, preference queries
- RLHF: reward modeling, PPO, DPO, KTO
- Assistance games: human-AI collaboration, bandits, recommendation
- Behavioral biases and style control: preference heterogeneity, prompt/style
- Aggregation and fairness: social choice, aggregating group preferences
Prerequisites
Graduate-level ML (CS 229 level); probability and optimization.
Assessment & workload
Assignments plus a project/paper reproduction, combining theory and experiments. Check the current syllabus for details.
Who it's for
Researchers in alignment, RLHF, recommendation, or HCI; engineers who want to understand why preference data is hard.
Study advice
Understand Bradley-Terry and PPO first. Pair it with the RLHF chapter of CS 224N.
Related courses
Official course description
Machine learning from human preferences (MLHP) provides mechanisms for optimizing systems that are used by humans - with applications in personalization, finetuning, and assistance. MLHP has been important to successes in modern language modeling, but the technical and conceptual underpinnings are spread across fields: psychometrics, discrete choice, reinforcement learning, information retrieval, and normative theory. This course will provide a careful and practical introduction to the main methods and models of MLHP. It will cover: choice modeling and its axiomatic approaches; imitation and static and active learning of choice; reinforcement learning from human feedback; assistance games, bandits, and recommendation; behavioral biases, style control; aggregation and fairness. This is a graduate-level course. By the end of the course, students should be able to build, analyze, and optimize systems that are used by humans.