Stanford AI 课程地图 EN GitHub ↗
← 返回课程地图
Stanford · 2026—27 CS 329H LLM 与 Agent ★★★★☆

核心课程 · LLM 与 Agent

从人类偏好中学习

Machine Learning from Human Preferences

RLHF 的理论与实作:从 choice model、偏好学习到 assistance games 和公平性。

3 学分 2026 秋 Letter or Credit/No Credit ★★★★☆
RLHF偏好对齐推荐

课程定位

CS 329H 由 Sanmi Koyejo 讲授,系统讲解“如何优化被人使用的系统”。它把心理测量、离散选择、强化学习、信息检索和规范理论串起来,是理解 RLHF/DPO/推荐系统偏好建模的硬核课程。

主要内容

  1. Choice modeling:随机效用、Plackett-Luce、公理化方法
  2. 模仿与主动学习:behavior cloning、偏好查询
  3. RLHF:奖励建模、PPO、DPO、KTO
  4. Assistance games:人机协作、bandit、推荐
  5. 行为偏差与风格控制:偏好异质性、prompt/style
  6. 聚合与公平:社会选择、群体偏好聚合

课程大纲 / 课件

#主题 / 讲者
01Course introduction & motivation
02Choice data, generalization, and Bradley–Terry
03Random utility models and axiomatizations
04Factor models
05Learning to predict choice; latent-variable estimation with MLE
06Bayesian estimation and finetuning
07Fisher information & optimal experimental design
08Reinforcement learning from human feedback
09Direct preference optimization
10Active elicitation for LLMs, and case studies
11Assistance games; assisted vs. contextual bandits
12Acquisition functions & Thompson sampling
13Dueling bandits & preferential Bayesian optimization
14Style control, surrogates
15Bounded rationality; computationally tractable choice
16Impossibilities: Gibbard–Satterthwaite and Arrow
17Fairness and information aggregation
18Summary and practical tips

Autumn 2026 18 sessions;slides 随课程发布。

先修

研究生级 ML 基础(CS 229 级);概率与优化基础。

考核与工作量

作业 + 项目/论文复现,偏理论与实验结合。具体以当季官网为准。

适合谁

做 alignment、RLHF、推荐系统、人机交互的研究者;想理解“偏好数据为什么难”的工程师。

学习建议

先理解 Bradley-Terry 模型和 PPO。配合 CS 224N 的 RLHF 章节一起学。

相关课程

官方课程描述(英文原文)

Machine learning from human preferences (MLHP) provides mechanisms for optimizing systems that are used by humans - with applications in personalization, finetuning, and assistance. MLHP has been important to successes in modern language modeling, but the technical and conceptual underpinnings are spread across fields: psychometrics, discrete choice, reinforcement learning, information retrieval, and normative theory. This course will provide a careful and practical introduction to the main methods and models of MLHP. It will cover: choice modeling and its axiomatic approaches; imitation and static and active learning of choice; reinforcement learning from human feedback; assistance games, bandits, and recommendation; behavioral biases, style control; aggregation and fairness. This is a graduate-level course. By the end of the course, students should be able to build, analyze, and optimize systems that are used by humans.