Course positioning
CS 234 covers the main line of RL from tabular methods to deep RL, with an emphasis on how autonomous systems make decisions through trial and error. It fits sequential decision-making in robotics, games, recommendation, and healthcare, and is a prerequisite for CS 224R/CS 332.
What it covers
- MDP basics: returns, Bellman equations, policy evaluation
- Dynamic programming: value iteration, policy iteration
- Model-free methods: Monte Carlo, TD, Q-learning, SARSA
- Function approximation: linear approximation, deep Q-networks
- Policy gradients: REINFORCE, actor-critic, PPO
- Exploration: ε-greedy, UCB, Thompson sampling
- Deep RL and offline RL: stability issues in practice
- Project: usually train an agent on a control or game task
Prerequisites
Fluent Python; CS 229 or equivalent ML background; linear algebra and probability.
Assessment & workload
Typically 3–4 assignments plus a midterm and a project; at least one deep RL assignment. Check the current syllabus for details.
Who it's for
Robotics, RL research, game AI, and recommender systems; anyone working on embodied AI.
Study advice
Master MDPs and dynamic programming before deep RL. For the exploration part, read Chapters 2–3 of Sutton & Barto.
Related courses
Official course description
To realize the dreams and impact of AI requires autonomous systems that learn to make good decisions. Reinforcement learning is one powerful paradigm for doing so, and it is relevant to an enormous range of tasks, including robotics, game playing, consumer modeling and healthcare. This class will briefly cover background on Markov decision processes and reinforcement learning, before focusing on some of the central problems, including scaling up to large domains and the exploration challenge. One key tool for tackling complex RL domains is deep learning and this class will include at least one homework on deep reinforcement learning. Prerequisites: proficiency in python, CS 229 or equivalents or permission of the instructor; linear algebra, basic probability.