Course positioning
Benchmarks, psychometrics, and validity theory for systematically evaluating AI capabilities. This page keeps the official record and entry points; check the current syllabus for details.
What it covers
This is a related-course page: the official description below is the main content.
Prerequisites
See the official description (related courses do not have a separate prerequisite write-up).
Assessment & workload
Check the current syllabus.
Who it's for
Anyone interested in AI Measurement Science.
Study advice
Read the official description and course site before deciding.
Related courses
Official course description
Artificial Intelligence (AI) measurement science provides frameworks and methodologies for evaluating, benchmarking, and understanding AI systems. As AI systems become increasingly powerful and deploy into high-stakes domains, the need for rigorous measurement approaches has grown in importance. Current measurement approaches are often ad hoc, lacking theoretical grounding, and failing to connect to real-world use cases. This has led to a measurement crisis characterized by benchmark saturation, inconsistent evaluation methodologies, and difficulty in making valid claims about AI capabilities. This course will cover the foundations of AI measurement science from first principles and outline connections to the growing literature on the topic. This includes: validity theory as applied to AI evaluation, focusing on content, criterion, construct, external, and consequential validity; psychometric models for AI measurement, including item response theory and latent variable models; scaling laws and intervention effects, predicting the impacts of data, computing, and architecture choices; synthetic data generation for evaluation and its implications; governance and policy considerations around AI measurement. This is a graduate-level course. By the end of the course, students should be able to understand, implement, and critique state-of-the-art AI measurement approaches and be ready to conduct research on these topics.