Course positioning
Taught by Christos Kozyrakis, CS 349D is a research seminar focused on inference rather than training. It argues that future AI systems are compound systems (multiple models + retrieval + tools + traditional software) and discusses the cloud infrastructure, scheduling, and architecture patterns that support them.
What it covers
- Inference serving: batching, continuous batching, KV cache
- Compound AI systems: multi-component orchestration, dataflow
- Cloud infrastructure: elasticity, multi-tenancy, cost
- Scheduling and routing: model routing, priorities, SLAs
- Latency/throughput/cost: trade-offs and optimization
- Industry practice: guest lectures from Databricks, OpenAI, NVIDIA, etc.
- Quarter-long project
Prerequisites
Systems/architecture/ML basics; distributed-systems experience helps.
Assessment & workload
Paper discussion plus guest lectures and a 2–3 person team project. Check the current syllabus for details.
Who it's for
People who want to work on inference platforms, AI infrastructure, or cloud systems; systems-oriented graduate students.
Study advice
Pair with CS 349E for single-machine/cluster performance and CS 349F for networking. Base the project on a real serving stack.
Related courses
Official course description
The emergence of foundation models has revolutionized AI applications, but the most powerful AI systems today are increasingly compound systems - compositions of multiple models, retrievers, tools, and traditional software components. This research seminar explores the intersection of cloud computing infrastructure and compound AI systems, covering both the infrastructure required to build and deploy these systems at scale and the architectural patterns that make them effective. Students will participate in guest lectures from leading experts across academia and industry, including leaders of companies like Databricks, OpenAI, Foundry, NVIDIA, Microsoft, and more, read and discuss papers, and complete a quarter-long project in groups of 2-3.