Stanford AI Course Map 中文 GitHub ↗
← Back to the course map
Stanford · 2026—27 CS 349D ML Systems & Hardware ★★★★☆

Core course · ML Systems & Hardware

AI Inference Infrastructure

中文:AI 推理基础设施

AI inference infrastructure: how compound AI systems are deployed, scheduled, and scaled in the cloud.

Draft translation: official facts are checked; wording may change after native review.

3 units Spring 2027 Letter or Credit/No Credit ★★★★☆
InferenceInfrastructureCloudSystems

Course positioning

Taught by Christos Kozyrakis, CS 349D is a research seminar focused on inference rather than training. It argues that future AI systems are compound systems (multiple models + retrieval + tools + traditional software) and discusses the cloud infrastructure, scheduling, and architecture patterns that support them.

What it covers

  1. Inference serving: batching, continuous batching, KV cache
  2. Compound AI systems: multi-component orchestration, dataflow
  3. Cloud infrastructure: elasticity, multi-tenancy, cost
  4. Scheduling and routing: model routing, priorities, SLAs
  5. Latency/throughput/cost: trade-offs and optimization
  6. Industry practice: guest lectures from Databricks, OpenAI, NVIDIA, etc.
  7. Quarter-long project

Prerequisites

Systems/architecture/ML basics; distributed-systems experience helps.

Assessment & workload

Paper discussion plus guest lectures and a 2–3 person team project. Check the current syllabus for details.

Who it's for

People who want to work on inference platforms, AI infrastructure, or cloud systems; systems-oriented graduate students.

Study advice

Pair with CS 349E for single-machine/cluster performance and CS 349F for networking. Base the project on a real serving stack.

Related courses

Official course description

The emergence of foundation models has revolutionized AI applications, but the most powerful AI systems today are increasingly compound systems - compositions of multiple models, retrievers, tools, and traditional software components. This research seminar explores the intersection of cloud computing infrastructure and compound AI systems, covering both the infrastructure required to build and deploy these systems at scale and the architectural patterns that make them effective. Students will participate in guest lectures from leading experts across academia and industry, including leaders of companies like Databricks, OpenAI, Foundry, NVIDIA, Microsoft, and more, read and discuss papers, and complete a quarter-long project in groups of 2-3.