PKU Class 2026 Fall: Optimization for Deep Learning

Instructor: Kun Yuan (kunyuan@pku.edu.cn)

Teaching assistants:

• Shuchen Zhu (shuchenzhu@stu.pku.edu.cn) Lead TA
  
• Ming Sun (2401111521@stu.pku.edu.cn)
  
• Feiyue Ye (2501111527@stu.pku.edu.cn)
  

Classroom: 3pm - 6pm Tuesday, 三教203

Office hour: 2pm - 3pm Tuesday, 静园六院220

References

Martin Jaggi and Nicolas Flammarion, Optimization for Machine Learning, EPFL Class CS-439
Chris De Sa, Advanced Machine Learning Systems, Cornell CS6787
Zaiwen Wen, Optimization Methods, PKU 2024 Fall
Kun Yuan, Introduction to LLM, PKU 2025 Spring

Materials

Lecture 1: Introduction

• Part I: Introduction to Deep Learning [Slides]
  
• Part II: Introduction to LLM [Slides]

Lecture 2: Linear Algebra I

• Part I: Gradient and Hessian [Notes]
  
• Part II: Linear Transform; Eigenvalue and Eigenvector [Notes]
  

Lecture 3: Linear Algebra II

• Part I: Jacobian matrix; Hessian matrix; Chain Rule [Notes]
  
• Part II: Inner product; Positive-definite matrix; Matrix functions; Forward-backward Propagation [Notes]
  
• Part III: Norms; Singular-value decomposition [Notes]

Lecture 4: Probability and Statistics

• Part I: Probability; Random variables; Distribution
• Part II: Expectation; Variance; Marginal/Conditional distribution
• Part III: Basics in Machine Learning

Lecture 5: Gradient Descent

• Convex set; Convex functions; Convex problems; Gradient descent [Slides] [Notes]
• Momentum gradient descent; Nesterov acceleration; Anderson acceleration [Slides] [Notes]
  

Lecture 6: Stochastic Gradient Descent

• Stochastic gradient descent [Slides] [Notes]
  
• Momentum SGD [Slides] [Notes]
  
• AdaGrad; RMSProp; Adam [Slides]

Lecture 7: Basics in Transformer and LLM I

• Part I: Word Embedding; Seq2Seq; Attention; Transformer [Slides]

Lecture 8: Basics in Transformer and LLM II

• Parameters, Computations, and Memory Costs in Transformer [Slides01] [Slides02]
• Memory access cost; Kernel fusion; Flash attention [Slides]

Lecture 9: Advanced Optimizer for LLM

• Shampoo; SOAP; Muon; Lite; MuonM [Slides]

Lecture 10: Subspace Optimizer for LLM

• Adam-mini; Memory-efficient Optimizers [Slides]
• Block coordinate descent; Coordinate friendly structure [Slides]
• Block-wise training in LLMs [Slides]

Lecture 11: Zeroth-Order Optimizer

• Finite difference; linear interpolation; sphere smoothing [Slides] [Notes]
• Memory-efficient zeroth-order optimization; Low-rank zeroth-order optimization [Slides]

Lecture 12: Distributed Optimizer

• Data Parallelism; Pipeline Parallelism; Tensor Parallelism [Slides]
• Decentralized Learning [Slides]
• Communication Compression; Local Learning [Slides]

Lecture 13: Mixed-Precision Training

• FP16; FP32; Mixed-Precision [Slides]
• FP8; NVFP4 [Slides]

Lecture 14: Efficient Fine-tuning and Inference

• Motivation; Adapter; LoRA; DoRA; LISA [Slides]
• KV Cache; Streaming LLM; Quest; vLLM [Slides]