PKU Class 2025 Fall: Optimizaiton for Deep Learning

Instructor: Kun Yuan (kunyuan@pku.edu.cn)

Teaching assistants:

• Yilong Song (2301213059@pku.edu.cn) [Leading Teaching Assistant]
  
• Mian Xiao (xiaom@stu.pku.edu.cn)
  
• Ruoxi Yu (2301111452@stu.pku.edu.cn)
  

Classroom: 3pm - 6pm Tuesday, 理教206

Office hour: 4pm - 5pm Thursday, 静园六院220

References

Martin Jaggi and Nicolas Flammarion, Optimization for Machine Learning, EPFL Class CS-439
Chris De Sa, Advanced Machine Learning Systems, Cornell CS6787
Zaiwen Wen, Optimization Methods, PKU 2024 Fall
Kun Yuan, Introduction to LLM, PKU 2025 Spring

Announcement

1. The lecture scheduled for November 25 has been canceled. It will be rescheduled for November 28, from 3:10 p.m. to 6:00 p.m., in Room 403 of the Third Teaching Building (三教403).

2. The Midterm Exam is scheduled on November 4.

3. The lecture scheduled for October 28 has been canceled. It will be rescheduled for October 31, from 3:10 p.m. to 6:00 p.m., in Room 理教306.

4. The lecture scheduled for September 9 has been canceled. It will be rescheduled for September 18, from 6:40 p.m. to 9:30 p.m., in Room 407 of the Third Teaching Building (三教407).

Materials

Lecture 1: Introduction

• Warm up: Preliminary [Notes]
  
• Part I: Introduction to Deep Learning [Slides]
  
• Part II: Introduction to LLM [Slides]

Lecture 2: Basics in Machine Learning and Langugae Models

• Part I: Basics in Machine Learning [Slides]
  
• Part II: Basics in Language models [Slides]

Lecture 3: Transformer

• Part I: Seq2Seq; Attention; Transformer [Slides]
• Part II: Parameters, Computations, and Memory Costs in Transformer [Slides01] [Slides02]

Lecture 4: Some LLM Models

• Teacher forcing; Pretrain and Finetuning; BERT; GPTs [Slides]
• DeepSeek [Slides]

Lecture 5: Gradient Descent

• Convex set; Convex functions; Convex problems; Gradient descent [Slides] [Notes]
• Forward-backward propagation [Notes]

Lecture 6: Accelerated Gradient Descent

• Momentum gradient descent; Nesterov acceleration; Anderson acceleration [Slides] [Notes]
  

Lecture 7: Stochastic Gradient Descent

• Stochastic gradient descent [Slides] [Notes]
  
• Sampling in SGD [Slides]
• Stability in SGD [Slides]

Lecture 8: Momentum and Adaptive SGD

• Momentum SGD [Slides] [Notes]
  
• AdaGrad; RMSProp; Adam [Slides]

Lecture 9: Midterm

Lecture 10: FlashAttention

• Memory access cost; Kernal fusion; Flash attention [Slides]

Lecture 11: Block Coordinate Descent

• Block coordinate descent; Coordinate friendly structure [Slides]
• Block-wise training in LLMs [Slides]

Lecture 12: Zeroth-Order Optimization

• Finite difference; linear interpolation; sphere smoothing [Slides] [Notes]
• Memory-efficient zeroth-order optimization; Low-rank zeroth-order optimization [Slides]

Lecture 13: Distributed Optimization

• Data Parallelism; Pipeline Parallelism; Tensor Parallelism [Slides]
• Decentralized Learning [Slides]
• Communication Compression; Local Learning [Slides]