PKU Class 2025 Fall: Optimizaiton for Deep Learning

Instructor: Kun Yuan (kunyuan@pku.edu.cn)

Teaching assistants:

• Yilong Song (2301213059@pku.edu.cn) [Leading Teaching Assistant]
  
• Mian Xiao (xiaom@stu.pku.edu.cn)
  
• Ruoxi Yu (2301111452@stu.pku.edu.cn)
  

Classroom: 3pm - 6pm Tuesday, 理教206

Office hour: 4pm - 5pm Thursday, 静园六院220

References

Martin Jaggi and Nicolas Flammarion, Optimization for Machine Learning, EPFL Class CS-439
Chris De Sa, Advanced Machine Learning Systems, Cornell CS6787
Zaiwen Wen, Optimization Methods, PKU 2024 Fall
Kun Yuan, Introduction to LLM, PKU 2025 Spring

Announcement

1. The Midterm Exam is scheduled on November 4.

2. The lecture scheduled for October 28 has been canceled. It will be rescheduled for October 31, from 3:10 p.m. to 6:00 p.m., in Room 理教306.

3. The lecture scheduled for September 9 has been canceled. It will be rescheduled for September 18, from 6:40 p.m. to 9:30 p.m., in Room 407 of the Third Teaching Building (三教407).

Materials

Lecture 1: Introduction

• Warm up: Preliminary [Notes]
  
• Part I: Introduction to Deep Learning [Slides]
  
• Part II: Introduction to LLM [Slides]

Lecture 2: Basics in Machine Learning and Langugae Models

• Part I: Basics in Machine Learning [Slides]
  
• Part II: Basics in Language models [Slides]

Lecture 3: Transformer

• Part I: Seq2Seq; Attention; Transformer [Slides]
• Part II: Parameters, Computations, and Memory Costs in Transformer [Slides01] [Slides02]

Lecture 4: Some LLM Models

• Teacher forcing; Pretrain and Finetuning; BERT; GPTs [Slides]
• DeepSeek [Slides]

Lecture 5: Gradient Descent

• Convex set; Convex functions; Convex problems; Gradient descent [Slides] [Notes]
• Forward-backward propagation [Notes]

Lecture 6: Accelerated Gradient Descent

• Momentum gradient descent; Nesterov acceleration; Anderson acceleration [Slides] [Notes]
  

Lecture 7: Stochastic Gradient Descent

• Stochastic gradient descent [Slides] [Notes]
  
• Sampling in SGD [Slides]
• Stability in SGD [Slides]