PKU Class 2024 Fall: Optimizaiton for Deep Learning

Instructor: Kun Yuan (kunyuan@pku.edu.cn)

Teaching assistants:

• Boao Kong (kongboao@stu.pku.edu.cn)
  
• Ziling Lin (linzl@stu.pku.edu.cn)
  
• Yuxi Liu (yuxiliu666@stu.pku.edu.cn)
  
• Yongqi Qiao (miraclefalls@stu.pku.edu.cn)
  
• Yilong Song (2301213059@pku.edu.cn)
  
• Baihao Wu (baihaowu@pku.edu.cn)
  
• Guangzhengao Yang (gzayang@stu.pku.edu.cn)
  
• Yuan Zhang (zy1002@stu.pku.edu.cn)
  

Classroom: 3pm - 6pm Tuesday, 三教403

Office hour: 4pm - 5pm Thursday, 静园六院220

References

Martin Jaggi and Nicolas Flammarion, Optimization for Machine Learning, EPFL Class CS-439
Chris De Sa, Advanced Machine Learning Systems, Cornell CS6787
Kun Yuan, Introduction to LLM, PKU

Materials

Lecture 1: Introduction

• Warm up: Preliminary [Notes]
  
• Part I: Introduction to Deep Learning [Slides]
  
• Part II: Introduction to LLM [Slides]

Lecture 2: Basics in Machine Learning and Langugae Models

• Part I: Basics in Machine Learning [Slides]
  
• Part II: Basics in Language models [Slides]

Lecture 3: Transformers

• Part I: Transformers [Slides]
  
• Part II: BERT and GPTs [Slides]

Lecture 4: Parameters and Computations in Decoder-only LLMs

• Parameters and Computations analysis [Slides]
  

Lecture 5: Stochastic Gradient Descent

• Gradient descent [Slides] [Notes]
  
• Stochastic gradient descent [Slides] [Notes]