Lectures 2026: Optimization for Large Language Models

Instructor: Kun Yuan (kunyuan@pku.edu.cn)

This is a 10-hour intensive course on Optimization for Large Language Models. I would like to express my sincere gratitude to the Operations Research Society of China for the invitation and excellent organization.

Classroom: To be announced

Time: To be annouced

References

Martin Jaggi and Nicolas Flammarion, Optimization for Machine Learning, EPFL Class CS-439
Chris De Sa, Advanced Machine Learning Systems, Cornell CS6787
Zaiwen Wen, Optimization Methods, PKU 2024 Fall
Kun Yuan, Introduction to LLM, PKU 2025 Spring

Materials

Lecture 1: Gradient Descent

• Introduction; Gradient Descent; Forward-Backward Splitting [Slides01] [Slides02] [Notes]

Lecture 2: Stochastic Gradient Descent

• SGD; Momentum SGD; Adaptive SGD [Slides01] [Slides02]

Lecture 3: LLM Foundations - Part I

• Word embedding; Language Models; RNN; Seq2Seq [Slides01] [Slides02]

Lecture 4: LLM Foundations - Part II

• Attention; Transformers [Slides]

Lecture 5: Costs in LLM Pre-Training

• Parameters, Computations, and Memory Costs [Slides01] [Slides02]

Lecture 6: Perturbed SGD and Mixed-Precision Training

• Perturbed SGD; Mixed-Precision Training [Slides]

Lecture 7: Coordinate Descent and Layer-wise Training

• Coordinate Descent; Layer-wise Training; LISA; BAdam; MISA [Slides01] [Slides02]

Lecture 8: Subspace Optimization and Low-Rank Training

• Subspace Optimization; Low-rank gradient projection

Lecture 9: Zeroth-order Optimization and Activation-Free LLM Training

• Zeros-Order Optimization; MeZO; LoZO [Slides01] [Slides02]

Lecture 10: Distributed Optimization for LLM Training

• Distributed SGD; Decentralized SGD; Local SGD; Communication Compression [Slides01] [Slides02] [Slides03]