PKU Class 2026 Spring: Introduction to Foundation Models

Instructor: Kun Yuan (kunyuan@pku.edu.cn)

Teaching assistants:

• Daibo Li (2501210088@stu.pku.edu.cn)
  
• Yilong Song (2301213059@pku.edu.cn)
  
• Zhoutong Wu (2501111519@stu.pku.edu.cn)
  

Classroom: 6:30pm - 8:30pm Tuesday, 1:00pm - 3:00pm Thursday, 三教506

Office hour: 4pm - 5pm Wednesday, 静园六院220

References

Stanford CS224n: Natural Language Processing with Deep Learning

Lectrures

Lecture 1: Introduction to LLM

• Introduction to deep learning [Slides1]
  
• Introduction to large language model [Slides2]
• Reading:
  
  • Andrej Karpathy, State of GPT
    
  • Andrej Karpathy, The busy person’s intro to LLM
    

Lecture 2: Machine Learning Basics

• Preliminary [Notes]
  
• Linear regression; Logistic regression; Multi-classification; Neural network [Slides]
• Reading:
  
  • Stanford CS231n, Linear classification
    
  • Stanford CS231n, Neural netowrk part I

Lecture 3: Language Models

• Word embedding; Language models; Recurrent neural networks [Slides]
  
• Seq2Seq; Attention; Transformer [Slides]
• Reading:
  
  • Stanford CS224N: Week 4
    
  • Ilya Sutskever et. al., Sequence to Sequence Learning with Neural Networks
    
  • Ashish Vaswani et. al., Attention Is All You Need
    

Lecture 4: Parameters, Computations, and Memories in Language Models

• Parameters, Computations, and Memory Costs in Transformer [Slides01] [Slides02]

Lecture 5: Popular LLM Models

• Teacher forcing; Pretrain and Finetuning; BERT; GPTs [Slides]
• DeepSeek [Slides]
• Reading:
  
  • J. Devlin et.al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
  • A. Radford et.al., Improving Language Understanding by Generative Pre-Training
  • A. Radford et.al., Language models are unsupervised multitask learners
  • T. B. Brown et.al., Language Models are Few-Shot Learners
  • DeepSeek, DeepSeek-V3 Technical Report

Lecture 6: Gradient descent

• Convex set; Convex functions; Convex problems; Gradient descent [Slides] [Notes]
• Forward-backward propagation [Notes]