Advanced Topics in Statistics (461-0-20)
Topic
Information Theory and Statistics
Instructors
Feng Ruan
Meeting Info
Parkes Hall 214: Mon, Wed 2:00PM - 3:20PM
Overview of class
Topic: Information Theory and Statistics
Information theory was initially developed to answer the fundamental questions in communication theory, but its
connections to statistics and machine learning are profound. This course explores how the fundamental concepts in information theory—entropy and mutual information used to characterize the limits of data compression and communication—can be borrowed to understand the fundamental limits in statistical modeling and machine learning procedures, allowing guidance on optimal schemes among many off-the-shelf algorithmic choices. More concretely, we shall see how information-theoretic tools allow the spot of unrealistic statistics targets and provide optimality guarantees for constructive procedures in a wide range of modern applications, including optimization, private estimation, regressions, coding, etc.
Registration Requirements
Mathematical maturity of undergraduate-level calculus, probability theory and linear algebra.
Evaluation Method
30% Homework, 70% Class Project
Class Materials (Required)
No Textbooks. The course will largely follow the course notes by John Duchi
https://web.stanford.edu/class/stats311/lecture-notes.pdf