Advanced Topics in Statistics (461-0-20)

Topic

Information Theory and Statistics

Instructors

Feng Ruan

Meeting Info

Parkes Hall 214: Mon, Wed 2:00PM - 3:20PM

Overview of class

Topic: Information Theory and Statistics
Information theory was initially developed to answer the fundamental questions in communication theory, but its
connections to statistics and machine learning are profound. This course explores how the fundamental concepts in information theory—entropy and mutual information used to characterize the limits of data compression and communication—can be borrowed to understand the fundamental limits in statistical modeling and machine learning procedures, allowing guidance on optimal schemes among many off-the-shelf algorithmic choices. More concretely, we shall see how information-theoretic tools allow the spot of unrealistic statistics targets and provide optimality guarantees for constructive procedures in a wide range of modern applications, including optimization, private estimation, regressions, coding, etc.

Registration Requirements

Mathematical maturity of undergraduate-level calculus, probability theory and linear algebra.

Evaluation Method

30% Homework, 70% Class Project

Class Materials (Required)

No Textbooks. The course will largely follow the course notes by John Duchi
https://web.stanford.edu/class/stats311/lecture-notes.pdf