Books
- “Elements of information theory” by T. M. Cover and J. A. Thomas – perfect intro book to the topic.
- “Information Theory, Inference, and Learning Algorithms” by David J.C. MacKay
- “Information Theory From Coding to Learning” by Yury Polyanskiy and Yihong Wu
Lecture Notes & Articles
- “A Mathematical Theory of Communication” by C. Shannon (2948) – entertaining and readable, even 70+ years later!
- “Lecture Notes on Statistics and Information Theory” by John Duchi
- “Information-theoretic methods for high-dimensional statistics” by Yihong Wu