ECE 563  Information Theory (Fall 2019)
Lecturer: Olgica Milenkovic (Office hours: Thursday 3:305:00pm, 313 CSL or by appointment as needed)
Teaching Assistants: Pattabiraman, Srilakshmi (Office hours, Tuesday 3:004:00pm, 3036 ECE; sp16@illinois.edu)
Lectures: Tuesday and Thursday, 12:30pm, 2015 Electrical and Computer Engineering Building
Problem Solving Sessions: The sessions will start on September 13th, 2019 Friday, 2:00pm, 141 Coordinated Science Laboratory [optional]
Course Objectives:
Catalog Description
Mathematical models for channels and sources; entropy, information, data compression, channel capacity, Shannon's theorems, and ratedistortion theory.
Prerequisites: Solid background in probability (ECE 534, MATH 464, or MATH 564).
Textbook: T. M. Cover and J. A. Thomas, Elements of Information Theory, 2nd ed., Wiley, 2006.
Grading: Homework (25%), Midterm exam [in class] (25%), Final exam [at a date determined by the university] (25%), Group project/paper (25%)
Midterm I: October 24th, 6pm, Room 2017
Midterm I Review: October 22nd, 5:006:30pm, Room 2017
Research Project Topics
Genomic Data Compression Review paper
The Information Bottleneck Problem Research paper
Lovasz number of a graph Lecture notes
Quantized Deep Learning Blog
Capacity of DNAStorage Channels Research paper
Polar codes Research paper
Network coding Text
Quantum information theory Text
Renyi entropy NeurIPS paper
Deletion errorcorrection and capacity of deletion channels Review paper by Sloane
Channel dispersion: finite blocklength regime Y. Polyansky et al., see also the prior work by Strassen.
Additional Instructional Material
Entropy in Physics (Video, TEDed)
Operational Characterization of Entropy (Video, Khan Academy)
The first lecture on the axiomatic derivation of Shannon's entropy is based on R. Ash, Information Theory, pp. 512. More on axiomatic approaches can be found here Entropy axioms
Homeworks, Fall 2019
Homeworks, Fall 2018 with Solutions
Problem Solving Sessions, Fall 2018
Exams
Juxtaposition Paper
Course Schedule
Date  Topic  Reading Assignment  Learning Objectives  Multimedia Supplements 
8/28 
1. The problem of communication, information theory beyond communication [slides] 


8/30 
2. The idea of errorcontrol coding and linear codes [slides] [handwritten] 


9/4  3. Information measures and their axiomatic derivation 



4. Basic inequalities with information measures 


9/11  5. Asymptotic Equipartition Property 



9/13  6. Source Coding Theorem 



9/18  7. Variablelength Codes 


9/20  8. Entropy Rate of Stochastic Processes 


9/25  9. Distributed Source Coding 


9/27  10. Universal Source Coding 



10/2  11. Method of Types 


10/4  12. Allerton Conference [no lecture]  
10/9  13. Hypothesis Testing 


10/11  14. Channel Coding Theorem: Converse and Joint AEP 



10/16  15. Channel Coding Theorem: Achievability and Examples 


10/18  16. Midterm [no lecture]  
10/23  17. SourceChannel Separation 


10/25  18. Differential Entropy, Maximum Entropy, and Capacity of RealValued Channels 


10/30  19. RateDistortion Theorem: Converse and Examples 



11/1  20. RateDistortion Theorem: Achievability and More Examples 


11/6  21. Quantization Theory 


11/8  22. BlahutArimoto 


11/13  23. Strong Data Processing Inequalities 


11/15  24. Large Deviations 


11/27  25. Error Exponents for Channel Coding 


11/29  26. Error Exponents for Channel Coding 


12/4  27. Multiple Access Channel: Achievability 


12/6  28. Quantum Information Theory [guest lecture]  
12/11  29. Multiple Access Channel: Converse, Examples, and Duality 

Topics: