Description: Statistical learning theory is a burgeoning research field at the intersection of probability, statistics, computer science, and optimization that studies the performance of computer algorithms for making predictions on the basis of training data. The following topics will be covered: basics of statistical decision theory; concentration inequalities; supervised and unsupervised learning; empirical risk minimization; complexity-regularized estimation; generalization bounds for learning algorithms; VC dimension and Rademacher complexities; minimax lower bounds; online learning and optimization. Along with the general theory, we will discuss a number of applications of statistical learning theory to signal processing, information theory, and adaptive control.
Problem sets:
Problem set 1 solutions .tex notes
Problem set 2 solutions .tex notes
Problem set 3 solutions .tex notes
Problem set 4 solutions .tex notes
Problem set 5 solutions .tex notes
Problem set 6 solutions .tex notes
Problem set 7 .tex notes
Exams:
Exam 1 solutions
Exam 2 will be held Wednesday, May 10, 7-9 pm in usual classroom (or classroom next to it, if available).
Project presentations will be scheduled MWF, May 8-12, 10am-noon. Please select a time slot and enter topic
(when you know it) in the Wiki under the tools section of the Compass page for the course.
Project write-ups are due 5pm Friday, May 12.
Grading scheme: Homework (40%), two ninety minute midterms (20% each), project (20%).
Prerequisite: ECE 534, Random Processes
Credit: 4 graduate hours
Lecture times and location: TuTh 2:00-3:20 p.m. in Room 2015 ECE Building
Assigned Reading: The reading will mainly be from notes prepared for this course by Prof. Max Raginsky . See the Fall 2013, Fall 2014, or Fall 2015 websites for earlier versions of the notes and related references. Additional reading may be assigned from other books or articles, including:
Course Staff and Office Hours:
Bruce Hajek, Instructor
b-hajek AT illinois dot edu |
Office Hours: Wednesdays, 1-3 pm in Room 105 CSL |
Harsh Gupta, TA hgupta10 AT illinois dot edu |
Office Hours: Wednesdays, 11 am - 1 pm in Room 2036 ECEB |
Muhammed Sayin, TA sayin2 AT illinois dot edu |
Office Hours: Mondays, 2-4 pm in Room 2036 ECEB |
Amir Taghvaei, TA taghvae2 AT illinois dot edu |
Office Hours: Mondays 4-6 pm in Room 2036 ECEB |
Optional recitation sessions. Staffed by TAs. |
Fridays, 1pm-2pm in Room 3020 ECEB (north tower)
TAs will work out sample problems and examples, review background material as needed. |
Question and answer site: Piazza
About the project: For the project you are to choose a topic related to the course content and understand and critically evaluate two or three major papers in that area. Then demonstrate knowledge of the papers by working an example based on a paper or possibly extending the theory of a paper. You will need to write a project report of five to ten pages in length, and prepare a fifteen minute presentation.
Additional policy:
Collaboration on the homework is permitted, however each student must write and submit independent solutions. Homework is due within the first 5 minutes of the class period on the due date. No late homework will be accepted (unless an extension is granted in advance by the instructor).
You are encouraged to do
your homework in Latex. The .tex source of the problem set is provided for your convenience. Also, you can upload
your homework to the compass/blackboard system instead of turning in a hard copy. If your
solutions are hand written, points may be deducted if the handwriting is difficult to read.
You may bring two sheets of notes plus a copy of the supplementary notes to the first exam and three sheets of notes plus the supplementary notes to the second exam. You may use both sides of the sheets, with font size 10 or larger printing (or similar handwriting size). The examinations are closed book otherwise. Calculators, laptop computers, tables of integrals, etc. are not permitted.