CSE 599 D1 Theoretical Deep Learning

Summary

Instructor: Simon S. Du
Teaching Assistant: Ruoqi Shen
Lecture: Mon and Wed 10:00 - 11:20 PT on Zoom. Zoom link is on Canvas. You need to use your UW account.
Office Hour: By appointment, email me at ssdu@cs.washington.edu

This is a graduate course focused on research in theoretical aspects of deep learning. Deep learning has become the central paradigm of machine learning. However, a mathematical understanding is still lacking. How many samples are needed? How fast does training succeed? Why are convolutional neural networks suitable for image data? This course will cover recent advances in answering these questions.

Announcements

• First lecture: Jan. 4th 10AM

Prerequisite

This is an advanced and theory-heavy course: there is no programming assignment and students are required to work on a theory-focused course project. Students entering the class should have a working knowledge of machine learning, probability and statistics, optimization and linear algebra. Backgrounds on deep learning is a bonus but not necessary.

Grading Policies

Assignments 50% (25% each) and Project 50% (10% proposal, 15% presentation, 25% final report).

Homework

There are two homework, which will be released on Canvas.

• No late homework.

• Homework must be typed. You can use any typesetting software you wish (latex, markdown, ms word, etc).

• You may discuss assignments with others, but you must write down the solutions by yourself.

• We follow the standard UW CSE policy for academic integrity.

Timeline:

• First homework release: Feb. 1st. First homework due: Feb. 15th.

• Second homework release: Feb. 15th. Second homework due: Mar. 1st.

Course Project

Projects should be done in groups of 2-3, with the intention of exploring one direction in greater detail. If you cannot find team members for a group, send an email to the instructor. The projects can be either a literature review or original research. We list ideas for a few project topics on Canvas. It is okay to do projects on topic not listed.

Timeline:

• Proposal (due Jan. 29th): submit a short report (2 pages) stating the papers you plan to survey or the research problems that you plan to work on. Describe why they are interesting and provide a list appropriate references. You are encouraged to connect this pojrect with your current research. You will receive feedback from the instructor.

• Presentation (Mar. 8th and Mar. 10th): more details soon.

• Final report (due Mar. 15th): You are expected to submit a written project report (~8pages) describing your findings.

Format: You must use the NeurIPS Latex format.

Reference Readings

• Deep Learning Theory (draft)

• Understanding Machine Learning

Related Courses

Sanjeev Aora: Theoretical Deep Learning
Soheil Feizi: Foundations of Deep Learning
Matus Telgarsky: Deep Learning Theory
Konstantinos Daskalakis, Aleksander Madry: Science of Deep Learning: Bridging Theory and Practice
David Donoho, Vardan Papyan, Yiqiao Zhong: Analyses of Deep Learning