CS 294-131: Special Topics in Deep Learning
Fall, 2016


Instructor and co-Instructors:

Teaching Assistant:

Lectures:

    Wed 10:00am-noon (First class starts on Aug 31)
    location: SDH250

Course mailing list:

    cs294-dl-f16@googlegroups.com
    Please sign up for the course mailing list for future updates.

    If you do not plan to take the class, but are interested in getting announcements about guest speakers in class, and more generally, deep learning talks at Berkeley, please sign up for the talk announcement mailing list for future announcements.

    Please also sign up for our Piazza.

Syllabus:

Lecture 1
8/31/16

Kaiming He: Deep Learning Gets Way Deeper [slides, introductory slides]

Reading:

  • Deep Learning, by Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.
Lecture 2
9/7/16

Ilya Sutskever: Why does deep learning actually work?

Main readings:

Background readings:
Lecture 3
9/14/16
Wojciech Zaremba: Turing-complete neural-network based models

Main readings:

Background readings:

Lecture 4
9/21/16
Kunal Talwar: Deep Learning with Differential Privacy

Main readings:

Background readings:
Lecture 5
9/27/16
(Special date)

Location: Wozniak Lounge (430-8 Soda Hall)
Time: 12:30 pm

Lecture 6
9/28/16
Samy Bengio: Sequence models

Main readings:

Background readings:

Lecture 7
10/5/16
Ian Goodfellow: Adversarial Examples and Adversarial Training [slides]

Main readings:

Background readings:

Lecture 8
10/12/16
Oriol Vinyals: Sequences and one-shot learning

Main readings:

Lecture 9
10/19/16
Bill Dally: Hardware for Deep Learning

Main readings:

Background readings:
Lecture 10
10/26/16
Hugo Larochelle: Autoregressive Generative Models with Deep Learning

Main readings:

Background readings:
  • Deep Learning, Chapter 20 (Generative Models)
    • Sections 20.1 to 20.4 (background on undirected graphical models)
    • Sections 20.9 and 20.10.1 to 20.10.5 (background on directed graphical models)

Lecture 11
11/2/16
Jitendra Malik: Embodied Cognition: Phylogeny and Ontogeny

Main readings:

Background readings:

Lecture 12
11/9/16
Surya Ganguli: Deep learning theory and physics-inspired generative models

Main readings:

Background readings:

Lecture 13
11/16/16
Timothy Lillicrap: Recent advances in deep reinforcement learning

Main readings:

Background readings:
Lecture 14
11/23/16
CANCELLED! No class 11/23
Diedrik Kingma: Generative models

Main readings:

Lecture 15
11/30/16
In-class project presentations

Course Description:

    In recent years, deep learning has enabled huge progress in many domains including computer vision, speech, NLP, and robotics. It has become the leading solution for many tasks, from winning the ImageNet competition to winning at Go against a world champion. This class is designed to help students develop a deeper understanding of deep learning and explore new research directions and applications of deep learning. It assumes that students already have a basic understanding of deep learning. In particular, we will explore a selected list of new, cutting-edge topics in deep learning:

    • Security and privacy issues in deep learning. First, we will explore attack methods and defenses in the area of adversarial deep learning, where attackers can purposefully generate adversarial examples to fool state-of-the-art deep learning systems. Second, we will explore the area of privacy-preserving deep learning. A deep learning system trained over private data could memorize and leak private information undesirably. We will explore areas including model-inversion attacks and how to provide differential privacy guarantees for deep learning algorithms. Finally, we will explore the use of deep learning in security applications such as malware and fraud detection.
    • Novel application domains of deep learning, beyond the mainstays of computer vision and speech recognition. First, we will explore new techniques in deep reinforcement learning, involving both applications of reinforcement learning to traditionally supervised learning problems and applications of deep learning to tasks that involve decision making and control. Second, we will explore new domains at the intersection of deep learning and program synthesis and formal verification. We will also explore other new application domains such as using deep learning for graph analysis.
    • Recent advances in the theoretical and systems aspects of deep learning. First, we will cover the recent advances in generative models, including variational autoencoders and generative adversarial networks. Second, we will explore new theoretical advances in understanding deep learning such as the Deep Rendering Model. Third, we will explore new system and architectural advances in scaling up deep learning and new architectural designs.

Class Format and Project:

    This is a lecture, discussion, and project oriented class. Each lecture will focus on one of the topics, including a survey of the state-of-the-art in the area and an in-depth discussion of the topic. Each week, students are expected to complete reading assignments before class and participate actively in class discussion.

    Students will also form project groups and complete a research-quality class project. Groups will consist of one to three students.

    Course #: COMPSCI 294-131 (Class #34939)

Deadlines:

  • Reading assignment deadlines:
    • For students,
      • Submit questions by Sunday noon
      • Vote on the poll of discussion questions by Monday 11:59 pm
    • For discussion leads,
      • Send form to collect questions from students by Friday 11:59 pm
      • Summarize questions proposed by students to form the poll and send it by Sunday 11:59 pm
      • Summarize the poll to generate a ranked & categorized discussion question list and send the list to teaching staff by Tuesday 7pm

  • Project deadlines:
    • 9/21: project groups due
    • 9/28: project proposal due
    • 10/26: project milestone report due
    • 11/30: in-class presentation
    • 12/16: final project report due

Background reading:

    Deep Learning, by Ian Goodfellow, Yoshua Bengio, and Aaron Courville

Grading:

  • 20% class participation
  • 35% weekly reading assignment
    • 10% discussion leads
    • 25% individual reading assignments
  • 45% project

All information is tentative and subject to change.