Deep Learning: Classics and Trends



Coming up

Date Presenter Topic or Paper
2020.07.31
2020.08.14 Zhongqi Miao, Ziwei Liu Open Compound Domain Adaptation
2020.08.21 Anna Goldie, Azalia Mirhoseini Chip Placement with Deep Reinforcement Learning
2020.08.28 Arianna Ornaghi Stereotypes in High-Stakes Decisions: Evidence from U.S. Circuit Courts
2020.09.04 Shrimai Prabhumoye Controllable Text Generation
2020.09.11 Jesse Mu Compositional Explanations of Neurons
2020.09.18 Katherine Ye Penrose: From Mathematical Notation to Beautiful Diagrams
2020.09.25 Sidak Pal Singh Model Fusion via Optimal Transport


The Story

“A super influential reading group that has achieved cult-like status.” —John Sears

Deep Learning: Classics and Trends (DLCT) is a reading group I have been running since 2018. It started within Uber AI Labs, with the support of Zoubin, Ken and Jason, and the help of many, when we felt the need of a space to sample the overwhelmingly large amount of papers, and to hold free-form, judgemental (JK) discussions; or as Piero puts it, to “ask a million questions”.

Since then, it has grown much larger, first opened up to the broader machine learning community in Uber, then to the general public in 2019. Starting March 2020, in light of COVID-19, we hold all meetings virtually, making it radically accessible to anyone from anywhere. From June 2020, DLCT operates under ML Collective, with a mission of making researchers more connected.

To me, it’s more than reading papers and attending presentations. It has started to serve as an anchor for all of us to connect every once in a while amidst all the changes, shifts of emphasis, and chaos, in Bay Area, in AI research, and generally in this fast-paced world.

The best thing about it is the group of people that it enables to connect—seriously, the smartest and kindest researchers that I feel so lucky to have known and have worked with.

  • Time: Every Friday, 12pm - 1pm Pacific Time
  • Place: Virturally on Zoom (up to 100 participants)
  • Format: Presentation based. An invited speaker would talk about a paper with slides, a lot of the time themselves being the author of the paper.
  • Scope: Deep learning, old (a.k.a. “let’s revisit the 2014 GAN paper”) and new (a.k.a. “look at this blog post from yesterday”).
  • Join the mailing list to receive weekly notifications of the upcoming talk.
  • Nominate a speaker, a paper, or just tell me what you think.

If you are an ML researcher and have a paper that you are proud of: tell us (besides posting on Twitter)! This could be yet another platform for feedbacks and engagements with a community of 600+ students, scientists and ML engineers and enthusiasts.


Past events

Date Presenter Topic or Paper
2020.07.24
[Paper] [Slides]
2020.07.17
[Paper]
2020.07.10
[Paper] [Slides]
2020.07.03
[Slido]
2020.06.26
[Paper] [Slides]
2020.06.19
[Paper] [Slides]
2020.06.12
[Paper] [Slides]
2020.06.08
[Paper] [Slides] [Recording]
2020.05.22
[Slides & Recording on Piero’s website]
2020.05.08
[Paper 1] [Paper 2] [Paper 3] [Slides]
2020.05.01
[Paper] [Slides]
2020.04.24
[Paper 1] [Paper 2] [Paper 3] [Slides]
2020.04.17 Nikhil Dev Deshmudre [AlphaGo], [AlphaGo Zero], [Alpha Zero], [MuZero] [Slides]
2020.04.10
[AlphaGo], [AlphaGo Zero], [Alpha Zero], [MuZero] [Slides] [Recording]
2020.04.03 Alyssa Dayan Mode-Adaptive Neural Networks for Quadruped Motion Control [Slides]
2020.03.27 Michela Paganini Empirical Observations in Pruned Networks & Tools for Reproducible Pruning Research
2020.03.20 Rapha Gontijo Lopes Affinity and Diversity: Quantifying Mechanisms of Data Augmentation [Slides] [Recording]
2020.03.13 Ian Thompson A Good View Is All You Need: Deep InfoMax (DIM) and Augmented Multiscale Deep InfoMax (AMDIM) [Slides] [Recording]
2020.02.28 Ashley Edwards Estimating Q(s, s’) with Deep Deterministic Dynamics Gradients [Slides]
2020.02.14 Xinchen Yan Conditional generative modeling and adversarial learning
2020.02.07 Yaroslav Bulatov einsum is all you need [Slides] [Recording]
2020.01.31 Rosanne Liu Selective Brain Damage: Measuring the Disparate Impact of Model Pruning
2020.01.24 Jeff Coggshall ReMixMatch and FixMatch
2020.01.17 Rosanne Liu Improving sample diversity of a pre-trained, class-conditional GAN by changing its class embeddings [Slides] [Recording]
2020.01.10 Zhuoyuan Chen Why Build an Assistant in Minecraft?
2019.11.22 Rosanne Liu On the “steerability” of generative adversarialnetworks [Slides] [Recording]
2019.11.15 Polina Binder Learning Deep Sigmoid Belief Networks with Data Augmentation
2019.11.08 Sanyam Kapoor Policy Search & Planning: Unifying Connections [1][2]
2019.11.01 Chris Olah Zoom in: Features and circuits as the basic unit of neural networks
2019.10.25 Renjie Liao Efficient Graph Generation with Graph Recurrent Attention Networks
2019.10.18 Nitish Shirish Keskar, Bryan McCann CTRL: A Conditional Transformer Language Model for Controllable Generation
2019.10.11 Subutai Ahmad Sparsity in the neocortex, and its implications for machine learning
2019.10.04 Eli Bingham Multiple Causes: A Causal Graphical View
2019.09.27 Xinyu Hu Learning Representations for Counterfactual Inference
2019.09.04 Jonathan Frankle The Latest Updates on the Lottery Ticket Hypothesis
2019.08.23 Ankit Jain Knowledge-aware Graph Neural Networks with Label Smoothness Regularization for Recommender Systems [Slides]
2019.08.16 Jiale Zhi Meta-Learning Neural Bloom Filters
2019.08.16 Ted Moskovitz Lookahead Optimizer: k steps forward, 1 step back
2019.07.26 Rui Wang Off-Policy Evaluation for Contextual Bandits and RL [1][2][3][4]
2019.07.19 Rosanne Liu Weight Agnostic Neural Networks [Slides] [Recording]
2019.07.12 Joost Huizinga A Distributional Perspective on Reinforcement Learning
2019.06.28 Ashley Edwards [ICML Preview] Learning Values and Policies from Observation [1][2]
2019.06.21 Stanislav Fořt [ICML Preview] Large Scale Structure of Neural Network Loss Landscapes
2019.06.07 Joey Bose [ICML Preview] Compositional Fairness Constraints for Graph Embeddings
2019.05.31 Yulun Li IntentNet: Learning to Predict Intention from Raw Sensor Data
2019.05.24 Thomas Miconi, Rosanne Liu, Janice Lan ICLR Recap, cont.
2019.05.17 Aditya Rawal, Jason Yosinski ICLR Recap
2019.04.26 JP Chen 3D-Aware Scene Manipulation via Inverse Graphics [Slides]
2019.04.19 Felipe Petroski Such Relational Deep Reinforcement Learning
2019.04.12 Piero Molino, Jason Yosinski Open mic
2019.04.05 Joel Lehman The copycat project: A model of mental fluidity and analogy-making
2019.03.29 Rosanne Liu Non-local Neural Networks [Slides]
2019.03.22 Yariv Sadan Learning deep representations by mutual information estimation and maximization [Slides]
2019.03.15 Chandra Khatri Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation
2019.03.01 Nikhil Dev Deshmudre BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
2019.02.22 Vashisht Madhavan Neural Turing Machines
2019.02.15 Open discussion GPT-2
2019.02.08 Adrien Ecoffet HyperNetworks
2019.02.01 Jiale Zhi Non-delusional Q-learning and value iteration
2019.01.25 Yulun Li Relational Recurrent Neural Networks
2019.01.18 Rui Wang Neural Ordinary Differential Equations
2019.01.11 Jonathan Simon Generating Humorous Portmanteaus using Word Embeddings [1][2] [Slides]
2018.12.21 Christian Perez Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles [Slides]
2018.12.14 Alexandros Papangelis Two trends in dialog [1][2]
2018.10.26 Aditya Rawal Stochastic Weight Averaging [Slides]
2018.10.12 Mahdi Namazifar Troubling Trends in Machine Learning Scholarship
2018.09.28 Yariv Sadan MINE: Mutual Information Neural Estimation [Slides]
2018.09.21 Jan-Matthis Lueckmann Glow and RealNVP [Slides]
2018.09.14 Jane Hung The YOLO series: v1, v2, v3
2018.09.07 Rosanne Liu Pooling is Neither Necessary nor Sufficient for Appropriate Deformation Stability in CNNs [Slides]
2018.08.31 Alican Bozkur Multimodal Unsupervised Image-to-Image Translation [Slides]
2018.08.24 Janice Lan The Lottery Ticket Hypothesis: Finding Small, Trainable Neural Networks [Slides]
2018.08.17 Yariv Sadan Opening the black box of Deep Neural Networks via Information [Slides]
2018.08.10 Joost Huizinga Learning to Reinforcement Learn, and RL 2: Fast Reinforcement Learning via Slow Reinforcement Learning [Slides]
2018.08.03 JP Chen Deep Convolutional Inverse Graphics Network [Slides]
2018.07.27 Lei Shu Attention is all you need [Slides]
2018.07.06 Neeraj Pradhan Auto encoding Variational Bayes, and ELBO
2018.06.29 Ankit Jain Dynamic Routing Between Capsules [Slides]
2018.06.22 Xinyu Hu Self Normalizing Neural Networks [Slides]
2018.06.15 John Sears The Decline and Fall of Adam: [1][2] [Slides]
2018.06.08 Alex Gajewski GANs, etc. [1][2] [Slides]
2018.06.01 Jason Yosinski Sensitivity and Generalization in Neural Networks: an Empirical Study [Slides]

Congratulations! Now that you have scrolled all the way down here, you get the reward of reading more text about the scope and vision of this reading group.

  • Q: What was the initial idea of organizing a reading group like this?
    A: It started with the rather selfish idea that I wanted to know about papers that I don’t have time to read, and learn about topics my individual intelligence limits me from fully understanding. Besides, I enjoy being around people that are smarter and more knowledgable than me, faster than me working out twelve math equations on one slide, braver than me to ask stupid questions, and more patient than me answering them, as well as those who value great presentations as much as I do.

  • Q: How much work is it for you?
    A: I never travel on Fridays now.

  • Q: Where do you see it going?
    A: I envision building a community where people work hard to tell science stories well. Each paper is a story. A great paper, apart from solid results and technical and scientific advances, stands out particularly in the way it tells the story. I hope we all value storytelling and talk-giving slightly more than we do now. This ties to an eventual wish that scientific writing moves towards being lucid and understandable. This reading group is a start.

    Here is how I see different levels of storytelling, in the format of an one-hour presentation, could happen in this group.

    You can give a Level 0 talk, which is going through someone else’s paper—the storyline is already there. This is perhaps the most basic and involves the least work: you just need to understand it and retell it to others. (I assume as a researcher you already read papers, and this additional work of making it into a presentation would only help you understand it better yourself.) And best of all, when the audience asks hard questions, you can just say “I don’t know—not my work.”

    A Level 1 talk, could mean presenting one of your own papers. The bar is higher because you are expected to know every detail of the project, but also lower because you probably already do. And a good background coverage to lead to the exact problem and idea always helps.

    Then we have Level 2 talks, which are usually a topic formed by understanding a field (however small it is) thoroughly well, and having in mind a hierarchical chart or spiderweb of a number of fields leading to that particular one. You might be citing multiple papers, drawing connections and coming up with conclusions that are mainly your own.

  • Q: Do you have a high bar for talks given there?
    A: Yes I do. But I also know we all have to start somewhere. And I myself was a horrible presenter not too long ago (likely still am). But we all get better.

2020-06-19 00:00:00 +0000 UTC

2020-06-11 00:00:00 +0000 UTC