Deep Learning: Classics and Trends

In the past decade, computer systems and chips have played a key role in the success of AI. Our vision in Google Brain’s ML for Systems team is to use AI to transform the way in which computer systems and chips are designed. Many core problems in systems and hardware design are combinatorial optimization or decision making tasks with state and action spaces that are orders of magnitude larger than that of standard AI benchmarks in robotics and games. In this talk, we will describe some of our latest learning based approaches to tackling such large-scale optimization problems. We will discuss our work on a new domain-transferable reinforcement learning method for optimizing chip placement, a long pole in hardware design. Our approach is capable of learning from past experience and improving over time, resulting in more optimized placements on unseen chip blocks as the RL agent is exposed to a larger volume of data. Our objective is to minimize PPA (power, performance, and area), and we show that, in under 6 hours, our method can generate placements that are superhuman or comparable on modern accelerator chips, whereas existing baselines require human experts in the loop and can take several weeks.

The success of modern deep learning techniques is based on standardized and balanced datasets such as ImageNet. However, methods developed from these datasets often fail on realistic datasets under realistic scenarios. In this presentation, we summarize three characteristics of realistic datasets: 1) long-tailed; 2) open-ended; and 3) multi-domain. We also discuss two CVPR projects that are dealing with these features: 1) Open Long-Tailed Recognition (OLTR) and 2) Open Compound Domain Adaptation (OCDA).

In OLTR: we address open long-tailed recognition with an integrated algorithm that handles imbalanced classification, few-shot learning, and open-set recognition at the same time, whereas existing classification approaches focus only on one aspect and deliver poorly over the entire class spectrum. Our OLTR algorithm maps an image to a feature space such that visual concepts can relate to each other based on a learned metric that respects the closed-world classification while acknowledging the novelty of the open world.

In OCDA: we consider the open compound domain adaptation problem, in which the compound target domain is a combination of multiple traditional target domains without domain labels, reflecting realistic data collection in various mixed as well as novel conditions. Contrary to existing single- or multi-target domain adaptation works, where known clear distinctions between domains are often assumed, OCDA does not rely on domain boundaries and continuously adapts the model within a learned domain space. Our model consists of two technical insights into OCDA: 1) a curriculum domain adaptation strategy to bootstrap generalization across domain distinction in a data-driven, self-organizing, and continuous fashion and 2) a memory module to increase the model’s agility towards novel domains.

Although ResNets and BERT achieve high accuracy on in-distribution examples, do they generalize to new distributions? In this talk I survey benchmarks in vision and NLP that measure how well models hold up when there is a discrepancy between the train and test set. The talk will draw on results in NLP from http://arxiv.org/abs/2004.06100 and recent vision results from http://arxiv.org/abs/2006.16241

I’ll describe our major contributions in this paper, as well as where we fell short. My work was mainly on training data, eval memorization, and the eval suite. I’ll offer deep dives on these sections.

We study the phenomenon that some modules of deep neural networks (DNNs) are more critical than others. Meaning that rewinding their parameter values back to initialization, while keeping other modules fixed at the trained parameters, results in a large drop in the network’s performance. Our analysis reveals interesting properties of the loss landscape which leads us to propose a complexity measure, called module criticality, based on the shape of the valleys that connect the initial and final values of the module parameters. We formulate how generalization relates to the module criticality, and show that this measure is able to explain the superior generalization performance of some architectures over others, whereas earlier measures fail to do so. I will also cover our recent results on extension to transfer learning setting, and how module criticality predicts which layers of the network play an important role for successful transfer.

Deep learning algorithms are well-known to have a propensity for fitting the training data very well and often fit even outliers and mislabeled data points. Such fitting requires memorization of training data labels, a phenomenon that has attracted significant research interest but has not been given a compelling explanation so far. A recent work of Feldman [Fel19] proposes a theoretical explanation for this phenomenon based on a combination of two insights. First, natural image and data distributions are (informally) known to be long-tailed, that is have a significant fraction of rare and atypical examples. Second, in a simple theoretical model such memorization is necessary for achieving close-to-optimal generalization error when the data distribution is long-tailed. However, no direct empirical evidence for this explanation or even an approach for obtaining such evidence were given.

In this work we design experiments to test the key ideas in this theory. The experiments require estimation of the influence of each training example on the accuracy at each test example as well as memorization values of training examples. Estimating these quantities directly is computationally prohibitive but we show that closely-related subsampled influence and memorization values can be estimated much more efficiently. Our experiments demonstrate the significant benefits of memorization for generalization on several standard benchmarks. They also provide quantitative and visually compelling evidence for the theory put forth in [Fel19].

Why did you go to grad school? Looking back, what would you have done differently? What are today’s grad school students/applicants facing, and how can they be better supported? We wish to touch upon all these during the panel, and address any other questions from the public via Slido 👇👇

Sparsity in Deep Neural Networks (DNNs) is studied extensively with the focus of maximizing prediction accuracy given an overall parameter budget. Existing methods rely on uniform or heuristic non-uniform sparsity budgets which have sub-optimal layer-wise parameter allocation resulting in a) lower prediction accuracy or b) higher inference cost (FLOPs). We propose Soft Threshold Reparameterization (STR), a novel use of the soft-threshold operator on DNN weights. STR smoothly induces sparsity while learning pruning thresholds thereby obtaining a non-uniform sparsity budget. Our method achieves state-of-the-art accuracy for unstructured sparsity in CNNs (ResNet50 and MobileNetV1 on ImageNet-1K), and, additionally, learns non-uniform budgets that empirically reduce the FLOPs by up to 50%. Notably, STR boosts the accuracy over existing results by up to 10% in the ultra sparse (99%) regime and can also be used to induce low-rank (structured sparsity) in RNNs. In short, STR is a simple mechanism which learns effective sparsity budgets that contrast with popular heuristics. Code, pretrained models and sparsity budgets are at https://github.com/RAIVNLab/STR.

Distributed optimization is essential for training large models on large datasets. Multiple approaches have been proposed to reduce the communication overhead in distributed training, such as synchronizing only after performing multiple local SGD steps, and decentralized methods (eg, using gossip algorithms) to decouple communications among workers. Although these methods run faster than AllReduce-based methods, which use blocking communication before every update, the resulting models may be less accurate after the same number of updates. Inspired by the BMUF method of Chen & Huo (2016), we propose a slow momentum (SlowMo) framework, where workers periodically synchronize and perform a momentum update, after multiple iterations of a base optimization algorithm. Experiments on image classification and machine translation tasks demonstrate that SlowMo consistently yields improvements in optimization and generalization performance relative to the base optimizer, even when the additional overhead is amortized over many updates so that the SlowMo runtime is on par with that of the base optimizer. We provide theoretical convergence guarantees showing that SlowMo converges to a stationary point of smooth non-convex losses. Since BMUF is a particular instance of the SlowMo framework, our results also correspond to the first theoretical convergence guarantees for BMUF.

Attention has been studied in psychology for over a hundred years and studies that record from neurons have aimed to understand the physical underpinnings of attentional processes for several decades. More recently, attention mechanisms have been added to artificial neural networks to enhance their performance. In this talk, I will briefly overview the study of attention in these different domains, with a focus on visual attention. I will then describe my own work using findings from neurophysiology to add feature-based attention to convolutional neural networks (CNNs). CNNs are currently some of the best models available of the primate visual system and they allow neuroscientists to probe the relationship between neural activity and task performance “in silico”. I will share how studying attention in these models can lead to a rethinking of how biological attention works.

To improve generalisation in supervised learning, it is common to encourage invariance in the solution, i.e. keeping the output relatively constant to irrelevant transformations of the input. Many techniques can be seen as introducing invariance, such as data augmentation, convolutional structure, or more general group structure.
But how do we learn what invariances should be used for a dataset? In this talk, we will discuss why the usual training loss is not the right objective function. We instead use the marginal likelihood as suggested by Bayesian inference, and develop a procedure which learns a useful invariance through gradient-based optimisation. Our model learns to be invariant to perturbations that are commonly hand-crafted in data augmentation, and learns very different perturbations depending on the dataset. We finish by speculating on how procedures like these can help automate the creation of network architectures.

In this talk I’ll propose a historical perspective that traces the origin of current self-supervision and word embedding trends in machine learning to the structuralist ideas proposed by Ferdinand de Saussure and Ludwig Wittgenstein in the early 20th century. I will also showcase several distributional semantic models (pre deep learning approaches to learn word representations) and connect them with more modern approaches up to recent self-supervised models for language, vision and graph structured data. The intent is that by showing the origins of these ideas the audience would be better equipped to both put the current self-supervision research in perspective with respect to the broader cultural context, and learn from past research as it contained deep insights that can help inform future directions for the field.
WARNING: there will not be a lot of deep learning in the talk, but potentially a lot of food for thought.

In this talk, I review a new class of genotype-to-phenotype encodings, which are not manually defined but learned from the data itself. For example, we can train a GAN on Super Mario Bros levels, allowing levels to be evolved in the latent space of a GAN that maximize desired properties such as difficulty. When the GAN is trained on a specific target domain, it becomes a compact and robust genotype-to-phenotype mapping allowing for target-based evolution. This Latent Variable Evolution (LVE) approach can also be combined with interactive evolution, allowing users to breed their own video game levels and play those discovered levels. I’ll also present our latest results on CPPN2GAN, in which a Compositional Pattern Producing Network (CPPN) can define latent vector GAN inputs as a function of geometry, which provides a way to organize level segments output by a GAN into large-scale patterns. The benefit of these data-driven encodings is that they make it easy to explore the space of high-quality solutions, for both humans and optimization algorithms.

There is a fundamental gap between how humans understand and use language — in open-ended, real-world situations — and today’s NLP benchmarks for language understanding. To narrow this gap, we propose to evaluate machines by their success at real-world language use – which greatly expands the scope of language tasks that can be measured and studied.
We introduce TuringAdvice, a new challenge for language understanding systems. Given a complex situation faced by a real person, a machine must generate helpful advice. We make our challenge concrete by introducing RedditAdvice, a dataset and leaderboard for measuring progress. Though we release a training set with 600k examples, our evaluation is dynamic, continually evolving with the language people use: models must generate helpful advice for recently-written situations.
Empirical results show that today’s models struggle at our task, even those with billions of parameters. The best model, a finetuned T5, writes advice that is at least as helpful as human-written advice in only 9% of cases. This low performance reveals language understanding errors that are hard to spot outside of a generative setting, showing much room for progress.

Human intelligence exhibits systematic compositionality (Fodor & Pylyshyn, 1988), the capacity to understand and produce a potentially infinite number of novel combinations of known components, i.e., to make “infinite use of finite means” (Chomsky, 1965). In the context of learning from a set of training examples, we can observe compositionality as compositional generalization, which we take to mean the ability to generalize to composed test examples from one distribution after being exposed to the necessary components during training on a different distribution.
In this talk, I will discuss several papers on the topic of compositional generalization. We will first look at the compositional generalization abilities of seq2seq models as measured on the SCAN tasks, and then look at some ideas that have been proposed to improve compositional generalization.

In this talk, I’ll trace the evolution of the main ideas in Deepmind’s Go playing ML work you’ve surely heard of. We’ll start with the original model free AlphaGo paper and work our way through to the recent model based MuZero.