Deep Learning Summit schedule

What Do Your Neural Networks Learn? A Peek Inside the Black Box

Deep neural networks are famously difficult to interpret. We'll take a tour of their inner workings to build an intuition of what's inside the black box and how all those cogs fit together. Then we'll use those insights as we step through a image processing problem with deep learning, showing at every step what the neural network is "thinking".

Brandon love solving puzzles and building things. Applied machine learning gives him the opportunity to do both in equal measure. He started by studying robotics and human rehabilitation at MIT (MS '99, PhD '02), moved on to machine vision and complex system modeling at Sandia National Laboratories, then to predictive modeling of agriculture DuPont Pioneer, and cloud data science at Microsoft. At Facebook he works to get internet and electrical power to those in the world who don't have it, using deep learning and satellite imagery, and to do a better job identifying topics reliably in unstructured text. In his spare time he likes to rock climb, write robot learning algorithms, and go on walks with his wife and our dog, Reign of Terror.

AI Neuroscience: Can we Understand the Neural Networks we Train?

Deep neural networks have recently made a bit of a splash, enabling machines to learn to solve problems that had previously been easy for humans but difficult for computers, like playing Atari games or identifying lions and jaguars in photos. But how do these neural nets actually work? What concepts do they learn en route to their goals? We built and trained the networks, so on the surface these questions might seem trivial to answer. However, network training dynamics, internal representations, and mechanisms of computation turn out to be surprisingly tricky to study and understand, because networks have so many connections — often millions or more — that the resulting computation is fundamentally complex.

This high fundamental complexity enables the models to master their tasks, but we find now that we need something like neuroscience just to understand the AI that we’ve constructed! As we continue to train more complex networks on larger and larger datasets, the gap between what we can build and what we can understand will only grow wider. This gap both inhibits progress toward more competent AI and bodes poorly for a society that will increasingly be run by learned algorithms that are poorly understood. In this talk, we’ll look at a collection of research aimed at shrinking this gap, with approaches including interactive model exploration, optimization, and visualization.

Jason Yosinski is a machine learning researcher, founding member of Uber AI Labs, and scientific adviser to Recursion Pharmaceuticals. His work focuses on building more capable and more understandable AI. He suspects scientists and engineers will build increasingly powerful AI systems faster than we can understand them, motivating much of his work on what has been called AI Neuroscience -- an emerging field of study that investigates fundamental properties and behaviors of AI systems. Dr. Yosinski was previously a PhD student and NASA Space Technology Research Fellow working at the Cornell Creative Machines Lab, the University of Montreal, Caltech/NASA Jet Propulsion Laboratory, and Google DeepMind. His work on AI has been featured on NPR, Fast Company, the Economist, TEDx, and on the BBC. Prior to his academic career, Jason cofounded two web technology companies and started a program in the Los Angeles school district that teaches student algebra via hands-on robotics. In his free time, Jason enjoys cooking, sailing, reading, paragliding, and sometimes pretending he's an artist.

Visualizing and Understanding Generative Adversarial Networks

The remarkable success of Generative Adversarial Networks in generating nearly photorealistic images leads to the question: how do they work? Are GAN just memorization machines, or do they learn semantic structures? What do these networks learn? I introduce the method of Network Dissection to test the semantics captured by neurons in the middle layers of a network, and show how recent state-of-the-art GANs learn a remarkable amount of structure. Even without any labels in the training data, neurons in a GAN trained to draw scenes will separately code for objects such as trees, furniture, and other meaningful objects. The causal effects such neurons are strong enough that we can add and remove objects and paint pictures directly by manipulating the neurons of a GAN. These methods provide insights about the a GAN's errors as well as the contextual relationships learned by a GAN. By cracking open the black box, we can see how deep networks learn meaningful structure, and we can gain understandable insights about a network’s inner workings.

David Bau is a PhD student at MIT CSAIL, advised by Professor Antonio Torralba. David previously worked at Google and Microsoft where he has contributed to several widely used products including Google Image Search and Microsoft Internet Explorer. David believes that complex systems should be built to be transparent, and his research focuses on the interpretability of deep networks.

Deep Learning Based Visual Scene and Object Recognition in Machine and Human Visual Systems

In this talk, we present deep learning solutions for three visual scene perception and object recognition problems. The goal is to investigate to which extent deep convolutional neural networks resemble the human visual system for scene perception and object recognition: (1) classification of scenes based on their global properties, (2) deploying multi-resolution technique for object recognition, and (3) evaluating the influence of the high-level context of scene grammar for object and scene recognition. The first problem proposes to drive global properties of a scene as high-level scene descriptions from deep features of convolutional neural networks in scene classification tasks. The second problem shows that fine-tuning the Faster-RCNN to multi-resolution data inspired by human multi-resolution visual system improves the network performance and robustness over a range of spatial frequencies. Finally, the third problem studies the effects of violating the high level scene syntactic and semantic rules on human eye-movement behavior and deep neural scene and object recognition networks.

Akram Bayat is a Research Assistant at the University of Massachusetts Boston where she also received her Ph.D. in computer Science at the Visual Attention Laboratory of the advised by Professor Marc Pomplun. Akram received both the master of Electrical Engineering and the master of Computer Science prior to joining Ph.D. program. She is currently working on how to apply human attentional mechanism to deep neural network for the scene and object recognition. Akram has conducted several projects on Human activity recognition and eye-movement based user classification. She is also interested in computer vision, machine learning, data mining, and human-user interface design.

Predicting What Drives Human Attention in Photographs, Visualizations, and Graphic Designs

Knowing where a person looks in an image can provide us with important clues about what captures their attention and what may eventually enter their memory. Aggregating the attention patterns of a group of people can help us make conclusions about the effectiveness of a design. Computational models of attention help guide image processing algorithms like automatic image resizing and thumbnailing, they can direct a model to compose more meaningful image captions, and they can be used to provide feedback within graphic design tools. In this talk, I will cover what we know about human attention and how we capture human attention and interest in images at a large data scale using novel crowdsourcing interfaces. I will then demonstrate how we use this data to build computational models of attention for photographs, visualizations, and graphic designs, along with the applications that these models make possible.

Zoya Bylinskii is a Research Scientist in the Creative Intelligence Lab at Adobe Research in Cambridge and an Associate of the Institute of Applied Computational Science at Harvard University. She received a Ph.D. and an M.Sc. in Computer Science from the Massachusetts Institute of Technology in 2018 and 2015, respectively, and an Hon. B.Sc. in Computer Science and Statistics from the University of Toronto in 2012. Zoya is a 2018 Rising Star in EECS, a 2016 Adobe Research Fellow, a 2014 NSERC Postgraduate Scholar, a 2013 Julie Payette Research Scholar, and a 2011 Anita Borg Scholar. Zoya works at the interface of human vision, computer vision, and human-computer interaction.

Childhood's End: Maturation of Deep Speech and Common Voice

We’ll talk about the blossoming of Deep Speech, an open deep learning based speech-to-text engine, and Common Voice, an open crowd-sourced speech corpora. We will cover recent Deep Speech advancements (streaming, small platform support, and product integrations) as well as Common Voice advancements (multi-language support, multi-language corpora, and profiles). Also we’ll give a overview of future plans and how to get involved.

Kelly Davis has many irons in the fire. He studied Mathematics and Physics at MIT, then went on to do graduate work in Superstring Theory/M-Theory. He then jumped ship, coding at a startup that eventually went public in the late 90's. When the bubble burst, he jumped back into an academic setting and joined the Max Planck Institute for Gravitational Physics where he worked on software systems used to help simulate black hole mergers. Jumping ship yet again, he went back into industry, writing 3D rendering software at Mental Images/NVIDIA. When that lost its charm, he founded a NLU at a startup, 42, that created a system, based off of IBM'S Watson, able to answer general knowledge questions. After a brief stint as the Director of Machine Learning at another Berlin startup, he joined Mozilla where he now leads the machine learning group.

Neural Network Force Field for Molecular Dynamics of Multi-Element Atomistic System

Neural network-based force field has recently emerged as a way to bypass expensive quantum mechanics calculation in molecular dynamics simulation, which enables us to study material properties and physical mechanisms at the atomistic level. Despite fundamental advances in rotation-invariant symmetry function “fingerprint” data representation, the derivative fingerprints required for the atomic force calculation significantly increases the training and execution runtime required in this approach. In this talk, we present an algorithm to bypass the need for fingerprint derivatives and perform direct atomic force prediction which significantly reduces the computation efforts required for training and executing the neural network force field for molecular dynamics simulations.

Jonathan Mailoa is currently a research engineer at Bosch Research and Technology Center, where he works on atomistic computational material science simulation of materials relevant for energy applications such as batteries and fuel cells. Prior to that, he completed his PhD in Electrical Engineering and Computer Science at MIT, developing novel tandem solar cell device architectures. He is currently interested in developing molecular dynamics force field based on machine learning methods.

Robot Learning via Human Adversarial Games

Much work in robotics has focused on “human-in-the-loop” learning techniques that improve the efficiency of the learning process. However, these algorithms have made the strong assumption of a cooperating human that assists the robot. In reality, people tend to act also in an adversarial manner towards deployed robotic systems. We show that this can in fact improve the robustness of the learned models by proposing a physical framework that leverages perturbations applied by a human adversary, guiding the robot towards more robust models. In a manipulation task, we show that grasping success improves significantly when the robot trains with a human adversary as compared to training in a self-supervised manner. This work opens a range of exciting potential applications in other domains as well, such as in autonomous driving.

Stefanos Nikolaidis is an Assistant Professor of Computer Science at the University of Southern California, where he directs the Interactive and Collaborative Autonomous Robotic Systems (ICAROS) Lab. Research in ICAROS spans the whole spectrum of human-robot interaction science: from distilling the fundamental mathematical principles that govern interactive behaviors, to developing approximation algorithms for deployed robotic systems and testing them "in the wild" with actual end users. Previously, Stefanos completed his PhD at Carnegie Mellon's Robotics Institute and received his MS from MIT. He has also a MEng from the University of Tokyo and a BS from the National Technical University of Athens. Stefanos has worked as a research associate at the University of Washington, as a research specialist at MIT and as a researcher at Square Enix in Tokyo. He has received a Best Enabling Technologies Paper Award from the IEEE/ACM International Conference on Human-Robot Interaction in 2015, a best paper nomination from the same conference in 2018 and was a best paper award finalist in the International Symposium on Robotics 2013.

Robot Learning through Motion and Interaction

Robots today are typically confined to operate in relatively simple, controlled environments. One reason for these limitations is that current methods for robotic perception and control tend to break down when faced with occlusions, viewpoint changes, poor lighting, unmodeled dynamics, and other challenging but common situations that occur when robots are placed in the real world. I argue that, in order to handle these variations, robots need to learn to understand how the world changes over time: how the environment can change as a result of the robot’s own actions or from the actions of other agents in the environment. I will show how we can apply this idea of understanding changes to a number of robotics problems, such as object tracking and safe robot learning. By learning how the environment can change over time, we can enable robots to operate in the complex, cluttered environments of our daily lives.

David Held is an assistant professor at Carnegie Mellon University in the Robotics Institute. His research focuses on robotic perception for autonomous driving and object manipulation. Prior to coming to CMU, David was a post-doctoral researcher at U.C. Berkeley, and he completed his Ph.D. in Computer Science at Stanford University where he developed methods for perception for autonomous vehicles. David also has a B.S. and M.S. in Mechanical Engineering at MIT. David is a recipient of the Google Faculty Research Award in 2017.

How Chick-fil-A uses AI to Spot Food Safety Trends in Social Media

Social media is an amazing way for companies to connect directly to their customers. At Chick-fil-A, it’s also one of many tools we use to help improve food safety at more than 2,000 locations around North America. To derive food safety insights from the hundreds of customer reviews received each day, we had to address a number of challenges inherent in analyzing social media data. After all, social media often contains broken grammar, mixed sentiments, and off-topic musings. To address these challenges, we developed a cloud-based service that uses artificial intelligence to help spot potential food safety issues from restaurant level customer review data. In this presentation, we’ll cover how the service works, using natural language processing (NLP) in combination with common serverless cloud computing tools. We’ll also cover how the data collected from social media can be used with other data sets (e.g. health department data) to help show possible correlations to better identify food safety trends.

Davis Addy is the Principal Technology Leader for Food Safety & Product Quality at Chick-fil-A. His team is responsible for internal Restaurant assessment and evaluation platforms in addition to developing digital solutions that leverage AI, advanced analytics, and connected devices in the Restaurant. Prior to joining Chick-fil-A in 2016, Davis spent eight years with General Electric (GE) supporting enterprise application rollouts across North America, Europe, and Asia. Davis holds a BS in Computer Engineering from the University of Florida and a MS in Information Systems Management from Georgia State University. He currently resides in Atlanta, GA with his wife Kim and their two daughters Brooklyn (6) and Hollyn (2).

The New Social Contract - Humanizing Artificial Intelligence

Artificial Intelligence is quickly becoming mainstream, engrained in the fabric of our lives, acting on our behalf – helping us get things done faster, more efficiently, giving us deeper insights, maybe even helping us be happier and healthier. AI is taking on tasks that were traditionally done by humans – from acting as our personal assistants and hiring our next co-worker, to driving our cars and assisting with our healthcare. But AI today has high IQ but no EQ, no emotional intelligence. We’re forging a new kind of partnership with technology. A new social contract that is based on mutual trust. In this talk, Dr. el Kaliouby will discuss the 5 tenets of this new social contract including how to build AI that has empathy, the ethical considerations of AI and the importance of guarding against data and algorithmic bias.

A pioneer in artificial emotional intelligence (Emotion AI), Rana el Kaliouby, PhD, is co-founder and CEO of MIT spinoff and category-defining company Affectiva. Rana led the innovation of the company’s award-winning technology, which uses deep learning and massive amounts of data to analyze complex and nuanced emotions and cognitive states from face and voice, for industries like automotive, market research, HR video recruitment, and mental health. Rana is now paving the way for Human Perception AI: software that can detect all things human, from nuanced human emotions and complex cognitive states, to behaviors, activities and the objects people use. Rana is one of few women leading a disruptive AI company. A Muslim-American and passionate advocate, she frequently speaks in press and on stage about innovation, women in technology, ethics in AI and leadership. Forbes recently included Rana in their list of America’s Top 50 Women in Tech, Fortune Magazine included her in their 2018 40 under 40 and she was named one of the three Global Business pioneers by Bloomberg in 2017. Rana is also a member of the World Economic Forum’s Young Global Leaders and a member of the Partnership on AI. Rana holds a BSc and MSc in Computer Science from the American University in Cairo, a PhD from the Computer Laboratory at the University of Cambridge and a Post Doctorate at MIT.