Visualizing Convolutional Neural Networks with Open-source Picasso
Toolkits for standard neural network visualizations exist, along with tools for monitoring the training process, but are often tied to the deep learning framework. Could a general, easy-to-setup tool for generating standard visualizations provide a sanity check on the learning process?
While it’s easier than ever to define and train deep neural networks (DNNs), understanding the learning process remains somewhat opaque. Monitoring the loss or classification error during training won’t always prevent your model from learning the wrong thing or learning a proxy for your intended classification task. To understand what we mean, consider this (possibly apocryphal) story [1]:
Wisely, the researchers had originally taken 200 photos, 100 photos of tanks and 100 photos of trees. They had used only 50 of each for the training set. The researchers ran the neural network on the remaining 100 photos, and without further training the neural network classified all remaining photos correctly. Success confirmed! The researchers handed the finished work to the Pentagon, which soon handed it back, complaining that in their own tests the neural network did no better than chance at discriminating photos. It turned out that in the researchers’ dataset, photos of camouflaged tanks had been taken on cloudy days, while photos of plain forest had been taken on sunny days. The neural network had learned to distinguish cloudy days from sunny days, instead of distinguishing camouflaged tanks from empty forest.
Definitely a real image of a modern tank from the training set. Source: Wikipedia
Regardless of the veracity of this tale, the point is familiar to machine learning researchers: training metrics don’t always tell the whole story. And the stakes are higher than ever before: for rising applications of deep learning like autonomous vehicles, these kinds of training errors can be deadly [2].
Fortunately, standard visualizations like partial occlusion [3] and saliency maps[4] provide a sanity check on the learning process. Toolkits [5] for standard neural network visualizations exist, along with tools for monitoring the training process. They’re often tied to the deep learning framework, if not model-specific. Could a general, easy-to-setup tool for generating standard visualizations have saved these researchers from detecting sunny days instead of tanks?
Picasso is a free open-source (Eclipse Public License) DNN visualization tool that gives you partial occlusion and saliency maps with minimal fuss. At Merantix, we work with a variety of neural network architectures; we developed Picasso to make it easy to see standard visualizations across our models in our various verticals: including applications in automotive, such as understanding when road segmentation or object detection fail; advertisement, such as understanding why certain creatives receive higher click-through rates; and medical imaging, such as analyzing what regions in a CT or X-ray image contain irregularities.
Picasso is a Flask application that glues together a deep-learning framework with a set of default and user-defined visualizations. You can use the built-in visualizations and easily add your own. Picasso was developed to work with checkpointed Keras and Tensorflow neural networks. If you want to try it out but don’t have any trained models, we provide Tensorflow and Keras MNIST checkpoints and a Keras VGG16 checkpoint for you.
An overview of the application flow with the default settings. The user has loaded a Keras model trained on the MNIST dataset, and generates a partial occlusion visualization on a couple handwritten digit images. See below for an in-depth explanation of occlusion maps.
At Merantix, we are particularly interested in convolutional neural networks (CNNs) that take images as inputs and do classification. We developed Picasso with these parameters in mind. However, the framework is flexible enough to use on all kinds of models.
