IBM Boosts Deep Learning Accuracy on Memristive Chips
IBM researchers have taken another step towards making in-memory computing based on phase change (PCM) memory devices a reality. Papers in Nature and Frontiers in Neuroscience this month present IBM work using a mixed-signal architecture with PCM devices to run deep neural networks at high accuracy. IBM demonstrated a novel approach to coping with natural variations present in PCM (memristive) devices to improve accuracy by training models to accommodate noise.
The broad goal of these “neuromorphic” processing approaches is to ‘mimic’ the low-power processing techniques used by biological systems and eliminate some of the data movement required by traditional compute architectures. IBM researcher Manuel LeGallo, an author on both papers, described the work in a blog last week.
“While there has been significant progress in the development of hardware-accelerator architectures for inference, many of the existing set-ups physically split the memory and processing units. This means that DNN models are typically stored in off-chip memory, and that computational tasks require a constant shuffling of data between the memory and computing units – a process that slows down computation and limits the maximum achievable energy efficiency,” wrote LeGallo.
“Our research, featured in Nature Communications, exploits in-memory computing methods using resistance-based (memristive) storage devices as a promising non-von Neumann approach for developing hardware that can efficiently support DNN inference models. Specifically, we propose an architecture based on phase-change memory (PCM) that, like the human brain, has no separate compartments to store and compute data, and therefore consumes significantly less energy.”
A major challenge in using PCM devices is achieving and maintaining computational accuracy. PCM technology is analog in nature, and computational precision is limited due to device variability as well as read and write conductance noise. IBM was seeking a way to train the neural networks so that transferring the digitally trained weights to the analog resistive memory devices would not result in significant loss of accuracy.
Wrote LeGallo, “Our approach was to explore injecting noise to the synaptic weights during the training of DNNs in software as a generic method to improve the network resilience against analog in-memory computing hardware non-idealities. Our assumption was that injecting noise comparable to the device noise during the training of DNNs would improve the robustness of the models.”
It turned out they were correct. “Training ResNet-type networks this way resulted in no considerable accuracy loss when transferring weights to PCM devices. We achieved an accuracy of 93.7% on the CIFAR-10 dataset and a top-1 accuracy on the ImageNet benchmark of 71.6% after mapping the trained weights to analog PCM synapses. And after programing the trained weights of ResNet-32 on 723,444 PCM devices of a prototype chip, the accuracy computed from the measured hardware,” weights stayed above 92.6% over a period of 1 day. To the best of our knowledge, this is the highest accuracy experimentally reported to-date on the CIFAR-10 dataset by any analog resistive memory hardware.
