Using GPUs for Deep Learning
As AI technology advances and more organizations are implementing machine learning operations (MLOps), companies are looking for ways to speed up processes. This is especially true for organizations working with deep learning (DL) methods which can be incredibly time-intensive to run. This process can be sped up using graphical processing units (GPUs), either on-premises or in the cloud.
GPUs are microprocessors specially designed to perform graphics-oriented tasks. These units utilize parallel processing of tasks and can be optimized to increase performance in artificial intelligence and deep learning processes.
The GPU is a powerful tool for speeding up a data pipeline in combination with a deep neural network. There are principally two reasons to integrate a GPU into data processing. The first is that a deep neural network (DNN) inference runs up to 10 times faster on a GPU compared to a central processing unit (CPU) that retails for the same price. The second reason is that it allows the CPU to do more work simultaneously and reduces the overall network load by taking some of the workloads off of the CPU.
The most common bottleneck with this approach is data transfer between CPU RAM and GPU DRAM. Therefore there are two main focuses when building a data science pipeline architecture. The first is to reduce the number of data transference transactions by aggregating several samples (images) into a batch. The second is to reduce the size of a specific sample by filtering data before transferring.
The training and implementation of DL models require deep neural networks (DNN) and datasets with hundreds of thousands of data points. These DNNs need significant resources, including memory, storage, and processing power. While central processing units (CPUs) can provide this power, graphical processing units (GPUs) can substantially expedite the process.
There are three significant advantages a dedicated GPU brings to a system implementing a DNN:
GPU calculations are high-speed compared to a CPU’s, and GPUs are a must-have component for many AI applications. But in some cases, this processing power is overkill, and system architects should consider utilizing CPUs to minimize strain on their budget.
Here we also need to say a few words about the cost of GPU calculations. As mentioned before, GPUs perform their calculations significantly faster than CPUs. Still, the combined time costs of transferring and processing the data may outweigh the speed you gained by switching to GPU.
So, for example, at the beginning of development, i.e., while developing proof of concept (POC) or a minimum viable product (MVP), one could use CPUs for preliminary activities like development, testing, and staging servers. In cases where users are okay with longer response times, production servers can use the CPU for DNN training and inference, but ideally only for tasks with a short overall duration.
When using GPUs for on-premises implementations, multiple vendor options are available. Two of the most popular choices are NVIDIA and AMD.
NVIDIA is a popular option because of the first-party libraries it provides, known as the CUDA toolkit. These libraries enable the easy establishment of deep learning processes and form the base of a strong machine learning community using NVIDIA products. This is evidenced by the widespread support that many DL libraries and frameworks provide for NVIDIA hardware.
In addition to supporting their GPU hardware, NVIDIA also offers libraries supporting popular DL frameworks, including PyTorch. The Apex library is particularly useful and includes several fast, fused optimizers such as FusedAdam.
The downside of a system integrating NVIDIA GPUs is that the company has recently placed restrictions on the configurations in which CUDA can be used. These restrictions require that the libraries only be used with the Tesla line of GPUs and cannot be used with the less expensive RTX or GTX hardware lines.
