No cloud required: Why AI’s future is at the edge

For all the promise and peril of artificial intelligence, there’s one big obstacle to its seemingly relentless march: The algorithms for running AI applications have been so big and complex that they’ve required processing on powerful machines in the cloud and data centers, making a wide swath of applications less useful on smartphones and other “edge” devices.

Now, that concern is quickly melting away, thanks to a series of breakthroughs in recent months in software, hardware and energy technologies. That’s likely to drive AI-driven products and services even further away from a dependence on powerful cloud-computing services and enable them to move into every part of our lives — even inside our bodies.

By 2022, 80% of smartphones shipped will have AI capabilities on the device itself, up from 10% in 2017, according to market researcher Gartner Inc. And by 2023, that will add up to some 1.2 billion shipments of devices with on-device AI computing capabilities, up from 79 million in 2017, according to ABI Research .

A lot of startups and their backers smell a big opportunity. According to Jeff Bier, founder of the Embedded Vision Alliance, which held a conference this past week in Silicon Valley, investors have plowed some $1.5 billion into new AI chip startups in the past three years — more than was invested in all chip startups in the previous three years. Market researcher Yole

Développement forecasts that AI application processors will enjoy a 46% compound annual growth rate through 2023, when nearly all smartphones will have them, from fewer than 20% today.

And it’s not just startups. “Within the next two years, virtually every processor vendor will be offering some kind of competitive platform for AI,” Tom Hackett, principal analyst at IHS Markit, said at the alliance’s Embedded Vision Summit. “We are now seeing a next-generation opportunity.”

Those chips are finding their way into many more devices beyond smartphones. They’re also being used in millions of “internet of things” devices such as robots, drones, cars, cameras and wearables. Among the 75 or so companies developing machine learning chips, for instance, is Israel’s Hailo, which raised a $21 million funding round in January. In mid-May it released a processor that’s tuned for deep learning, a branch of machine learning responsible for recent breakthroughs in voice and image recognition.

More compact and capable software is paving the way for AI at the edge as well. Google LLC, for instance debuted its TensorFlow Lite machine learning library for mobile devices in late 2017, enabling the potential for smart cameras to can identify wildlife or imaging devices to can make medical diagnoses even where there’s no internet connection. Some 2 billion mobile now have TensorFlow Lite deployed on them, Google staff research engineer Pete Warden said at a keynote presentation at the Embedded Vision Summit.

And in March, Google rolled out an on-device speech recognizer to power speech input in Gboard, Google’s virtual keyboard app. The automatic speech recognition transcription algorithm is now down to 80 megabytes so it can run on the Arm Ltd. A-series chip inside a typical Pixel phone, and that means it works offline so there’s no network latency or spottiness.
Not least, rapidly rising privacy concerns about data traversing the cloud means there’s also a regulatory reason to avoid moving data off the devices.
“Virtually all the machine learning processing will be done on the device,” said Bier, who’s also co-founder and president of Berkeley Design Technology Inc., which provides analysis and engineering services for embedded digital signal processing technology. And there will be a whole lot of devices: Warden cited an estimate of 250 billion active embedded devices in the world today, and that number is growing 20% a year.