How Google makes custom cloud chips that power Apple AI and Gemini

[Pelican Press 0]

This is the hidden content, please

• Sign In
• or
• Sign Up

How

This is the hidden content, please

• Sign In
• or
• Sign Up

makes custom cloud chips that power Apple AI and Gemini

Inside a sprawling lab at

This is the hidden content, please

• Sign In
• or
• Sign Up

headquarters in Mountain View, California, hundreds of server racks hum across several aisles, performing tasks far less ubiquitous than running the world’s dominant search engine or executing workloads for

This is the hidden content, please

• Sign In
• or
• Sign Up

Cloud’s millions of customers.

Instead, they’re running tests on

This is the hidden content, please

• Sign In
• or
• Sign Up

’s own microchips, called Tensor Processing Units, or TPUs.

Originally trained for internal workloads,

This is the hidden content, please

• Sign In
• or
• Sign Up

’s TPUs have been available to cloud customers since 2018. In July, Apple revealed it uses TPUs to train AI models underpinning Apple Intelligence.

This is the hidden content, please

• Sign In
• or
• Sign Up

also relies on TPUs to train and run its Gemini chatbot.

“The world sort of has this fundamental belief that all AI, large language models, are being trained on Nvidia, and of course Nvidia has the lion’s share of training volume. But

This is the hidden content, please

• Sign In
• or
• Sign Up

took its own path here,” said Futurum Group CEO Daniel Newman. He’s been covering

This is the hidden content, please

• Sign In
• or
• Sign Up

’s custom cloud chips since they launched in 2015.

This is the hidden content, please

• Sign In
• or
• Sign Up

was the first cloud provider to make custom AI chips. Three years later,

This is the hidden content, please

• Sign In
• or
• Sign Up

Web Services announced its first cloud AI chip, Inferentia.

This is the hidden content, please

• Sign In
• or
• Sign Up

‘s first custom AI chip, Maia, wasn’t announced until the end of 2023. 

But being first in AI chips hasn’t translated to a top spot in the overall rat race of generative AI.

This is the hidden content, please

• Sign In
• or
• Sign Up

’s faced criticism for botched product releases, and Gemini came out more than a year after OpenAI’s ChatGPT.

This is the hidden content, please

• Sign In
• or
• Sign Up

Cloud, however, has gained momentum due in part to AI offerings.

This is the hidden content, please

• Sign In
• or
• Sign Up

parent company Alphabet reported cloud revenue rose 29% in the most recent quarter, surpassing $10 billion in quarterly revenues for the first time.

“The AI cloud era has completely reordered the way companies are seen, and this silicon differentiation, the TPU itself, may be one of the biggest reasons that

This is the hidden content, please

• Sign In
• or
• Sign Up

went from the third cloud to being seen truly on parity, and in some eyes, maybe even ahead of the other two clouds for its AI prowess,” Newman said.

‘A simple but powerful thought experiment’

In July, CNBC got the first on-camera tour of

This is the hidden content, please

• Sign In
• or
• Sign Up

’s chip lab and sat down with the head of custom cloud chips, Amin Vahdat. He was already at

This is the hidden content, please

• Sign In
• or
• Sign Up

when it first toyed with the idea of making chips in 2014. 

Amin Vahdat, VP of Machine Learning, Systems and Cloud AI at

This is the hidden content, please

• Sign In
• or
• Sign Up

, holds up TPU Version 4 at

This is the hidden content, please

• Sign In
• or
• Sign Up

headquarters in Mountain View, California, on July 23, 2024.

Marc Ganley

“It all started with a simple but powerful thought experiment,” Vahdat said. “A number of leads at the company asked the question: What would happen if

This is the hidden content, please

• Sign In
• or
• Sign Up

users wanted to interact with

This is the hidden content, please

• Sign In
• or
• Sign Up

via voice for just 30 seconds a day? And how much compute power would we need to support our users?”

The group determined

This is the hidden content, please

• Sign In
• or
• Sign Up

would need to

This is the hidden content, please

• Sign In
• or
• Sign Up

. So they looked for a better solution.

“We realized that we could build custom hardware, not general purpose hardware, but custom hardware — Tensor Processing Units in this case — to support that much, much more efficiently. In fact, a factor of 100 more efficiently than it would have been otherwise,” Vahdat said.

This is the hidden content, please

• Sign In
• or
• Sign Up

data centers still rely on general-purpose central processing units, or CPUs, and Nvidia’s graphics processing units, or GPUs.

This is the hidden content, please

• Sign In
• or
• Sign Up

’s TPUs are a different type of chip called an application-specific integrated circuit, or ASIC, which are custom-built for specific purposes. The TPU is focused on AI.

This is the hidden content, please

• Sign In
• or
• Sign Up

makes another ASIC focused on video called a Video Coding Unit. 

This is the hidden content, please

• Sign In
• or
• Sign Up

also makes custom chips for its devices, similar to Apple’s custom silicon strategy. The Tensor G4 powers

This is the hidden content, please

• Sign In
• or
• Sign Up

’s new AI-enabled Pixel 9, and its new A1 chip powers Pixel Buds Pro 2. 

The TPU, however, is what set

This is the hidden content, please

• Sign In
• or
• Sign Up

apart. It was the first of its kind when it launched in 2015.

This is the hidden content, please

• Sign In
• or
• Sign Up

TPUs still dominate among custom cloud AI accelerators, with 58% of the market share,

This is the hidden content, please

• Sign In
• or
• Sign Up

.

This is the hidden content, please

• Sign In
• or
• Sign Up

coined the term based on the algebraic term “tensor,” referring to the large-scale matrix multiplications that happen rapidly for advanced AI applications.

With the second TPU release in 2018,

This is the hidden content, please

• Sign In
• or
• Sign Up

expanded the focus from inference to training and made them available for its cloud customers to run workloads, alongside market-leading chips such as Nvidia’s GPUs.

“If you’re using GPUs, they’re more programmable, they’re more flexible. But they’ve been in tight supply,” said Stacy Rasgon, senior analyst covering semiconductors at Bernstein Research.

The AI ***** has sent Nvidia’s stock through the roof, catapulting the chipmaker to a $3 trillion market cap in June, surpassing Alphabet and jockeying with Apple and

This is the hidden content, please

• Sign In
• or
• Sign Up

for position as the world’s most valuable public company.

“Being candid, these specialty AI accelerators aren’t nearly as flexible or as powerful as Nvidia’s platform, and that is what the market is also waiting to see: Can anyone play in that space?” Newman said.

Now that we know Apple’s using

This is the hidden content, please

• Sign In
• or
• Sign Up

’s TPUs to train its AI models, the real test will come as those full AI features roll out on iPhones and Macs

This is the hidden content, please

• Sign In
• or
• Sign Up

.

Broadcom and TSMC

It’s no small feat to develop alternatives to Nvidia’s AI engines.

This is the hidden content, please

• Sign In
• or
• Sign Up

’s sixth generation TPU, called Trillium, is set to come out later this year.

This is the hidden content, please

• Sign In
• or
• Sign Up

showed CNBC the sixth version of its TPU, Trillium, in Mountain View, California, on July 23, 2024. Trillium is set to come out later in 2024.

Marc Ganley

“It’s expensive. You need a lot of scale,” Rasgon said. “And so it’s not something that everybody can do. But these hyperscalers, they’ve got the scale and the money and the resources to go down that path.”

The process is so complex and costly that even the hyperscalers can’t do it alone. Since the first TPU,

This is the hidden content, please

• Sign In
• or
• Sign Up

’s partnered with Broadcom, a chip developer that also helps Meta design its AI chips. Broadcom

This is the hidden content, please

• Sign In
• or
• Sign Up

to make these partnerships happen.  

“AI chips — they’re very complex. There’s lots of things on there. So

This is the hidden content, please

• Sign In
• or
• Sign Up

brings the compute,” Rasgon said. “Broadcom does all the peripheral stuff. They do the I/O and the

This is the hidden content, please

• Sign In
• or
• Sign Up

, all of the different pieces that go around that compute. They also do the packaging.”

Then the final design is sent off for manufacturing at a fabrication plant, or fab — primarily those owned by the world’s largest chipmaker, Taiwan Semiconductor Manufacturing Company, which makes 92% of the world’s most advanced semiconductors.

When asked if

This is the hidden content, please

• Sign In
• or
• Sign Up

has any safeguards in place should the worst happen in the geopolitical sphere between China and Taiwan, Vahdat said, “It’s certainly something that we prepare for and we think about as well, but we’re hopeful that actually it’s not something that we’re going to have to trigger.”

Protecting against those risks is the primary reason the White House is handing out $52 billion in CHIPS Act funding to companies building fabs in the U.S. — with the

This is the hidden content, please

• Sign In
• or
• Sign Up

going to Intel, TSMC, and Samsung to date.

Processors and power

This is the hidden content, please

• Sign In
• or
• Sign Up

showed CNBC its new Axion CPU,

Marc Ganley

“Now we’re able to bring in that last piece of the puzzle, the CPU,” Vahdat said. “And so a lot of our internal services, whether it’s

This is the hidden content, please

• Sign In
• or
• Sign Up

, whether it’s

This is the hidden content, please

• Sign In
• or
• Sign Up

,

This is the hidden content, please

• Sign In
• or
• Sign Up

advertising and more are running on Axion.”

This is the hidden content, please

• Sign In
• or
• Sign Up

is late to the CPU game.

This is the hidden content, please

• Sign In
• or
• Sign Up

launched its Graviton processor in 2018. Alibaba launched its server chip in 2021.

This is the hidden content, please

• Sign In
• or
• Sign Up

announced its CPU in November.

When asked why

This is the hidden content, please

• Sign In
• or
• Sign Up

didn’t make a CPU sooner, Vahdat said, “Our focus has been on where we can deliver the most value for our customers, and there it has been starting with the TPU, our video coding units, our networking. We really thought that the time was now.”

All these processors from non-chipmakers, including

This is the hidden content, please

• Sign In
• or
• Sign Up

’s, are made possible by Arm chip architecture — a more customizable, power-efficient alternative that’s gaining traction over the traditional x86 model from Intel and AMD. Power efficiency is crucial because, by 2027, AI servers are projected to use up

This is the hidden content, please

• Sign In
• or
• Sign Up

like Argentina.

This is the hidden content, please

• Sign In
• or
• Sign Up

’s latest environmental

This is the hidden content, please

• Sign In
• or
• Sign Up

showed emissions rose nearly 50% from 2019 to 2023 partly due to data center growth for powering AI.

“Without having the efficiency of these chips, the numbers could have wound up in a very different place,” Vahdat said. “We remain committed to actually driving these numbers in terms of carbon emissions from our infrastructure, 24/7, driving it toward zero.”

It takes a massive amount of water to cool the servers that train and run AI. That’s why

This is the hidden content, please

• Sign In
• or
• Sign Up

’s third-generation TPU started using direct-to-chip cooling, which uses far less water. That’s also how

This is the hidden content, please

• Sign In
• or
• Sign Up

GPUs.

Despite challenges, from geopolitics to power and water,

This is the hidden content, please

• Sign In
• or
• Sign Up

is committed to its generative AI tools and making its own chips. 

“I’ve never seen anything like this and no sign of it slowing down quite yet,” Vahdat said. “And hardware is going to play a really important part there.”

This is the hidden content, please

• Sign In
• or
• Sign Up

#

This is the hidden content, please

• Sign In
• or
• Sign Up

#custom #cloud #chips #power #Apple #Gemini

This is the hidden content, please

• Sign In
• or
• Sign Up

This is the hidden content, please

• Sign In
• or
• Sign Up