[STEAM] Why Google Gemini’s Pokémon success isn’t all it’s cracked up to be

[Steam 0]

Earlier this year, we took a look at how and why

This is the hidden content, please

• Sign In
• or
• Sign Up

(a game, let's remember, designed for young children). But while Claude 3.7 is

This is the hidden content, please

• Sign In
• or
• Sign Up

weeks later, a similar

This is the hidden content, please

• Sign In
• or
• Sign Up

managed to finally complete Pokémon Blue this weekend across over 106,000 in-game actions, earning accolades from followers,

This is the hidden content, please

• Sign In
• or
• Sign Up

.

Before you start using this achievement as a way to compare the relative performance of these two AI models—or even the advancement of LLM capabilities over time—there are some important caveats to keep in mind. As it happens, Gemini needed some fairly significant outside help on its path to eventual Pokémon victory.

Strap in to the agent harness

Gemini Plays Pokémon developer JoelZ (who's unaffiliated with

This is the hidden content, please

• Sign In
• or
• Sign Up

) will be the first to tell you that Pokémon is ill-suited as a reliable benchmark for LLM models. As he

This is the hidden content, please

• Sign In
• or
• Sign Up

, "please don't consider this a benchmark for how well an LLM can play Pokémon. You can't really make direct comparisons—Gemini and Claude have different tools and receive different information. ... Claude's framework has many shortcomings so I wanted to see how far Gemini could get if it were given the right tools."

This is the hidden content, please

• Sign In
• or
• Sign Up

This is the hidden content, please

• Sign In
• or
• Sign Up

This is the hidden content, please

• Sign In
• or
• Sign Up