OpenAI Five Benchmark Test results

The OpenAI Five team ran a benchmark match late last night that pitted their bots against a human team (Blitz, Cap, Fogged, MoonMeander, and Merlini). The human team are in the 99.95% percentile of Dota 2 players. This benchmark test against the best players in the world has been a long time coming, and winning against the human team would be a critical milestone for the OpenAI team. The match was casted on Twitch by Purge and ODPixel. We detailed how OpenAI works in an article a while back, so if you want to know more background please read this article.

OpenAI Five Benchmark Test details

The 5v5 Bo5 series against OpenAI Five was played with a few limitations:

• Pool of 18 heroes (Axe, Crystal Maiden, Death Prophet, Earthshaker, Gyrocopter, Lich, Lion, Necrophos, Queen of Pain, Razor, Riki, Shadow Fiend, Slark, Sniper, Sven, Tidehunter, Viper, or Witch Doctor)
• No summons/illusions
• 5 invulnerable couriers, no exploiting them by scouting or tanking
• No Scan
• No Divine Rapier or Bottle

How’d it go?

The OpenAI Five bots seem to be in a really strong place within the limitations outlined above. They dominated all the matches they played against the audience yesterday. By the time they came up against the human team of Blitz, Cap, Fogged, MoonMeander, and Merlini the bots had not lost a single match on the day. In all the matches against the human team the bots exhibited good basic mechanical skills.

What really makes the bots stand out is how aggressive they are (they’ll dive very early for kills), how good they are at using their abilities, and their rotations. The way they seem to dominate matches is to kill human players early, and just keep putting pressure so humans can’t farm efficiently. The bots are also great as grouping up as four or five and pushing towers or taking teamfights. This is where the bots really excel.

OpenAI Five team being interviewed by ODPixel.

Hello, Skynet?

However, Skynet (Eds: He means OpenAI Five) isn’t without weaknesses. Pulling is really effective against bots, and when it happens the bots don’t seem to know how to react. Bots don’t use complicated items well, but buy a lot of consumables (five couriers per team is advantageous for the bots). This means they’re almost always poor in the early stages, so if they don’t get kills on the map they could fall behind. However, even in the later stages of the game the bots still seem to prioritize consumables. Invisibility (and items that grant invisibility) also seem to heavily counter the way OpenAI Five bots play, which of course the human team attempted to take advantage of.

OpenAI Five bots are just so much better at basic mechanics (last hitting and using abilities) and tend to trade against the human team’s heroes so much more efficiently. For instance on multiple occasions the human team’s Earthshaker would blink in to initiate, but would be instantly hexed by the OpenAI Five Lion. However when the bots are forced out of their comfort zones they completely fall apart. In game 3 the audience and Twitch chat selected the heroes OpenAI had to use, and the bots suddenly weren’t at all confident.

Summary

It seems like the bots have learned very specific strategies around the heroes in the pool they have access to, but when they’re unable to get the heroes they want (or execute the strategy they’ve developed) they’re just terrible. The human players are significantly more adaptable, as human learning has access to more complicated interactions than bots randomly generate on their own. Bots struggle to use complicated items (and complicated interactions/strategies also seem beyond them). Thus in game 3 the bots tried their same aggressive strategies, but without optimal heroes. This time the humans were able to turn their fortunes around, winning the match.

The bots do some really interesting things, but there are certainly gaps in their knowledge. Bots just don’t get into complicated situations like humans tend to do. Their ability to innovate is certainly constrained, which is a significant cap on OpenAI Five. All in all the end result in the Bo5 series yesterday went 2-1 in favor of the OpenAI Five bots. You can watch the VODs here. This includes interviews with the OpenAI Five team, players, and some really fun matches to watch. I highly recommend paying tribute to your new AI overlords.

What about Ti8?

It’s my understanding that OpenAI Five will once again compete against pro players at Ti8. I am really excited to see how pro players manage against OpenAI Five, but I think that if there are no limitations (i.e. a normal Dota 2 Captain’s Mode match against pros) then the bots will struggle. Although it would be interesting to see how pros adapt to playing against the bots under limitations as well. I don’t think OpenAI Five would come out on top, though. Whichever way it goes, I’m excited to see how the matches shape up.