Pokémon Go has been in public release for slightly over a week now.  I had an opportunity to spend some time playing it in a rather dense area (literally dozens of players around a cluster of PokéStops every evening), and since I’ve been an Ingress player for over three years, have played plenty of Ingress, and have some background with large scale server systems, I decided to share some of my thoughts.

If you were really, really looking forward to some details on the inside of my Solar Shed (an off grid office I built out of a Tuff-Shed), here’s a teaser.  But that’s pushed back to next week.  Sorry.

Otherwise, read on for thoughts about scaling, servers, social aspects, and how everything relates to Ingress.

First: Ingress

The sad, sad, neglected game of Ingress…  I deliberately left some portals in a very popular Pokémon Go area sitting with a single L1 resonator.  They’ve lasted over 36 hours as literally hundreds of Pokémon Go players interacted with the PokéStops.

And if you have no idea what the above means, keep reading, because it’s relevant.

Niantic (the company who released Pokémon Go) isn’t a new company.  Their first game was called Ingress, and I’ve been playing it for over three years.

Ingress is more or less geocaching with two teams (and sound effects).  The points of interest are called portals, and players were allowed to submit new and interesting locations as portals (though submitting new portals was removed a while back).

To capture a portal, you need to attach resonators to it.  If the other team owns a portal, you need to first destroy their resonators, then deploy your own resonators on it to capture it for your team.

You can make links between portals, and build triangular fields, but that’s not particularly relevant here.

What is relevant is that the portals of Ingress are now the PokéStops and Gyms of Pokémon Go.  They’re often interesting pieces of art, unique local businesses, parks, or memorial benches.  Not all Ingress portals are in Pokémon Go, but every Pokémon point of interest is an Ingress portal.

Which, by the way, means that the Ingress Intel Map (https://www.ingress.com/intel/) is also useful for finding local clusters of PokéStops in your area.  And it also means you can switch between Ingress and Pokémon Go to play two games for the price of one!

Global Consistency: Ingress vs Pokémon Go

Despite coming from the same company, and using the same points of interest, there is a huge difference between Ingress and Pokémon Go.  Ingress is a globally consistent game.  Pokémon Go is not.

Before I moved out to my few acres in the country, I worked for a company that handled a little bit of data.  I was, among other things, a Site Reliability Engineer.  Discussions about consistency, scaling, communication, and ramping up over time came up quite regularly - especially since many of my coworkers also played Ingress during the rough early years.

Globally consistent data is hard.  If you want all users to see one and only one view of the game world, it’s very hard to scale this.  Storage is only so fast, and you will hit limits early and often as you try to scale a globally consistent set of data.

Ingress Consistency

Ingress has millions of portals, millions of players, and tens or hundreds of millions of resonators to keep track of at any given time.

Since Ingress is based “in the world,” as augmented reality, there’s no way to have multiple servers to play on like many other world-based games (World of Warcraft being just one example of this).  Every single player (ideally) sees the exact same state of the world, and this state can change very rapidly if you have lots of players from both teams battling over a portal.  This is just hard to do at scale.

Early on, Ingress server issues were common.  Lag was the most frequent issue - you’d try to perform an action, and it would just sit around doing nothing for a few seconds or longer, then take the action.  If you were on a good cell connection, it wasn’t your end - it was the servers, and if I had to guess, I’d wager most of the delays were waiting for the globally consistent storage backend involved to get through it’s queue of requests and commit whatever change needed to happen.

Eventually, Niantic worked out the details, and other than weak cell connections, significant lag in Ingress is rather rare these days.  But I doubt Ingress could take a 10x increase in players without running into more issues.

Pokémon Go Consistency

Pokémon Go has a very, very different level of global consistency - which is to say, it really doesn’t have it.

The world map is the same for everyone, and gyms have global state that everyone sees, and it seems that a new Pokémon will usually pop up for everyone in the area (if you have, say, a few dozen people around a cluster of PokéStops, they’ll all see the same type pop up), but everyone can capture it, and each captured one will have different skills/CP/HP.

What this means is that there is very little global state to deal with.  Even the gyms only have a tiny bit of global state - the battles are “you vs bots,” and updating the owners/prestige is trivial compared to keeping track of the hit points of every resonator, on every portal, and making sure everyone sees this.

Sharding

Since there’s so little global state to keep track of, it’s very easy to implement a technique known as “sharding” for a game like this.  This means that one has a variety of different servers, and they each keep track of data for different players.  All of my stuff can be on Shard 176, whereas the person playing next to me can be on Shard 793.  Nothing about the general gameplay requires any communication between those.

There are still global events (“Hey, a Pokémon just showed up!”), but these are infrequent (relatively speaking), and appear to be a basic broadcast to all servers, with the details worked out by each server with regards to skills/CP/HP.

This lack of global consistency means that it’s relatively easy to scale things up - just add more servers for more players.  Sharding based on geographic areas (which is likely how Ingress is set up) is much harder, as there will need to be processes for splitting areas, talking to multiple servers if you’re on the boundary, and you can still overwhelm a shard with a lot of players in one area.

Ramp Up and Player Count

Ingress, much like GMail, started out with an invite based system.  You couldn’t just sign up and play - you had to either know someone who could send you an invite, or sign up in the general invite queue from the website.

This approach offers strict controls on how many new people can play - and it was used that way.  Players would get one or two invites every few weeks, sometimes after a wave of invites went out the servers would slow down, Niantic would fix whatever bottleneck they just found, and after things were stable again, they’d give out another wave of invites to add more users.

Eventually, after a lot of work on scaling, Ingress went wide open.  There’s still an invite system, but it’s meaningless other than giving players a bit more gear when they start.  Anyone can sign up and play, and it seems that Niantic gave all players 90 invites to use.

Pokémon Go has none of that.  Anyone who wants to can sign up and start playing - and insane numbers of people are doing exactly that.

I don’t have enough data to estimate actual player numbers, but I can make some ballpark comparisons based on review counts.

As of around 4PM on Saturday, July 16, 2016, here are the review counts for the apps

Ingress has been public for nearly 3 years.

Pokémon Go has been public for 10 days.

Ingress/iOS: 2,308 Reviews

Pokémon Go/iOS: 87,805 Reviews

Ingress/Android: 340,558 Reviews

Pokémon Go/Android: 794,616 Reviews

On the Play Store, both Ingress and Pokémon Go are in the “10M-50M installs” range - one after three years, one after a week.

Based on the review counts alone, Pokémon Go has ~40x the iOS players, and ~2.5x the Android players as Ingress.  However, that’s deceptive, as Ingress has three years worth of reviews, and Pokémon Go has 10 days of reviews.

I’m comfortable estimating that Pokémon Go has somewhere around 50x-100x the current active daily player count as Ingress.  It’s got more active daily users right now than Twitter, Tinder, and Facebook.

Ingress took four years to scale to the current point.  Pokémon Go took a week.  That’s insane.

Saturday Server Outage

So, if it’s possible to shard it and scale quickly, what caused the Saturday outage on July 16th?

Well, they did launch it in 26 countries over in Europe…

I have no idea what actually failed.  It was quite slow very early in the morning, quit entirely by mid-morning, and showed again up late afternoon.

I’d guess it was either an issue with login/authentication, or an issue with the frontend load balancers (that direct users to their particular server).  Either one could cause the problem, but based on the very slow performance in the morning before it quit, I’d wager it was the load balancers not being able to keep up with traffic and falling over.

As is the case with any major service being down, some small hacker group or another has claimed responsibility, but I see no reason to believe them without further proof.

Social Aspects

I’m always interested in the “social and cultural” aspects of really weird games - and Niantic makes weird games!  I’m also familiar with a lot of Ingress culture.

Pokémon Go has no way to interact with other players directly in-game.  Despite this, or, in my opinion, because of this, it seems to be much more social than Ingress, and I expect it will remain so.

Social interaction in Ingress

Ingress often has social interaction, but it’s almost always limited to people on “your team,” as your team is incredibly core to the game.  Since play generally involves taking over the other team’s portals, and it requires multiple people to make a “good” high level portal, you can’t really play socially with people from the other team.

Also, in Ingress, there are many ways to grief people and just be a general jerk.  Global chat makes it easy to harass people remotely, and because you can drop items (into a global game state that everyone sees), you can do things like write words, or draw a “Gentleman’s Sausage” in front of the house or workplace of someone you don’t like.  I believe bot scripts existed at one point to do this automatically, without having to actually go to the location.

So social interaction is somewhat limited outside a small group you may play with regularly, and interaction with people on the other team generally ranges from annoyance (“You’re here.  Darn.”) to various levels of hostility.

Social interaction in Pokémon Go

So far, my experience with social interaction in Pokémon Go has been radically different from Ingress.  First, there are more people.  Second, the game encourages hanging around areas with lots of Pokémon longer - you’re not just interacting with the stops, you’re collecting Pokémon that wander by.  And, everyone can see and collect their own copy of interesting Pokémon that wander past.

This leads to a much more social environment.  Large numbers of people out, sitting around, chatting, playing, and capturing.  It’s very different from Ingress, where this rarely happens.

Pokémon Go eliminates nearly all of the ways to interact directly with another player.  There’s no global chat.  You can’t drop things in patterns.  You really cannot interact directly with another player in any way in-game - even the gyms are “you vs bot” interactions.

Another major difference is that the team you’re on is a minor attribute, not terribly important for most gameplay.  You can sit around a PokéStop with people from all three colors, farming the PokéStop, luring rare Pokémon in, everyone can capture their own copy of it, and the fact that you’re all on different teams doesn’t matter a bit.  It comes into play for gyms, but even around them, “player vs machine” seems to reduce the hostility a good bit - the gyms I’ve seen with clusters of people clearly have different teams, but everyone is having fun, and it’s not nearly as aggressive as fighting over an Ingress portal.  That you can fight your own team for training means that the gym color is almost entirely irrelevant to gameplay as well (for most players).

With the removal of global state and the de-emphasis of your team, Niantic has made this a much more friendly game, and eliminated pretty much every possible way to grief another player in the game world.  Which is exactly what they needed to do to make a younger-player friendly game.

The “You must walk to hatch these eggs” mechanic is also very well done to encourage actual walking.

Final Thoughts

I’m really, really impressed with what Niantic has done.

They’ve taken the essence of Ingress (going out to places with an augmented reality game) and turned it into something that is more social, easier to scale, and still amazing amounts of fun.

Have fun, and don’t forget to look up on occasion!