Thread: Crazy day for servers

Completely unrelated to Java, but still a good engineering story - today has been a crazy day for server loads (please leave your politics at the door). There have apparently been two very publicized server load crashes today. It was launch day for both the US's ACA Healthcare Exchanges and for Grand Theft Auto 5 Online. Both of these launches have suffered pretty sever server disasters due to a lack of capacity. I just found it interesting that two such publicized server problems happened on the same day. It's a lesson on caution for engineers and developers, because we do see these sorts of problems far too often these days.

Here is an interesting article on Forbes. The author, to an acceptable extent, stays politically neutral on the obvious topic: 'GTA Online' And Obamacare's Health Care Exchange Both Entirely Borked By Traffic On First Day - Forbes

To be fair, the engineers and developers on both projects probably expected exactly this to happen. It would be wasteful to design an architecture to support day-one traffic, since in a week or two traffic will be a fraction of what it is at first. So they design it for the actual traffic they'll see on average and put checks in place that make people wait during the very beginning of the roll-out. This is pretty standard... have you ever tried to buy tickets to a concert that sells out in 10 minutes?

I'm not sure if Rockstar (for GTA 5) had expected this. They apparently didn't put in any queuing mechanism or anything to take into account for server overload. People are complaining that when they try to connect in, they are getting some "unable to create host" or something error and being aborted entirely out of the online component. Standard or not, they certainly are not handing the problem with grace.
At least the healthcare thing (as far as I know) included some sort of queuing to allow users to wait, instead of just kicking them away.

Fair enough, but I don't think either of these situations is a case of under-engineering or not thinking the problem through. They knew exactly what would happen, made decisions based on their goals, and implemented their particular solutions. You can disagree with the solutions, but painting either one as a disaster or what-not-to-do story misses the point a bit, imho.

Valid argument.
Personally, with all of the stories lately about game companies releasing to server crashes (in no small part due to the shift towards always-on DRMs and cloud computing over the past couple years), I find it surprising that these companies do not have some sort of release-period emergency server room set up, which could be used exclusively for taking care of the traffic spikes on the first few days of release for any of their products. I mean, most of these game companies only release one title every few months. They could just reconfigure the emergency server room every time they release a new title. By the time the new title is supposed to be released, the traffic on the previous title has probably calmed down enough to not need the emergency servers anymore. That way they can invest the money in the title-exclusive servers based on average long-term traffic, while still being able to cover the initial traffic spike without having to purchase new servers every time.

I think it would be more likely that companies would use some auto-scaling service in "the cloud". But it's not just a question of supporting sheer numbers. Even if they could let everybody in at once time, what if they then find an issue in a related product? Is it better to test it out on the first 1,000 people, or should all 300 million people encounter the bug at once? It's an issue of slowly increasing capacity to make sure everything works okay instead of just opening the floodgates.

When you take a shower, you don't turn it to 100% hot and jump in, do you? No, you start out kinda warm, test it out, warm it up a little more, test it again, and repeat that process until you're at the right temperature. Same concept applies to a service like these.

True, but isn't there the concern that the simple fact that the users and servers are having issues would propagate "bugs" which would not exist if the servers were operating properly? By "bugs" I mean bugs which could be the direct result of the user not having a solid connection. While these sorts of bugs would be obviously the fault of the server issue, the company will still be getting spammed with bug reports from users which could all be related to the server issues. And, as we know, users are pretty damn ambiguous when they report bugs, so reproducing or "solving" such bugs would be difficult for developers, or near impossible considering they are most likely operating off of a developer server which is not suffering from such server issues. So you could have god knows how many resources dedicated to debugging issues which they may be unaware has a common issue, since the product is unstable to begin with. And, even once they have the server issue resolved, if the bug just disappears, they cannot simply ignore it.
It just seems to me that the top priority, when considering the existence of other bugs, would be to ensure you have a stable product first. Otherwise you could just end up wasting time chasing ghosts...

I'm not a server guy, especially for something as big as what we're talking about, but what I meant was issues that the development team could detect on their end. Questions like, "how is our matchmaking server holding up with 10,000 people? Can we increase it to 50? Now how about 100? Okay, now we're seeing an issue on the east coast, let's scale back and take a look at that region..."

Dealing with a huge software suite like that is a completely different beast compared to how us regular developers think about things. It's not so much bug reports and debugging as it is monitoring entire systems to see how they hold up against growing waves of people. With a brand new system, simply letting everybody in at once isn't the way to go- you have to make sure your system can handle 1000 people before you let a million people on it.