NA and EU stability update

First Riot Post
Comment below rating threshold, click here to show it.

sonicdeathmonk

Vice President of Operations

02-23-2014

We are sorry for the ongoing server issues in many parts of the world including very recent North America and Europe problems. Lots of work has been done to help improve things and more work is still being done.

We have seen significant increases in hackers running distributed denial of service attacks (DDoS) against game and platform servers. League of Legends has not been the only target, most large game companies have been feeling the pain along with other online businesses, but we have been a more frequent target than most. In the last 6 months over half of all DDoS attacks we have seen against LoL have been in the last 2 months. We are currently seeing 2 to 4 attacks every day against one or more regions. The strength of these attacks has increased significantly over the last few months as the arms race against hackers continues to grow in scale.

League of Legends is made up of dozens of components, some of these are easier to protect against DDoS than others. Typically websites are easiest to protect, while game servers are hardest (most standard forms of protection would create lag in game like League of Legends).

In the last 2 weeks world record breaking DDoS attacks have been seen by several companies including Riot. These attacks are so big that the ISPs and Telecom's that service League of Legends have not been able to keep their Internet circuits up while the attack is underway. This means that our servers often do not even see or feel the malicious traffic itself because the Internet connections feeding our data centers are being cut off due to the size of the attacks filling the very large Internet circuits the telecoms and ISPs have.

When our data centers lose the Internet, your connections to game servers and/or PVP.net services like Chat and Login get disconnected in the worst case or lots of players will see serious in-game lag for the duration of the attack. If lots of players get disconnected, you will often see a login queue to get back into PVP.net or the game.

We expect the current attacks to get better soon. We already have been able to fend off many problems before they create problems for you. Our network team has been working almost non-stop for several weeks both fighting off DDoS attacks around the world and strengthening our Internet services. We have also been working with telecoms and ISPs to help them understand how to block these attacks so that their services are not affected. Some fixes have required new equipment and ISP circuits to be ordered which can take weeks to get shipped and installed. Some telecoms and ISPs are slower than others in applying the fixes needed to ensure they do not fall over from these attacks, we depend on many ISPs so its important that all ISPs make the fix or the bad traffic finds a way to knock off our servers.

Of course this is an arms race so as one defense gets put in place, eventually we can expect new methods to launch attacks that will often require time to put a defense in place. About half of the equipment investment in League of Legends is network gear with a large part of that in place to just handle the ongoing rise in frequency and size of DDoS attacks. We are investing even more every day around the world and will continue to work hard to win the fight. We are also working closely with our equipment providers and other services to make sure we are doing everything we can to protect the many components that make up League of Legends.

Sometimes we lose the fight and that sucks because it means players like you feel the pain too. It also means that instead of our engineers building cool stuff for players we have to fight off bad stuff from hackers, so players lose again. It's frustrating, but the reality of large scale Internet services, so we have to get smarter and better to make sure players around the world can have fun when you want to have fun.

I apologize for the pain and promise that we will continue to get better. New data centers with stronger and even more Internet connections are getting close to being live for Europe and North American players, with all other regions scheduled for complete overhauls as 2014 goes on. These new data centers will help us keep up the fight in the hacker arms race, they won't solve all the problems we deal with all the time, but they should level up our side of the fight a great deal.

In the meantime we are not helpless, teams are still working long hours, expect daily improvement.

See you on the Fields of Justice,

sonicdeathmonk
VP of Network Operations
Riot Games

P.S. for the more technically curious here is some background reading on the current attacks:

http://krebsonsecurity.com/2014/02/t...-ddos-attacks/


Comment below rating threshold, click here to show it.

I love beer

Senior Member

02-23-2014

Sweet

Side note, what is the bounty for derptrolling?


Comment below rating threshold, click here to show it.

Made In China

Member

02-23-2014

Much appreciated, Keep working hard Riot.


Comment below rating threshold, click here to show it.

Dealyth

This user has referred a friend to League of Legends, click for more information

Senior Member

02-23-2014

So no 1000 RP?


Comment below rating threshold, click here to show it.

ThePlagueborn

Member

02-23-2014

Thanks for the update


Comment below rating threshold, click here to show it.

Spells

Senior Member

02-23-2014

ok so can i get my lp back now?


Comment below rating threshold, click here to show it.

ThePlagueborn

Member

02-23-2014

Quote:
Originally Posted by Dealyth View Post
So no 1000 RP?
I would assume no. I would consider the fact that riot is fixing and improving their servers as a good enough. I'd rather be able to play then have RP.


Comment below rating threshold, click here to show it.

Giggle

Senior Member

02-23-2014

stfu 3 weeks of lag is unacceptable


Comment below rating threshold, click here to show it.

Shmookpup

Senior Member

02-23-2014

I really appreciate the fact that you took the time to write this giant wall of text. This will calm all of us down and be back on your side again.

Thank you for letting us know about the situation and not being silent.


Comment below rating threshold, click here to show it.

ScarPirate

This user has referred a friend to League of Legends, click for more information

Senior Member

02-23-2014

I like this post.
I hope riot can out pace the hackers.
maybe riot could look into highering social engineers to try and get inside the heads of these hackers