Comment below rating threshold, click here to show it.
Vice President of Operations
2 Weeks Ago
We are sorry for the ongoing server issues in many parts of the world including very recent North America and Europe problems. Lots of work has been done to help improve things and more work is still being done.
We have seen significant increases in hackers running distributed denial of service attacks (DDoS) against game and platform servers. League of Legends has not been the only target, most large game companies have been feeling the pain along with other online businesses, but we have been a more frequent target than most. In the last 6 months over half of all DDoS attacks we have seen against LoL have been in the last 2 months. We are currently seeing 2 to 4 attacks every day against one or more regions. The strength of these attacks has increased significantly over the last few months as the arms race against hackers continues to grow in scale.
League of Legends is made up of dozens of components, some of these are easier to protect against DDoS than others. Typically websites are easiest to protect, while game servers are hardest (most standard forms of protection would create lag in game like League of Legends).
In the last 2 weeks world record breaking DDoS attacks have been seen by several companies including Riot. These attacks are so big that the ISPs and Telecom's that service League of Legends have not been able to keep their Internet circuits up while the attack is underway. This means that our servers often do not even see or feel the malicious traffic itself because the Internet connections feeding our data centers are being cut off due to the size of the attacks filling the very large Internet circuits the telecoms and ISPs have.
When our data centers lose the Internet, your connections to game servers and/or PVP.net services like Chat and Login get disconnected in the worst case or lots of players will see serious in-game lag for the duration of the attack. If lots of players get disconnected, you will often see a login queue to get back into PVP.net or the game.
We expect the current attacks to get better soon. We already have been able to fend off many problems before they create problems for you. Our network team has been working almost non-stop for several weeks both fighting off DDoS attacks around the world and strengthening our Internet services. We have also been working with telecoms and ISPs to help them understand how to block these attacks so that their services are not affected. Some fixes have required new equipment and ISP circuits to be ordered which can take weeks to get shipped and installed. Some telecoms and ISPs are slower than others in applying the fixes needed to ensure they do not fall over from these attacks, we depend on many ISPs so its important that all ISPs make the fix or the bad traffic finds a way to knock off our servers.
Of course this is an arms race so as one defense gets put in place, eventually we can expect new methods to launch attacks that will often require time to put a defense in place. About half of the equipment investment in League of Legends is network gear with a large part of that in place to just handle the ongoing rise in frequency and size of DDoS attacks. We are investing even more every day around the world and will continue to work hard to win the fight. We are also working closely with our equipment providers and other services to make sure we are doing everything we can to protect the many components that make up League of Legends.
Sometimes we lose the fight and that sucks because it means players like you feel the pain too. It also means that instead of our engineers building cool stuff for players we have to fight off bad stuff from hackers, so players lose again. It's frustrating, but the reality of large scale Internet services, so we have to get smarter and better to make sure players around the world can have fun when you want to have fun.
I apologize for the pain and promise that we will continue to get better. New data centers with stronger and even more Internet connections are getting close to being live for Europe and North American players, with all other regions scheduled for complete overhauls as 2014 goes on. These new data centers will help us keep up the fight in the hacker arms race, they won't solve all the problems we deal with all the time, but they should level up our side of the fight a great deal.
In the meantime we are not helpless, teams are still working long hours, expect daily improvement.
See you on the Fields of Justice,
VP of Network Operations
P.S. for the more technically curious here is some background reading on the current attacks: