Which matchmaking 'issue' is the most important to you? (See post for more details!)

1) AFKs in Champion Select Lobby 4,880 36.22%
2) Duo-Queue Elo Disparities in Ranked 1,010 7.50%
3) Skilled Ranked Players in Normal Modes 668 4.96%
4) Premade Matching 674 5.00%
5) Transitioning from Normal to Ranked Mode 1,348 10.01%
6) Free to Play Champions in Ranked Mode 802 5.95%
7) Random Champions in Ranked Mode 647 4.80%
8) Provisional Matches in Ranked 723 5.37%
9) Duo Queue Prevalence in Ranked 423 3.14%
10) Level Disparities 652 4.84%
11) Team Margin of Victory 1,645 12.21%
Voters: 13472. You may not vote on this poll

After Hours with Matchmaking and Lyte

First Riot Post
Comment below rating threshold, click here to show it.

ThirdTimeLucky

Junior Member

08-12-2012

Quote:
Originally Posted by isobold View Post
If you want to dismiss Elo for such a scenario, give mathematical reasoning or don't talk. Matchmaking is a mathematical problem and should be treated as such. "X hasn't been designed for Y" is no reason at all.

Show us with math why http://lol.noamik.de/No-Elo-Hell.png is wrong for starters (because you basically stated it would be wrong).

Elo is actually working despite not being designed for the use case, and it's mathematically guaranteed to work! The real issue with multiplayer is that it slows down the Elo rating process. As Lyte has repeatedly stated: the problem isn't that Elo wouldn't converge, convergence is mathematically guaranteed here! The real issue is, that it takes up to 300 games to converge. So the real question is: how to make a rating algorithm converge faster all the while not losing the guarantee of convergence ...
1. Lyte's comment is only half correct. The convergence math assume a static population. If a significant percentage of the population is churning, this result cannot be assumed.

2. Actually, ELO does NOT converge well in the mutli-player scenario

http://www.moserware.com/2010/03/com...our-skill.html

"Although the Elo model will get you far, there are a few notable things it doesn’t handle well:
... Teams - Elo was explicitly designed for two players. Efforts to adapt it to work for multiple people on multiple teams have primarily been unsophisticated hacks. One such approach is to treat teams as individual players that duel against the other players on the opposing teams and then apply the average of the duels. This is the “duelling heuristic” mentioned in the TrueSkill paper. I implemented it in the accompanying project. It’s ok, but seems a bit too hackish and doesn’t converge well."

3. The most important argument against ELO as implemented here is a bit more subtle -
It creates a terrible gaming experience. To quote again:

"ELO has a widely acknowledged flaw (http://www.codinghorror.com/blog/200...g-systems.html) in that new players shouldn't be ranked average. This causes games around the 1200-1700 bracket to be determined by roughly 97% luck. It causes new players to face incredible hostility – the weakest point in S2’s F2P model. And it encourages smurfing."

http://forums.heroesofnewerth.com/ar.../t-326174.html

Simply put, newer players are not average. What the system is virtually guaranteed to do is to have new players sink like a rock when they start playing ranked. This "downdraft" makes it more difficult to move up. (because if you end up below the starting level, you are more likely than average to be paired with players who should be going down". Finally, the new player finds themselves stuck with players who have richly "earned" their low ELO over time due to trolling, frequent afking, or other issues.

I am NOT saying it is impossible to get out, over a long period time, a better player may be able to get out. The problem is it is could be a very long time, and the question is, should a player have to commit to playing hundreds and hundreds of games with repeated afks and trolls just to have an enjoyable gaming experience playing with people who are actually at the same level of skill?

4. Some challenges to lazy ELO defenders (and RIOT)

- If this system is working properly, it should be possible to document "lift" from the ELO difference between teams - - that is, if team A has an average ELO of 1100 and team B has an average ELO of 1150, there should be a small but statistically detectable higher probability of "B" winning. RIOT has all the data - - over millions of games it should be easy to show us that in various bands (especially in the lower half of the distribution) that the teams with the higher average ELO show a statistically significant higher probability of winning.

- here are plenty of documented cases of huge streaks. If the system was actually working, the probability of say, a 14 losing streak occurring is just over 0.006% - that is 6 chances in 100,000. Show us that such streaks are as rare as ELO theory says it should be.

- If the system is working properly, approximately 50% of all players with a starting ranking of 1200 should rise, and 50% should fall. Let's see the stats, RIOT.

***
Addendum

I'm not "pro" and maybe I'm not even above average - - I know this and this is my point - - the system as it exists today creates a bad gaming experience for a significant majority of average players. I don't really give a rat's *** if ELO works at 1500+, I'm sure it probably does (due to the lack of a downdraft effect).

My humble request to riot is to take seriously the damage this system does to the enjoyability of the game, and make a serious and concerted effort to make improvements.


Comment below rating threshold, click here to show it.

albert2006xp

Junior Member

08-13-2012

Quote:
Originally Posted by ThirdTimeLucky View Post
- If the system is working properly, approximately 50% of all players with a starting ranking of 1200 should rise, and 50% should fall. Let's see the stats, RIOT.
I can actually point to that: http://lolmatches.com/charts/rankedsolo
as you can see 50% of players are above 1250 and 50% below.

The problems that are in ranked right now are:
1. The starting matches can cause you to lose/gain way too much and if you are unlucky (and let's face it most games don't depend on your skills,especially at 1200) you can lose so much elo that you will have to gain 12-14 at a time and it will take too long.
I had to play all my first 20-30 matches of season 2 only in duo because i couldn't take my chances. Fortunately i stomped all up to 1600s with only 1 loss.

2.AND THE BIGGEST ISSUE RIGHT NOW = New dodge penalty => people can actually dodge => people pick revive/heal evelynn to force someone to dodge
I see this happen so often. Some guy is 4 or 5th pick, his favorite role is already taken, so he trolls hoping that someone will dodge. Sometimes they have the decency to dodge themselves if nobody else will, but most of the time they will just happily take the loss and troll you.
Even when i get to play a match, i have to go through 2 or 3 champion draft picks because of dodges.
Solution: Bring back the old dodge penalty or find some other penalty that is so bad nobody will dare to dodge. ( 30 min off lets be serious...)


Comment below rating threshold, click here to show it.

Ilzhahkha

Junior Member

08-13-2012

Quote:
Originally Posted by ThirdTimeLucky View Post
1. Lyte's comment is only half correct. The convergence math assume a static population. If a significant percentage of the population is churning, this result cannot be assumed.

2. Actually, ELO does NOT converge well in the mutli-player scenario

http://www.moserware.com/2010/03/com...our-skill.html

"Although the Elo model will get you far, there are a few notable things it doesn’t handle well:
... Teams - Elo was explicitly designed for two players. Efforts to adapt it to work for multiple people on multiple teams have primarily been unsophisticated hacks. One such approach is to treat teams as individual players that duel against the other players on the opposing teams and then apply the average of the duels. This is the “duelling heuristic” mentioned in the TrueSkill paper. I implemented it in the accompanying project. It’s ok, but seems a bit too hackish and doesn’t converge well."

3. The most important argument against ELO as implemented here is a bit more subtle -
It creates a terrible gaming experience. To quote again:

"ELO has a widely acknowledged flaw (http://www.codinghorror.com/blog/200...g-systems.html) in that new players shouldn't be ranked average. This causes games around the 1200-1700 bracket to be determined by roughly 97% luck. It causes new players to face incredible hostility – the weakest point in S2’s F2P model. And it encourages smurfing."

http://forums.heroesofnewerth.com/ar.../t-326174.html

Simply put, newer players are not average. What the system is virtually guaranteed to do is to have new players sink like a rock when they start playing ranked. This "downdraft" makes it more difficult to move up. (because if you end up below the starting level, you are more likely than average to be paired with players who should be going down". Finally, the new player finds themselves stuck with players who have richly "earned" their low ELO over time due to trolling, frequent afking, or other issues.

I am NOT saying it is impossible to get out, over a long period time, a better player may be able to get out. The problem is it is could be a very long time, and the question is, should a player have to commit to playing hundreds and hundreds of games with repeated afks and trolls just to have an enjoyable gaming experience playing with people who are actually at the same level of skill?

4. Some challenges to lazy ELO defenders (and RIOT)

- If this system is working properly, it should be possible to document "lift" from the ELO difference between teams - - that is, if team A has an average ELO of 1100 and team B has an average ELO of 1150, there should be a small but statistically detectable higher probability of "B" winning. RIOT has all the data - - over millions of games it should be easy to show us that in various bands (especially in the lower half of the distribution) that the teams with the higher average ELO show a statistically significant higher probability of winning.

- here are plenty of documented cases of huge streaks. If the system was actually working, the probability of say, a 14 losing streak occurring is just over 0.006% - that is 6 chances in 100,000. Show us that such streaks are as rare as ELO theory says it should be.

- If the system is working properly, approximately 50% of all players with a starting ranking of 1200 should rise, and 50% should fall. Let's see the stats, RIOT.

***
Addendum

I'm not "pro" and maybe I'm not even above average - - I know this and this is my point - - the system as it exists today creates a bad gaming experience for a significant majority of average players. I don't really give a rat's *** if ELO works at 1500+, I'm sure it probably does (due to the lack of a downdraft effect).

My humble request to riot is to take seriously the damage this system does to the enjoyability of the game, and make a serious and concerted effort to make improvements.
Very intresting links, thanks!
The lvl 30 requierment for ranked games is a barrier that counteracts the starting elo issue since that makes sure that when someone starts playing ranked they have at least played a reasonable number of games. For players who started playing before bot-games where implemented that holds true, but now it seems likely that when a player reaches lvl 30 he probably got less actual games and likely spread of several modes.

I for one would prefer a larger barrier for entering ranked games that would be in the form of champions and runes. You got a pretty large setback in the draftportion if you get a teammate that got ~3 choices for each position and runes for half of those.

One thing that I find challenging is that given that you are not equally good with all champions and roles in a team your communication skill as well as what draftposition you get is largely affecting what ELO you "should" have in a given match. For someone that mostly play on a EU server that is a even larger problem since there is a pretty large playersbase that's not very fluent in english which creates alot of frustration at times.


Comment below rating threshold, click here to show it.

isobold

This user has referred a friend to League of Legends, click for more information

Senior Member

08-13-2012

Quote:
Originally Posted by ThirdTimeLucky View Post
1. Lyte's comment is only half correct. The convergence math assume a static population. If a significant percentage of the population is churning, this result cannot be assumed.
Plain wrong. Even if 100% of the population is churning, except you, Elo would still converge for you!

Yet again http://lol.noamik.de/No-Elo-Hell.png already tells you why. If ALL players except one are trolls, the team with the one players that is no troll will win significantly more often than the other teams. Hence the one player will be Top 1 player of the ladder.

Btw.: if that wouldn't be the case, the rating algorithm would be plain broken and not be working for a static population either.

This would only change in case those "churning" players somehow agree to make it hard for this one person and to make him lose on purpose. But do you really think several hundred thousand of players allied to make YOU lose? Is that the impression you have of matchmaking? Go get psychiatric help then ...


Quote:
Originally Posted by ThirdTimeLucky View Post
2. Actually, ELO does NOT converge well in the mutli-player scenario

http://www.moserware.com/2010/03/com...our-skill.html
Your quote doesn't support your claim. Read up the article again and you will understand why Elo is actually the right approach. The article vouches for True Skill which is actually just a variation on Elo (designed to converge faster with varying skill, which is important for X-Box, since a X-Box is shared!!!, while a LoL-account is supposed to be used by a single player).

The main difference between Elo and True-Skill is that True-Skill won't decay the K-factor, but have it change as well. Thus: if Elo isn't working for Teams, True-Skill isn't either. Yet a good number of players would want to favor True-Skill claiming Elo wouldn't be working, thus basically displaying not having understood either of those algorithms ...

Citing from your own source:
Quote:
Finding universal units of skill is too hard, so we’ll just give up and not use any units. The only thing we really care about is roughly who’s better than whom and by how much.
This is EXACTLY what Elo does.
What the writer means by:
Quote:
It’s ok, but seems a bit too hackish and doesn’t converge well.
is NOT that Elo wouldn't converge, or that it's convergence wouldn't be guaranteed. What he means is exactly what I have been telling you in my last post: It's convergence is ****ing slow!

Also the next bit in this article is even plain wrong:
Quote:
This addresses the newbie problem because it removes the need to have “provisional” games.
Just plain No. Of course you still have to have provisional matches. The algorithm knowing that the current skill value is uncertain basically translates into a higher k-factor, which is exactly what LoL does with the newbie-island. So the main advantage of True-Skill is void in the newbie area. I honestly wonder how the author of such a brilliant article managed to explain for math noobs how and why Elo works this well, but then fails to see this small bit by himself.

Btw.: most people who criticize Riot-Elo blame the noob-island to be in parts responsible of their faith. In your article you can actually read up why such high k-factors in the beginning are necessary.


Quote:
Originally Posted by ThirdTimeLucky View Post
3. The most important argument against ELO as implemented here is a bit more subtle -
It creates a terrible gaming experience. To quote again:

"ELO has a widely acknowledged flaw (http://www.codinghorror.com/blog/200...g-systems.html) in that new players shouldn't be ranked average. This causes games around the 1200-1700 bracket to be determined by roughly 97% luck. It causes new players to face incredible hostility – the weakest point in S2’s F2P model. And it encourages smurfing."
Sorry, but the claim is plain wrong. Who ever wrote that understood little of either: Elo and LoL. Between 1200 and 1700 Elo we are talking of top 33% of the actual population and top 1% of the population. While I give you that having new players start at 1200 introduces noise into that area, I can outright guarantee you that 1700 Elo isn't noticeably affected by this happening. There might have been ONE player in the history of LoL that got dragged there by luck, but even that is highly unprobable.

But again: it's not Elo that creates a terrible gaming experience here but the need to have players start somewhere while not knowing anything about their skill (the need of placement matches). I read a good "solution" just recently to the issue. LoL should have players start at 200 Elo instead of 1200 and than just statically add Elo after each match till the player would have had 1200 starting Elo. This way we would maintain the median all the while the players would be "playing up the ladder".


Quote:
Originally Posted by ThirdTimeLucky View Post
Simply put, newer players are not average. What the system is virtually guaranteed to do is to have new players sink like a rock when they start playing ranked. This "downdraft" makes it more difficult to move up.
This is yet again plain wrong from a mathematical view point. This down-draft actually makes you swim up even faster. The real issue with this is the already mentioned noise it introduces. So while it makes it actually easier for you to climb the ladder, it makes it feel a lot harder.


Quote:
Originally Posted by ThirdTimeLucky View Post
(because if you end up below the starting level, you are more likely than average to be paired with players who should be going down".
You guys are always forgetting the enemies are suffering the same fate but worse: http://lol.noamik.de/No-Elo-Hell.png
That's what I meant with: while it feels bad, it actually helps you.


Quote:
Originally Posted by ThirdTimeLucky View Post
Finally, the new player finds themselves stuck with players who have richly "earned" their low ELO over time due to trolling, frequent afking, or other issues.
This isn't an issue that can be solved by matchmaking. This is an issue that has to be tackled by tribunal and other similar mechanisms.


Quote:
Originally Posted by ThirdTimeLucky View Post
- If this system is working properly, it should be possible to document "lift" from the ELO difference between teams - - that is, if team A has an average ELO of 1100 and team B has an average ELO of 1150, there should be a small but statistically detectable higher probability of "B" winning. RIOT has all the data - - over millions of games it should be easy to show us that in various bands (especially in the lower half of the distribution) that the teams with the higher average ELO show a statistically significant higher probability of winning.
It's not only possible, it's actually happening. If it wouldn't, the ladder wouldn't be that stable.


Quote:
Originally Posted by ThirdTimeLucky View Post
- here are plenty of documented cases of huge streaks. If the system was actually working, the probability of say, a 14 losing streak occurring is just over 0.006% - that is 6 chances in 100,000. Show us that such streaks are as rare as ELO theory says it should be.
How many of your friends did have such a streak recently?


Quote:
Originally Posted by ThirdTimeLucky View Post
- If the system is working properly, approximately 50% of all players with a starting ranking of 1200 should rise, and 50% should fall. Let's see the stats, RIOT.
Just plain nope. You failed to understand Elo here. I give you so, that it would be good for the algorithm if that was the case. However the system can be working perfectly fine all the while 100% of all players rise after starting with 1200 Elo and it can be working perfectly fine with 100% of all players falling after starting with 1200 Elo. The Elo gained or lost is drained from or distributed among the population.


Quote:
Originally Posted by ThirdTimeLucky View Post
I don't really give a rat's *** if ELO works at 1500+, I'm sure it probably does (due to the lack of a downdraft effect).
I'm currently doing an Elo-Hell-experiment and I can assure you the system has been working perfectly fine from 185-700 Elo so far. I've not met a single player yet who didn't belong onto his Elo (except maybe some, who didn't drop low enough yet). I've not met a single player who was actually stuck in his Elo. On the other hand nearly every single player I talked to believes he would be stuck and hold back by his mates. None of this players would even accept his mistakes having the same impact than his mates ones, even after long talks where I explained them where they went wrong. Elo isn't perfect: but the self-perception of players is a lot more flawed than that ...


Quote:
Originally Posted by ThirdTimeLucky View Post
My humble request to riot is to take seriously the damage this system does to the enjoyability of the game, and make a serious and concerted effort to make improvements.
You must be joking. Lyte discusses the matter with us for half a year now, this thread has >300 red posts on the matter, we have had many of our concerns tackled and yet this thread hasn't stopped bringing up new solutions for issues ... and your concern is that Riot wouldn't take the system seriously?


Comment below rating threshold, click here to show it.

naotasan

Senior Member

08-13-2012

I don't see a "ranked dominion" option, thats the only thing I want


Comment below rating threshold, click here to show it.

TeamTooHeavy

Member

08-13-2012

Matchmaking at night is horrendous...

Honestly, I win 70%+ of my games during the day. This falls down to about 5% (maybe less) at night.

Jungle Singed without smite loses blue buff and sits at mid at lvl 1 until everyone else is lvl 5.

GG Matchmaking.


Comment below rating threshold, click here to show it.

Live Lavish

Senior Member

08-13-2012

Quote:
Originally Posted by TeamTooHeavy View Post
Matchmaking at night is horrendous...

Honestly, I win 70%+ of my games during the day. This falls down to about 5% (maybe less) at night.

Jungle Singed without smite loses blue buff and sits at mid at lvl 1 until everyone else is lvl 5.

GG Matchmaking.
This is so true.... i'm so afraid to enter queue at night...


Comment below rating threshold, click here to show it.

Goumindong

This user has referred a friend to League of Legends, click for more information

Senior Member

08-13-2012

Quote:
Originally Posted by ThirdTimeLucky View Post
1. Lyte's comment is only half correct. The convergence math assume a static population. If a significant percentage of the population is churning, this result cannot be assumed.
Yes and no. In that at any particular point things may not be perfect, but that they are always moving towards it. This isn't a problematic issue and exists in all rating systems which continually add people because adding people adds uncertainty, at the very least, to those people who recently enter. Less so to others if the population is large, which it is.

We can see how it adds less uncertainty by pretending that your ranking wasn't a number, but rather a partial order with each person lined up in order of ranking. If there are three people in line and one person enters and then find their proper spot, the total order has moved significantly relative to the middle point. The middle person, who was previously ranked in the middle, is no ranked either the second or third quartile. If three hundred people are in line the total order has moved very little relative to the middle point.

Since ratings are more or less enforced to average to 1200 adding new people doesn't do much to increase uncertainty in your ELO.

That is to say that new entrants are only problematic in that they consistently make it more difficult to predict the games which have the new entrants.

Quote:
2. Actually, ELO does NOT converge well in the mutli-player scenario
ELO, like true skill, is a heuristic to achieve Bayesian updating. ELO is just a common name for it. There are limits to how fast we can become more certain of a value with Bayesian updating which is a function of the amount of information contained in the newly added data points. Less information relative to the amount of information currently possessed means slower updating. Adding players to a game reduces the amount of information that we have available on the skill level of players. Such there is no system which "converges well" for multiplayer games where the players are not static. This is not a function of the ranking system you use, it is a function of the information contained in the data points.

Many people think that because the original ELO system did not have a system to measure uncertainty that it is not a heuristic bayesian updater, which is wrong, it simply requires less computations and few more assumptions. It also is not likely as swift in updating as others.

To understand, roughly, why this is so consider Baysian updating.

P(A|B)P(B)=P(B|A)P(A). Or written in the more common form P(A|B)=P(A)P(B|A)/P(B)

Or as we are discussing it. The posterior distribution is proportional to the prior distribution multiplied by the likelihood. I say proportional because we don't actually know P(B) in many cases.

There is one interesting thing that TrueSkill does, but its not actually pertinent to how it matches [well, it may be, but probably not]. In that it lists your skill not as the mean, but as the mean minus three standard deviations. I.E. Your "True Skill" rating is the systems "I am 99% confident you're better than this" point.

From there all we're doing is tuning our likelihood function until we converge as swift as possible. Which for a game like league with millions of players and low information content, will be quite a while.

It is also important to note that "TrueSkill" as described in that link, is not showing convergence for games like League. Its showing convergence for games like Halo, where the final score determines the winner and provides valuable information about who is better than whom. In league we can't detect easily when one side wins handily, both because the nexus is the only thing that matters, and because its much to easy to game stats in increase your "skill" without actually being better at the game. Whereas in Halo if you go 50/1 it means you dominated, in League if you go 50/1 you can still lose, and in going 50/1 probably were not playing as efficiently as you could[for various reasons not worth discussing here]

Note also that while the name that League of Legends uses for its heuristic is ELO, it is not free from uncertainty. And of course that there must be a minimum level of uncertainty in ratings, because without it it becomes impossible to move your rating.

Quote:
3. The most important argument against ELO as implemented here is a bit more subtle -
It creates a terrible gaming experience. To quote again:
[...]

Simply put, newer players are not average.
Actually in League "new" players to ranked are pretty average. Its one of the advantages of having to play 200+ games and own 16 champions before starting ranked. There are no "newbies"

Quote:
This "downdraft" makes it more difficult to move up. (because if you end up below the starting level, you are more likely than average to be paired with players who should be going down". Finally, the new player finds themselves stuck with players who have richly "earned" their low ELO over time due to trolling, frequent afking, or other issues.
If newer players experience an unfair downdraft then they're more skilled than their teammates and the probability of them winning matches increases. ELO hell does not exist. More trolls and AFK's will exist on the other team, creating a net advantage for the player who is legitimately more skilled than his ranking suggests


Quote:

- here are plenty of documented cases of huge streaks. If the system was actually working, the probability of say, a 14 losing streak occurring is just over 0.006% - that is 6 chances in 100,000. Show us that such streaks are as rare as ELO theory says it should be.
No. The probability of seeing a 14 game losing streak is not .0006%. The probability of seeing a 14 game losing stream in any particular 14 games assuming that the probability of winning the game is stable at 50% is about .0006%. But... and this is a big but. The probability of seeing at least one 14 game losing streak in a large number of ranked games, making the same stable win probability assumption is actually quite high. I don't have numbers since IIRC doing this requires simulation and I am lazy. But I can tell you that the probability of seeing 9 heads or 9 tails in a row if you flip a coin 1000 times is about 80%, 10 in a row is about 60%.

Since large streaks can "feed" on themselves due to poor play from psychological factors its not really a surprise for this to be seen in "large" numbers.

It would be interesting to know if large streaks were common or uncommon [i suspect large streaks of losses are more common than large streaks of wins] but I am not sure that knowing that we see large streaks of losses more often than large streaks of wins tells us anything particularly interesting about our match making system.


Comment below rating threshold, click here to show it.

isobold

This user has referred a friend to League of Legends, click for more information

Senior Member

08-13-2012

Quote:
Originally Posted by Goumindong View Post
...
Big wall of text, but man: worth reading. If you skipped this, go back and read it.

+1!


Comment below rating threshold, click here to show it.

Yegg

Game Designer

Follow Yegg on Twitter

08-13-2012
338 of 362 Riot Posts

Quote:
Originally Posted by ThirdTimeLucky
Simply put, newer players are not average. What the system is virtually guaranteed to do is to have new players sink like a rock when they start playing ranked. This "downdraft" makes it more difficult to move up. (because if you end up below the starting level, you are more likely than average to be paired with players who should be going down". Finally, the new player finds themselves stuck with players who have richly "earned" their low ELO over time due to trolling, frequent afking, or other issues.
Quote:
Originally Posted by Goumindong View Post
Actually in League "new" players to ranked are pretty average. Its one of the advantages of having to play 200+ games and own 16 champions before starting ranked. There are no "newbies".
Goumindong is correct. We ran this analysis awhile back and players who are new to ranked finish their provisional matches with an average rating that's very close to 1200 (and it included some players with sub-500 ratings because of heavy dodging). Now that players don't lose Elo from dodging I expect it would be even closer.