@Lyte: A small concern about the Tribunal

First Riot Post
Comment below rating threshold, click here to show it.

Lyte

This user has referred a friend to League of Legends, click for more information

Lead Social Systems Designer

Follow RiotLyte on Twitter

04-06-2013
6 of 12 Riot Posts

Quote:
Originally Posted by Fomorian27 View Post
I actually have a question about this. Someone posted implying that your quote

"only about 0.7% of accounts in North America were eligible for a Tribunal case" took into account ALL accounts on NA server, both inactive and active.

Can you comment on this?
We always compare against active accounts; it'd be silly to compare against all-time accounts. Players tend to forget that the Tribunal was designed to only handle the worst of the worst. Now that the Tribunal is pretty accurate and effective at reforming or removing the worst of the worst, we have been focusing on new features and systems that try to address behavior across the entire spectrum.


Comment below rating threshold, click here to show it.

Mirror

Senior Member

04-06-2013

In these kinds of conversations you guys should remember that there's really no motive for Lyte & Pals to be cheating on the stats - as they'd only be cheating themselves. It's in their best interest to design systems that run in the most intelligent ways possible, not ones that produce the best one-line statistics for infographics.


Comment below rating threshold, click here to show it.

loooooooololol

Senior Member

04-06-2013

Quote:
Originally Posted by Lyte View Post
I agree, "Correct" and "Incorrect" aren't the ideal terms to be using in the Justice Reviews, but we had to start somewhere and iterate over time.

In the social sciences, there's a statistic called inter-rater reliability, which basically measures the level of agreement among a number of raters (voters, in Tribunal's case). Conceptually, that's the number we are presenting when we refer to "Accuracy" but we need to work on better ways to define and explain this in Justice Reviews.

I must stress that there's no "wrong" opinion in the Tribunal. Everyone has a unique opinion on what behavior is OK or not OK, and every opinion is extremely valuable to the system and the shaping of online norms over time in the community.
Lol are you getting a psychologist to develop the tribunal system? I've seen posts of you using methods I learned from 1st year Psychology classes.


Comment below rating threshold, click here to show it.

BrohannesJahms

Senior Member

04-06-2013

Quote:
Originally Posted by Metronomotopoeia View Post
In these kinds of conversations you guys should remember that there's really no motive for Lyte & Pals to be cheating on the stats - as they'd only be cheating themselves. It's in their best interest to design systems that run in the most intelligent ways possible, not ones that produce the best one-line statistics for infographics.
I don't understand the cynical mentality that some people have in claiming that the PB&J team is bending the stats to make the Tribunal look good. I like your attitude, though.


Comment below rating threshold, click here to show it.

Doct0r Phil

This user has referred a friend to League of Legends, click for more information

Senior Member

04-06-2013

Quote:
Originally Posted by Lyte View Post
We always compare against active accounts; it'd be silly to compare against all-time accounts. Players tend to forget that the Tribunal was designed to only handle the worst of the worst. Now that the Tribunal is pretty accurate and effective at reforming or removing the worst of the worst, we have been focusing on new features and systems that try to address behavior across the entire spectrum.
Lyte, can you give us some helpful statistics and information (about the effectiveness of the tribunal), so that we can relay that information to those on the tribunal forum pages. You had posted a while ago that you would give us some helpful stats/data.

I believe the tribunal is helpful and accurate, it's just nice to have more information than just your word on it and my beliefs.


Comment below rating threshold, click here to show it.

Fomorian27

This user has referred a friend to League of Legends, click for more information

Senior Member

04-06-2013

Quote:
Originally Posted by Lyte View Post
We always compare against active accounts; it'd be silly to compare against all-time accounts. Players tend to forget that the Tribunal was designed to only handle the worst of the worst. Now that the Tribunal is pretty accurate and effective at reforming or removing the worst of the worst, we have been focusing on new features and systems that try to address behavior across the entire spectrum.
Yes, of course, given your previous quote it seems silly to even ask.

BTW, when can we expect your GDC talk to be online?


Comment below rating threshold, click here to show it.

Lyte

This user has referred a friend to League of Legends, click for more information

Lead Social Systems Designer

Follow RiotLyte on Twitter

04-06-2013
7 of 12 Riot Posts

Quote:
Originally Posted by Doct0r Phil View Post
Lyte, can you give us some helpful statistics and information (about the effectiveness of the tribunal), so that we can relay that information to those on the tribunal forum pages. You had posted a while ago that you would give us some helpful stats/data.

I believe the tribunal is helpful and accurate, it's just nice to have more information than just your word on it and my beliefs.
Quite a few players have done some neat analysis on the Tribunal data, might be nice for them to link their posts

In terms of new Tribunal data, we've done some inter-rater reliability assessments comparing Tribunal judges to in-house Player Support staff. It turns out that at the beginning of the Tribunal, the level of agreement between players and Player Support was pretty high with over 80% of cases having unanimous agreement compared to a team of up to 6 Player Support. As we tuned the Tribunal and optimized the math behind the system, the level of agreement increased into the extremely high 90% range which is why we have recently focused Player Support reviews on the 'close' cases instead of reviewing every single case.

In addition to the recent findings that the community has taken a pretty firm stance against homophobic and racial slurs, something else that is interesting is that the usage of those terms have been dropping, and the most commonly used negative word or term is now "report" and "bg noobs."


Comment below rating threshold, click here to show it.

Lyte

This user has referred a friend to League of Legends, click for more information

Lead Social Systems Designer

Follow RiotLyte on Twitter

04-06-2013
8 of 12 Riot Posts

For those interested, I'm also having a discussion on Reddit at the moment about feedback loops for when players receive reports and whether it can be a viable idea.

http://www.reddit.com/r/leagueoflege...hy_not/c99ufsl


Comment below rating threshold, click here to show it.

BrohannesJahms

Senior Member

04-06-2013

Quote:
Originally Posted by Lyte View Post
For those interested, I'm also having a discussion on Reddit at the moment about feedback loops for when players receive reports and whether it can be a viable idea.

http://www.reddit.com/r/leagueoflege...hy_not/c99ufsl
Lots of good points being made here, but I'm really interested to know if there are any feedback loops that you've noticed in player behavior that result in more positivity. Does PB&J have plans to exploit any such loops?


Comment below rating threshold, click here to show it.

LaDemonius

Junior Member

04-06-2013

I think the tribunal is over ban happy and will be the cause of LoL to fall apart in the long run because they don't care they just get their jollies by banning folks.