@Lyte- Why incorporate a "rating" system for Tribunal voting results

Comment below rating threshold, click here to show it.

Charltonheston

Junior Member

12-10-2012

Hey whoever wants to read this. Just something thats been bugging me a bit since I started participating in the Tribunal.
I think the Tribunal is a great idea and seems to be very successful in fixing/eliminating toxic behavior/players. The thing I don't really understand is why Riot decided to incorporate a rating system based on cases you voted "correctly" (aka coinciding with the majority) on. To me this seems like giving a jury in criminal cases some kind of incentive to come to a unanimous agreement on a verdict.

I have a problem with this not because my 'accuracy' rating is terrible (It's around 80 percent after my first 3 days of participating) but because I feel that having a rating system that rewards you for agreeing with the majority of voters encourages players to vote to simply how they think other people vote based on trends rather than simply voting according their own opinions and interpretation of the Summoner's Code.

When participating with anything that has a score or rating system, there is always a desire to improve that score. In the Tribunal's case, the easiest way to do this is just to vote the way you've seen the majority vote when you review past cases. In my case, this would mean being just a little bit more harsh on some cases that I consider to be on the fringe of being pardonable.

I understand the importance on having some system that encourages players to take the time to think about their votes. But I do think that this system is potentially abusable- players could simply decide to all vote or pardon 100 percent of all cases- thereby achieving 100 percent accuracy rating. At the very least, I think the system really does encourage you to simply blindly follow trends you notice the majority voting.

Lyte- out of curiosity. This seems like a new system. Have you found that having this rating system has improved general voter thoughtfulness, or improved the results of the Tribunal? Just curious.

Thanks to whoever bothered to sift through this horrible wall of text. I really don't expect anyone to bother haha. Happy gaming!


Comment below rating threshold, click here to show it.

ChaoticStorm

Junior Member

12-10-2012

I like the rating system. It doesn't hurt anyone, except maybe rankings because it is just a cosmetic number and doesn't effect anything else.

There will always be a desire to improve on anything throughout life, same with the Tribunal.


Comment below rating threshold, click here to show it.

Wheremi

Member

12-10-2012

Off the top of my head, the reason there is a rating system is the same reason they took out the gift IP that I would quickly like to address before responding to your question. It incentivizes those who want to put the time in to actually do the tribunal properly to judge well. Without an IP gift there is no reason to hang arond and do cases unless you're either there to try to skew results, Which will eventually result in your votes having less weight due to voting AGAINST the logical majority trying to do their job. Or because you want to legitimately judge cases fairly to which this change wouldn't have effected you anyway realistically.

With a Rating system this is following the same goal, It incentivizes you to judge well if you want to have that high ranking, even tribunal members can be won over by being able to say how good they are at making sure they call right. Now that being said certainly you can say this pushes people who want the highest rank to judge based on what the community around them would do, but I also would say this is not a horrible thing that is a giant problem if you think about it.

If you knew before hand that a case was going to be Punish or Pardon, you voting the same way would change nothing because a majority would have already had to be decided for you to know what way it was going to trend. The end result would get the same judgement regardless, and all that would change is your score would go up. With this in mind the rating system would not hurt any cases that have a strong majority in either direction. However the fact that we still have cases that are only "Majority" in either direction implies to me that there are still plenty of cases where there are people voting on both sides fairly evenly, so clearly there are still people that don't always vote one way and the tribunal culture has not gotten to a point where we are all voting like mindedly just to save face

I will however say that as the more I tried to write this post, I started to lose my argument for why having the ranking was a good idea admittedly. The system was indeed put in place so that the community could shape the grey area of the code on what was pushing those boundaries too far at times and while this does call for everyone's personal feelings to help shape that part of what we judge as fair game and crossing the line. The rating system does jeopardize this a bit once you know how the community at large considers a certain infraction compared to how you yourself would vote.

As a final note, Riot does audit the system every so often supposedly, If the entire tribunal community suddenly got together and all decided to 100% Punish or Pardon just to boost their scores, I can guarantee someone over there would notice. Anyone caught doing such would probably have their scores reset and may even be up for further penalties, would that really be worth the risk just to boost a virtual tribunal score? I don't think so personally compared to the allure of actual in game rank.

All that aside I am admittedly curious myself to hear what the the changes to the system have shown for accuracy, but I am certain we will find out more about that when there is something they feel is worth letting us know, all will be revealed in due time.


Comment below rating threshold, click here to show it.

Charltonheston

Junior Member

12-11-2012

Quote:
Originally Posted by ChaoticStorm View Post
I like the rating system. It doesn't hurt anyone, except maybe rankings because it is just a cosmetic number and doesn't effect anything else.

There will always be a desire to improve on anything throughout life, same with the Tribunal.

I definitely agree that if you have a ranking system, it encourages people to participate and gives them the desire to improve the score. The only 'problem' I have with it is that improving your score is really easy if you just vote with the majority based on trends you see.

The ranking itself isn't hurtful, just the fact that it can perhaps encourage people to vote slightly differently than if the rating didn't exist.
Thanks for the reply!


Comment below rating threshold, click here to show it.

Charltonheston

Junior Member

12-11-2012

@Wheremi
I agree that the rating system is a good way to provide some incentive to vote thoughtfully. But what does judging 'well' really mean? If it means voting along with the majority then I think the system is doing a great job. But if it is to vote according to your own ideas and interpretation of the Summoners Code, maybe it doesn't do that as well necessarily.

You're definitely right that it's not a huge problem, and probably does a great job in eliminating grey areas. I would say that having some degree if swing either way with results if you were intentionally trying to vote according to trends could just be random coincidence- who knows.

I really don't have a problem with the system, I was just struck by the idea and was wondering if anyone was having similar thoughts. And yeah I think it would be really interesting to see any data Lyte and Riot have collected regarding this. As you say all in good time haha.


Comment below rating threshold, click here to show it.

Weimdog

Junior Member

12-11-2012

I am having the same thoughts Charlton. As I said in another thread, I'm wondering how they (Riot and it's Justice Review staff) are going to judge the people who actually care about these abuses, when we are supposedly being judged by our ability to agree with those who would pardon these cases.

That said, I am struggling to think of a better system of evaluating our performance as judges. Threads like this could be really helpful to the riot staff if we come up with a good idea for fixing the Justice Review.