How Justice Rating should work (an opinion).

Comment below rating threshold, click here to show it.

Lord Emamah

Junior Member

11-16-2012

Albeit farthest from my intention to tell you guys how you should be doing your job, I merely thought that .. well.. Alright. Here's how you should be doing your job, OK?

The justice rating as it is right now rewards people for participating, and that is all good and all. However, people are naturally inclined to compare it to in-game ratings, which is meaningless - I have seen people post justice rating with 5 digits - and if you do not have a point of reference, the number itself becomes meaningless.

Here's how I would have done it:

First step: We rate people in sampling windows. A week? X cases? Variable? Not important. The important thing is that we have a number of cases, and the person has a rating.

Second step, when the tribunal member rates a case, we record the verdict and the person's rating at the time of giving the verdict. End of that.

Third step, when we process the case. We determine the verdict based on whatever it is you guys use. Consensus, number of reports, whatever. We do not involve the rating here. This stays unchanged.

Fourth step, we rate the case. We go through every single verdict given to the case and give the case a performance rating against each verdict. If the player gave the right judgment, the case gets a performance rating of (Player Justice Rating - 450). If the player gave the wrong judgment, the case gets a performance rating of (Player Justice Rating + 450). At the end we give the case a Difficulty Rating, which is the average of the performance ratings.

In other words: If highly rated players get it wrong, it gets a high Difficulty Rating. If low rated players get it right, it gets a low Difficulty Rating. (Talking Justice Rating still, not game Elo)

Fifth step: At the end of the player sampling window, we go through the cases processed, and calculate the expected score by comparing the player Justice Rating to the Difficulty Ratings. We then cap the expected score to be maximum 0.95 x the number of cases. Finally we change the rating based on the difference between expected and achieved score and a sensible multiplier coefficient.

This will:

a) Keep the ratings comparable to ingame Elo's.

b) Reward people for getting the cases right, and punish them for getting them wrong.

c) Curb rating inflation. The only way you will get points, is if the case was hard, i.e. other people got it wrong and therefore lost rating.

d) The 95% cap will still cause a rating inflation, but will reward people for volume of voting.

Riotplz?


Comment below rating threshold, click here to show it.

SoresuMakashi

Senior Member

11-16-2012

Why should ratings be kept comparable or have features similar to in-game Elos? Different systems.


Comment below rating threshold, click here to show it.

vulgarwolf

Senior Member

11-16-2012

why should people be punished for getting a case wrong ??

Everybody has different moralistic views, some players may punish racism, but not other forms of offensive language, i punish what i feel is wrong


Comment below rating threshold, click here to show it.

LMQ UZl

Senior Member

11-18-2012

Quote:
Originally Posted by vulgarwolf View Post
why should people be punished for getting a case wrong ??

Everybody has different moralistic views, some players may punish racism, but not other forms of offensive language, i punish what i feel is wrong
^ This

I do believe if you are below 60% accuracy you should be banned from using tribunal for a month, that way it can be fair. But you shouldn't be punished in game for tribunal.

Think of Tribunal as jury duty, lets face it always a person there that just doesn't care or biased, so take him out of the list. =)


Comment below rating threshold, click here to show it.

dignitas Rampant

This user has referred a friend to League of Legends, click for more information

Senior Member

11-18-2012

Quote:
Originally Posted by Melicalol View Post
^ This

I do believe if you are below 60% accuracy you should be banned from using tribunal for a month, that way it can be fair. But you shouldn't be punished in game for tribunal.

Think of Tribunal as jury duty, lets face it always a person there that just doesn't care or biased, so take him out of the list. =)
I disagree Melic, because the Tribunal is supposed to be a group decision on how we want others in the community to behave. If we ostracize someone for having a differing opinion, it no longer is the community's decision, as we are excluding members of the community. If you have a low percentage, that is what is it is, it just shows that you do not agree with the majority view of behavior. It doesn't mean you should be removed, as these people act as a tempering force on the Tribunal.

I do agree however, if someone was to say attempt to disingenuously influence the Tribunal by, say, voting Punish or Pardon every single game, as these people add nothing to the group decision except the opinion "I don't like the Tribunal".


Comment below rating threshold, click here to show it.

Lord Emamah

Junior Member

11-18-2012

Sorry, I find it hard to find old threads in this forum, so I'm not very good at replying.

Quote:
Why should ratings be kept comparable or have features similar to in-game Elos? Different systems.
Well, if they used this system, it would be the same system!

The point I'm attempting to make is: My opinion of your question is, I think it merits a quality score of 2.84!

What do you think of that? If you're like me, you'd think nothing except that this is a completely meaningless statement. I claim that the Justice Rating is currently a meaningless statement. It is not obvious to me why you need a rating to begin with. But Riot made it, and if it's there then I think something should be made to make it intelligible. Putting it on an Elo-like calculation, it would make sense to people.

Quote:
why should people be punished for getting a case wrong ??
Yes, OK, "punish" was probably a badly chosen word. I apologise. What I meant was that it would reduce your rating (and assuming that people care). I was not advertising temporary bans for people not conforming to the general consensus.

But if you have a rating, shouldn't getting it "wrong" affect the rating?


Comment below rating threshold, click here to show it.

Great Pyrenees

Senior Member

11-19-2012

Intersting, personally I would be happy with a judge boarder on in game load screen. I mean I already know that my participation in the Tribunal is making a real difference. But I see your point. Take care friend =)