Comments, Suggestions and/or Feature Requests for the Soundevotion Competition.

Alternate Scoring System

Alternate Scoring System

Postby dilvie » Sat May 27, 2006 2:38 am

Hi Gang,

I'd like to propose an alternative scoring system. It's a tiny bit more complicated, but it reduces the likelyhood of ties (by introducing a total number of votes factor), and it emphasizes consistent votes over a single high vote (making it less random and arbitrary).

Under this system, this is the way round 3 would have turned out:

Code: Select all
1   GroovyOne
2   Mick Rippon
3   Harmony
4   Sonicade
5   Necto Ulin
6   Overthruster
7   Aged
8   Chotoro
9   DJ Status
10   robwilliamsjnr
11   Dilvie
12   fr0gr0ck
13   dj_io


To get these results, I awarded points like this:

1st - 5 points
2nd - 4 points
3rd - 3 points
4th - 2 points
5th - 1 point

This awards consistency more than a single high vote, which the current exponential system emphasizes (reducing the disruption caused by random voting, particularly in the lower ranks).

I normalized these results on a scale of 0 - 1.

Then I totalled the number of votes for each entry, and normalized those to the same scale.

The last step multiplies both scores, and scales the final score such that the top ranked tune gets 100 points. You still get an exponential scale with this scheme, with fewer ties, and better resistence to random voters and tampering.

Some observations:

The top 3 results didn't change, but Sonicade (deservedly) places a bit better. In terms of points, there is a 3-way tie near the bottom of the pack. robwilliamsjnr gets recognition for placing on 3 top lists, rather than two (as my entry placed), which pushes him out ahead of the other stragglers. Likewise, my entry beats fr0gr0ck, because I got two votes, rather than one.

I hate to come off sounding like a sore loser, but when I looked beyond the top 3 places, the results seemed totally random to me. When Mick pointed out that the current system was based on an exponential simple tally scale, it seemed like something that could be easily corrected.

I challenge each of you to compare these results to the official results, honestly. I think you'll find that my list makes more sense when you listen to the music and think about how each contestant placed.

Either way, I placed very low, but I don't think any amount of math tinkering is going to help that. ;)

- Eric
dilvie
Regular
Regular
 
Posts: 86
Joined: Tue May 16, 2006 4:40 am

Postby Sonicade » Sat May 27, 2006 2:47 am

Hey Dilvie, I'm glad you brought this up. I'm perfectly content with the round 3 results but there may be some improvements we can make to the point system. I'm about to go to sleep now so I can't fully comment tonight but lets open up some discussion about this.

Firstly, dj_io voiced his concern about this in the past and I can see now that he might have been right to do so. So dj_io my appologies and feel free to share with us any new thoughts you might have on this issue.

Secondly I just finished discussing this a little with Mick Rippon and it seems like it might be wise to make some adjustments.

The key ideas here are simplicity and fairness. It seems like such a simple task to tally up some votes and list the entries in order but it's turning out to be quite challenging. Insights welcome. :)
User avatar
Sonicade
Compo Admin
Compo Admin
 
Posts: 773
Joined: Fri Mar 17, 2006 6:42 pm
Location: California, USA

Postby dilvie » Sat May 27, 2006 3:14 am

Sonicade wrote:Hey Dilvie, I'm glad you brought this up. I'm perfectly content with the round 3 results but there may be some improvements we can make to the point system.


Frankly, I was shocked by the round 3 results -- first, I expected my entry to do much better, but I can take the ego bruising. ;) On the other hand, you had more votes and better consistency than Overthruster's entry.

Even without a bonus for getting more votes, you would have beat him on a linear point scale. Under the current system, he got a significant boost from one (IMO) overly enthusiastic voter. Sure, the track was stylistically the most unique of the bunch, but the track wasn't musically as strong. It didn't deserve to beat you, regardless of the novelty value.

The key ideas here are simplicity and fairness.


A linear point scale is simpler than an exponential point scale, and factoring the number of votes serves mostly as a tie-breaker, which improves the fairness. ;)

It seems like such a simple task to tally up some votes and list the entries in order but it's turning out to be quite challenging. Insights welcome. :)


It should stay fairly simple. I don't think my scheme complicates things much at all. I like the tie-breaker element, and I think it's worth a little bit of complexity trade-off, and IMO, the linear scale is much more stable and fair for the tail places.

- Eric
dilvie
Regular
Regular
 
Posts: 86
Joined: Tue May 16, 2006 4:40 am

Postby dilvie » Sat May 27, 2006 2:14 pm

I took a look at the objections in the older thread that suggested a linear point distribution. It seems to me that the exponential distribution places undue emphasis on individual voters -- where one person's opinion over-rides the opinions of (potentially many) other people. Consider, for instance, if the compo grows to 20 participants. One person is rushed for time, and votes randomly. His 1st place vote would count more heavily than a majority voting another song into 4th or 5th place. I find that idea silly, frankly. I don't think one person's opinion should count more than 15 other people.

As you can see in my alternate distribution, really good songs are not likely to be affected. Somebody deserving 1st place is still going to get 1st place. What really changes are the votes beyond the top 3. If you have doubts, I can prepare results for each of the previous compos, for comparison.

True, under this system, popularity is rewarded, but even if only 2 people vote a song into first place, it's still going to get more points than 3 people voting a song into 3rd. I don't think that's unfair. I think that the bottom line for me is, random voting by people rushing at the last minute is an unavoidable reality, and with the current system, a couple of random votes can really corrupt the results.
dilvie
Regular
Regular
 
Posts: 86
Joined: Tue May 16, 2006 4:40 am

Postby organic io » Sat May 27, 2006 3:15 pm

where are you seeing the data for how many people voted for each contestant? i didn't think that info was publicly available

edit: nevermind.. found it :)
User avatar
organic io
Compo Admin
Compo Admin
 
Posts: 1535
Joined: Thu Apr 06, 2006 11:55 am

Postby Sonicade » Sun May 28, 2006 3:15 am

I'm going to sit down and go over the point system again when I get back home. Right now I'm traveling so it's hard to get into restructuring it.
Originally the compo worked on an individual entry vote from 1 to 10. That system seemed to have some inherent flaws and so we switched to the Top 5 voting system. A large concern when we setup the top 5 was the possibility of two 2nd place votes over riding a 1st place vote. With that in mind it looks like we went a bit too heavily with the weighting.
Thanks Dilvie for clearly laying it out again and to DJ_io for bringing it up in the first place.
User avatar
Sonicade
Compo Admin
Compo Admin
 
Posts: 773
Joined: Fri Mar 17, 2006 6:42 pm
Location: California, USA

Postby dilvie » Sun May 28, 2006 4:21 pm

Is it really so important that two second place votes don't over-ride a single first place vote? Keep in mind that generally, there are more than 3 people voting. I sincerely doubt that the truly deserving songs are going to suffer with this proposed change. A daring song may not get all 1st place votes, but if it really has merrit, it might get a bunch of votes for 3rd and 4th, along with one or two votes for 1st -- it's still going to do well. Should it beat a song that gets more votes for 2nd place, but no votes for 1st? This scenario is much more realistic -- but even this is pushing it.

In reality, this change probably won't have a much impact on the top spots -- particularly as the compo grows and we see more voters weighing in. What it will do is stabalize the tail positions and (with the number of voters factor) break point ties.
dilvie
Regular
Regular
 
Posts: 86
Joined: Tue May 16, 2006 4:40 am

Postby MickRip » Mon May 29, 2006 8:15 pm

Using the exponential (16,8,4,2,1) system is too elitist, and it doesn't reward consistency.

If someone votes your tune as 5th, you are getting a 16th of the points if you received a 1st place vote...

This means that if 15 people thought your tune was 5th best, all it would take is *one* person to vote a tune as 1st place for it to beat you - even if that tune was regarded as crap by everyone else.
This system fails to consider that 5th place still beat 7 or so over tunes. (depending on how many entries there are).

I am not even sure if "5,4,3,2,1" system is 100% accurate either. I believe that 3 x 5th votes should beat a 1 x 1st vote.

The other way to work is to score it based on the number of tunes that entry beat+1 . If there are 15 tunes into the compo, then "15,14,13,12,11" may work. This system rewards consistency. (Maybe even 15,13,11,9,7?).
User avatar
MickRip
Regular
Regular
 
Posts: 32
Joined: Wed Apr 26, 2006 6:31 pm
Location: Newcastle, Australia

Postby Sonicade » Tue May 30, 2006 12:53 am

We need a statistics major in here. I tried rerunning the results with the 5,4,3,2,1 system and this is what it looks like.

Place Author Title Filename Score
1st groovyone 80's State Of Mind sdc03-groovyone-80s-state-of-mind.rar 38
2nd mickrip Lingo Rollercoaster sdc3_MickRip-LingoRollercoaster.rar 37
3rd Harmony Dreamer sdc3_Harmony-Dreamer.zip 28
4th Sonicade The Break of Dawn sdc3_Sonicade-The_Break_Of_Dawn.rar 16
5th Necto_Ulin Ultrasonic motorway (Necto Ulin) sdc3_Necto_Ulin-Ultrasonic_Motorway.rar 15
6th overthruster itsukigoshi sdc03_OVERTHRUSTER_itskukigoshi.zip 14
7th aged grotesk party sdc3_aged-grotesk-party.rar 8
8th chotoro First permutation sdc03_Chotoro-First_permutation.zip 7
9th dj_status eBallad sdc3_DJStatus-eBallad.rar 6
10th dilvie Detour sdcROUND3_Dilvie-Detour.zip 3
10th robwilliamsjnr Careful Where You Land sdc3_RobWilliamsJnr_CarefulWhereYouLand.rar 3
10th fr0gr0ck EV CAFE sdc03-fr0gr0ck-EVCafe-.rar 3
11th dj_io unfinished mess hall sdc3_djio-unfinished-mess-hall.rar 2

Three people are tied for 10th place. Dilvie is (2,1), RobWilliamsJr is (1,1,1) and fr0gr0ck is (3).

I've noticed that the more you weight the points (ie: 16,8,4,2,1) the farther down the list you defer the ties. I'm not that much of a mathematician so this stuff makes my head hurt. The (5,4,3,2,1) system seems to reward consistency more but it may have issues of it's own and the triple tie concerns me.

I'm still undecided here. Does everyone feel the 5,4,3,2,1 system is fair?
User avatar
Sonicade
Compo Admin
Compo Admin
 
Posts: 773
Joined: Fri Mar 17, 2006 6:42 pm
Location: California, USA

Postby organic io » Tue May 30, 2006 9:04 am

Sonicade wrote:I'm still undecided here. Does everyone feel the 5,4,3,2,1 system is fair?


i'm fine with it, but i think that there will actually end up being more ties with it than less... simply because if the average numerical values are lower, the chances of more than 1 person getting the same score are higher.
User avatar
organic io
Compo Admin
Compo Admin
 
Posts: 1535
Joined: Thu Apr 06, 2006 11:55 am

Postby dilvie » Tue May 30, 2006 11:48 am

You left out the number of votes per entry component. You'll get ties with any single-factor scoring system. When you normalize the results, and multiply the normalized value with a normalized representation of the number of votes, ties become significantly less likely, without making significant changes to the outcome:

Total points

40
33
30
16
15
14
8
7
6
3
3
3
2

Points Normalized
(entry points / biggest score)

1
0.83
0.75
0.4
0.38
0.35
0.2
0.18
0.15
0.08
0.08
0.08
0.05

Votes

10
9
8
6
6
5
3
3
2
3
2
1
1

Votes Normalized
(entry votes / biggest vote score)

1
0.9
0.8
0.6
0.6
0.5
0.3
0.3
0.2
0.3
0.2
0.1
0.1

Scores
((points normalized * votes normalized) * 100)
100.00
74.25
60.00
24.00
22.50
17.50
6.00
5.25
3.00
2.25
1.50
0.75
0.50

Note: No ties, and no significant change in the outcome, either.
dilvie
Regular
Regular
 
Posts: 86
Joined: Tue May 16, 2006 4:40 am

Postby dilvie » Tue May 30, 2006 11:50 am

One more thing -- if an entry gets zero votes, just leave it out of the equation and tack it on to the end -- there's no way with this system to give it a meaningful score other than zero.
dilvie
Regular
Regular
 
Posts: 86
Joined: Tue May 16, 2006 4:40 am

Postby Sonicade » Tue May 30, 2006 3:58 pm

That looks great Dilvie. Do you think you could run down a basic scenario of maybe 3 votes and the math steps? I'm not seeing how your doing the normalization.
User avatar
Sonicade
Compo Admin
Compo Admin
 
Posts: 773
Joined: Fri Mar 17, 2006 6:42 pm
Location: California, USA

Postby dilvie » Tue May 30, 2006 5:35 pm

Sonicade wrote:That looks great Dilvie. Do you think you could run down a basic scenario of maybe 3 votes and the math steps? I'm not seeing how your doing the normalization.


Normalization is easy. First, you need to find the largest score in the group:

Song 1: 8
Song 2: 6
Song 3: 5

Song 1 is the largest, so the Biggest Score is 8.

Here are the normalized values:

Song 1: 8/8 = 1
Song 2: 6/8 = .75
Song 3: 5/8 = .63

Do the same thing for the number of votes:

Song 1: 3
Song 2: 2
Song 3: 3

Biggest score in this case is 3. Keep in mind that with more songs and more voters, the lowest scoring item is very unlikely to have the most votes. This is just for fun.

Song 1: 3/3 = 1
Song 2: 2/3 = .67
Song 3: 3/3 = 1

The last step multiplies the results. Since we want to build a scale of 0 - 100, we'll multiply the final result by 100, just for scaling purposes:

OOPS! I wrote Song 1: (1*.75)*100 = 75! This should be:
Song 1: (1*1)*100 = 100
Song 2: (.75*.67)*100 = 50.25
Song 3: (.63*1)*100 = 63

So the final ranks are:
Song 1: 75
Song 3: 63
Song 2: 50.25

Note that Song 3 takes 2nd place with a slightly lower point score, because it has more votes. Also note that the more votes there are, the less this factor will influence results, because it will hold better that more votes = more points. For the same reason, this is not likely to disturb the top ranks. The more participants and voters there are, the more this serves as a tie-breaker, rather than rank-maker.

If you prefer it to serve strictly as a tie breaker, you could put both values into a two dimensional array, and sort it by descending score first, and vote score second using a nested loop.

I personally prefer it as a factor, because it only matters when there are few votes, anyway, the scoring is less complex (you do a multiplication and get a neat combined score), and a better song is more likely to be recognized by more than one voter.
Last edited by dilvie on Fri Jun 02, 2006 3:55 pm, edited 2 times in total.
dilvie
Regular
Regular
 
Posts: 86
Joined: Tue May 16, 2006 4:40 am

Postby Sonicade » Tue May 30, 2006 8:03 pm

Great, I will consult this when I dig into the code again. Thanks Dilvie this is very helpful.
User avatar
Sonicade
Compo Admin
Compo Admin
 
Posts: 773
Joined: Fri Mar 17, 2006 6:42 pm
Location: California, USA
Next

Return to Feedback / Requests

Who is online

Users browsing this forum: No registered users and 1 guest