SRB2 Message Board

SRB2 Message Board (https://mb.srb2.org/index.php)
-   Outdated Official Level Design Contest (2.0.X) (https://mb.srb2.org/forumdisplay.php?f=99)
-   -   Voting Changes For July/August 2012 (https://mb.srb2.org/showthread.php?t=37096)

Mystic 07-03-2012 08:07 AM

Voting Changes For July/August 2012
 
Well, this is a bit early but it's safe to say we're going to be continuing the contest at least for a few more attempts, so for the next contest (July/August 2012), I'm giving everyone a heads-up that the voting system will be changing completely.

To vote, all you do is order the maps in a division in the order of preference, from best to worst. For purposes of demonstration, let's say that there are four maps in the division: A Zone, B Zone, C Zone, and D Zone, and you liked C, followed by B, A, and then finally D was the worst of the group. You would simply put the following at the top of your voting post:

C > B > A > D

With real zone names, using line breaks instead of ">" symbols is just fine, just as long as the best map is first and the worst map is last. That's all you need to do for your vote to be counted. It's still preferable if you also provide more detailed feedback to the author, but please separate it from the vote list so it's easier to tally the votes at the end. Instead of dropping the lowest vote on your map when you vote on the division you're entered in, you simply put your map at the top regardless of whether or not you think it's the best map in the division. This is the new benefit for voting, and you should always vote your own map at the top. Let's say that the author of map A liked maps C, D, A (his own map), and B, in that order. Author A would vote as follows:

A > C > D > B

Finally, at the end of voting period, the votes are tallied. Tallying votes is now simple addition. For each vote, the map ranked first gets n-1 points, where n is the number of maps in the division. Second place gets n-2 points, third gets n-3, all the way down to n-n (zero) points for last. For the two votes in our demonstration, here's how that would break down:

A Zone: (4-3) + (4-1) = 1 + 3 = 4
B Zone: (4-2) + (4-4) = 2 + 0 = 2
C Zone: (4-1) + (4-2) = 3 + 2 = 5
D Zone: (4-4) + (4-3) = 0 + 1 = 1

Should there be a tie, the votes are calculated again with only the tied maps included (C > A instead of C > B > A > D). If this still fails to break the tie, the tie stands and both maps are winners. Since C Zone has more points than the other maps, C Zone's author is the winner of the division.

I'm sure the question on a lot of your minds is "why?". Why change a system we've been using for just short of a decade? The simple answer is to make the system more fair, while also avoiding some of the drama that has plagued the OLDC voting in the past. The old system relied a lot on everyone voting correctly. Because it worked by averages, a single person voting dramatically differently from the rest of the voters skewed the scores overall quite heavily. In this system, the weight of each person's vote is exactly equal. There is no longer a need to scream at the guy who voted all 9s on the maps he liked and 2s on everything else because his vote will no longer have significantly more weight. This also closes several loopholes in the system that people could exploit if they really wanted to.

However, let's say everyone is, in good faith, attempting to vote correctly and provide unbiased, fair ratings to everyone. Even then, the system has had problems in the past because it's really, really hard to provide a number based on your experience. I'm sure everyone who has voted in the OLDC has had situations where they sit there for a while trying to figure out whether a map is a 7 or an 8, even when they voted every other map in the division a 5 or less. The difference between a 7 or 8 could even make a difference way down the line when the votes are tallied, so not only is it a hard decision, it could even be important. This eliminates the need for such complexities and boils it down to the core question of which map you liked the most. Considering the goal of voting is to determine which map the community liked the best overall, it's time we have a voting system that reflects that.

As a final reminder this does NOT apply to the May/June 2012 contest that starts voting tonight. If you have any questions or concerns, or if I missed something blatantly obvious in my explanation of the new rules, please reply. I want it to be extremely clear how the system works BEFORE we start using it in two months, so the more questions, the better.

Internet Explorer 07-03-2012 09:52 AM

It's true that some people were playing with giving 0 to maps which had not chance to win, and reciprocally 10 for the map supposed to win. So, that idea solved this annoying madness. And also...

Quote:

Originally Posted by Mystic (Post 727104)
Instead of dropping the lowest vote on your map when you vote on the division you're entered in, you simply put your map at the top regardless of whether or not you think it's the best map in the division.

...I think I see what you wanted to do.

The system is pretty complex, but you explained it correctly, -actually- it's not easy to explain it. On my side, I understood it well after a second reading, but if some didn't paid enough attention, I can't tell you what confusion will happen for some if this occurs~

So, in overall, it's a pretty good idea to solve problems about judging the author and related stuff, and, I think that after some contests, we will be able to handle it without any problems.

Forgive me if I repeat myself about something you've probably answered in your post, but I want to make everything clear on it : Since we have to vote according to "the best to the worth", we don't have to add the note anywhere in the post, right ?

I'll Begin 07-03-2012 01:58 PM

Sounds good to me!

Fred 07-03-2012 03:28 PM

Quote:

Originally Posted by Mystic (Post 727104)
C > B > A > D

Wow!

Quote:

Originally Posted by Mystic (Post 727104)
I'm sure the question on a lot of your minds is "why?".

Why is right! Why did it take so long

Well, who cares! At least now I actually have the will to vote on this contest's m--

Quote:

Originally Posted by Mystic (Post 727104)
this does NOT apply to the May/June 2012 contest that starts voting tonight

Oh.

Wow!

Shadow Hog 07-03-2012 03:43 PM

I'm kind of curious as to how the votes would've turned out if this new voting scheme had been implemented for this contest.

Monster Iestyn 07-03-2012 03:56 PM

This new system sounds a neat idea at first glance, but there's one thing there you seem to have completely missed out:

Where does the actual quality of the maps, including the top ones, fit into this?

I mean, all's well with showing which of the bunch of maps is the best, yes, but how on earth does that tell us if the map's really good enough to play for anyone who goes back through OLDCs to play the entries?

And for that matter, the current sort of leaderboards we have just wouldn't work anymore - the scores would all be affected by the number of people who actually voted for each contest instead, which again says nothing of the quality of the map.

As in, we could end up getting a bad map right at the top of a division leaderboard, and better maps far lower down, perhaps because the bad map by chance had more people deciding it was the best that OLDC than with the OLDC the better maps were in.

In short, in closing the loopholes with the current voting system, you've completely broken the current leaderboard system instead. Great job there.

Shadow Hog 07-03-2012 03:58 PM

All thing considered, those leaderboards were kind of meaningless, anyway. I mean, Tree Ring was good, but do you really think we haven't seen better maps in later contests by now? If you follow those leaderboards verbatim, we sure as hell haven't.

Monster Iestyn 07-03-2012 04:33 PM

Well, if you haven't noticed already on the SRB2 Wiki, the leaderboards for 2.0, 1.09.4, and pre-1.09.4 have been seperated for at least a year or two now, given the OLDC standards have changed over the years, yes.

And at the end of the day, this proposed new system means there is no quick way to compare a level's quality to another in the same division anymore - not without having to play the levels themselves again to decide that, anyway.

Thompson 07-03-2012 05:00 PM

Considering of this new voting system, unlike the other voting system, it doesn't express the best of maps enough because it depends so heavily on the number of voters you have in that contest. The other expressed maps based on average which was more fair. To be able to achieve a decent high score, it depends on a decent high number of voters, and we all know how inconsistent voters come to vote.

Maybe I'd suggest replacing the leadership board with a winning board, seeming it would be the only score board I find to fit for this voting system. But I guess we'll just have to see how this plays out in the end.

Fred 07-03-2012 05:41 PM

Quote:

Originally Posted by Shadow Hog (Post 727120)
I mean, Tree Ring was good

This is a common myth.

Quote:

Originally Posted by Monster Iestyn (Post 727119)
Where does the actual quality of the maps, including the top ones, fit into this?

I mean, all's well with showing which of the bunch of maps is the best, yes, but how on earth does that tell us if the map's really good enough to play for anyone who goes back through OLDCs to play the entries?

The contest has always operated under the system that a single entry be elected winner and triumph over all the others in its division. That's the whole point. However, it's impossible to have a voting system that both accurately determines who should be the winner AND meaningfully, yet succinctly informs about the quality of each of the entries.

Rather befuddlingly, for the better part of a decade, we've used a system that falls in the latter end of the spectrum, and since your average SRB2 forum goer isn't the most adept of judges, what we've ended up with is essentially a glorified "Like" button.

Quote:

Originally Posted by Monster Iestyn (Post 727119)
And for that matter, the current sort of leaderboards we have just wouldn't work anymore - the scores would all be affected by the number of people who actually voted for each contest instead, which again says nothing of the quality of the map.

As in, we could end up getting a bad map right at the top of a division leaderboard, and better maps far lower down, perhaps because the bad map by chance had more people deciding it was the best that OLDC than with the OLDC the better maps were in.

In short, in closing the loopholes with the current voting system, you've completely broken the current leaderboard system instead. Great job there.

http://www.theghz.com/gibbon/waaah.jpg

Rob 07-03-2012 05:58 PM

I don't know why there's issue with this new system. It makes sense.

This forces people to have to actually think on quality of maps in comparison to the others, which is the whole point of the contest in the first place. It makes complete and utter sense to do it this way because of the reasons stated in the opening post. This way there's no arbitrary score system that's completely subjective. This forces a vote that puts the overall best quality map in first place more often than not.

fickleheart 07-03-2012 07:31 PM

...Wait, people paid attention to the overall leaderboard?

I mean, yeah, getting to the top of said leaderboard was all fun and good, but I'm personally completely fine with seeing that go away in favor of a voting system that's easy to understand. I've always had issues as it is assigning numbers to maps, so judging everything comparatively works wonders for making it easier to say which map was the best.

The leaderboard was never fair anyway, since every single person who votes has a different idea of what constitutes a 3, 8, etc, which skewed the scores toward a certain range depending on who participated.

I am curious as to what happens in the case of two maps of equal quality, but from what I can recall off-hand from IRC discussions it'd be fine to just mark them with an = sign?

Mystic 07-03-2012 08:34 PM

Quote:

Originally Posted by Internet Explorer (Post 727105)
Since we have to vote according to "the best to the worth", we don't have to add the note anywhere in the post, right ?

You never had to provide reasoning for your vote. It is preferred but just putting a number is and always has been acceptable.

Quote:

Originally Posted by Katmint (Post 727118)
Sounds fine to me, but one question: what if you like two maps equally? In the current system you could just vote them the same score, but there doesn't appear to be a way to do it in the new system.

This is correct. You DO have to decide which one is better. It might be hard to decide sometimes, but considering how much easier this is than deciding between a 7 and an 8, I think this'll work itself out in time. Instead of having 11n options, you now have n-1 options for the entire division. This makes the decision tree a lot simpler when voting.

Quote:

Originally Posted by Monster Iestyn (Post 727119)
Where does the actual quality of the maps, including the top ones, fit into this?

It doesn't, and it never has. The leaderboards have always been a terrible comparison of quality because of the reasons I outlined above. Different voters have different opinions of what the numbers mean, and often equivalently decent maps end up with an average 1-2 apart simply because different people voted in the contest. I've said time and time again that you're using numbers that were never meant for statistical purposes and making them into statistics, generating a meaningless table.

I am actually glad this removes the leaderboard, as that was another unfair part of the OLDC system that's needed to be removed for a long time.

Ezer.Arch 07-04-2012 05:45 AM

Quote:

Originally Posted by Rob (Post 727127)
I don't know why there's issue with this new system. It makes sense.

It does make sense. Once we get rid of numbers, it will no longer be possible to compare a contest with another, knowing how awful or how great a contest was. We just need to elect the best and the worst of each contest. Simple.

In my opinion, let it go.

Internet Explorer 07-04-2012 06:40 AM

Quote:

Originally Posted by Mystic (Post 727137)
You never had to provide reasoning for your vote. It is preferred but just putting a number is and always has been acceptable

No wait, I'm not talking about the justification, but about the mark of a level, its note on 10. This note on 10 shouldn't appear ...?

Also, I noticed that this new system only solve the problem of people overusing numbers as "pushers of note on 10", but it replaces that with a system where each map is compared with the others, the maps depend of the others. That means they're framed at the same level, and the statistics disapear, since the quality disapears as well, if denoted to the individual quality scale.
So, the quality is also replaced with the quality according to the others levels. We just won't have an accurate and independant mark for the reviews on an individual map ; and on the other hand, we'll try to get closer to "The best map wins the contest" goal, that's it ?~

I'm okay with simplifying the choice of the mark, like "deciding between giving a 7 or a 8", but still, we lose some accuracy.

Mystic 07-04-2012 07:54 AM

Quote:

Originally Posted by Internet Explorer (Post 727158)
I'm okay with simplifying the choice of the mark, like "deciding between giving a 7 or a 8", but still, we lose some accuracy.

This is the thing I'm really trying to point out. We're GAINING accuracy. We are losing precision. Let's imagine you have two thermometers and one reads the temperature at 23 and the other reads it as -12.239. The actual temperature is 22.621. Which thermometer do you want, the one that provides the most digits (precision) or the one that provides the closest estimate to the actual temperature (accuracy)?

In the old system, it was very easy for the numbers to be thrown wildly off by only a few people. You could get a precise answer with tons of numbers (we rounded to 2), but for all that precision, it wasn't a very accurate representation of the overall opinion of the community. Because of how averages worked, the people with more extreme opinions had their votes count more than the people whose opinions were in line with the rest of the community. Let's take the following example of a group of votes on a single map on the old system:

0, 1, 1, 1, 1, 1, 1, 2

Overall, the votes here are very consistent. The average is 1 for this set of votes. Now imagine a 10th voter comes in and votes a 10. The average suddenly goes up to 2. This one voter changed the score by a full point. Now let's imagine someone else comes and votes a 1. The score now drops to 1.9. The voter that voted the 10 and the voter who voted a 1 are both people and should be counted equally, but the system is counting the voter that voted 10 to be ten times more important than the voter who voted a 1. Why should any one person's opinion be more important than everyone else?

While the new system lacks decimals and cool (irrelevant) statistics comparing all the maps we've ever made, it has the advantage of actually making every voter equal under the system, and therefore the end result will actually be an accurate representation of the overall opinions of the community. I think that's infinitely more important than any decimal point could be.

ZarroTsu 07-05-2012 04:44 AM

It's really quite simple.

If you want to let people know about the quality of a map, say it in your post.

All (or at least most) of the past OLDC topics are linked to by the wiki. If people want to be dorks and just line up their A's and B's, then they can fistbump with the SuperChris'es. If people want to actually explain WHY it's better or worse, then they can get to work making SpiritCrusher blush.

Not that difficult a concept to grasp. Unless, you know, you're 80% of the forum and don't explain your vote.

Quote:

<@Prime_2> The issue isn't that the information on which maps or contests are good isn't available, it's about convenience of that availability
<ZarroTsu> then fucking quip witten reviews on the wiki page
<ZarroTsu> just single-sentence quotes
<ZarroTsu> and put them in mouse-over explaination boxes
<ZarroTsu> if that's too inconvenient I can only question how they even know the contest exists
<@Prime_2> Great! That's a productive suggestion!

MascaraSnake 07-08-2012 08:44 PM

Quote:

Originally Posted by ZarroTsu (Post 727188)
If you want to let people know about the quality of a map, say it in your post.

Actually, if you want to let people know about the absolute (as opposed to relative) quality of a map, why don't you just give a score on the 0-10 scale anyway, even if it won't count in the results? I know that's what I will do, because even though I fully support the new tallying system, I still want the mappers to know how much I liked their map on an absolute scale, not just in relation to the other ones. The 0-10 scale is an excellent way of summarizing your opinions in a concise number, it's just when these numbers are tallied and averaged that the problems start.

And regarding the leaderboard: The leaderboard has always been a "fun fact" kind of thing. It has no actual informational value. In fact, once the next contest starts, I will remove the leaderboards from the OLDC article altogether.

CobaltBW 08-02-2012 11:40 PM

Is the voter allowed to tie his/her votes for certain maps? For instance, can they send their vote count as 1,2,2,3 instead of 1,2,3,4? I'd like to be able to vote for tie placements in case I think two maps are of equal quality, which happens quite often.

Charybdizs 08-02-2012 11:46 PM

Theoretically, it should be incredibly rare that any maps are actually equal in quality. I'm sure you can make up your mind. :)


All times are GMT. The time now is 01:54 PM.

Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2019, vBulletin Solutions, Inc.