SRB2 Message Board  

Go Back   SRB2 Message Board > Archived > Outdated Releases (2.0.X) > Outdated Official Level Design Contest (2.0.X)

 
 
Thread Tools Search this Thread
Old 07-03-2012   #1
Mystic
チェン!
Administrator
 
Mystic's Avatar
Default Voting Changes For July/August 2012

Well, this is a bit early but it's safe to say we're going to be continuing the contest at least for a few more attempts, so for the next contest (July/August 2012), I'm giving everyone a heads-up that the voting system will be changing completely.

To vote, all you do is order the maps in a division in the order of preference, from best to worst. For purposes of demonstration, let's say that there are four maps in the division: A Zone, B Zone, C Zone, and D Zone, and you liked C, followed by B, A, and then finally D was the worst of the group. You would simply put the following at the top of your voting post:

C > B > A > D

With real zone names, using line breaks instead of ">" symbols is just fine, just as long as the best map is first and the worst map is last. That's all you need to do for your vote to be counted. It's still preferable if you also provide more detailed feedback to the author, but please separate it from the vote list so it's easier to tally the votes at the end. Instead of dropping the lowest vote on your map when you vote on the division you're entered in, you simply put your map at the top regardless of whether or not you think it's the best map in the division. This is the new benefit for voting, and you should always vote your own map at the top. Let's say that the author of map A liked maps C, D, A (his own map), and B, in that order. Author A would vote as follows:

A > C > D > B

Finally, at the end of voting period, the votes are tallied. Tallying votes is now simple addition. For each vote, the map ranked first gets n-1 points, where n is the number of maps in the division. Second place gets n-2 points, third gets n-3, all the way down to n-n (zero) points for last. For the two votes in our demonstration, here's how that would break down:

A Zone: (4-3) + (4-1) = 1 + 3 = 4
B Zone: (4-2) + (4-4) = 2 + 0 = 2
C Zone: (4-1) + (4-2) = 3 + 2 = 5
D Zone: (4-4) + (4-3) = 0 + 1 = 1

Should there be a tie, the votes are calculated again with only the tied maps included (C > A instead of C > B > A > D). If this still fails to break the tie, the tie stands and both maps are winners. Since C Zone has more points than the other maps, C Zone's author is the winner of the division.

I'm sure the question on a lot of your minds is "why?". Why change a system we've been using for just short of a decade? The simple answer is to make the system more fair, while also avoiding some of the drama that has plagued the OLDC voting in the past. The old system relied a lot on everyone voting correctly. Because it worked by averages, a single person voting dramatically differently from the rest of the voters skewed the scores overall quite heavily. In this system, the weight of each person's vote is exactly equal. There is no longer a need to scream at the guy who voted all 9s on the maps he liked and 2s on everything else because his vote will no longer have significantly more weight. This also closes several loopholes in the system that people could exploit if they really wanted to.

However, let's say everyone is, in good faith, attempting to vote correctly and provide unbiased, fair ratings to everyone. Even then, the system has had problems in the past because it's really, really hard to provide a number based on your experience. I'm sure everyone who has voted in the OLDC has had situations where they sit there for a while trying to figure out whether a map is a 7 or an 8, even when they voted every other map in the division a 5 or less. The difference between a 7 or 8 could even make a difference way down the line when the votes are tallied, so not only is it a hard decision, it could even be important. This eliminates the need for such complexities and boils it down to the core question of which map you liked the most. Considering the goal of voting is to determine which map the community liked the best overall, it's time we have a voting system that reflects that.

As a final reminder this does NOT apply to the May/June 2012 contest that starts voting tonight. If you have any questions or concerns, or if I missed something blatantly obvious in my explanation of the new rules, please reply. I want it to be extremely clear how the system works BEFORE we start using it in two months, so the more questions, the better.
Mystic is offline  
Old 07-03-2012   #2
Internet Explorer
IT'S NO USE
 
Internet Explorer's Avatar
Default

It's true that some people were playing with giving 0 to maps which had not chance to win, and reciprocally 10 for the map supposed to win. So, that idea solved this annoying madness. And also...

Quote:
Originally Posted by Mystic View Post
Instead of dropping the lowest vote on your map when you vote on the division you're entered in, you simply put your map at the top regardless of whether or not you think it's the best map in the division.
...I think I see what you wanted to do.

The system is pretty complex, but you explained it correctly, -actually- it's not easy to explain it. On my side, I understood it well after a second reading, but if some didn't paid enough attention, I can't tell you what confusion will happen for some if this occurs~

So, in overall, it's a pretty good idea to solve problems about judging the author and related stuff, and, I think that after some contests, we will be able to handle it without any problems.

Forgive me if I repeat myself about something you've probably answered in your post, but I want to make everything clear on it : Since we have to vote according to "the best to the worth", we don't have to add the note anywhere in the post, right ?
__________________
Bind out of the "life"
Curiosity : It's alright, go on, you'll never know anything if you don't try

Last edited by Internet Explorer; 07-03-2012 at 09:56 AM. Reason: Some English grammar~
Internet Explorer is offline  
Old 07-03-2012   #3
I'll Begin
 
I'll Begin's Avatar
Default

Sounds good to me!
I'll Begin is offline  
Old 07-03-2012   #4
Fred
Retired
 
Fred's Avatar
Default

Quote:
Originally Posted by Mystic View Post
C > B > A > D
Wow!

Quote:
Originally Posted by Mystic View Post
I'm sure the question on a lot of your minds is "why?".
Why is right! Why did it take so long

Well, who cares! At least now I actually have the will to vote on this contest's m--

Quote:
Originally Posted by Mystic View Post
this does NOT apply to the May/June 2012 contest that starts voting tonight
Oh.

Wow!
Fred is offline  
Old 07-03-2012   #5
Shadow Hog
Administrator
 
Shadow Hog's Avatar
Default

I'm kind of curious as to how the votes would've turned out if this new voting scheme had been implemented for this contest.
Shadow Hog is offline  
Old 07-03-2012   #6
Monster Iestyn
Friendly Neighbour-Hood
Moderator
 
Monster Iestyn's Avatar
Default

This new system sounds a neat idea at first glance, but there's one thing there you seem to have completely missed out:

Where does the actual quality of the maps, including the top ones, fit into this?

I mean, all's well with showing which of the bunch of maps is the best, yes, but how on earth does that tell us if the map's really good enough to play for anyone who goes back through OLDCs to play the entries?

And for that matter, the current sort of leaderboards we have just wouldn't work anymore - the scores would all be affected by the number of people who actually voted for each contest instead, which again says nothing of the quality of the map.

As in, we could end up getting a bad map right at the top of a division leaderboard, and better maps far lower down, perhaps because the bad map by chance had more people deciding it was the best that OLDC than with the OLDC the better maps were in.

In short, in closing the loopholes with the current voting system, you've completely broken the current leaderboard system instead. Great job there.

Last edited by Monster Iestyn; 07-03-2012 at 04:29 PM.
Monster Iestyn is offline  
Old 07-03-2012   #7
Shadow Hog
Administrator
 
Shadow Hog's Avatar
Default

All thing considered, those leaderboards were kind of meaningless, anyway. I mean, Tree Ring was good, but do you really think we haven't seen better maps in later contests by now? If you follow those leaderboards verbatim, we sure as hell haven't.
Shadow Hog is offline  
Old 07-03-2012   #8
Monster Iestyn
Friendly Neighbour-Hood
Moderator
 
Monster Iestyn's Avatar
Default

Well, if you haven't noticed already on the SRB2 Wiki, the leaderboards for 2.0, 1.09.4, and pre-1.09.4 have been seperated for at least a year or two now, given the OLDC standards have changed over the years, yes.

And at the end of the day, this proposed new system means there is no quick way to compare a level's quality to another in the same division anymore - not without having to play the levels themselves again to decide that, anyway.

Last edited by Monster Iestyn; 07-03-2012 at 04:36 PM.
Monster Iestyn is offline  
Old 07-03-2012   #9
Thompson
Endokukennnn
 
Thompson's Avatar
Default

Considering of this new voting system, unlike the other voting system, it doesn't express the best of maps enough because it depends so heavily on the number of voters you have in that contest. The other expressed maps based on average which was more fair. To be able to achieve a decent high score, it depends on a decent high number of voters, and we all know how inconsistent voters come to vote.

Maybe I'd suggest replacing the leadership board with a winning board, seeming it would be the only score board I find to fit for this voting system. But I guess we'll just have to see how this plays out in the end.
__________________
Solar Egglipse in progress.
Thompson is offline  
Old 07-03-2012   #10
Fred
Retired
 
Fred's Avatar
Default

Quote:
Originally Posted by Shadow Hog View Post
I mean, Tree Ring was good
This is a common myth.

Quote:
Originally Posted by Monster Iestyn View Post
Where does the actual quality of the maps, including the top ones, fit into this?

I mean, all's well with showing which of the bunch of maps is the best, yes, but how on earth does that tell us if the map's really good enough to play for anyone who goes back through OLDCs to play the entries?
The contest has always operated under the system that a single entry be elected winner and triumph over all the others in its division. That's the whole point. However, it's impossible to have a voting system that both accurately determines who should be the winner AND meaningfully, yet succinctly informs about the quality of each of the entries.

Rather befuddlingly, for the better part of a decade, we've used a system that falls in the latter end of the spectrum, and since your average SRB2 forum goer isn't the most adept of judges, what we've ended up with is essentially a glorified "Like" button.

Quote:
Originally Posted by Monster Iestyn View Post
And for that matter, the current sort of leaderboards we have just wouldn't work anymore - the scores would all be affected by the number of people who actually voted for each contest instead, which again says nothing of the quality of the map.

As in, we could end up getting a bad map right at the top of a division leaderboard, and better maps far lower down, perhaps because the bad map by chance had more people deciding it was the best that OLDC than with the OLDC the better maps were in.

In short, in closing the loopholes with the current voting system, you've completely broken the current leaderboard system instead. Great job there.

(Fred received a warning for this post: Act like a mature adult. Use some common sense when posting.)

Last edited by Fred; 07-03-2012 at 05:43 PM. Reason: covering my own ass here
Fred is offline  
Old 07-03-2012   #11
Rob
Administrator
 
Rob's Avatar
Default

I don't know why there's issue with this new system. It makes sense.

This forces people to have to actually think on quality of maps in comparison to the others, which is the whole point of the contest in the first place. It makes complete and utter sense to do it this way because of the reasons stated in the opening post. This way there's no arbitrary score system that's completely subjective. This forces a vote that puts the overall best quality map in first place more often than not.
Rob is offline  
Old 07-03-2012   #12
RedEnchilada
happy, pretty, lovely!
Developer
 
RedEnchilada's Avatar
Default

...Wait, people paid attention to the overall leaderboard?

I mean, yeah, getting to the top of said leaderboard was all fun and good, but I'm personally completely fine with seeing that go away in favor of a voting system that's easy to understand. I've always had issues as it is assigning numbers to maps, so judging everything comparatively works wonders for making it easier to say which map was the best.

The leaderboard was never fair anyway, since every single person who votes has a different idea of what constitutes a 3, 8, etc, which skewed the scores toward a certain range depending on who participated.

I am curious as to what happens in the case of two maps of equal quality, but from what I can recall off-hand from IRC discussions it'd be fine to just mark them with an = sign?
__________________
Quote:
<@toaster> '"sonic robot explosion 2", the hot new sensation for teens looking for a good time, tempts many children away from god each year'

Last edited by RedEnchilada; 07-03-2012 at 07:33 PM.
RedEnchilada is offline  
Old 07-03-2012   #13
Mystic
チェン!
Administrator
 
Mystic's Avatar
Default

Quote:
Originally Posted by Internet Explorer View Post
Since we have to vote according to "the best to the worth", we don't have to add the note anywhere in the post, right ?
You never had to provide reasoning for your vote. It is preferred but just putting a number is and always has been acceptable.

Quote:
Originally Posted by Katmint View Post
Sounds fine to me, but one question: what if you like two maps equally? In the current system you could just vote them the same score, but there doesn't appear to be a way to do it in the new system.
This is correct. You DO have to decide which one is better. It might be hard to decide sometimes, but considering how much easier this is than deciding between a 7 and an 8, I think this'll work itself out in time. Instead of having 11n options, you now have n-1 options for the entire division. This makes the decision tree a lot simpler when voting.

Quote:
Originally Posted by Monster Iestyn View Post
Where does the actual quality of the maps, including the top ones, fit into this?
It doesn't, and it never has. The leaderboards have always been a terrible comparison of quality because of the reasons I outlined above. Different voters have different opinions of what the numbers mean, and often equivalently decent maps end up with an average 1-2 apart simply because different people voted in the contest. I've said time and time again that you're using numbers that were never meant for statistical purposes and making them into statistics, generating a meaningless table.

I am actually glad this removes the leaderboard, as that was another unfair part of the OLDC system that's needed to be removed for a long time.
Mystic is offline  
Old 07-04-2012   #14
Ezer.Arch
ArchPack developer
 
Ezer.Arch's Avatar
Default

Quote:
Originally Posted by Rob View Post
I don't know why there's issue with this new system. It makes sense.
It does make sense. Once we get rid of numbers, it will no longer be possible to compare a contest with another, knowing how awful or how great a contest was. We just need to elect the best and the worst of each contest. Simple.

In my opinion, let it go.
__________________
Ezer'Arch
ArchPack v2.1 level pack | EzerArch.com | YouTube | ERZ2 FTW!

Last edited by Ezer.Arch; 07-05-2012 at 10:58 AM.
Ezer.Arch is offline  
Old 07-04-2012   #15
Internet Explorer
IT'S NO USE
 
Internet Explorer's Avatar
Default

Quote:
Originally Posted by Mystic View Post
You never had to provide reasoning for your vote. It is preferred but just putting a number is and always has been acceptable
No wait, I'm not talking about the justification, but about the mark of a level, its note on 10. This note on 10 shouldn't appear ...?

Also, I noticed that this new system only solve the problem of people overusing numbers as "pushers of note on 10", but it replaces that with a system where each map is compared with the others, the maps depend of the others. That means they're framed at the same level, and the statistics disapear, since the quality disapears as well, if denoted to the individual quality scale.
So, the quality is also replaced with the quality according to the others levels. We just won't have an accurate and independant mark for the reviews on an individual map ; and on the other hand, we'll try to get closer to "The best map wins the contest" goal, that's it ?~

I'm okay with simplifying the choice of the mark, like "deciding between giving a 7 or a 8", but still, we lose some accuracy.
__________________
Bind out of the "life"
Curiosity : It's alright, go on, you'll never know anything if you don't try

Last edited by Internet Explorer; 07-04-2012 at 06:53 AM. Reason: Quality, quality, quality.
Internet Explorer is offline  
Old 07-04-2012   #16
Mystic
チェン!
Administrator
 
Mystic's Avatar
Default

Quote:
Originally Posted by Internet Explorer View Post
I'm okay with simplifying the choice of the mark, like "deciding between giving a 7 or a 8", but still, we lose some accuracy.
This is the thing I'm really trying to point out. We're GAINING accuracy. We are losing precision. Let's imagine you have two thermometers and one reads the temperature at 23 and the other reads it as -12.239. The actual temperature is 22.621. Which thermometer do you want, the one that provides the most digits (precision) or the one that provides the closest estimate to the actual temperature (accuracy)?

In the old system, it was very easy for the numbers to be thrown wildly off by only a few people. You could get a precise answer with tons of numbers (we rounded to 2), but for all that precision, it wasn't a very accurate representation of the overall opinion of the community. Because of how averages worked, the people with more extreme opinions had their votes count more than the people whose opinions were in line with the rest of the community. Let's take the following example of a group of votes on a single map on the old system:

0, 1, 1, 1, 1, 1, 1, 2

Overall, the votes here are very consistent. The average is 1 for this set of votes. Now imagine a 10th voter comes in and votes a 10. The average suddenly goes up to 2. This one voter changed the score by a full point. Now let's imagine someone else comes and votes a 1. The score now drops to 1.9. The voter that voted the 10 and the voter who voted a 1 are both people and should be counted equally, but the system is counting the voter that voted 10 to be ten times more important than the voter who voted a 1. Why should any one person's opinion be more important than everyone else?

While the new system lacks decimals and cool (irrelevant) statistics comparing all the maps we've ever made, it has the advantage of actually making every voter equal under the system, and therefore the end result will actually be an accurate representation of the overall opinions of the community. I think that's infinitely more important than any decimal point could be.
Mystic is offline  
Old 07-05-2012   #17
ZarroTsu
Lead Kart Dev
 
ZarroTsu's Avatar
Default

It's really quite simple.

If you want to let people know about the quality of a map, say it in your post.

All (or at least most) of the past OLDC topics are linked to by the wiki. If people want to be dorks and just line up their A's and B's, then they can fistbump with the SuperChris'es. If people want to actually explain WHY it's better or worse, then they can get to work making SpiritCrusher blush.

Not that difficult a concept to grasp. Unless, you know, you're 80% of the forum and don't explain your vote.

Quote:
<@Prime_2> The issue isn't that the information on which maps or contests are good isn't available, it's about convenience of that availability
<ZarroTsu> then fucking quip witten reviews on the wiki page
<ZarroTsu> just single-sentence quotes
<ZarroTsu> and put them in mouse-over explaination boxes
<ZarroTsu> if that's too inconvenient I can only question how they even know the contest exists
<@Prime_2> Great! That's a productive suggestion!

Last edited by ZarroTsu; 07-05-2012 at 05:08 AM.
ZarroTsu is offline  
Old 07-08-2012   #18
MascaraSnake
aka SpiritCrusher
Moderator
 
MascaraSnake's Avatar
Default

Quote:
Originally Posted by ZarroTsu View Post
If you want to let people know about the quality of a map, say it in your post.
Actually, if you want to let people know about the absolute (as opposed to relative) quality of a map, why don't you just give a score on the 0-10 scale anyway, even if it won't count in the results? I know that's what I will do, because even though I fully support the new tallying system, I still want the mappers to know how much I liked their map on an absolute scale, not just in relation to the other ones. The 0-10 scale is an excellent way of summarizing your opinions in a concise number, it's just when these numbers are tallied and averaged that the problems start.

And regarding the leaderboard: The leaderboard has always been a "fun fact" kind of thing. It has no actual informational value. In fact, once the next contest starts, I will remove the leaderboards from the OLDC article altogether.
MascaraSnake is offline  
Old 08-02-2012   #19
CobaltBW
I do things sometimes
 
CobaltBW's Avatar
Default

Is the voter allowed to tie his/her votes for certain maps? For instance, can they send their vote count as 1,2,2,3 instead of 1,2,3,4? I'd like to be able to vote for tie placements in case I think two maps are of equal quality, which happens quite often.
__________________
~CobaltBW

Check out my soundcloud profile for music stuff
CobaltBW is offline  
Old 08-02-2012   #20
Charybdizs
Excelsior
Judge
 
Charybdizs's Avatar
Default

Theoretically, it should be incredibly rare that any maps are actually equal in quality. I'm sure you can make up your mind. :)
Charybdizs is offline  
 

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT. The time now is 09:43 AM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2017, vBulletin Solutions, Inc.