no more secrets Forum Index no more secrets
security consultancy forum
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

Scoring and the Leaderboard

 
Post new topic   Reply to topic    no more secrets Forum Index -> NMS Discussion
View previous topic :: View next topic  
Author Message
CrystalChen-GC



Joined: 21 May 2007
Posts: 7

PostPosted: Tue May 22, 2007 3:13 pm    Post subject: Re: Awards! Reply with quote

richb wrote:

That sounds awesome! I think it might have been even more awesome still to have allowed teams to see the leaderboard during the game as well. Did you ever consider doing that? If so, why didn't you?


Haha, why am I not surprised that Rich is excited about the statistics? Smile

I think there are different philosophies on this, but I tend to be more of the mindset that teams should focus on solving puzzles and enjoying themselves rather than obsessing over their placement among other teams. I, for one, would rather not know Smile
Back to top
View user's profile Send private message
richb



Joined: 22 May 2007
Posts: 8

PostPosted: Tue May 22, 2007 3:58 pm    Post subject: Re: Awards! Reply with quote

CrystalChen-GC wrote:

Haha, why am I not surprised that Rich is excited about the statistics? Smile

I think there are different philosophies on this, but I tend to be more of the mindset that teams should focus on solving puzzles and enjoying themselves rather than obsessing over their placement among other teams. I, for one, would rather not know Smile


Yeah, I guess I'm getting a too bit predictable in that regard. Smile I just really believe that teams shouldn't feel bad about enjoying the competitive aspects of The Game, since I don't believe it is in the least bit mutually exclusive from the other enjoyable things about the experience that you mentioned. I would hope that teams that really just didn't care about that sort of thing would just not bother to look at the leaderboard, but perhaps the temptation is just too great...

But the, of course, I can't rule out the possibility that maybe I'm just trying to make myself feel better about enjoying the competitive aspects, when really I should get with program and accept that this is just not supposed to be a competitive venture, and I should save that for the athletic field. Smile
Back to top
View user's profile Send private message
JustinSantamaria-GC
Site Admin


Joined: 14 May 2007
Posts: 4
Location: Mountain View

PostPosted: Tue May 22, 2007 4:28 pm    Post subject: Re: Awards! Reply with quote

richb wrote:
I think it might have been even more awesome still to have allowed teams to see the leaderboard during the game as well. Did you ever consider doing that? If so, why didn't you?


At GC, we talked about this possibility -- with the right URL and username/password, you certainly could have checked out the leaderboard in realtime.

We had made a distinct decision to keep the competition aspects of this game down to a minimum while playing (we intentionally never explained our scoring system), but still had many debates (it was probably mainly between Yar and myself) over how to score this thing.

There's a balance when it comes to 'scoring' an event such as this, and we may have erred on the 'warm and fuzzy' side than the 'hardcore competition' side -- but in the end our philosophy was to allow teams to have as much fun as possible while still having a coherent scoring mechanism.
Back to top
View user's profile Send private message Visit poster's website AIM Address
richb



Joined: 22 May 2007
Posts: 8

PostPosted: Tue May 22, 2007 4:47 pm    Post subject: Re: Awards! Reply with quote

JustinSantamaria-GC wrote:

There's a balance when it comes to 'scoring' an event such as this, and we may have erred on the 'warm and fuzzy' side than the 'hardcore competition' side -- but in the end our philosophy was to allow teams to have as much fun as possible while still having a coherent scoring mechanism.


For the record, I'm actually really happy with the balance you came up with. Smile
Back to top
View user's profile Send private message
YARRRRR-GC



Joined: 21 May 2007
Posts: 10

PostPosted: Tue May 22, 2007 4:53 pm    Post subject: Reply with quote

Quote:
But the, of course, I can't rule out the possibility that maybe I'm just trying to make myself feel better about enjoying the competitive aspects, when really I should get with program and accept that this is just not supposed to be a competitive venture


I don't think there's anything wrong with enjoying the competitive aspects of the Game... I enjoy them a lot (in comparison to the rest of my team, it seems...)! But I do think it's easier for competitive people to enjoy themselves in a fuzzy Game than it is for fuzzy people to enjoy themselves in a rigidly competitive Game (IF YOU DO NOT SOLVE THIS CLUE IN TWO HOURS YOU WILL BE ELIMINATED).
Back to top
View user's profile Send private message
red



Joined: 31 May 2007
Posts: 10

PostPosted: Thu May 31, 2007 9:33 am    Post subject: Hmmm Reply with quote

Our team philosophy is more anti-scoring than anything -- you can see by our bottom-of-the-barrel score. While we choked on some clues (refusing to call for well over an hour), we felt that we rocked others. Our team was as cohesive as ever and not once did we 'have words' with one-another. I'd definitely play with the same group again -- regardless of leader boards, scoring systems, or pack position.

Most important was that we had a great time, and hope that we helped teams around us have fun as well. Laughing

Scoring systems never really properly account for how long a team chooses to work on a puzzle before they call (a *big* variable), whether they choose to work with/against and/or help/hinder other teams, what type of help they ask for, how much food they provide the GC helpers along the way, or for other random variables (tampered clues, broken vans, etc) Speaking of which, shouldn't we get bonus points for solving the "van swap" clue without missing a beat. Wink

As a self-evaluation metric, I think we'd prefer to see the number of calls for each given clue on the board. We'd also prefer to have team names encoded -- simply to head off the trend of the Game heading further down the road to being more and more of a competitive puzzle challenge. Competition can make people do crazy/unfriendly things. Since this historically has been a gentleman's Game, we'll be the anti-competition at the back-of-the-bus doing the zany stuff. Y'all are welcome to join us anytime!

We were bummed to have been skipped by clues -- especially the MS Text adventure. We kept getting sent to field offices within 15 minutes of their closing, and by the time we found a seat, we were moving on.... We'd much prefer the opportunity to solve all the clues at our pace and help level and come in hours later having played our Game.

My 2 cents
-Red
Orange Snood (Snood + Orange Crush)
Back to top
View user's profile Send private message Visit poster's website
DaleNeal-GC



Joined: 21 May 2007
Posts: 14

PostPosted: Thu May 31, 2007 4:19 pm    Post subject: Reply with quote

Hi Red,

I re-read this post just now before submitting it, and it's probably way more information than you cared to know. But, you brought up a number of topics (competitiveness, scoring, skipping, etc) that various people on coed astronomy care a lot about and that we've discussed and debated at great length.

Our philosophy when putting together NMS was to try to make a game that was fun to play, and which allowed teams to be competitive *if they wanted to*, but that didn't require it. In particular, we tried to offload the competition aspect onto the field offices rather than the main route, and we intentionally did not bring up any aspects of scoring prior to or during the game. Our aim was to allow teams to choose their own level of competitiveness and take as many or as few hints as they felt like on the main route clues.

With respect to the leaderboard and scoring, I definitely debated with myself about publishing all of the data. Any scoring system we could come up with would be imperfect for all of the reasons you mentioned and more. My hope is that the leaderboard data can be used both for competitive teams to "see how well they did" as well as for all teams to better understand their own solving abilities. I think it's neat to have the data to see things like which puzzle types you did better than average on, or whether you get more affected by sleepiness and late night solving than other teams.

Encoding the team names is an idea I hadn't thought of, and the cat's already out of the bag on this one, so to speak. I will consider it in the future if coed astronomy is ever crazy enough to run a game again. It seems to potentially be a nice balance, as it allows teams who do want to be competitive to reveal their times while allowing all teams to see their times within the context of all of the other teams.

Regarding skipping, we're very sensitive to it, and I think NMS had one of the lowest skip rates of any game I can recall, in part due to the field offices. As I mentioned in a previous thread, teams saw much more of the field offices than we expected, and we didn't intend for teams to be repeatedly arriving 15 min before closing.

Our skips were not imposed out of any desire to force teams to be competitive or solve clues more quickly, but out of purely logistical constraints. Unfortunately, a number of our clues had very hard cutoff times caused by a number of factors--public parks close any time between 7 and 10pm, Marini's Candy closes at 11pm, rangers kick you out of Cuesta park if you're running a clue after 11am, there was a birthday party at Twister's Gym after us, etc. We felt especially bad about skipping you on the Microsoft Text Adventure clue, but it had high enough staffing requirements that most of core GC would have had to skip our own end-party to continue to staff that clue.

There is an argument that we could have restructured our route to use only sites with low staffing demands that we could have kept open all weekend. We could also have had staggered finishing times, which I feel takes away from the positive social aspect of hanging out with all of the other teams. This would have been a very different, and in my mind at least, a less compelling game, but I understand that there is a very large variance in what different people and different teams want out of a game.

Lastly, regarding per-clue GC help line call volume, we recorded this manually, so we don't have a very accurate count. I can try to get you some numbers on that when I get home tonight, though.

So that's my 3 cents or thereabouts. I'll shut up now. =)

-Dale
_________________
AnEelLad
Back to top
View user's profile Send private message Send e-mail Visit poster's website
red



Joined: 31 May 2007
Posts: 10

PostPosted: Thu May 31, 2007 4:37 pm    Post subject: Reply with quote

DaleNeal-GC wrote:
and we intentionally did not bring up any aspects of scoring prior to or during the game. Our aim was to allow teams to choose their own level of competitiveness and take as many or as few hints as they felt like on the main route clues.


Thanks for the detailed response, actually. We were very glad not to have heard about the scoring thing, actually. It's hard to explain, but we have a very visceral and negative reaction to the various scoring systems and forced help systems that have been imposed in the past -- even on Games where we have 'placed' in the top 5. Your approach of "not telling" is appreciated.

That said, your help was *PERFECT* for our needs and every single member of our team appreciated that you would not give more than we asked for (even at the expense of watching us increase the spread). We definitely appreciated the way in which you provided help and had a great time talking on the phone with you guys [I swear that one of those anagrams worked out to BOLTCUTTERS].

Quote:
... I think it's neat to have the data to see things like which puzzle types you did better than average on, or whether you get more affected by sleepiness and late night solving than other teams....


I actually like using the leader board to confirm what we kind of already knew (we rocked, we choked, etc). Typically I look at number of calls (not time) required for us to solve a clue. For instance, if it takes teams on average 4 calls but we did it in 2, I would be happy. Likewise, if it takes us more than the average -- I wonder what we did wrong (groupthink, sleepiness, etc). If we used the same number of calls as the average but it took us longer, I would assume we waited too long to call in the first place. So, the leader board is definitely appreciated from our end for self-evaluation.

Quote:
...if coed astronomy is ever crazy enough to run a game again....


Please please run another Game. You guys brought some great innovations and a unique style to NMS. I would love to see what you guys had in store for next time.

Quote:
So that's my 3 cents or thereabouts.


True that. GC gets more cents than teams!
Back to top
View user's profile Send private message Visit poster's website
sparCKL



Joined: 21 May 2007
Posts: 14
Location: Mountain View, CA

PostPosted: Fri Jun 01, 2007 11:33 am    Post subject: Reply with quote

I'll second what Red said:
Quote:
Please please run another Game. You guys brought some great innovations and a unique style to NMS. I would love to see what you guys had in store for next time.

This was a great event, and Snout had tons of fun! We'd also love to see more from coed astronomy.

Re: scoring: We tried something different with Hogwarts, which was only possible because of the theme (four houses, house points, etc.), and we did obfuscate team names when we published the final results, even if the encoding was rather trivial. I don't think our scheme was entirely successful; among the unintended side effects, we heard from at least one team who were annoyed that the other teams in their house kept trying to help them solve their clues. Smile

(We also experimented in the Dry Run with awarding arbitrary bonus points at the banquet--as happens in the Harry Potter books--so that Gryffindor would win no matter what, but that flew like a ton of bricks, so we just left it alone for the actual event--which turned out to be even more dramatic than we could have planned!)

I suspect that one-point-per-solved-clue isn't quite enough scoring granularity for the more competitive teams. We actually met with Rich Bragg before Hogwarts to discuss whether tracking more detailed statistics would be a sufficient substitute for a single overall score per team, but didn't spend enough time developing that idea. Maybe another GC will pick it up and do something interesting with it...
_________________
I am not an aardvark.
http://snout.org/stories
Back to top
View user's profile Send private message Visit poster's website
YARRRRR-GC



Joined: 21 May 2007
Posts: 10

PostPosted: Fri Jun 01, 2007 12:32 pm    Post subject: Reply with quote

Quote:
I suspect that one-point-per-solved-clue isn't quite enough scoring granularity for the more competitive teams.


From my perspective (again, as the big point/competition whore on the team) , I was pretty happy with how this turned out. I felt that because of the way the cold cases turned out, we actually had a fairly solid ordering at the end -- in almost all the cases, a team finishing behind another team solved a proper subset of the latter team's clues. The only other thing that we could have done, I think, is imposed some sort of penalty for hints, but after seeing how well Hogwarts' "free hints" policy worked, we decided to not do that on the route. In practice, as noted somewhere else, we actually didn't get a lot of calls to GC, so I don't think that was a huge issue either.
Back to top
View user's profile Send private message
DaleNeal-GC



Joined: 21 May 2007
Posts: 14

PostPosted: Sat Jun 02, 2007 2:52 pm    Post subject: Reply with quote

DaleNeal-GC wrote:
Lastly, regarding per-clue GC help line call volume, we recorded this manually, so we don't have a very accurate count. I can try to get you some numbers on that when I get home tonight, though.


As promised here are some numbers. All calls were logged by a GC member manually adding a note through a website I set up. Since we had a lot of people rotating through GC, and since people sometimes had to answer calls back to back, these numbers likely under-represent the calls in some cases. Also, no distinction was made between calls where the team was simply asking for confirmation that they were doing the right thing and calls where the team was stuck. So please take this with a grain of salt.

Tickets had no calls (except for people calling because my phone server blew up)
Triangles had 18 calls from 12 teams.
Chess had 19 calls from 11 teams.
Rocks had 3 calls from 3 teams.
Bugged (No Morse Egrets) had 3 calls from 2 teams.
Plumber had 3 calls from 3 teams.
Virus had no calls.
Wingding Crossword had 1 call from 1 team.
Lego had 2 calls from 2 teams.
Bank Heist had 1 call from 1 team.
Vowels had 1 call from 1 team.
Surveillance had 4 calls from 4 teams.
Rhyme and Reason had no calls.
Blinkenlights had 7 calls from 6 teams.
Elements parts 1 and 2 combined had 31 calls from 15 teams.
Charades had no calls.
Cube had 1 call from 1 team.
Scrabble had 12 calls from 7 teams.
Sound Maze had no calls.
Propagation had 12 calls from 7 teams.
Karaoke had no calls.
Text Adventure had 21 calls from 10 teams.
_________________
AnEelLad
Back to top
View user's profile Send private message Send e-mail Visit poster's website
neilfred



Joined: 27 May 2007
Posts: 9

PostPosted: Sat Jun 02, 2007 8:32 pm    Post subject: Reply with quote

Looking at the leaderboard.html page is interesting, but it's hard to digest the data. I hacked up a little javascript to allow you to watch the weekend playback at high speed with times appearing in order:

http://www.derf.net/nomoresecrets/

I made it show only the completion times so you can fit more on the screen at a time, and of course reduced font size. Hopefully the buttons are self-explanatory. I should also say that I'm not a javascript expert, and I only tested it in Firefox.

Enjoy!
Back to top
View user's profile Send private message
YARRRRR-GC



Joined: 21 May 2007
Posts: 10

PostPosted: Sun Jun 03, 2007 12:37 pm    Post subject: Reply with quote

This javascript thing is pretty cool!
Back to top
View user's profile Send private message
sparCKL



Joined: 21 May 2007
Posts: 14
Location: Mountain View, CA

PostPosted: Sun Jun 03, 2007 6:38 pm    Post subject: Open Source Software Reply with quote

neilfred wrote:
http://www.derf.net/nomoresecrets/

I agree with Yar: that's pretty damn cool. :) Thanks, NeilFred!

And speaking of code... Between all the automated phone systems, PalmPilot apps, and stat-tracking spreadsheets, it seems like there exists a pretty good collection of software that GCs can use to run a Game. How many people would be willing to publish their code as open source software? Just as GCs learn from each other's logistical mistakes in past Games, we should take advantage of existing software that can be reused and even improved.
_________________
I am not an aardvark.
http://snout.org/stories
Back to top
View user's profile Send private message Visit poster's website
DaleNeal-GC



Joined: 21 May 2007
Posts: 14

PostPosted: Mon Jun 04, 2007 3:46 pm    Post subject: Re: Open Source Software Reply with quote

sparCKL wrote:
And speaking of code... Between all the automated phone systems, PalmPilot apps, and stat-tracking spreadsheets, it seems like there exists a pretty good collection of software that GCs can use to run a Game. How many people would be willing to publish their code as open source software? Just as GCs learn from each other's logistical mistakes in past Games, we should take advantage of existing software that can be reused and even improved.


Cleaning up the code we used for our phone line/GC web frontend and putting it out there under a permissive (GPL/BSD/public domain) license for any future GCs to use is definitely on my grand list of things to do. I've just gotta make sure to remove the hardcoded login to my mysql database, among other things... It probably would also be kind to future programmers if I actually threw in a comment or two, as well.

In a similar vein, I have a mess of python puzzle solving/creating libraries that I've been meaning to turn into a coherent open source project. We'll see if I have the time and energy now that NMS is over. =)
_________________
AnEelLad
Back to top
View user's profile Send private message Send e-mail Visit poster's website
Display posts from previous:   
Post new topic   Reply to topic    no more secrets Forum Index -> NMS Discussion All times are GMT - 7 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group