The Good and Bad Kinds of Crowds

by Andrew McAfee on March 21, 2009

This past week I rolled out a couple Enterprise 2.0-ish experiments in my MBA Class Managing in the Information Age. First, I attempted to use crowd wisdom to outsmart my students. Second, I let them form their own online crowd during a single class.

The previous week I had thrown down a challenge: any students that outpicked me in the men’s NCAA college basketball tournament (aka “March Madness”) would win freedom from cold calls for the rest of the semester. I use an Excel-based random cold call generator in class and my students absolutely hate it, so they had ample incentive to fill out an entry within the ESPN group I set up.

I told my students that there was plenty of help and advice about the tournament available online, as well as many, many freely available brackets completed by different flavors of expert. I also told of them none of this would do them much good, though, ’cause I was such an ardent college hoops fan that I would surely outpick them.

This was a baldfaced lie.  I haven’t watched a basketball game in years, and have no idea who’s any good these days. I’m certainly not better informed than the tournament organizers, who seed the 64 teams based on their expected performance. So for me, a smart strategy is to just pick the team seeded higher in each game, thereby taking advantage of all the intelligence baked into the tournament.

I also think this is a pretty smart strategy no matter how well informed you are. I wonder how many ‘experts’ predict the tournament’s results better each year than the seeds alone do. I bet it’s not many. I bet even fewer experts would have a track record over many years of outpredicting the seeds.

But I also thought it would be possible to do better than just picking the seeds by tapping into crowd wisdom –  seeing, in other words, if a crowd thought that the tournament organizers got it wrong in any cases. And my preferred way to do this is to look at prediction markets.

NewsFutures set up prediction markets for each of the first round games in the tournament. When I checked them shortly before the deadline for submitting a bracket to ESPN, I saw that they were predicting 3 upsets: #10 Maryland over #7 California, #9 Tennessee over #8 Oklahoma St., and #10 USC over #7 Boston College. In all other cases, the collective prediction of the NewsFutures traders was that the higher-seeded team (the one with the lower number) was more than 50% likely to win in the tournament’s first round.

So in all cases except the three listed above, I picked the higher seed. With one exception: #12 Arizona had, according to the market a 48% chance of beating #5 Utah, and I couldn’t pass up the chance to correctly call a big upset like that. So I went with Arizona.

In all rounds after the first I simply picked the higher seed. I could have set up my challenge to students so that we had to pick one round at a time. Setting it up this way would have let me use the markets established for later rounds, but I’m pretty sure my students would have caught on to this strategy before too long, and we would have all converged.  Maybe next year…

I don’t know any of the traders in the NewsFutures markets, and don’t know exactly how to interpret the data they provide on trading volume – the number of contracts held for each game (beyond knowing that more trading is better). I just have a lot of faith in prediction markets‘ ability to forecast real-world events, and wanted to put that faith to the test in a visible way.

None of my students guessed correctly that I had used prediction markets to make my picks. I asked them to describe their strategies; they relied on their knowledge, seeds, odds from Las Vegas, and hometown affiliations (a very bad strategy). None of them said that they’d used the markets as I had. The most intriguing strategy I heard was to use the results of the video game company EA Sports‘ simulated tournament to make picks.

The first round of the tournament is now over, and the markets correctly predicted the upset victories of Maryland and USC. In addition, #12 seed Arizona did win, so the markets did an excellent job of highlighting this possibility. They were wrong about Tennessee over Oklahoma St. There were seven additional first round upsets that the markets did not correctly predict.

Overall, my strategy of relying on the markets left me in significantly better shape –  two victories worth -  than would have been the case if I’d relied only on seeds.  I’m currently tied for fourth place among the 39 brackets submitted as part of my class, and in pretty good shape for the rest of the tournament; no one has more possible points remaining than I do. The bracket using results from the EA sports simulated tournament is tied with me at present, but has fewer possible points remaining.

My second experiment with emergent social software platforms was much simpler: I just told all my students to get Twitter accounts, then allowed them to tweet freely during class on Friday, March 20. I also told them to send one tweet using the hashtag #HBSMIA2009.

Twitter expert and celebrity and Pistachio Consulting founder Laura Fitton (@pistachio) is coming to class later this semester, will encourage students to tweet during class, and will also (I believe) display these their tweets on a screen throughout class. I found myself unwilling to take that last step, but did want to get my students comfortable with the practice of live tweeting, and also wanted to see what it felt like to teach class while that was going on.

So I told them that class on the 20th would be an exception to HBS’s standard ‘screens down’ policy (i.e. no use of digital devices during class), and that they could tweet using whatever device they preferred.

I’ll ask my students what they thought about the experience, but I thought it was miserable. Class discussion limped along at well below its normal levels of engagement, interest, and insight. I thought it was due to my bad class plan, a comparatively weak case, and/or the fact that the 20th was the last day before spring break.

Any or all of these could have been part of the explanation, but I’m quite sure that another part was the tweeting that went on. When I reviewed students’ tweets after class, I found that a lot of them remarked on how difficult it was to pay attention to what was going on in the room and on their screens. And it was very clear that the screens won.

Speaking to an audience that’s tweeting away is now a fact of life at most technology conferences (as clearly evidenced by this year’s South by Southwest). Laura says she likes it, and I’m eager to learn from her why this is and how I can turn live tweeting to my advantage when speaking. So far it feels to me like trying to talk to people who all have TVs in front of them. I realize that live tweeting might be beneficial to some constituencies (like the tweeters’ followers), but it feels to me like a pure negative for speakers. We’re now competing for attention with a very compelling interactive activity.

I know this is the new reality of public speaking, and I know I’ve got to get good at it, but I’m not sure how I’m ever going to come to like it. And I know my classrooms are going to remain unwired. I want my students to concentrate on the discussion taking place in meatspace, not the ones in cyberspace.  I want to be clear: I like twitter a lot and use it a fair bit myself (follow me at @amcafee if you like), but I don’t like it in a classroom when a live discussion is (supposed to be) taking place.

Two questions to wind up this post. First, how could I have made better use of crowd wisdom and other available information to make better March Madness picks? And do you have any tips on how to be a good Twitter-assisted public speaker?  Leave a comment, please, and let us know.

  • http://engineerswithoutfears.blogspot.com/ Matt Moore

    Very interesting experiments Andrew – please keep 'em coming.
    Twitter public speaking:
    - A lot of it may depend the experience of your audience. A random sample of 3 of your students yields total updates of: 13, 110 & 61. They are still coping with the “Hey, I'm Tweeting” thing. If you run this again, you may get a better result.
    - If people are using Twitter then was it built into the session? Strong structure may help (esp. for newbies): “Focus on me for a few minutes, now Tweet a bit, now I want you to interact F2F for a few minutes”
    - Did you open up the session to non-participants? Could you advertise the time, date, hashtag & lecture topic on the blog a few days beforehand and so involve those not in the room.

    Personally, I think Twitter has a role to play in lectures, conferences, etc. But it can be distracting & de-energizing. It's like riding a bike – we (as presenters & participants) need to become adept at it – and good structure certainly helps with that at the start…

  • http://www.twitter.com/aclearbluesky Arjan

    This might be an interesting way of using Twitter during an event: http://www.avc.com/a_vc/2009/03/if-you-are-doin…. I think the thing is that it has its most value when you get the interaction back in the discussion.

  • Anonymous

    It’s fascinating that Arizona is in the Sweet 16. Well predicted, Prof. Prediction Market.

    As for using Twitter during live events, you’re fortunate that HBS has an established closed screen policy and that you consistently engage your students to this extent. Every event I’ve attended of late, especially in the conference or panel discussion world, now has a Twitter back channel. There’s a double-edged sword to such engagement.

    Live attendees can share news, quotes, links to resources, commentary or other significant “value adds” t=with each other, allowing remote readers to gain useful insights. On the other hand, such focus on cyberspace almost necessarily detracts from engagement in the presentation. Kudos, BTW, for using the term “meatspace” in your post — very cyberpunky.

    I’ve tried note-taking with Word or a notebook, like a traditional student, live blogging, live-tweeting, using my reporter’s notebook or simply sitting and listening. Only the last truly allows for full engagement. I find, however, that using a portion of my cognitive bandwidth to take notes and monitor the digital back channel to be quite useful in terms of capturing key elements, taking notes for later use and — critically — leveraging my online readership to gain more insight into the relevancy, use or meaning of a given statement or assertion. The issue of combining a virtual experience and classroom with the in-person lecture is sure to be a thorny one for years to come. I suspect that Laura’s use of a projected screen to pull the heads out of the screens and into the conversation occurring in real-time will be helpful in showing how merging the two can be effectively managed, as the shared experience of watching collaborative discussion emerge is more compelling than trying to manage the conversation in isolation.

    I’m sure we’ll all hear about it from @pistachio & @amcafee down the road.

  • http://marchmadness.hubdub.com Nigel Eccles

    Nice idea on the competition. We are also running markets on March Madness here: http://marchmadness.hubdub.com

    Unusually, there has been fewer upsets than usual this year (the favorites are winning) which means your strategy is probably performing better than would be expected.

  • http://kyle.mathews2000.com/blog Kyle Mathews

    I agree with Matt that digital interactions work best if the digital interactions + lecture + class time is structured. Last semester, a professor once posted a question on the class group site and asked us to comment. Our comments then led into a class discussion around different people's answers.

    https://island.byu.edu/content/performance-mana

  • Quotes

    I could relate to all the points you have covered here, thanks for such an interesting article, I just enjoyed reading it.

  • http://confusedofcalcutta.com JP

    Hi Andy, I've commented at length on your Twitter question in my blog, it was too long to fit simply here.

    http://confusedofcalcutta.com/2009/03/27/twitte

  • http://blog.strategicheading.com Oliver Young

    Hey Professor, I finally got around to reading your blog post and it turns out you and I had the exact same plan for March Madness this year (though in my case I entered brackets in two leagues, one with no upsets and one using my normal bracket selection process). I'm not quite as extreme as you, but am a very casual college basketball fan — I don't watch any games until the tournament starts.

    Well the results are in: my normal bracket selection process has won out in a squeaker (as you know the “no upset” bracket is now busted, with no more points to be had). If North Carolina or MSU make it to the final the regular bracket selection will win big. I'll admit, a lot of this success has been a homer pick of MSU — I grew up just outside Detroit — though so far I have correctly picked 47 of 60 winners.

    In the league where I entered the no-upset bracket things are considerably less decided. Of 12 players the “no upsets” bracket is currently tied for second and has a chance to stay there, though if UNC or UCONN advance the “no upsets” bracket will fall to the bottom half of the league.

    All in all it was a fun experiment, though if I were to do it again i would likely give the “popular picks” option Yahoo! provides a shot. It is a more “true” wisdom of crowds experiment since the crowd is bigger and the decisions are made in complete isolation, politics and “agreement” can cloud the selection committee process.

    In case you were wondering, my “normal” bracket has UNC beating MSU in the final and is currently in first place for my league.

  • http://zerobeta.wordpress.com zerobeta

    I work for a company, http://yoonew.com, that uses prediction market algorithms to dynamically price sports tickets, including the Final Four. I used some of our information to produce one bracket this year and used my standard “random-ish” method for another bracket.

    Our markets were high on Villanova and Missouri earlier this year, but completely missed the boat on MSU (which is odd because one would think there would be an expected premium on their ticket due to the Final Four being in Detroit). My bracket where I used the information did better than the other one, but will not win due to a few points missed in the earlier round. This makes sense since our markets are used to predict the Final Four Field (which I got 3 out of 4 correct), but not the early upsets, etc.

    You mention that the NCAA committee picks teams based on their “expected performance”. i would disagree on one level. I believe they pick a field that maximizes their ability to sell tickets along the way and for the final four. The notion of “home field” advantage only applies if fans are in the arenas, therefore using it as a reward is the same thing as saying we are going to sell more tickets for your team. Thats why you sometimes see a projected 1 seed get a 2 seed w/ regional advantage, etc. They know by matching up UNC in the south, they have a better chance of taking advantage of UNC's “edge” on its competition in the place in which their edge is most valuable.

    As for using prediction markets to pick a bracket, I think its no different an edge of any other “system”. It works for a while, until the “market” realizes it and then it doesn't. Any large pools will have the same properties of the crowd, therefore using the information of the prediction markets gives you the average pick of the pool before the brackets are submitted. While this is useful information, you must be wiser than the crowd in order to win. Thus it may be a good system in order to compete but a lousy one if you want to win. Also, a crowd that begins to use its own “wisdom” in order to make picks will suffer from the same reflexive properties that Soros argues about in “The Alchemy of Finance” et al. So I'm not too sure that as an absolute strategy prediction markets are of much use if the end goal is to win.

  • choozm

    Here are tips from Pistachio on presenting while audience are using twitter: http://pistachioconsulting.com/twitter-presenta

  • ronga

    Having been at SXSW and not in MIA during tweet day, I feel as though I have been influenced by HBS policy of no technology allowed in classroom. See one of my first tweets at SXSW (http://twitter.com/ronga/status/1323577721). However, these people I spoke of were checking email, IM'ing or doing other things.

    I do feel, that Twitter can be harnessed in a valuable way both in the classroom and at conferences if done correctly. It would enhance the panels, if there a live screen featuring everyone's tweets (w/ panel hashtag) were included during the panel. Hopefully it would self-regulate and keep out the nonsense tweets such as “I don't like what @so-and-so is wearing.” Twitter is hardly anonymous and the community will retaliate. Additionally, people paid big money to go to SXSW and learn. I feel I would have learned a ton from the audience's thoughts and feel I'd still be able to concentrate on the panel itself.

    Similarly in your class, it seems there were a lot of “nonsense” tweets. Had you included a live feed on the screen, perhaps they may have been more relevant to the discussion and enhanced the discussion.

    Another reason I like using Twitter during conferences or class is because, for me, it is helpful to take notes of profound comments or quotes (as there were at SXSW). I did this during @garyvee's speech from my phone and tweeted as he was talking. It helped me absorb the information and went into my catalog of tweets immediately so I could reference them anytime. That surely has value for me and for others who were debating whether to come see @garyvee's keynote.

    Similarly in your class, if I were to Tweet someone's comment and reference the person, they know they've made an impact on one person in the class. This day and age (unfortunately), we rarely approach the person to say, “your comment really made me think…”. We might say, “hey nice comment.” But tweeting their comment and referencing them… well, that SAYS something.

  • PowerPres

    Fascinating. I am very interested: 1., that you did this experiment, and 2. seeing the results. I would have predicted such results, having taught at the University level, now working as a coach and holding workshops, and as a person who uses Twitter.
    We are all still learning about the use and best-use of many technologies today. However, regardless of one's learning style, focus is critical. In our programs and workshops we stress the importance and value of concentration.
    I will now read you blog more often.
    Thanks…

  • http://blog.strategicheading.com Oliver Young

    So, how did you finish up?

  • robinbordoli

    Andy

    I've been in the audience for half a dozen presentations with a live twitter feed on a 2nd screen, but I recently gave my first presentation where there was a live twitter feed, so I thought I'd share my perspective as both from the audience and presenter perspective

    Here's the presentation I gave “Outlook vs Twitter: who wins the war inside the enterprise?” (http://bit.ly/3ueiMq)

    Advantages as an audience participant

    + I can participate in a presentation/discussion without being in the room with #hashtags
    + I can ask a question at the point of inspiration rather than wait until the end
    + I can add to or build on another person's comment so it's not just a 1 to 1 interaction but 1 to many
    + I can see whether the audience is reacting the same way I am to the presentation (bored? focused on piece of the presentation, etc)

    Disadvantages as an audience participant
    - Some twitterati can flood and dominate the stream

    Advantages as a presenter
    + I can gauge the level of interest in the audience
    + I can adjust my presentation in realtime (although this is a tough skill to master)
    + After the presentation I have a record of all the comments
    + Extending my reach beyond the room
    + Shows humility that you don't know it all. As Socrates paradoxically said “I know I know nothing”
    + If you ask the audience to retweet the questions that are most interesting to them you can do a better job of addressing your audience
    + As an advocate for the power of transparency and sharing as CEO of HeadMix (http://www.headmix.com/blog), the use of this medium reinforces my message

    Disdvantages as a presenter
    - Reading the twitter feed can be disruptive. Do I do it in real time, at specific breaks, at the end?
    - Seeing people in the audience with their heads down and not being able to establish eye contact can be disturbing

    In both cases, as a member of the audience and as a presenter, I feel the advantages outweigh the disadvantages. That's not to say it doesn't force you out of a comfort zone as presenter – it does – but the payoff is worth it.

    My top 5 tips would be

    1. Twitter feeds are best used for presentations that are image/idea heavy and data light (ie Lawrence Lessig/Dick Hardt style)
    2. Advertise the #hashtag in advance and at the beginning of your presentation
    3. Ask audience members to retweet the questions they see that they want you to answer
    4. Wait until the end of your presentation to answer questions
    5. Have a person dedicated to scanning the twitter feed and ask you the questions

    http://headmix.com/blog/2009/04/11/outlook-vs-t

  • http://www.mollybob.wordpress.com Mollybob

    I am really interested in the twitter backchannel, but cannot say I have spoken while there's been one running. as a class member, I find it more engaging because I have to think about what is being said and provide some sort of opinion. So I guess it helps to to start reflecting and thinking critically about what's happening. But in saying this, there has been no other dialogue while I've been tweeting, it's been a sage on the stage setup so I considered the tweets to be my way of creating dialogue. Some other questions would be- did you have the tweets showing your hashtag up on a screen near you while you were talking so students could see each others thoughts? Were the students using Twitter to speak with each other? Did they have to cncentrate on Twitter because it was new to them?
    Your post has certainly provided more insight into why some may not like the twitter backchannel than others I have read – thanks.

  • Victor Chin

    I really love the insight provided on the need for an evolution for our presentation styles, this really provided me insight on how to manage a presentation while Twittering is going on in the audience and how to use it to our advantage. Keep up the good work.

  • http://www.tourtravelchina.com/beijing-tour.html Beijing Tour

    That Sounds interesting, I agree with you.Please keep at your good work, I would come back often.*

  • pixbook

    I could relate to all the points you have covered here, thanks for such an interesting article, I just enjoyed reading it.

    Ways to make money

  • sheenadorie

    what are the four classification of crowd?

  • sheenadorie

    what are the four classification of crowd?

  • http://tomaschavez.com Tomas Chavez

    Using twitter in presentations is very smart! It will definitely help the attendees understand better. However, as what Robinbordoli said in a previous comment. Another twitter user can definitely spam the twitter feed. 
    Overall it’s a very smart way of presenting.

    Cheers!
    Tom

  • Anonymous

    Andrew,

    I’m very late of stumbling upon your blog since Damn Me!.

    Do you have any updates on this? I’m very interested in learning about this..An update/twist on the approach? @TomChavez:disqus said, “Another twitter user can definitely spam the twitter feed.”  A new twist will definitely help a lot! :)

  • http://www.saintsticketshotspot.com/saints-tickets/ Saints Tickets

    I’d love to know the answer too…”what are the four classification of crowd?”

  • http://www.tomaschavez.com/ Tomas Chavez

    Wow! This blogsite is very busy..Got lots of visitors!

    Keep up the good work!
    gallstone symptoms

    Saints Football Tickets

  • Anonymous

    It’s pretty amazing how social media can help in such presentation.

Previous post:

Next post: