The Wisdom of Crowdsourcing

In 1906, British scientist Francis Galton took a day trip to the country fair that would uncover a principle critical to the idea of crowd-listening. It was the heyday of social Darwinism, and Galton believed very little in the common man; he was of the opinion that only proper breeding and the preservation of power for the elite could maintain a healthy society; few, to his mind, were suitable to make decisions or lead others.

Happening upon a contest to guess the weight of a fat ox, he decided to put the common man’s judgment to the test: he collected all 800 guesses at the end and ran statistical tests to see how far off this random collection was. To his great surprise, the average of all the answers was just one pound away from the cow’s actual weight – much closer than any individual’s guess, including the livestock experts’.

The wisdom of crowds… or the danger of mobs?

This stunning display of accuracy gave birth to the idea of the wisdom of crowds: that the judgments of a large number of people, averaged together, tend towards a high degree of correctness. The concept has been applied to a number of fields, to explain and defend such things as democratic governance, or Wikipedia. Crowdsourcing too relies heavily on the principle.

Some, though, question whether it is wisdom that tends to define large groups, or a dangerous mob mentality. Throughout history, large groups have committed appalling acts as individuals lose all sense of personal responsibility among the mob. Where is the line? What drives a crowd to one or the other end of the spectrum? And how can we use this knowledge to better the way we do crowdsourcing?

1. Diversity: Having diversity in a group is critical to getting ‘wise’ judgments. The reason for this is integral to the reason why the wisdom of the crowd works at all – when so many different viewpoints and ideas are combined together and aggregated, underestimations and overestimations cancel out; support and opposition strike a balance; every stretched and skewed and miscalculated piece of input is accounted for by another. The more different ideas there are, the more likely this process is to happen.

In the context of crowdsourcing technology, the best way to ensure diversity is to build welcoming communities that invite people and ideas of every kind. More importantly, the platform must be accessible to a wide range of people. It can be all too easy, when you’re building a website or system, to create something that specially fits your own mental models, but to encourage diversity your creation has to be easy for anybody to understand. It’s important that no group feels like someone else is being catered to or being treated with preference, and so testing your system with a wide variety of demographics – different incomes, education levels, ethnic backgrounds, web experience, and so on – is critical.

2. Independent generation of ideas: People in groups can be prone to bandwagoning and groupthink. The loudest voice grab followers, and the most popular opinion gets more popular. Of course, when this happens, all of the different viewpoints of the individuals in the crowd are lost as they throw in their lot with someone else. To preserve the crowd’s insight and wisdom, members must make their judgments independently, with minimal social influencing. That doesn’t mean they have to be an isolated hermit to have a valid opinion, but simply that when it comes time to speak their mind, they don’t feel overly pressured. For example, come election time we are unavoidably subject to all sorts of media and popular debate, but when we cast our ballot, it is done alone in a private booth.

For crowdsourcing, since most web users are already alone and less likely to be unduly swayed by their peers, the important thing is to give every idea exposure. When many users are submitting ideas and views, it is easy for submissions that gain early traction to get all the attention and sweep away less prominent or later submissions. It’s a tricky problem, but there are solutions, including letting crowd members vote or comment before seeing the successfulness of posts, or displaying a randomized selection of submissions, or providing rewards for reading or voting on more submissions.

The main thing is to be conscious of how users experience and are affected by the system, so that the flaws that sometimes surface in crowd thinking can be dealt with and prevented from turning your crowd into a mob. With the right framework, the crowd can be a powerful tool to generate quality feedback and can be harnessed to do a wide variety of work.

To hear about how the wisdom of crowds is being harnessed to take the work out of website usability research, sign up for the UXCrowd webinar on December 9th with IdeaScale CEO Rob Hoehn, former Zynga user research director Rob Aseron, and usability testing service TryMyUI.