The Wisdom of Mechanical Turks

Recently Martin and I conducted a crowdsourcing experiment to test the wisdom of the crowd. The basic idea is that the collective input of a group can, in some instances, provide a more accurate result than the input of an individual belonging to the group. Using a jar of coins and Mechanical Turks, we were able to demonstrate this:

jar of coins

Experiment

The setup was pretty basic. After convincing the bank to sell us a lot of coins, we filled a jar and took the picture you see above. We then hired 100 mechanical turks to guess how many coins were in the jar. After all of the guesses had been submitted, we simply took an average of the guesses from the 100 individuals to obtain the “guess of the crowd”.

Result

Normally, a very large group is required to provide an estimate with low error. With a group of only 100 Mechanical Turks we were surprisingly able to obtain an accuracy of 95%. In theory as the size of the group increases, the estimation error will decrease. The graph below shows the how the average of the guesses became closer to the actual number of coins as more guesses were collected:

estimation error vs. number of guesses

This was just a very basic (and cheap) experiment which was nevertheless entertaining. If you’re interested in other (much cooler) crowdsourced projects you should check out: The Sheep Market and Ten Thousands Cents.