High-speed idea filtering with the bag of lemons

• an approach for using the crowd to filter ideas submitted in open innovation engagements
• increases filtering accuracy and reduces rater time compared to conventional (5-point Likert scale) approach
• key insight: crowds are better at eliminating bad ideas than identifying good ones.

Open innovation platforms (web sites where crowds post ideas in a shared space) enable us to elicit huge volumes of potentially valuable solutions for problems we care about, but identifying the best ideas in these collections can be prohibitively expensive and time-consuming. This paper presents an approach, called the “bag of lemons”, which enables crowd to filter ideas with accuracy superior to conventional (Likert scale) rating approaches, but in only a fraction of the time. The key insight behind this approach is that crowds are much better at eliminating bad ideas than at identifying good ones.