Crowdsourcing & Open Innovation

The Laboratory of Innovation Science at Harvard is currently working on a number of studies, experiments, and projects that center around Crowdsourcing & Open Innovation. Such initiatives allow for the optimization of research for solutions to complex problems by calling on the crowd, instead of limiting knowledge to what is available within an organization.

Listed below are some examples of research which have greatly benefited from utilizing exterior knowledge and not limiting resources to those employed in-house, including a number of crowdsourcing challenges from our partners at NASA, the Broad Institute, and others. Browse LISH’s Crowdsourcing & Open Innovation projects and papers below.

If you are looking to run a crowdsourcing challenge, LISH's Find A Crowdsourcing Platform can help you find a platform that meets your needs.

Publications

Olivia S Jung, Fahima Begum, Andrea Dorbu, Sara J Singer, and Patricia Satterstrom. 7/17/2023. “Ideas from the Frontline: Improvement Opportunities in Federally Qualified Health Centers.” Journal of General Internal Medicine. Publisher's VersionAbstract

Background

Engaging frontline clinicians and staff in quality improvement is a promising bottom-up approach to transforming primary care practices. This may be especially true in federally qualified health centers (FQHCs) and similar safety-net settings where large-scale, top-down transformation efforts are often associated with declining worker morale and increasing burnout. Innovation contests, which decentralize problem-solving, can be used to involve frontline workers in idea generation and selection.

Objective

We aimed to describe the ideas that frontline clinicians and staff suggested via organizational innovation contests in a national sample of 54 FQHCs.

Interventions

Innovation contests solicited ideas for improving care from all frontline workers—regardless of professional expertise, job title, and organizational tenure and excluding those in senior management—and offered opportunities to vote on ideas.

Participants

A total of 1,417 frontline workers across all participating FQHCs generated 2,271 improvement opportunities.

Approaches

We performed a content analysis and organized the ideas into codes (e.g., standardization, workplace perks, new service, staff relationships, community development) and categories (e.g., operations, employees, patients).

Key Results

Ideas from frontline workers in participating FQHCs called attention to standardization (n = 386, 17%), staffing (n = 244, 11%), patient experience (n = 223, 10%), staff training (n = 145, 6%), workplace perks (n = 142, 6%), compensation (n = 101, 5%), new service (n = 92, 4%), management-staff relationships (n = 82, 4%), and others. Voting results suggested that staffing resources, standardization, and patient communication were key issues among workers.

Conclusions

Innovation contests generated numerous ideas for improvement from the frontline. It is likely that the issues described in this study have become even more salient today, as the COVID-19 pandemic has had devastating impacts on work environments and health/social needs of patients living in low-resourced communities. Continued work is needed to promote learning and information exchange about opportunities to improve and transform practices between policymakers, managers, and providers and staff at the frontlines.

Milena Tsvetkova, Sebastian Müller, Oana Vuculescu, Haylee Ham, and Rinat A Sergeev. 11/11/2022. “Relative Feedback Increases Disparities in Effort and Performance in Crowdsourcing Contests: Evidence from a Quasi-Experiment on Topcoder.” Proceedings of the ACM on Human-Computer Interaction, 6, CSW2, Pp. 1-27. Publisher's VersionAbstract
Rankings and leaderboards are often used in crowdsourcing contests and online communities to motivate individual contributions but feedback based on social comparison can also have negative effects. Here, we study the unequal effects of such feedback on individual effort and performance for individuals of different ability. We hypothesize that the effects of social comparison differ for top performers and bottom performers in a way that the inequality between the two increases. We use a quasi-experimental design to test our predictions with data from Topcoder, a large online crowdsourcing platform that publishes computer programming contests. We find that in contests where the submitted code is evaluated against others' submissions, rather than using an absolute scale, top performers increase their effort while bottom performers decrease it. As a result, relative scoring leads to better outcomes for those at the top but lower engagement for bottom performers. Our findings expose an important but overlooked drawback from using gamified competitions, rankings, and relative evaluations, with potential implications for crowdsourcing markets, online learning environments, online communities, and organizations in general.
Ademir Vrolijk and Zoe Szajnfarber. 12/20/2022. “The Opportunists in Innovation Contests Understanding Whom to Attract and How to Attract Them.” Research-Technology Management, 66, 1, Pp. 30-40. Publisher's VersionAbstract
Organizations increasingly turn to innovation contests for solutions to their complex problems. But these contests still face a fundamental inefficiency: they need to attract many participants to find the right solution, resulting in high costs and uncertainty. Studies have identified multiple dichotomies of successful and unsuccessful solver types, but these diverge. These studies also offer little guidance on how to attract successful solver types. We introduce the opportunist-transactor dichotomy, bridging whom to attract and how to attract them. Opportunists view the contest as a onramp to a new pursuit instead of a temporary undertaking. Characterizing solvers according to this new dichotomy was a better predictor of success than existing ones: in our context, most winners were opportunists. This type of solver was also reliably attracted by the seeker’s in-kind incentives, unlike those described by the other dichotomies. Our insights provide a deeper understanding of participants in complex contests and a concrete lever for influencing who shows up to solve.
Frank Nagle, James Dana, Jennifer Hoffman, Steven Randazzo, and Yanuo Zhou. 3/2/2022. Census II of Free and Open Source Software — Application Libraries. The Linux Foundation. Harvard Laboratory for Innovation Science (LISH) and Open Source Security Foundation (OpenSSF). Publisher's VersionAbstract

Free and Open Source Software (FOSS) has become a critical part of the modern economy. There are tens of millions of FOSS projects, many of which are built into software and products we use every day. However, it is difficult to fully understand the health, economic value, and security of FOSS because it is produced in a decentralized and distributed manner. This distributed development approach makes it unclear how much FOSS, and precisely what FOSS projects, are most widely used. This lack of understanding is a critical problem faced by those who want to help enhance the security of FOSS (e.g., companies, governments, individuals), yet do not know what projects to start with. This problem has garnered widespread attention with the Heartbleed and log4shell vulnerabilities that resulted in the susceptibility of hundreds of millions of devices to exploitation.

This report, Census II, is the second investigation into the widespread use of FOSS and aggregates data from over half a million observations of FOSS libraries used in production applications at thousands of companies, which aims to shed light on the most commonly used FOSS packages at the application library level. This effort builds on the Census I report that focused on the lower level critical operating system libraries and utilities, improving our understanding of the FOSS packages that software applications rely on. Such insights will help to identify critical FOSS packages to allow for resource prioritization to address security issues in this widely used software.

The Census II effort utilizes data from partner Software Composition Analysis (SCA) companies including Snyk, the Synopsys Cybersecurity Research Center (CyRC), and FOSSA, which partnered with Harvard to advance the state of open source research. Our goal is to not only identify the most widely used FOSS, but to also provide an example of how the distributed nature of FOSS requires a multi-party effort to fully understand the value and security of the FOSS ecosystem. Only through data-sharing, coordination, and investment will the value of this critical component of the digital economy be preserved for generations to come.

In addition to the detailed results on FOSS usage provided in the report, we identified five high-level findings: 1) the need for a standardized naming schema for software components, 2) the complexities associated with package versions, 3) much of the most widely used FOSS is developed by only a handful of contributors, 4) the increasing importance of individual developer account security, and 5) the persistence of legacy software in the open source space.

Philip Brookins, Dmitry Ryvkin, and Andrew Smyth. 3/8/2021. “Indefinitely repeated contests: An experimental study.” Experimental Economics . Publisher's VersionAbstract
We experimentally explore indefinitely repeated contests. Theory predicts more cooperation, in the form of lower expenditures, in indefinitely repeated contests with a longer expected time horizon. Our data support this prediction, although this result attenuates with contest experience. Theory also predicts more cooperation in indefinitely repeated contests compared to finitely repeated contests of the same expected length, and we find empirical support for this. Finally, theory predicts no difference in cooperation across indefinitely repeated winner-take-all and proportional-prize contests, yet we find evidence of less cooperation in the latter, though only in longer treatments with more contests played. Our paper extends the experimental literature on indefinitely repeated games to contests and, more generally, contributes to an infant empirical literature on behavior in indefinitely repeated games with “large” strategy spaces.
  •  
  • 1 of 12
  • »