Science of Science

Working Paper
Misha Teplitskiy, Hardeep Ranu, Gary Gray, Michael Menietti, Eva Guinan, and Karim Lakhani. Working Paper. “Do Experts Listen to Other Experts? Field Experimental Evidence from Scientific Peer Review.” HBS Working Paper Series. Publisher's VersionAbstract
Organizations in science and elsewhere often rely on committees of experts to make important decisions, such as evaluating early-stage projects and ideas. However, very little is known about how experts influence each other’s opinions and how that influence affects final evaluations. Here, we use a field experiment in scientific peer review to examine experts’ susceptibility to the opinions of others. We recruited 277 faculty members at seven U.S. medical schools to evaluate 47 early stage research proposals in biomedicine. In our experiment, evaluators (1) completed independent reviews of research ideas, (2) received (artificial) scores attributed to anonymous “other reviewers” from the same or a different discipline, and (3) decided whether to update their initial scores. Evaluators did not meet in person and were not otherwise aware of each other. We find that, even in a completely anonymous setting and controlling for a range of career factors, women updated their scores 13% more often than men, while very highly cited “superstar” reviewers updated 24% less often than others. Women in male-dominated subfields were particularly likely to update, updating 8% more for every 10% decrease in subfield representation. Very low scores were particularly “sticky” and seldom updated upward, suggesting a possible source of conservatism in evaluation. These systematic differences in how world-class experts respond to external opinions can lead to substantial gender and status disparities in whose opinion ultimately matters in collective expert judgment.
Jacqueline N. Lane, Eva C. Guinan, Ina Ganguli, Karim R. Lakhani, and Patrick Gaule. Working Paper. “Engineering Serendipity: The Role of Cognitive Similarity in Knowledge Sharing and Knowledge Production.” HBS Working Paper Series. Publisher's Version 20-058_.pdf
Kyle R. Myers, Wei Yang Tham, Yian Yin, Nina Cohodes, Jerry G. Thursby, Marie C. Thursby, Peter E. Schiffer, Joseph T. Walsh, Karim R. Lakhani, and Dashun Wang. Working Paper. “Quantifying the Immediate Effects of the COVID-19 Pandemic on Scientists”. Publisher's VersionAbstract
The COVID-19 pandemic has undoubtedly disrupted the scientific enterprise, but we lack empirical evidence on the nature and magnitude of these disruptions. Here we report the results of a survey of approximately 4,500 Principal Investigators (PIs) at U.S.- and Europe-based research institutions. Distributed in mid-April 2020, the survey solicited information about how scientists' work changed from the onset of the pandemic, how their research output might be affected in the near future, and a wide range of individuals' characteristics. Scientists report a sharp decline in time spent on research on average, but there is substantial heterogeneity with a significant share reporting no change or even increases. Some of this heterogeneity is due to field-specific differences, with laboratory-based fields being the most negatively affected, and some is due to gender, with female scientists reporting larger declines. However, among the individuals' characteristics examined, the largest disruptions are connected to a usually unobserved dimension: childcare. Reporting a young dependent is associated with declines similar in magnitude to those reported by the laboratory-based fields and can account for a significant fraction of gender differences. Amidst scarce evidence about the role of parenting in scientists' work, these results highlight the fundamental and heterogeneous ways this pandemic is affecting the scientific workforce, and may have broad relevance for shaping responses to the pandemic's effect on science and beyond.
Misha Teplitskiy, Eamon Duede, Michael Menietti, and Karim R. Lakhani. Working Paper. “Status drives how we cite: Evidence from thousands of authors”. Publisher's VersionAbstract
Researchers cite works for a variety of reasons, including some having nothing to do with acknowledging influence. The distribution of different citation types in the literature, and which papers attract which types, is poorly understood. We investigate high-influence and low-influence citations and the mechanisms producing them using 17,154 ground-truth citation types provided via survey by 9,380 authors systematically sampled across academic fields. Overall, 54% of citations denote little-to-no influence and these citations are concentrated among low status (lightly cited) papers. In contrast, high-influence citations are concentrated among high status (highly cited) papers through a number of steps that resemble a pipeline. Authors discover highly cited papers earlier in their projects, more often through social contacts, and read them more closely. Papers' status, above and beyond any quality differences, directly helps determine their pipeline: experimentally revealing or hiding citation counts during the survey shows that low counts cause lowered perceptions of quality. Accounting for citation types thus reveals a "double status effect": in addition to affecting how often a work is cited, status affects how meaningfully it is cited. Consequently, highly cited papers are even more influential than their raw citation counts suggest.
2020
Kyle R. Myers, Wei Yang Tham, Yian Yin, Nina Cohodes, Jerry G. Thursby, Marie C. Thursby, Peter Schiffer, Joseph T. Walsh, Karim R. Lakhani, and Dashun Wang. 7/15/2020. “Unequal effects of the COVID-19 pandemic on scientists.” Nature Human Behavior. Publisher's Version
2019
Andrea Blasco, Olivia S. Jung, Karim R. Lakhani, and Michael E. Menietti. 4/2019. “Incentives for Public Goods Inside Organizations: Field Experimental Evidence.” Journal of Economic Behavior & Organization, 160, Pp. 214-229. Publisher's VersionAbstract

We report results of a natural field experiment conducted at a medical organization that sought contribution of public goods (i.e., projects for organizational improvement) from its 1200 employees. Offering a prize for winning submissions boosted participation by 85 percent without affecting the quality of the submissions. The effect was consistent across gender and job type. We posit that the allure of a prize, in combination with mission-oriented preferences, drove participation. Using a simple model, we estimate that these preferences explain about a third of the magnitude of the effect. We also find that these results were sensitive to the solicited person’s gender.

Incentives_for_Public_Goods_Inside_Orgs.pdf
2018
Michael Menietti, M.P. Recalde, and L. Vesterlund. 2018. “Charitable Giving in the Laboratory: Advantages of the Piecewise Linear Public Good Game.” In The Economics of Philanthropy: Donations and Fundraising, edited by Mirco Tonin and Kimberley Scharf. MIT Press. Publisher's Version
2017
Kevin Boudreau, Tom Brady, Ina Ganguli, Patrick Gaule, Eva Guinan, Tony Hollenberg, and Karim R. Lakhani. 2017. “A Field Experiment on Search Costs and the Formation of Scientific Collaborations.” The Review of Economics and Statistics, 99, 4, Pp. 565-576. Publisher's VersionAbstract

Scientists typically self-organize into teams, matching with others to collaborate in the production of new knowledge. We present the results of a field experiment conducted at Harvard Medical School to understand the extent to which search costs affect matching among scientific collaborators. We generated exogenous variation in search costs for pairs of potential collaborators by randomly assigning individuals to 90-minute structured information-sharing sessions as part of a grant funding opportunity for biomedical researchers. We estimate that the treatment increases the baseline probability of grant co-application of a given pair of researchers by 75% (increasing the likelihood of a pair collaborating from 0.16 percent to 0.28 percent), with effects higher among those in the same specialization. The findings indicate that matching between scientists is subject to considerable frictions, even in the case of geographically-proximate scientists working in the same institutional context with ample access to common information and funding opportunities.

Field_Experiment_on_Search_Costs.pdf
Olivia Jung, Andrea Blasco, and Karim R. Lakhani. 2017. “Perceived Organizational Support For Learning and Contribution to Improvement by Frontline Staff.” Academy of Management Proceedings, 2017, 1. Publisher's VersionAbstract

Utilizing suggestions from clinicians and administrative staff is associated with process and quality improvement, organizational climate that promotes patient safety, and added capacity for learning. However, realizing improvement through innovative ideas from staff depends on their ability and decision to contribute. We hypothesized that staff perception of whether the organization promotes learning is positively associated with their likelihood to engage in problem solving and speaking up. We conducted our study in a cardiology unit in an academic hospital that hosted an ideation contest that solicited frontline staff to suggest ideas to resolve issues encountered at work. Our primary dependent variable was staff participation in ideation. The independent variables measuring perception of support for learning were collected using the validated 27-item Learning Organization Survey (LOS). To examine the relationships between these variables, we used analysis of variance, logistic regression, and predicted probabilities. We also interviewed 16 contest participants to explain our quantitative results. The study sample consisted of 30% of cardiology unit staff (n=354) that completed the LOS. In total, 72 staff submitted 138 ideas, addressing a range of issues including patient experience, cost of care, workflow, utilization, and access. Figuring out the cost of procedures in the catheterization laboratory and creating a smartphone application that aids patients to navigate through appointments and connect with providers were two of the ideas that won the most number of votes and funding to be implemented in the following year. Participation in ideation was positively associated with staff perception of supportive learning environment. For example, one standard deviation increase in perceived welcome for differences in opinions was associated with a 43% increase in the odds of participating in ideation (OR=1.43, p=0.04) and 55% increase in the odds of suggesting more than one idea (OR=1.55, p=0.09). Experimentation, a practice that supports learning, was negatively associated with ideation (OR=0.36, p=0.02), and leadership that reinforces learning was not associated with ideation. The perception that new ideas are not sufficiently considered or experimented could have motivated staff to participate, as the ideation contest enables experimentation and learning. Interviews with ideation participants revealed that the contest enabled systematic bottom-up contribution to quality improvement, promoted a sense of community, facilitated organizational exchange of ideas, and spread a problem-solving oriented mindset. Enabling frontline staff to feel that their ideas are welcome and that making mistakes is permissible may increase their likelihood to engage in problem solving and speaking up, contributing to organizational improvement.

2016
Kevin J. Boudreau and Karim R. Lakhani. 2016. “Innovation Experiments: Researching Technical Advance, Knowledge Production, and the Design of Supporting Institutions.” In Innovation Policy and the Economy, 16: Pp. 135-167. Chicago, IL. Publisher's VersionAbstract

This paper discusses several challenges in designing field experiments to better understand how organizational and institutional design shapes innovation outcomes and the production of knowledge. We proceed to describe the field experimental research program carried out by our Crowd Innovation Laboratory at Harvard University to clarify how we have attempted to address these research design challenges. This program has simultaneously solved important practical innovation problems for partner organizations, like NASA and Harvard Medical School (HMS), while contributing research advances, particularly in relation to innovation contests and tournaments. We conclude by proceeding to highlight the opportunity for the scholarly community to develop a “science of innovation” that utilized field experiments as means to generate knowledge.

Innovation_Experiments.pdf
Kevin J. Boudreau, Eva C. Guinan, Karim R. Lakhani, and Christoph Riedl. 2016. “Looking Across and Looking Beyond the Knowledge Frontier: Intellectual Distance, Novelty, and Resource Allocation in Science.” Management Science, 62, 10, Pp. 2765-2783. Publisher's VersionAbstract

Selecting among alternative projects is a core management task in all innovating organizations. In this paper, we focus on the evaluation of frontier scientific research projects. We argue that the “intellectual distance” between the knowledge embodied in research proposals and an evaluator’s own expertise systematically relates to the evaluations given. To estimate relationships, we designed and executed a grant proposal process at a leading research university in which we randomized the assignment of evaluators and proposals to generate 2,130 evaluator–proposal pairs. We find that evaluators systematically give lower scores to research proposals that are closer to their own areas of expertise and to those that are highly novel. The patterns are consistent with biases associated with boundedly rational evaluation of new ideas. The patterns are inconsistent with intellectual distance simply contributing “noise” or being associated with private interests of evaluators. We discuss implications for policy, managerial intervention, and allocation of resources in the ongoing accumulation of scientific knowledge.

Looking_Across_and_Looking_Beyond_the_Knowledge_Frontier.pdf
Andrea Blasco, Olivia S. Jung, Karim R. Lakhani, and Michael Menietti. 2016. Motivating Effort in Contributing to Public Goods Inside Organizations: Field Experimental Evidence. National Bureau of Economic Research. Publisher's VersionAbstract

We investigate the factors driving workers’ decisions to generate public goods inside an organization through a randomized solicitation of workplace improvement proposals in a medical center with 1200 employees. We find that pecuniary incentives, such as winning a prize, generate a threefold increase in participation compared to non-pecuniary incentives alone, such as prestige or recognition. Participation is also increased by a solicitation appealing to improving the workplace. However, emphasizing the patient mission of the organization led to countervailing effects on participation. Overall, these results are consistent with workers having multiple underlying motivations to contribute to public goods inside the organization consisting of a combination of pecuniary and altruistic incentives associated with the mission of the organization.

Motivating_Effort_in_Contributing_to_Public_Good_Inside_Organizations.pdf
2015
Karim R. Lakhani and Greta Friar. 2015. Nivea (A) and (B). Harvard Business School Teaching Notes. Harvard Business School. Publisher's VersionAbstract

Teaching Note for HBS Cases 614-042 and 614-043.

The first case describes the efforts of Beiersdorf, a worldwide leader in the cosmetics and skin care industries, to generate and commercialize new R&D through open innovation using external crowds and "netnographic" analysis. Beiersdorf, best known for its consumer brand Nivea, has a rigorous R&D process that has led to many successful product launches, but are there areas of customer need that are undervalued by the traditional process? A novel online customer analysis approach suggests untapped opportunities for innovation, but can the company justify a launch based on this new model of research?
The supplementary case follows up on an innovative R&D approach by Beiersdorf, a skin care and cosmetics company. The case relates what happened to the product launched by Beiersdorf, to its Nivea line, following the events of the first case, and how the commercial success of the product informed thinking by leaders in R&D for the future.

2014
Michael L. Tushman, Hila Lifshitz-Assaf, and Kerry Herman. 2014. Houston, We Have a Problem: NASA and Open Innovation (A). Harvard Business School Case. Harvard Business School. Publisher's VersionAbstract
Jeff Davis, director of Space Life Sciences Directorate at NASA, has been working for several years to raise awareness amongst scientists and researchers in his organizations of the benefits of open innovation as a successful and efficient way to collaborate on difficult research problems regarding health and space travel. Despite a number of initiatives, SLSD members have been skeptical about incorporating the approach into their day-to-day research and work, and have resisted Davis's and his strategy team's efforts. The (A) case outlines these efforts and the organization members' reactions. The (B) case details what Davis and the SLSD strategy team learned, and how they adapted their efforts to successfully incorporate open innovation as one of many tools used in collaborative research at NASA.
Michael L. Tushman, Hila Lifshitz-Assaf, and Kerry Herman. 2014. Houston, We Have a Problem: NASA and Open Innovation (B). Harvard Business School Case. Harvard Business School. Publisher's VersionAbstract
Jeff Davis, director of Space Life Sciences Directorate at NASA, has been working for several years to raise awareness amongst scientists and researchers in his organizations of the benefits of open innovation as a successful and efficient way to collaborate on difficult research problems regarding health and space travel. Despite a number of initiatives, SLSD members have been skeptical about incorporating the approach into their day-to-day research and work, and have resisted Davis's and his strategy team's efforts. The (A) case outlines these efforts and the organization members' reactions. The (B) case details what Davis and the SLSD strategy team learned, and how they adapted their efforts to successfully incorporate open innovation as one of many tools used in collaborative research at NASA.
Karim R. Lakhani, Johann Fuller, Volker Bilgram, and Greta Friar. 2014. Nivea (A). Harvard Business School Case. Harvard Business School. Publisher's VersionAbstract

The case describes the efforts of Beiersdorf, a worldwide leader in the cosmetics and skin care industries, to generate and commercialize new R&D through open innovation using external crowds and "netnographic" analysis. Beiersdorf, best known for its consumer brand Nivea, has a rigorous R&D process that has led to many successful product launches, but are there areas of customer need that are undervalued by the traditional process? A novel online customer analysis approach suggests untapped opportunities for innovation, but can the company justify a launch based on this new model of research?

Karim R. Lakhani, Johann Fuller, Volker Bilgram, and Greta Friar. 2014. Nivea (B). Harvard Business School Case Supplement. Harvard Business School. Publisher's VersionAbstract

This supplementary case follows up on an innovative R&D approach by Beiersdorf,a skin care and cosmetics company. The case relates what happened to the product launched by Beiersdorf, to its Nivea line, following the events of the A case, and how the commercial success of the product informed thinking by leaders in R&D for the future.

2013
Eva C. Guinan, Karim R. Lakhani, and Kevin J. Boudreau. 2013. “Experiments in Open Innovation at Harvard Medical School.” MIT Sloan Management Review 54 (3). Publisher's VersionAbstract

This article examines an experiment in open innovation applied to scientific research on Type 1 diabetes at Harvard Medical School. In the traditional research process in academic medicine, a single research team typically carries through each stage of the process — from generating the idea to carrying out the research and publishing the results. Harvard Catalyst, a pan-Harvard agency with a mission to speed biomedical research from the lab to patients' bedsides, modified the traditional grant proposal process as an experiment in bringing greater openness into every stage of research. Participation was successfully extended to nontraditional actors. With support from Dr. William Chin, the executive dean for research at Harvard Medical School and a former vice president of research at Eli Lilly (an early adopter of open innovation), Harvard Catalyst started with the front end of the innovation system by opening up the process of generating research questions. Instead of focusing on identifying individuals who might tackle a tough research problem, Harvard Catalyst wanted to allow an open call for ideas in the form of a prize-based contest to determine the direction of the academic research. This might lead to potentially relevant questions not currently under investigation or largely ignored by the Type 1 diabetes research community. Harvard Catalyst partnered with the InnoCentive online contest platform to initiate the idea generation process. Participants had to formulate well-defined problems and/or hypotheses to advance knowledge about Type 1 diabetes research in new and promising directions. In the end, 150 new hypotheses and research pathways were proposed. Teams were invited to propose projects on the 12 most promising of these; today, seven teams are carrying out the research. The Harvard Catalyst experience suggests that open-innovation principles can be adopted even within a well-established and experienced innovation-driven organization.

Karim R. Lakhani, Katja Hutter, Stephanie Healy Pokrywa, and Johann Fuller. 2013. Open Innovation at Siemens. Harvard Business School Case. Harvard Business School. Publisher's VersionAbstract

The case describes Siemens, a worldwide innovator in the Energy, Healthcare, Industry, and Infrastructure & Cities sectors, and its efforts to develop and commercialize new R&D through open innovation, including internal and external crowdsourcing contests. Emphasis is placed on exploring actual open innovation initiatives within Siemens and their outcomes. These include creating internal social- and knowledge-sharing networks and utilzing third party platforms to host internal and external contests. Industries discussed include energy, green technology, infrastructure and cities, and sustainability. In addition, the importance of fostering a collaborative online environment and protecting intellectual property is explored.

Karim R. Lakhani, Kevin J. Boudreau, Po-Ru Loh, Lars Backstrom, Carliss Y. Baldwin, Eric Lonstein, Mike Lydon, Alan MacCormack, Ramy A. Arnaout, and Eva C. Guinan. 2013. “Prize-based Contests Can Provide Solutions to Computational Biology Problems.” Nature Biotechnology, 31, 2, Pp. 108-111. Publisher's VersionAbstract

In summary, we show that a prize-based contest on a commercial platform can effectively recruit skilled individuals to apply their knowledge to a big-data biomedical problem. Deconstruction and transformation of problems for a heterogeneous solver community coupled with adequate data to produce and validate results can support solution diversity and minimize the risk of sub-optimal solutions that may arise from limited searches. In addition to the benefits of generating new knowledge, this strategy may be particularly useful in situations where the computational or algorithmic problem, or potentially any science problem, represents a barrier to rapid progress but where finding the solution is not itself the major thrust of the investigator’s scientific effort. The America Competes Act passed by the US Congress provides funding agencies with the authority to administer their own prize-based contests and paves the way for establishing how grant recipients might access commercial prize platforms to accelerate their own research.

Prize-based_Contests_Can_Provide_Solutions.pdf

Pages