Publications

Working Paper
Misha Teplitskiy, Hardeep Ranu, Gary Gray, Michael Menietti, Eva Guinan, and Karim Lakhani. Working Paper. “Do Experts Listen to Other Experts? Field Experimental Evidence from Scientific Peer Review.” HBS Working Paper Series. Publisher's VersionAbstract
Organizations in science and elsewhere often rely on committees of experts to make important decisions, such as evaluating early-stage projects and ideas. However, very little is known about how experts influence each other’s opinions and how that influence affects final evaluations. Here, we use a field experiment in scientific peer review to examine experts’ susceptibility to the opinions of others. We recruited 277 faculty members at seven U.S. medical schools to evaluate 47 early stage research proposals in biomedicine. In our experiment, evaluators (1) completed independent reviews of research ideas, (2) received (artificial) scores attributed to anonymous “other reviewers” from the same or a different discipline, and (3) decided whether to update their initial scores. Evaluators did not meet in person and were not otherwise aware of each other. We find that, even in a completely anonymous setting and controlling for a range of career factors, women updated their scores 13% more often than men, while very highly cited “superstar” reviewers updated 24% less often than others. Women in male-dominated subfields were particularly likely to update, updating 8% more for every 10% decrease in subfield representation. Very low scores were particularly “sticky” and seldom updated upward, suggesting a possible source of conservatism in evaluation. These systematic differences in how world-class experts respond to external opinions can lead to substantial gender and status disparities in whose opinion ultimately matters in collective expert judgment.
Jana Gallus, Olivia S. Jung, and Karim R. Lakhani. Working Paper. “Managerial Recognition as an Incentive for Innovation Platform Engagement: A Field Experiment and Interview Study at NASA.” HBS Working Paper Series. Publisher's Version 20-059.pdf
Forthcoming
Jacqueline N. Lane, Ina Ganguli, Patrick Gaule, Eva C. Guinan, and Karim R. Lakhani. Forthcoming. “Engineering Serendipity: When Does Knowledge Sharing Lead to Knowledge Production?” Strategic Management Journal. Publisher's VersionAbstract

We investigate how knowledge similarity between two individuals is systematically related to the likelihood that a serendipitous encounter results in knowledge production. We conduct a field experiment at a medical research symposium, where we exogenously varied opportunities for face‐to‐face encounters among 15,817 scientist‐pairs. Our data include direct observations of interaction patterns collected using sociometric badges, and detailed, longitudinal data of the scientists' postsymposium publication records over 6 years. We find that interacting scientists acquire more knowledge and coauthor 1.2 more papers when they share some overlapping interests, but cite each other's work between three and seven times less when they are from the same field. Our findings reveal both collaborative and competitive effects of knowledge similarity on knowledge production outcomes.

Click here to view the video abstract.

 

Engineering_serendipity.pdf
2023
Olivia S Jung, Fahima Begum, Andrea Dorbu, Sara J Singer, and Patricia Satterstrom. 7/17/2023. “Ideas from the Frontline: Improvement Opportunities in Federally Qualified Health Centers.” Journal of General Internal Medicine. Publisher's VersionAbstract

Background

Engaging frontline clinicians and staff in quality improvement is a promising bottom-up approach to transforming primary care practices. This may be especially true in federally qualified health centers (FQHCs) and similar safety-net settings where large-scale, top-down transformation efforts are often associated with declining worker morale and increasing burnout. Innovation contests, which decentralize problem-solving, can be used to involve frontline workers in idea generation and selection.

Objective

We aimed to describe the ideas that frontline clinicians and staff suggested via organizational innovation contests in a national sample of 54 FQHCs.

Interventions

Innovation contests solicited ideas for improving care from all frontline workers—regardless of professional expertise, job title, and organizational tenure and excluding those in senior management—and offered opportunities to vote on ideas.

Participants

A total of 1,417 frontline workers across all participating FQHCs generated 2,271 improvement opportunities.

Approaches

We performed a content analysis and organized the ideas into codes (e.g., standardization, workplace perks, new service, staff relationships, community development) and categories (e.g., operations, employees, patients).

Key Results

Ideas from frontline workers in participating FQHCs called attention to standardization (n = 386, 17%), staffing (n = 244, 11%), patient experience (n = 223, 10%), staff training (n = 145, 6%), workplace perks (n = 142, 6%), compensation (n = 101, 5%), new service (n = 92, 4%), management-staff relationships (n = 82, 4%), and others. Voting results suggested that staffing resources, standardization, and patient communication were key issues among workers.

Conclusions

Innovation contests generated numerous ideas for improvement from the frontline. It is likely that the issues described in this study have become even more salient today, as the COVID-19 pandemic has had devastating impacts on work environments and health/social needs of patients living in low-resourced communities. Continued work is needed to promote learning and information exchange about opportunities to improve and transform practices between policymakers, managers, and providers and staff at the frontlines.

ideas_frontline_fqhcs.pdf
Steven Randazzo, Jin H Paik, and Yael Grushka-Cockayne. 2023. “Commonwealth Bank: Amplifying Customer Centricity with AI.” Management and Business Review, 3, 1&2, Pp. 140-143. Publisher's Version mbr_ai-customer-engagement-issue.pdf
2022
Ademir Vrolijk and Zoe Szajnfarber. 12/20/2022. “The Opportunists in Innovation Contests Understanding Whom to Attract and How to Attract Them.” Research-Technology Management, 66, 1, Pp. 30-40. Publisher's VersionAbstract
Organizations increasingly turn to innovation contests for solutions to their complex problems. But these contests still face a fundamental inefficiency: they need to attract many participants to find the right solution, resulting in high costs and uncertainty. Studies have identified multiple dichotomies of successful and unsuccessful solver types, but these diverge. These studies also offer little guidance on how to attract successful solver types. We introduce the opportunist-transactor dichotomy, bridging whom to attract and how to attract them. Opportunists view the contest as a onramp to a new pursuit instead of a temporary undertaking. Characterizing solvers according to this new dichotomy was a better predictor of success than existing ones: in our context, most winners were opportunists. This type of solver was also reliably attracted by the seeker’s in-kind incentives, unlike those described by the other dichotomies. Our insights provide a deeper understanding of participants in complex contests and a concrete lever for influencing who shows up to solve.
Milena Tsvetkova, Sebastian Müller, Oana Vuculescu, Haylee Ham, and Rinat A Sergeev. 11/11/2022. “Relative Feedback Increases Disparities in Effort and Performance in Crowdsourcing Contests: Evidence from a Quasi-Experiment on Topcoder.” Proceedings of the ACM on Human-Computer Interaction, 6, CSW2, Pp. 1-27. Publisher's VersionAbstract
Rankings and leaderboards are often used in crowdsourcing contests and online communities to motivate individual contributions but feedback based on social comparison can also have negative effects. Here, we study the unequal effects of such feedback on individual effort and performance for individuals of different ability. We hypothesize that the effects of social comparison differ for top performers and bottom performers in a way that the inequality between the two increases. We use a quasi-experimental design to test our predictions with data from Topcoder, a large online crowdsourcing platform that publishes computer programming contests. We find that in contests where the submitted code is evaluated against others' submissions, rather than using an absolute scale, top performers increase their effort while bottom performers decrease it. As a result, relative scoring leads to better outcomes for those at the top but lower engagement for bottom performers. Our findings expose an important but overlooked drawback from using gamified competitions, rankings, and relative evaluations, with potential implications for crowdsourcing markets, online learning environments, online communities, and organizations in general.
Misha Teplitskiy, Eamon Duede, Michael Menietti, and Karim R. Lakhani. 5/2022. “How status of research papers affects the way they are read and cited”. Publisher's VersionAbstract
Although citations are widely used to measure the influence of scientific works, research shows that many citations serve rhetorical functions and reflect little-to-no influence on the citing authors. If highly cited papers disproportionately attract rhetorical citations then their citation counts may reflect rhetorical usefulness more than influence. Alternatively, researchers may perceive highly cited papers to be of higher quality and invest more effort into reading them, leading to disproportionately substantive citations. We test these arguments using data on 17,154 randomly sampled citations collected via surveys from 9,380 corresponding authors in 15 fields. We find that most citations (54%) had little-to-no influence on the citing authors. However, citations to the most highly cited papers were 2–3 times more likely to denote substantial influence. Experimental and correlational data show a key mechanism: displaying low citation counts lowers perceptions of a paper's quality, and papers with poor perceived quality are read more superficially. The results suggest that higher citation counts lead to more meaningful engagement from readers and, consequently, the most highly cited papers influence the research frontier much more than their raw citation counts imply.
Frank Nagle, James Dana, Jennifer Hoffman, Steven Randazzo, and Yanuo Zhou. 3/2/2022. Census II of Free and Open Source Software — Application Libraries. The Linux Foundation. Harvard Laboratory for Innovation Science (LISH) and Open Source Security Foundation (OpenSSF). Publisher's VersionAbstract

Free and Open Source Software (FOSS) has become a critical part of the modern economy. There are tens of millions of FOSS projects, many of which are built into software and products we use every day. However, it is difficult to fully understand the health, economic value, and security of FOSS because it is produced in a decentralized and distributed manner. This distributed development approach makes it unclear how much FOSS, and precisely what FOSS projects, are most widely used. This lack of understanding is a critical problem faced by those who want to help enhance the security of FOSS (e.g., companies, governments, individuals), yet do not know what projects to start with. This problem has garnered widespread attention with the Heartbleed and log4shell vulnerabilities that resulted in the susceptibility of hundreds of millions of devices to exploitation.

This report, Census II, is the second investigation into the widespread use of FOSS and aggregates data from over half a million observations of FOSS libraries used in production applications at thousands of companies, which aims to shed light on the most commonly used FOSS packages at the application library level. This effort builds on the Census I report that focused on the lower level critical operating system libraries and utilities, improving our understanding of the FOSS packages that software applications rely on. Such insights will help to identify critical FOSS packages to allow for resource prioritization to address security issues in this widely used software.

The Census II effort utilizes data from partner Software Composition Analysis (SCA) companies including Snyk, the Synopsys Cybersecurity Research Center (CyRC), and FOSSA, which partnered with Harvard to advance the state of open source research. Our goal is to not only identify the most widely used FOSS, but to also provide an example of how the distributed nature of FOSS requires a multi-party effort to fully understand the value and security of the FOSS ecosystem. Only through data-sharing, coordination, and investment will the value of this critical component of the digital economy be preserved for generations to come.

In addition to the detailed results on FOSS usage provided in the report, we identified five high-level findings: 1) the need for a standardized naming schema for software components, 2) the complexities associated with package versions, 3) much of the most widely used FOSS is developed by only a handful of contributors, 4) the increasing importance of individual developer account security, and 5) the persistence of legacy software in the open source space.

lfresearch_harvard_census_ii_1.pdf
Jaehan Cho, Timothy DeStefano, Hanhin Kim, Inchul Kim, and Jin Hyun Paik. 2/19/2022. “What's driving the diffusion of next-generation digital technologies?” Technovation. Publisher's VersionAbstract
The recent development and diffusion of next-generation digital technologies (NGDTs) such as artificial intelligence, the Internet of Things, big data, 3D printing, and so on are expected to have an immense impact on businesses, innovation, and society. While we know from extant research that a firm's R&D investment, intangible assets, and productivity are factors that influence technology use more generally, to date there is little known about the factors that determine how these emerging tools are used, and by who. Using Probit and OLS modeling on a survey of 12,579 South Korean firms in 2017, we conduct one of the first comprehensive examinations highlighting various firm characteristics that drive NGDT implementation. While much of the literature assesses the use of individual technologies, our research attempts to unveil the extent to which firms implement NGDTs in bundles. Our investigation shows that more than half of the firms that use NGDTs deployed multiple technologies simultaneously. One of the insightful complementarities identified in this research exists amongst technologies that generate, facilitate and demand large sums of data, including big data, IoT, cloud computing and AI. Such technologies also appear important for innovative tools such as 3D printing and robotics.
2021
Karim R. Lakhani, Yael Grushka-Cockayne, Jin H. Paik, and Steven Randazzo. 10/2021. “Customer-Centric Design with Artificial Intelligence: Commonwealth Bank”. Publisher's VersionAbstract
As Commonwealth Bank (CommBank) CEO Matt Comyn delivered the full financial year results in August 2021 over videoconference, it took less than two minutes for him to make his first mention of the organization's Customer Engagement Engine (CEE), the AI-driven customer experience platform. With full cross-channel integration, CEE operated using 450 machine learning models that learned from a total of 157 billion data points. Against the backdrop of a once-in-a century global pandemic, CEE had helped the Group deliver a strong financial performance while also supporting customers with assistance packages designed in response to the coronavirus outbreak. Six years earlier, in 2015, financial services were embarking on a transformation driven by the increased availability and standardization of data and artificial intelligence (AI). Speed, access and price, once key differentiators for attracting and retaining customers, had been commoditized by AI, and new differentiators such as customization and enhanced interactions were expected. Seeking to create value for customers through an efficient, data-driven practice, CommBank leveraged existing channels of operations. Angus Sullivan, Group Executive of Retail Banking, remarked, "How do we, over thousands of interactions, try and generate the same outcomes as from a really in-depth, one-to-one conversation?" The leadership team began to make key investments in data and infrastructure. While some headway had been made, newly appointed Chief Data and Analytics Officer, Andrew McMullan, was brought in to catalyze the process and progress of the leadership's vision for a new customer experience. Success would depend on continued drive from leadership, buy-in from frontline staff, and a reliable team of passionate and knowledgeable data professionals. How did Comyn and McMullan bring their vision to life: to deliver better outcomes through a new approach to customer-centricity? How did they overcome internal resistance, data sharing barriers, and requirements for technical capabilities?
Iavor Bojinov, Prithwiraj Choudhury, and Jacqueline N. Lane. 5/2021. “Virtual Watercoolers: A Field Experiment on Virtual Synchronous Interactions and Performance of Organizational Newcomers.” SSRN, Harvard Business School Technology & Operations Mgt. Unit Working Paper , Pp. 21-125. Publisher's VersionAbstract
Do virtual, yet informal and synchronous, interactions affect individual performance outcomes of organizational newcomers? We report results from a randomized field experiment conducted at a large global organization that estimates the performance effects of “virtual water coolers” for remote interns participating in the firm’s flagship summer internship program. Findings indicate that interns who had randomized opportunities to interact synchronously and informally with senior managers were significantly more likely to receive offers for full-time employment, achieved higher weekly performance ratings, and had more positive attitudes toward their remote internships. Further, we observed stronger results when the interns and senior managers were demographically similar. Secondary results also hint at a possible abductive explanation of the performance effects: virtual watercoolers between interns and senior managers may have facilitated knowledge and advice sharing. This study demonstrates that hosting brief virtual water cooler sessions with senior managers might have job and career benefits for organizational newcomers working in remote workplaces, an insight with immediate managerial relevance.
SSRN Virtual Watercoolers: A Field Experiment on Virtual Synchronous Interactions and Performance of Organizational Newcomers
Andrea Blasco, Ted Natoli, Michael G. Endres, Rinat A. Sergeev, Steven Randazzo, Jin Paik, Max Macaluso, Rajiv Narayan, Karim R. Lakhani, and Aravind Subramaniam. 4/6/2021. “Improving Deconvolution Methods in Biology through Open Innovation Competitions: An Application to the Connectivity Map.” Bioinformatics. Publisher's VersionAbstract
Do machine learning methods improve standard deconvolution techniques for gene expression data? This article uses a unique new dataset combined with an open innovation competition to evaluate a wide range of approaches developed by 294 competitors from 20 countries. The competition’s objective was to address a deconvolution problem critical to analyzing genetic perturbations from the Connectivity Map. The issue consists of separating gene expression of individual genes from raw measurements obtained from gene pairs. We evaluated the outcomes using ground-truth data (direct measurements for single genes) obtained from the same samples.
Henry Eyring, Patrick J. Ferguson, and Sebastian Koppers. 3/30/2021. “Less Information, More Comparison, and Better Performance: Evidence from a Field Experiment.” Journal of Accounting Research , 59, 2, Pp. 657-711. Publisher's VersionAbstract
We use a field experiment in professional sports to compare effects of providing absolute, relative, or both absolute and relative measures in performance reports for employees. Although studies have documented that the provision of these types of measures can benefit performance, theory from economic and accounting literature suggests that it may be optimal for firms to direct employees’ attention to some types of measures by omitting others. In line with this theory, we find that relative performance information alone yields the best performance effects in our setting—that is, that a subset of information (relative performance information) dominates the full information set (absolute and relative performance information together) in boosting performance. In cross-sectional and survey-data analyses, we do not find that restricting the number of measures shown per se benefits performance. Rather, we find that restricting the type of measures shown to convey only relative information increases involvement in peer-performance comparison, benefitting performance. Our findings extend research on weighting of and responses to measures in performance reports.
eyring_et_al_2021.pdf
Philip Brookins, Dmitry Ryvkin, and Andrew Smyth. 3/8/2021. “Indefinitely repeated contests: An experimental study.” Experimental Economics . Publisher's VersionAbstract
We experimentally explore indefinitely repeated contests. Theory predicts more cooperation, in the form of lower expenditures, in indefinitely repeated contests with a longer expected time horizon. Our data support this prediction, although this result attenuates with contest experience. Theory also predicts more cooperation in indefinitely repeated contests compared to finitely repeated contests of the same expected length, and we find empirical support for this. Finally, theory predicts no difference in cooperation across indefinitely repeated winner-take-all and proportional-prize contests, yet we find evidence of less cooperation in the latter, though only in longer treatments with more contests played. Our paper extends the experimental literature on indefinitely repeated games to contests and, more generally, contributes to an infant empirical literature on behavior in indefinitely repeated games with “large” strategy spaces.
Brookins - Indefinitely Repeated Contests
Philip Brookins and Paan Jindapon. 2/20/2021. “Risk preference heterogeneity in group contests.” Journal of Mathematical Economics. Publisher's VersionAbstract
We analyze the first model of a group contest with players that are heterogeneous in their risk preferences. In our model, individuals’ preferences are represented by a utility function exhibiting a generalized form of constant absolute risk aversion, allowing us to consider any combination of risk-averse, risk-neutral, and risk-loving players. We begin by proving equilibrium existence and uniqueness under both linear and convex investment costs. Then, we explore how the sorting of a compatible set of players by their risk attitudes into competing groups affects aggregate investment. With linear costs, a balanced sorting (i.e., minimizing the variance in risk attitudes across groups) always produces an aggregate investment level that is at least as high as an unbalanced sorting (i.e., maximizing the variance in risk attitudes across groups). Under convex costs, however, identifying which sorting is optimal is more nuanced and depends on preference and cost parameters.
Brookins - Risk Preference Heterogeneity
2020
Hannah Mayer. 7/2020. “AI in Enterprise: AI Product Management.” Edited by Jin H. Paik, Jenny Hoffman, and Steven Randazzo.Abstract

While there are dispersed resources to learn more about artificial intelligence, there remains a need to cultivate a community of practitioners for cyclical exposure and knowledge sharing of best practices in the enterprise. That is why Laboratory for Innovation Science at Harvard launched the AI in the Enterprise series, which exposes managers and executives to interesting applications of AI and the decisions behind developing such tools. 

Moderated by HBS Professor and co-author of Competing in the Age of AI, Karim R. Lakhani, the July virtual session featured Peter Skomoroch from DataWrangling and formerly at LinkedIn. Together, they discussed what differentiates AI product management from managing other tech products and how to adapt to the uncertainty in the AI product lifecycle.

AI in Enterprise - AI Product Management (P Skomoroch).pdf
Gerard George, Karim R. Lakhani, and Phanish Puranam. 12/2020. “What Has Changed? The Impact of COVID Pandemic on the Technology and Innovation Management Research Agenda.” Journal of Management Studies, 57, 8, Pp. 1754-1758. Publisher's VersionAbstract
Whereas the pandemic has tested the agility and resilience of organizations, it forces a deeper look at the assumptions underlying theoretical frameworks that guide managerial decisions and organizational practices. In this commentary, we explore the impact of the Covid‐19 pandemic on technology and innovation management research. We identify key assumptions, and then, discuss how new areas of investigation emerge based on the changed reality.
Karim R. Lakhani, Anne-Laure Fayard, Manos Gkeredakis, and Jin Hyun Paik. 10/5/2020. “OpenIDEO (B)”. Publisher's VersionAbstract
In the midst of 2020, as the coronavirus pandemic was unfolding, OpenIDEO - an online open innovation platform focused on design-driven solutions to social issues - rapidly launched a new challenge to improve access to health information, empower communities to stay safe during the COVID-19 crisis, and inspire global leaders to communicate effectively. OpenIDEO was particularly suited to challenges which required cross-system or sector-wide collaboration due to its focus on social impact and ecosystem design, but its leadership pondered how they could continue to improve virtual collaboration and to share their insights from nearly a decade of running online challenges. Conceived as an exercise of disruptive digital innovation, OpenIDEO successfully created a strong open innovation community, but how could they sustain - or even improve - their support to community members and increase the social impact of their online challenges in the coming years?
Hannah Mayer. 10/2020. “Data Science is the New Accounting.” Edited by Jin H. Paik and Jenny Hoffman.Abstract

In the October session of the AI in Enterprise series, HBS Professor and co-author of Competing in the Age of AI, Karim R. Lakhani and Roger Magoulas (Data Science Advisor) delved into O'Reilly's most recent survey of AI adoption in larger companies. The discussion explored common risk factors, techniques, tools, as well as the data governance and data conditioning that large companies are using to build and scale their AI practices. 

 

Read Hannah Mayer's recap of the event to learn more about what senior managers in enterprises need to know about AI - particularly, if they want to adopt at scale. 

 

AI in Entreprise - Data is the New Accounting (R Magoulas)

Pages