Publications

Organizations in science and elsewhere often rely on committees of experts to make important decisions, such as evaluating early-stage projects and ideas. However, very little is known about how experts influence each other’s opinions and how that influence affects final evaluations. Here, we use a field experiment in scientific peer review to examine experts’ susceptibility to the opinions of others. We recruited 277 faculty members at seven U.S. medical schools to evaluate 47 early stage research proposals in biomedicine. In our experiment, evaluators (1) completed independent reviews of research ideas, (2) received (artificial) scores attributed to anonymous “other reviewers” from the same or a different discipline, and (3) decided whether to update their initial scores. Evaluators did not meet in person and were not otherwise aware of each other. We find that, even in a completely anonymous setting and controlling for a range of career factors, women updated their scores 13% more often than men, while very highly cited “superstar” reviewers updated 24% less often than others. Women in male-dominated subfields were particularly likely to update, updating 8% more for every 10% decrease in subfield representation. Very low scores were particularly “sticky” and seldom updated upward, suggesting a possible source of conservatism in evaluation. These systematic differences in how world-class experts respond to external opinions can lead to substantial gender and status disparities in whose opinion ultimately matters in collective expert judgment.

We investigate how knowledge similarity between two individuals is systematically related to the likelihood that a serendipitous encounter results in knowledge production. We conduct a field experiment at a medical research symposium, where we exogenously varied opportunities for face‐to‐face encounters among 15,817 scientist‐pairs. Our data include direct observations of interaction patterns collected using sociometric badges, and detailed, longitudinal data of the scientists' postsymposium publication records over 6 years. We find that interacting scientists acquire more knowledge and coauthor 1.2 more papers when they share some overlapping interests, but cite each other's work between three and seven times less when they are from the same field. Our findings reveal both collaborative and competitive effects of knowledge similarity on knowledge production outcomes.

Click here to view the video abstract.

 

Although citations are widely used to measure the influence of scientific works, research shows that many citations serve rhetorical functions and reflect little-to-no influence on the citing authors. If highly cited papers disproportionately attract rhetorical citations then their citation counts may reflect rhetorical usefulness more than influence. Alternatively, researchers may perceive highly cited papers to be of higher quality and invest more effort into reading them, leading to disproportionately substantive citations. We test these arguments using data on 17,154 randomly sampled citations collected via surveys from 9,380 corresponding authors in 15 fields. We find that most citations (54%) had little-to-no influence on the citing authors. However, citations to the most highly cited papers were 2–3 times more likely to denote substantial influence. Experimental and correlational data show a key mechanism: displaying low citation counts lowers perceptions of a paper's quality, and papers with poor perceived quality are read more superficially. The results suggest that higher citation counts lead to more meaningful engagement from readers and, consequently, the most highly cited papers influence the research frontier much more than their raw citation counts imply.

Free and Open Source Software (FOSS) has become a critical part of the modern economy. There are tens of millions of FOSS projects, many of which are built into software and products we use every day. However, it is difficult to fully understand the health, economic value, and security of FOSS because it is produced in a decentralized and distributed manner. This distributed development approach makes it unclear how much FOSS, and precisely what FOSS projects, are most widely used. This lack of understanding is a critical problem faced by those who want to help enhance the security of FOSS (e.g., companies, governments, individuals), yet do not know what projects to start with. This problem has garnered widespread attention with the Heartbleed and log4shell vulnerabilities that resulted in the susceptibility of hundreds of millions of devices to exploitation.

This report, Census II, is the second investigation into the widespread use of FOSS and aggregates data from over half a million observations of FOSS libraries used in production applications at thousands of companies, which aims to shed light on the most commonly used FOSS packages at the application library level. This effort builds on the Census I report that focused on the lower level critical operating system libraries and utilities, improving our understanding of the FOSS packages that software applications rely on. Such insights will help to identify critical FOSS packages to allow for resource prioritization to address security issues in this widely used software.

The Census II effort utilizes data from partner Software Composition Analysis (SCA) companies including Snyk, the Synopsys Cybersecurity Research Center (CyRC), and FOSSA, which partnered with Harvard to advance the state of open source research. Our goal is to not only identify the most widely used FOSS, but to also provide an example of how the distributed nature of FOSS requires a multi-party effort to fully understand the value and security of the FOSS ecosystem. Only through data-sharing, coordination, and investment will the value of this critical component of the digital economy be preserved for generations to come.

In addition to the detailed results on FOSS usage provided in the report, we identified five high-level findings: 1) the need for a standardized naming schema for software components, 2) the complexities associated with package versions, 3) much of the most widely used FOSS is developed by only a handful of contributors, 4) the increasing importance of individual developer account security, and 5) the persistence of legacy software in the open source space.

The recent development and diffusion of next-generation digital technologies (NGDTs) such as artificial intelligence, the Internet of Things, big data, 3D printing, and so on are expected to have an immense impact on businesses, innovation, and society. While we know from extant research that a firm's R&D investment, intangible assets, and productivity are factors that influence technology use more generally, to date there is little known about the factors that determine how these emerging tools are used, and by who. Using Probit and OLS modeling on a survey of 12,579 South Korean firms in 2017, we conduct one of the first comprehensive examinations highlighting various firm characteristics that drive NGDT implementation. While much of the literature assesses the use of individual technologies, our research attempts to unveil the extent to which firms implement NGDTs in bundles. Our investigation shows that more than half of the firms that use NGDTs deployed multiple technologies simultaneously. One of the insightful complementarities identified in this research exists amongst technologies that generate, facilitate and demand large sums of data, including big data, IoT, cloud computing and AI. Such technologies also appear important for innovative tools such as 3D printing and robotics.

As Commonwealth Bank (CommBank) CEO Matt Comyn delivered the full financial year results in August 2021 over videoconference, it took less than two minutes for him to make his first mention of the organization's Customer Engagement Engine (CEE), the AI-driven customer experience platform. With full cross-channel integration, CEE operated using 450 machine learning models that learned from a total of 157 billion data points. Against the backdrop of a once-in-a century global pandemic, CEE had helped the Group deliver a strong financial performance while also supporting customers with assistance packages designed in response to the coronavirus outbreak. Six years earlier, in 2015, financial services were embarking on a transformation driven by the increased availability and standardization of data and artificial intelligence (AI). Speed, access and price, once key differentiators for attracting and retaining customers, had been commoditized by AI, and new differentiators such as customization and enhanced interactions were expected. Seeking to create value for customers through an efficient, data-driven practice, CommBank leveraged existing channels of operations. Angus Sullivan, Group Executive of Retail Banking, remarked, "How do we, over thousands of interactions, try and generate the same outcomes as from a really in-depth, one-to-one conversation?" The leadership team began to make key investments in data and infrastructure. While some headway had been made, newly appointed Chief Data and Analytics Officer, Andrew McMullan, was brought in to catalyze the process and progress of the leadership's vision for a new customer experience. Success would depend on continued drive from leadership, buy-in from frontline staff, and a reliable team of passionate and knowledgeable data professionals. How did Comyn and McMullan bring their vision to life: to deliver better outcomes through a new approach to customer-centricity? How did they overcome internal resistance, data sharing barriers, and requirements for technical capabilities?

Do virtual, yet informal and synchronous, interactions affect individual performance outcomes of organizational newcomers? We report results from a randomized field experiment conducted at a large global organization that estimates the performance effects of “virtual water coolers” for remote interns participating in the firm’s flagship summer internship program. Findings indicate that interns who had randomized opportunities to interact synchronously and informally with senior managers were significantly more likely to receive offers for full-time employment, achieved higher weekly performance ratings, and had more positive attitudes toward their remote internships. Further, we observed stronger results when the interns and senior managers were demographically similar. Secondary results also hint at a possible abductive explanation of the performance effects: virtual watercoolers between interns and senior managers may have facilitated knowledge and advice sharing. This study demonstrates that hosting brief virtual water cooler sessions with senior managers might have job and career benefits for organizational newcomers working in remote workplaces, an insight with immediate managerial relevance.

Do machine learning methods improve standard deconvolution techniques for gene expression data? This article uses a unique new dataset combined with an open innovation competition to evaluate a wide range of approaches developed by 294 competitors from 20 countries. The competition’s objective was to address a deconvolution problem critical to analyzing genetic perturbations from the Connectivity Map. The issue consists of separating gene expression of individual genes from raw measurements obtained from gene pairs. We evaluated the outcomes using ground-truth data (direct measurements for single genes) obtained from the same samples.

We use a field experiment in professional sports to compare effects of providing absolute, relative, or both absolute and relative measures in performance reports for employees. Although studies have documented that the provision of these types of measures can benefit performance, theory from economic and accounting literature suggests that it may be optimal for firms to direct employees’ attention to some types of measures by omitting others. In line with this theory, we find that relative performance information alone yields the best performance effects in our setting—that is, that a subset of information (relative performance information) dominates the full information set (absolute and relative performance information together) in boosting performance. In cross-sectional and survey-data analyses, we do not find that restricting the number of measures shown per se benefits performance. Rather, we find that restricting the type of measures shown to convey only relative information increases involvement in peer-performance comparison, benefitting performance. Our findings extend research on weighting of and responses to measures in performance reports.

We experimentally explore indefinitely repeated contests. Theory predicts more cooperation, in the form of lower expenditures, in indefinitely repeated contests with a longer expected time horizon. Our data support this prediction, although this result attenuates with contest experience. Theory also predicts more cooperation in indefinitely repeated contests compared to finitely repeated contests of the same expected length, and we find empirical support for this. Finally, theory predicts no difference in cooperation across indefinitely repeated winner-take-all and proportional-prize contests, yet we find evidence of less cooperation in the latter, though only in longer treatments with more contests played. Our paper extends the experimental literature on indefinitely repeated games to contests and, more generally, contributes to an infant empirical literature on behavior in indefinitely repeated games with “large” strategy spaces.

We analyze the first model of a group contest with players that are heterogeneous in their risk preferences. In our model, individuals’ preferences are represented by a utility function exhibiting a generalized form of constant absolute risk aversion, allowing us to consider any combination of risk-averse, risk-neutral, and risk-loving players. We begin by proving equilibrium existence and uniqueness under both linear and convex investment costs. Then, we explore how the sorting of a compatible set of players by their risk attitudes into competing groups affects aggregate investment. With linear costs, a balanced sorting (i.e., minimizing the variance in risk attitudes across groups) always produces an aggregate investment level that is at least as high as an unbalanced sorting (i.e., maximizing the variance in risk attitudes across groups). Under convex costs, however, identifying which sorting is optimal is more nuanced and depends on preference and cost parameters.

Whereas the pandemic has tested the agility and resilience of organizations, it forces a deeper look at the assumptions underlying theoretical frameworks that guide managerial decisions and organizational practices. In this commentary, we explore the impact of the Covid‐19 pandemic on technology and innovation management research. We identify key assumptions, and then, discuss how new areas of investigation emerge based on the changed reality.

In the midst of 2020, as the coronavirus pandemic was unfolding, OpenIDEO - an online open innovation platform focused on design-driven solutions to social issues - rapidly launched a new challenge to improve access to health information, empower communities to stay safe during the COVID-19 crisis, and inspire global leaders to communicate effectively. OpenIDEO was particularly suited to challenges which required cross-system or sector-wide collaboration due to its focus on social impact and ecosystem design, but its leadership pondered how they could continue to improve virtual collaboration and to share their insights from nearly a decade of running online challenges. Conceived as an exercise of disruptive digital innovation, OpenIDEO successfully created a strong open innovation community, but how could they sustain - or even improve - their support to community members and increase the social impact of their online challenges in the coming years?

In the October session of the AI in Enterprise series, HBS Professor and co-author of Competing in the Age of AI, Karim R. Lakhani and Roger Magoulas (Data Science Advisor) delved into O'Reilly's most recent survey of AI adoption in larger companies. The discussion explored common risk factors, techniques, tools, as well as the data governance and data conditioning that large companies are using to build and scale their AI practices. 

 

Read Hannah Mayer's recap of the event to learn more about what senior managers in enterprises need to know about AI - particularly, if they want to adopt at scale. 

 

In summer 2020, Stephane Bancel, CEO of biotech firm Moderna, faces several challenges as his company races to develop a vaccine for COVID-19. The case explores how a company builds a digital organization, and leverages artificial intelligence and other digital resources to speed its operations, manage its processes and ensure quality across research, testing and manufacturing. Built from the ground up as such a digital organization, Moderna was able to respond to the challenge of developing a vaccine as soon as the gene sequence for the virus was posted to the Web on January 11, 2020. As the vaccine enters Phase III clinical trials, Bancel considers several issues: How should Bancel and his team balance the demands of developing a vaccine for a virus creating a global pandemic alongside the other important vaccines and therapies in Moderna's pipeline? How should Moderna communicate its goals and vision to investors in this unprecedented time? Should Moderna be concerned it will be pegged as "a COVID-19 company?"

Supervised machine learning (ML) methods are a powerful toolkit for discovering robust patterns in quantitative data. The patterns identified by ML could be used for exploratory inductive or abductive research, or for post hoc analysis of regression results to detect patterns that may have gone unnoticed. However, ML models should not be treated as the result of a deductive causal test. To demonstrate the application of ML for pattern discovery, we implement ML algorithms to study employee turnover at a large technology company. We interpret the relationships between variables using partial dependence plots, which uncover surprising nonlinear and interdependent patterns between variables that may have gone unnoticed using traditional methods. To guide readers evaluating ML for pattern discovery, we provide guidance for evaluating model performance, highlight human decisions in the process, and warn of common misinterpretation pitfalls. The Supporting Information section provides code and data to implement the algorithms demonstrated in this article.