Organization & Processes

Working Paper
Misha Teplitskiy, Hardeep Ranu, Gary Gray, Michael Menietti, Eva Guinan, and Karim Lakhani. Working Paper. “Do Experts Listen to Other Experts? Field Experimental Evidence from Scientific Peer Review.” HBS Working Paper Series. Publisher's VersionAbstract
Organizations in science and elsewhere often rely on committees of experts to make important decisions, such as evaluating early-stage projects and ideas. However, very little is known about how experts influence each other’s opinions and how that influence affects final evaluations. Here, we use a field experiment in scientific peer review to examine experts’ susceptibility to the opinions of others. We recruited 277 faculty members at seven U.S. medical schools to evaluate 47 early stage research proposals in biomedicine. In our experiment, evaluators (1) completed independent reviews of research ideas, (2) received (artificial) scores attributed to anonymous “other reviewers” from the same or a different discipline, and (3) decided whether to update their initial scores. Evaluators did not meet in person and were not otherwise aware of each other. We find that, even in a completely anonymous setting and controlling for a range of career factors, women updated their scores 13% more often than men, while very highly cited “superstar” reviewers updated 24% less often than others. Women in male-dominated subfields were particularly likely to update, updating 8% more for every 10% decrease in subfield representation. Very low scores were particularly “sticky” and seldom updated upward, suggesting a possible source of conservatism in evaluation. These systematic differences in how world-class experts respond to external opinions can lead to substantial gender and status disparities in whose opinion ultimately matters in collective expert judgment.
2023
Olivia S Jung, Fahima Begum, Andrea Dorbu, Sara J Singer, and Patricia Satterstrom. 7/17/2023. “Ideas from the Frontline: Improvement Opportunities in Federally Qualified Health Centers.” Journal of General Internal Medicine. Publisher's VersionAbstract

Background

Engaging frontline clinicians and staff in quality improvement is a promising bottom-up approach to transforming primary care practices. This may be especially true in federally qualified health centers (FQHCs) and similar safety-net settings where large-scale, top-down transformation efforts are often associated with declining worker morale and increasing burnout. Innovation contests, which decentralize problem-solving, can be used to involve frontline workers in idea generation and selection.

Objective

We aimed to describe the ideas that frontline clinicians and staff suggested via organizational innovation contests in a national sample of 54 FQHCs.

Interventions

Innovation contests solicited ideas for improving care from all frontline workers—regardless of professional expertise, job title, and organizational tenure and excluding those in senior management—and offered opportunities to vote on ideas.

Participants

A total of 1,417 frontline workers across all participating FQHCs generated 2,271 improvement opportunities.

Approaches

We performed a content analysis and organized the ideas into codes (e.g., standardization, workplace perks, new service, staff relationships, community development) and categories (e.g., operations, employees, patients).

Key Results

Ideas from frontline workers in participating FQHCs called attention to standardization (n = 386, 17%), staffing (n = 244, 11%), patient experience (n = 223, 10%), staff training (n = 145, 6%), workplace perks (n = 142, 6%), compensation (n = 101, 5%), new service (n = 92, 4%), management-staff relationships (n = 82, 4%), and others. Voting results suggested that staffing resources, standardization, and patient communication were key issues among workers.

Conclusions

Innovation contests generated numerous ideas for improvement from the frontline. It is likely that the issues described in this study have become even more salient today, as the COVID-19 pandemic has had devastating impacts on work environments and health/social needs of patients living in low-resourced communities. Continued work is needed to promote learning and information exchange about opportunities to improve and transform practices between policymakers, managers, and providers and staff at the frontlines.

ideas_frontline_fqhcs.pdf
2021
Karim R. Lakhani, Yael Grushka-Cockayne, Jin H. Paik, and Steven Randazzo. 10/2021. “Customer-Centric Design with Artificial Intelligence: Commonwealth Bank”. Publisher's VersionAbstract
As Commonwealth Bank (CommBank) CEO Matt Comyn delivered the full financial year results in August 2021 over videoconference, it took less than two minutes for him to make his first mention of the organization's Customer Engagement Engine (CEE), the AI-driven customer experience platform. With full cross-channel integration, CEE operated using 450 machine learning models that learned from a total of 157 billion data points. Against the backdrop of a once-in-a century global pandemic, CEE had helped the Group deliver a strong financial performance while also supporting customers with assistance packages designed in response to the coronavirus outbreak. Six years earlier, in 2015, financial services were embarking on a transformation driven by the increased availability and standardization of data and artificial intelligence (AI). Speed, access and price, once key differentiators for attracting and retaining customers, had been commoditized by AI, and new differentiators such as customization and enhanced interactions were expected. Seeking to create value for customers through an efficient, data-driven practice, CommBank leveraged existing channels of operations. Angus Sullivan, Group Executive of Retail Banking, remarked, "How do we, over thousands of interactions, try and generate the same outcomes as from a really in-depth, one-to-one conversation?" The leadership team began to make key investments in data and infrastructure. While some headway had been made, newly appointed Chief Data and Analytics Officer, Andrew McMullan, was brought in to catalyze the process and progress of the leadership's vision for a new customer experience. Success would depend on continued drive from leadership, buy-in from frontline staff, and a reliable team of passionate and knowledgeable data professionals. How did Comyn and McMullan bring their vision to life: to deliver better outcomes through a new approach to customer-centricity? How did they overcome internal resistance, data sharing barriers, and requirements for technical capabilities?
2020
Hannah Mayer. 7/2020. “AI in Enterprise: AI Product Management.” Edited by Jin H. Paik, Jenny Hoffman, and Steven Randazzo.Abstract

While there are dispersed resources to learn more about artificial intelligence, there remains a need to cultivate a community of practitioners for cyclical exposure and knowledge sharing of best practices in the enterprise. That is why Laboratory for Innovation Science at Harvard launched the AI in the Enterprise series, which exposes managers and executives to interesting applications of AI and the decisions behind developing such tools. 

Moderated by HBS Professor and co-author of Competing in the Age of AI, Karim R. Lakhani, the July virtual session featured Peter Skomoroch from DataWrangling and formerly at LinkedIn. Together, they discussed what differentiates AI product management from managing other tech products and how to adapt to the uncertainty in the AI product lifecycle.

AI in Enterprise - AI Product Management (P Skomoroch).pdf
Karim R. Lakhani, Anne-Laure Fayard, Manos Gkeredakis, and Jin Hyun Paik. 10/5/2020. “OpenIDEO (B)”. Publisher's VersionAbstract
In the midst of 2020, as the coronavirus pandemic was unfolding, OpenIDEO - an online open innovation platform focused on design-driven solutions to social issues - rapidly launched a new challenge to improve access to health information, empower communities to stay safe during the COVID-19 crisis, and inspire global leaders to communicate effectively. OpenIDEO was particularly suited to challenges which required cross-system or sector-wide collaboration due to its focus on social impact and ecosystem design, but its leadership pondered how they could continue to improve virtual collaboration and to share their insights from nearly a decade of running online challenges. Conceived as an exercise of disruptive digital innovation, OpenIDEO successfully created a strong open innovation community, but how could they sustain - or even improve - their support to community members and increase the social impact of their online challenges in the coming years?
Hannah Mayer. 10/2020. “Data Science is the New Accounting.” Edited by Jin H. Paik and Jenny Hoffman.Abstract

In the October session of the AI in Enterprise series, HBS Professor and co-author of Competing in the Age of AI, Karim R. Lakhani and Roger Magoulas (Data Science Advisor) delved into O'Reilly's most recent survey of AI adoption in larger companies. The discussion explored common risk factors, techniques, tools, as well as the data governance and data conditioning that large companies are using to build and scale their AI practices. 

 

Read Hannah Mayer's recap of the event to learn more about what senior managers in enterprises need to know about AI - particularly, if they want to adopt at scale. 

 

AI in Entreprise - Data is the New Accounting (R Magoulas)
Hannah Mayer. 9/2020. “AI in Enterprise: In Tech We Trust.. Maybe Too Much?Edited by Jin H. Paik and Jenny Hoffman.Abstract

While there are dispersed resources to learn more about artificial intelligence, there remains a need to cultivate a community of practitioners for cyclical exposure and knowledge sharing of best practices in the enterprise. That is why Laboratory for Innovation Science at Harvard launched the AI in the Enterprise series, which exposes managers and executives to interesting applications of AI and the decisions behind developing such tools. 

In the September session of the AI in Enterprise series, HBS Professor and co-author of Competing in the Age of AI, Karim R. Lakhani spoke with Latanya Sweeney about algorithmic bias, data privacy, and the way forward for enterprises adopting AI. They explored how AI and ML can impact society in unexpected ways and what senior enterprise leaders can do to avoid negative externalities. Professor of the Practice of Government and Technology at the Harvard Kennedy School and in the Harvard Faculty of Arts and Sciences, director and founder of the Data Privacy Lab, and former Chief Technology Officer at the U.S. Federal Trade Commission, Latanya Sweeney pioneered the field known as data privacy and launched the emerging area known as algorithmic fairness.

AI in Enterprise - In Tech We Trust - Maybe Too Much (L Sweeney)
Hannah Mayer, Jin H. Paik, Timothy DeStefano, and Jenny Hoffman. 8/2020. “From Craft to Commodity: The Evolution of AI in Pharma and Beyond”.Abstract

While there are dispersed resources to learn more about artificial intelligence, there remains a need to cultivate a community of practitioners for cyclical exposure and knowledge sharing of best practices in the enterprise. That is why Laboratory for Innovation Science at Harvard launched the AI in the Enterprise series, which exposes managers and executives to interesting applications of AI and the decisions behind developing such tools. 

Moderated by HBS Professor and co-author of Competing in the Age of AI, Karim R. Lakhani, the August virtual session featured Reza Olfati-Saber, an experienced academic researcher currently managing teams of data scientists and life scientists across the globe for Sanofi. Together, they discussed the evolution of AI in life science experimentation and how it may become the determining factor for R&D success in pharma and other industries.

AI in Enterprise - From Craft to Commodity (R Olfati-Saber).pdf
Jin H. Paik, Steven Randazzo, and Jenny Hoffman. 6/2020. “AI in the Enterprise: How Do I Get Started?”.Abstract

While there are dispersed resources to learn more about artificial intelligence, there remains a need to cultivate a community of practitioners for cyclical exposure and knowledge sharing of best practices in the enterprise. That is why Laboratory for Innovation Science at Harvard launched the AI in the Enterprise series, which exposes managers and executives to interesting applications of AI and the decisions behind developing such tools. 

Moderated by HBS Professor and co-author of Competing in the Age of AI, Karim R. Lakhani, the most recent virtual session with over 240 attendees featured Rob May, General Partner at PJC, an early-stage venture capital firm, and founder of Inside AI, a premier source for information on AI, robotics and neurotechnology. Together, they discussed why we have seen a rise in interest in AI, what managers should consider when wading into the AI waters, and what steps they can take when it is time to do so. 

AI in Enterprise - How Do I Get Started (R May).pdf
Marco Iansiti and Karim R. Lakhani. 3/3/2020. “From Disruption to Collision: The New Competitive Dynamics.” MIT Sloan Management Review.Abstract
In the age of AI, traditional businesses across the economy are being attacked by highly scalable data-driven companies whose operating models leverage network effects to deliver value.
2018
Michael Menietti, M.P. Recalde, and L. Vesterlund. 2018. “Charitable Giving in the Laboratory: Advantages of the Piecewise Linear Public Good Game.” In The Economics of Philanthropy: Donations and Fundraising, edited by Mirco Tonin and Kimberley Scharf. MIT Press. Publisher's Version
2017
Kevin Boudreau, Tom Brady, Ina Ganguli, Patrick Gaule, Eva Guinan, Tony Hollenberg, and Karim R. Lakhani. 2017. “A Field Experiment on Search Costs and the Formation of Scientific Collaborations.” The Review of Economics and Statistics, 99, 4, Pp. 565-576. Publisher's VersionAbstract

Scientists typically self-organize into teams, matching with others to collaborate in the production of new knowledge. We present the results of a field experiment conducted at Harvard Medical School to understand the extent to which search costs affect matching among scientific collaborators. We generated exogenous variation in search costs for pairs of potential collaborators by randomly assigning individuals to 90-minute structured information-sharing sessions as part of a grant funding opportunity for biomedical researchers. We estimate that the treatment increases the baseline probability of grant co-application of a given pair of researchers by 75% (increasing the likelihood of a pair collaborating from 0.16 percent to 0.28 percent), with effects higher among those in the same specialization. The findings indicate that matching between scientists is subject to considerable frictions, even in the case of geographically-proximate scientists working in the same institutional context with ample access to common information and funding opportunities.

Field_Experiment_on_Search_Costs.pdf
Olivia Jung, Andrea Blasco, and Karim R. Lakhani. 2017. “Perceived Organizational Support For Learning and Contribution to Improvement by Frontline Staff.” Academy of Management Proceedings, 2017, 1. Publisher's VersionAbstract

Utilizing suggestions from clinicians and administrative staff is associated with process and quality improvement, organizational climate that promotes patient safety, and added capacity for learning. However, realizing improvement through innovative ideas from staff depends on their ability and decision to contribute. We hypothesized that staff perception of whether the organization promotes learning is positively associated with their likelihood to engage in problem solving and speaking up. We conducted our study in a cardiology unit in an academic hospital that hosted an ideation contest that solicited frontline staff to suggest ideas to resolve issues encountered at work. Our primary dependent variable was staff participation in ideation. The independent variables measuring perception of support for learning were collected using the validated 27-item Learning Organization Survey (LOS). To examine the relationships between these variables, we used analysis of variance, logistic regression, and predicted probabilities. We also interviewed 16 contest participants to explain our quantitative results. The study sample consisted of 30% of cardiology unit staff (n=354) that completed the LOS. In total, 72 staff submitted 138 ideas, addressing a range of issues including patient experience, cost of care, workflow, utilization, and access. Figuring out the cost of procedures in the catheterization laboratory and creating a smartphone application that aids patients to navigate through appointments and connect with providers were two of the ideas that won the most number of votes and funding to be implemented in the following year. Participation in ideation was positively associated with staff perception of supportive learning environment. For example, one standard deviation increase in perceived welcome for differences in opinions was associated with a 43% increase in the odds of participating in ideation (OR=1.43, p=0.04) and 55% increase in the odds of suggesting more than one idea (OR=1.55, p=0.09). Experimentation, a practice that supports learning, was negatively associated with ideation (OR=0.36, p=0.02), and leadership that reinforces learning was not associated with ideation. The perception that new ideas are not sufficiently considered or experimented could have motivated staff to participate, as the ideation contest enables experimentation and learning. Interviews with ideation participants revealed that the contest enabled systematic bottom-up contribution to quality improvement, promoted a sense of community, facilitated organizational exchange of ideas, and spread a problem-solving oriented mindset. Enabling frontline staff to feel that their ideas are welcome and that making mistakes is permissible may increase their likelihood to engage in problem solving and speaking up, contributing to organizational improvement.

2016
Kevin J. Boudreau and Karim R. Lakhani. 2016. “Innovation Experiments: Researching Technical Advance, Knowledge Production, and the Design of Supporting Institutions.” In Innovation Policy and the Economy, 16: Pp. 135-167. Chicago, IL. Publisher's VersionAbstract

This paper discusses several challenges in designing field experiments to better understand how organizational and institutional design shapes innovation outcomes and the production of knowledge. We proceed to describe the field experimental research program carried out by our Crowd Innovation Laboratory at Harvard University to clarify how we have attempted to address these research design challenges. This program has simultaneously solved important practical innovation problems for partner organizations, like NASA and Harvard Medical School (HMS), while contributing research advances, particularly in relation to innovation contests and tournaments. We conclude by proceeding to highlight the opportunity for the scholarly community to develop a “science of innovation” that utilized field experiments as means to generate knowledge.

Innovation_Experiments.pdf
Kevin J. Boudreau, Eva C. Guinan, Karim R. Lakhani, and Christoph Riedl. 2016. “Looking Across and Looking Beyond the Knowledge Frontier: Intellectual Distance, Novelty, and Resource Allocation in Science.” Management Science, 62, 10, Pp. 2765-2783. Publisher's VersionAbstract

Selecting among alternative projects is a core management task in all innovating organizations. In this paper, we focus on the evaluation of frontier scientific research projects. We argue that the “intellectual distance” between the knowledge embodied in research proposals and an evaluator’s own expertise systematically relates to the evaluations given. To estimate relationships, we designed and executed a grant proposal process at a leading research university in which we randomized the assignment of evaluators and proposals to generate 2,130 evaluator–proposal pairs. We find that evaluators systematically give lower scores to research proposals that are closer to their own areas of expertise and to those that are highly novel. The patterns are consistent with biases associated with boundedly rational evaluation of new ideas. The patterns are inconsistent with intellectual distance simply contributing “noise” or being associated with private interests of evaluators. We discuss implications for policy, managerial intervention, and allocation of resources in the ongoing accumulation of scientific knowledge.

Looking_Across_and_Looking_Beyond_the_Knowledge_Frontier.pdf
Dietmar Harhoff and Karim R. Lakhani. 2016. Revolutionizing Innovation: Users, Communities, and Open Innovation. Cambridge, MA: MIT Press. Publisher's VersionAbstract

The last two decades have witnessed an extraordinary growth of new models of managing and organizing the innovation process, which emphasize users over producers. Large parts of the knowledge economy now routinely rely on users, communities, and open innovation approaches to solve important technological and organizational problems. This view of innovation, pioneered by the economist Eric von Hippel, counters the dominant paradigm, which casts the profit-seeking incentives of firms as the main driver of technical change. In a series of influential writings, von Hippel and colleagues found empirical evidence that flatly contradicted the producer-centered model of innovation. Since then, the study of user-driven innovation has continued and expanded, with further empirical exploration of a distributed model of innovation that includes communities and platforms in a variety of contexts and with the development of theory to explain the economic underpinnings of this still emerging paradigm. This volume provides a comprehensive and multidisciplinary view of the field of user and open innovation, reflecting advances in the field over the last several decades.

The contributors—including many colleagues of Eric von Hippel—offer both theoretical and empirical perspectives from such diverse fields as economics, the history of science and technology, law, management, and policy. The empirical contexts for their studies range from household goods to financial services. After discussing the fundamentals of user innovation, the contributors cover communities and innovation; legal aspects of user and community innovation; new roles for user innovators; user interactions with firms; and user innovation in practice, describing experiments, toolkits, and crowdsourcing and crowdfunding.

2015
Karim R. Lakhani and Greta Friar. 2015. Nivea (A) and (B). Harvard Business School Teaching Notes. Harvard Business School. Publisher's VersionAbstract

Teaching Note for HBS Cases 614-042 and 614-043.

The first case describes the efforts of Beiersdorf, a worldwide leader in the cosmetics and skin care industries, to generate and commercialize new R&D through open innovation using external crowds and "netnographic" analysis. Beiersdorf, best known for its consumer brand Nivea, has a rigorous R&D process that has led to many successful product launches, but are there areas of customer need that are undervalued by the traditional process? A novel online customer analysis approach suggests untapped opportunities for innovation, but can the company justify a launch based on this new model of research?
The supplementary case follows up on an innovative R&D approach by Beiersdorf, a skin care and cosmetics company. The case relates what happened to the product launched by Beiersdorf, to its Nivea line, following the events of the first case, and how the commercial success of the product informed thinking by leaders in R&D for the future.

2014
Michael L. Tushman, Hila Lifshitz-Assaf, and Kerry Herman. 2014. Houston, We Have a Problem: NASA and Open Innovation (A). Harvard Business School Case. Harvard Business School. Publisher's VersionAbstract
Jeff Davis, director of Space Life Sciences Directorate at NASA, has been working for several years to raise awareness amongst scientists and researchers in his organizations of the benefits of open innovation as a successful and efficient way to collaborate on difficult research problems regarding health and space travel. Despite a number of initiatives, SLSD members have been skeptical about incorporating the approach into their day-to-day research and work, and have resisted Davis's and his strategy team's efforts. The (A) case outlines these efforts and the organization members' reactions. The (B) case details what Davis and the SLSD strategy team learned, and how they adapted their efforts to successfully incorporate open innovation as one of many tools used in collaborative research at NASA.
Michael L. Tushman, Hila Lifshitz-Assaf, and Kerry Herman. 2014. Houston, We Have a Problem: NASA and Open Innovation (B). Harvard Business School Case. Harvard Business School. Publisher's VersionAbstract
Jeff Davis, director of Space Life Sciences Directorate at NASA, has been working for several years to raise awareness amongst scientists and researchers in his organizations of the benefits of open innovation as a successful and efficient way to collaborate on difficult research problems regarding health and space travel. Despite a number of initiatives, SLSD members have been skeptical about incorporating the approach into their day-to-day research and work, and have resisted Davis's and his strategy team's efforts. The (A) case outlines these efforts and the organization members' reactions. The (B) case details what Davis and the SLSD strategy team learned, and how they adapted their efforts to successfully incorporate open innovation as one of many tools used in collaborative research at NASA.
Karim R. Lakhani, Johann Fuller, Volker Bilgram, and Greta Friar. 2014. Nivea (A). Harvard Business School Case. Harvard Business School. Publisher's VersionAbstract

The case describes the efforts of Beiersdorf, a worldwide leader in the cosmetics and skin care industries, to generate and commercialize new R&D through open innovation using external crowds and "netnographic" analysis. Beiersdorf, best known for its consumer brand Nivea, has a rigorous R&D process that has led to many successful product launches, but are there areas of customer need that are undervalued by the traditional process? A novel online customer analysis approach suggests untapped opportunities for innovation, but can the company justify a launch based on this new model of research?

Pages