A serious study of generative AI used in business finds dramatic quantifiable benefits
The general hoopla around generative AI is finally supported by serious academic underpinnings. A recent blog post by Wharton professor Ethan Mollick, who was involved in a rigorous study of ChatGPT-4 use at the Boston Consulting Group (BCG), reports “for 18 different tasks selected to be realistic samples of the kinds of work done . . . consultants using ChatGPT-4 outperformed those who did not, by a lot. On every dimension. Every way we measured performance.”
Mollick writes that BCG consultants using AI “finished 12.2% more tasks on average, completed tasks 25.1% more quickly, and produced 40% higher quality results than those without.”
The study at BCG also found that generative AI works as a skill leveler – “the consultants who scored the worst when we assessed them at the start of the experiment had the biggest jump in their performance, 43%, when they got to use AI. The top consultants still got a boost, but less of one.”
The importance of optimizing generative AI for particular business use cases
Mollick’s post is a fascinating read, in part because the results researchers measured at BCG were obtained using the globally available consumer version of ChatGPT-4 – “no special fine-tuning or prompting,” as he puts it. We believe the opportunity for productivity and other gains will be even greater when generative AI is optimized for specific business use cases and the underlying large language model (LLM) is trained on more reliable content than is available on the internet.
For example, Northern Light is currently deploying generative AI at our Fortune 1000 clients, but it’s not “vanilla” ChatGPT. Rather, our generative AI is a “question answering” application (based on ChatGPT 3.5-Turbo) embedded in our knowledge management platform, optimized for market and competitive intelligence research, and that draws only from thoroughly vetted content – primary and licensed secondary market research, authoritative business news sources, government and industry databases – housed within a company’s strategic research portal.
Not all users of generative AI get it right
We also were struck by an observation from the BCG study that appears in the full paper written by the research team. The authors write, “Professionals who had a negative performance when using [generative] AI tended to blindly adopt its output and interrogate it less.”
We’re not surprised; in fact, we endorse an approach to using our own generative AI application with a similar idea in mind. Our guidance to Northern Light clients is: Generative AI results should be considered just the first step of a user’s research into their topic, not the last word. We advocate (and expect) users drill down into the source documents that provided the answer and read more about it “from the horse’s mouth”. Because at the end of the day, any competitive insights, strategic recommendations, or business decisions are the human being’s work product and responsibility, not the machine’s.
More study on the impact of generative AI in business is needed
It’s great to see academics taking a methodical approach to studying generative AI in the workplace. We expect more such investigations will follow, as the interest in and use of generative AI continues to accelerate in all sectors of business and industry, and across society at large.