Calculating the ROI for generative AI applied to business research

Technology analysts often mention knowledge management (KM) as one of the most practical early applications of Generative AI (GenAI) in business.  It’s a natural fit because KM involves aggregating, managing, and mining vast amounts of text.

Northern Light plays in this space with our generative AI-based question answering capability embedded in SinglePoint market and competitive intelligence (CI) research platforms.  Our clients have responded enthusiastically to this feature, but as with all investments, companies want more than anecdotal evidence to justify their expenditure.

Generative AI boosts worker performance

A study published in the Fall of 2023 by professors at the Harvard Business School (HBS) and Boston Consulting Group (BCG) using 758 BCG consultants performing competitive analysis assignments found that those consultants using Generative AI produced work 25% faster with 40% higher quality than consultants not using GenAI.

While it is difficult to model the impact of higher quality analysis, it is possible to model the ROI impact of being 25% faster.  But the first question is “faster than what?”

GenAI can be applied to competitive intelligence research

Let’s hypothesize a business research project in which the user is looking for a list of partnerships that a competitor has announced during the past few months.

Without Generative AI, the user executes a search query “[Competitor-A] strategic partnerships” and gets a search result of 20 documents.  To have a comprehensive answer, the user must examine search snippet of one sentence from each document and decide if there is a strategic partnership identified in the document.  Since snippets are not always the best sentence for the user’s intention, it may be that some partnerships are identified in some of the documents but the snippets for those documents do not give a sufficient hint that the document is of interest.  It is safe to say that some partnerships will be missed if the user doesn’t download all 20 search results and scan the document for partnership information.  Allow 1 minute per document for this process, or 20 minutes in total.  In addition, as partnerships are identified, some form of manual list must be maintained of those partnerships found, perhaps with links to source documents.  Perhaps some of the partnerships were very interesting to dig into, so the analyst spends another 60 minutes reading the relevant source documents that the search process surfaced, for a total time expended on the project of 80 minutes.

Using Generative AI, the user would execute a query, “What partnerships has [Competitor A] announced?”.  Twenty seconds later the user would have a list of the partnerships identified by the Generative AI in the 20 documents on the search result.  The list would likely be much more comprehensive than a manually compiled list since the full text of each document would have been consumed by the Generative AI models in generating the result, improving the quality of the research.  The list would be footnoted with links to contributing documents and could be copied and pasted into a report or email message.  Summaries of each partnership would be provided by the Generative AI application.

Generative AI speeds CI research by 25%

So, the first 20 minutes of the research project would be condensed into 20 seconds by the Generative AI.  Our hypothetical user will still spend another 60 minutes digging into the most important partnerships in the surfaced documents, for a total of 60 minutes on the assignment (ignoring the 20 seconds at the beginning for the Generative AI to respond).  Comparing this total to 80 minutes without Generative AI, then the user will have completed his or her work 25% faster, which is comparable to the HBS/BCG result.  Matching the HBS/BCG result gives us confidence in our hypothetical model of a typical business research project.

So, ignoring the issue of the higher quality of work done with Generative AI, we can make a reasonable estimate that for business research questions, Generative AI may save 20 minutes per user session.

All search queries will not need Generative AI, so we should not apply this metric to all user sessions.  At present Northern Light clients are using Generative AI in as much as 8% of user sessions.  Given how new the function is, we do not believe this is representative for the future.  Use of Generative AI will increase dramatically over the next year as users learn it is an option, are trained on how to use it, and experience it for themselves.  Northern Light projects that 20% or more of user sessions will use Generative AI soon; indeed, it may be far more than that.

For these purposes let’s assume that 20% of user sessions use Generative AI.  This 20% will save the 20 minutes (1/3 hour) per user session described above.

ROI of GenAI in SinglePoint: 14x/year

Accordingly, an ROI metric would be:

Number of annual user sessions x .20 x 1/3-hour x $100 per hour = time saving from Generative AI

To illustrate this calculation for an organization with 50,000 annual user sessions on SinglePoint (Northern Light’s average):

50,000 x .20 x 1/3 x 100 = $ 333,330 time savings per year

Northern Light prices our Generative AI add-on at $24,000 per year.  So, using these metrics, the ROI from the SinglePoint Generative AI option is 14x a year.