Billions of Conversations Served…

In 2025, Microsoft Foundry APIs alone (supporting custom AI Agents) have processed over 500 Trillion tokens (7x YoY).  In terms of conversations, 500 Trillion tokens is equivalent to hundreds of billions of multi-turn chats.  With as many as 5 billion visits to ChatGPT per a month in 2025 alone, estimates for a total number of human-to-AI conversations in 2025 easily surpass 100 billion.  

We’re not even at the peak yet, with widespread enterprise AI usage just coming online.  For example, only 1.8% of Microsoft's 400M+ Office 365 users are also using Copilot today. And enterprise flavors of Claude (like Claude for Finance) are only beginning to make in-roads.  

Conversational Use Cases

We can break enterprise AI conversations into common categories such as:

• Research & Summarization - Search the web or take existing documents and produce summarizations of the information found.

• Generating Communications - Drafting and refining emails and other documents to use in internal comms or with customers.

• Meeting Productivity - Meeting preparation through summarization & analysis of CRM data.  Post meeting summarization and analysis of meeting transcripts and notes.

• Conversations with Data - Providing data to the AI so that it can be queried and analyzed intuitively and on the fly.

• Knowledge Management - Query across internal knowledge databases.

• Operational Efficiency - Drafting and refining operational documents such as SOPs, Runbooks, job descriptions, and training materials.

Executing one of these workflows may be a single shot prompt such as: 

Please refine the following customer email for me:

Hello Bob, ...

Or, multi-turn flows using outside data sources and tools.   For example:

Please  use the attached document and extract all of the people, their company, and role…

[ response from LLM ]

Great!  Now put into a CSV format and save as a file in excel

[ LLM tool call ]

While there is some incremental value in being able to improve and automate each of these tasks, real value comes from putting these components together into compound workflows that can be automated from a trigger.  In other words, what makes for effective use of AI isn’t the prompt syntax, it is how different prompts and tools are leveraged together to solve a problem.

For example, if a salesperson receives an inquiry from a customer asking about their product roadmap along with an RFP for a specific feature, the following workflow might kick off: 

• execute  knowledge base searches using keywords from the query
• pull up several matching docs and summarize them
• create an RFP response from a template
• use the above context to create a draft response

It’s these kinds of workflows that distinguish the power users of conversational AI from everyone else.

Golden Needle in a Haystack

Complex sequences locked within conversations are the real use case treasures to capture.  But these can be like needles in a haystack, because we need to know:

• What the trigger for the workflow is
• What each of the steps within the workflow are
• When it ends and what the outcome was
• To what degree the outcome was successful or not

After we have this, AI can determine a canonical shape for the workflow and identify other workflow runs with the same shape.

Then the workflow becomes automatable, and we now are able to harness real, measurable value from unstructured exhaust data.  Imagine with the above example, that the salesperson’s workflow is automatically turned into an agent for RFP Response.  The next time a similar email comes in, the RFP Response Agent will automatically draft a reply to be reviewed asynchronously.  The approval or revision of the draft will then provide critical signals for the agent to refine future behavior.  Not only is time saved up front through automation, but a compounding effect occurs as intelligence is captured and harnessed for future decisions.

Once found, the golden needle leads us to more needles hidden in the stack.

‍