A company deploys an AI agent to handle customer support. Within a month, it has held hundreds of thousands of conversations.
Some went well. Some went badly. Some may have failed in ways nobody has caught yet.
That problem is becoming more urgent as agentic AI expands. The market is projected to grow from $7.29 billion in 2025 to $139.19 billion by 2034, but the ability to understand what AI agents are doing in production has not kept pace with their adoption.
Mohammad Al-Rasheed, co-founder and CTO of Moda, works on that second problem. Moda builds monitoring and debugging tools for AI agents, with a focus on turning large volumes of agent conversations into information product teams can use.
The logs nobody reads
The category built to address this problem is often called AI agent observability. The market for those tools is projected to grow from $0.4 billion in 2025 to $7.1 billion by 2035.
The harder problem begins at scale, when the question is not what happened in a single conversation but what is happening across millions of them.
That is the gap Moda set out to address.
When an agent handles millions of interactions, raw logs can become an archive nobody has time to open. Al-Rasheed’s view is that a trace can tell a team what an agent did on one request, but it cannot easily show what users are collectively trying to do.
“A trace is an autopsy. It tells you how one request died,” Al-Rasheed said. “The harder question is which failures are happening a thousand times a week, and there was no good way to answer that short of paying people to read transcripts all day.”
Conversations are the wrong unit
Moda now processes more than 50 million conversations across more than 30 companies. That scale changes the problem.
A single conversation with a consumer agent is rarely about one thing.
Several different intents become one broad label.
Al-Rasheed’s approach was to stop treating the conversation as the main unit of analysis. Moda’s system breaks each conversation into segments, or slices of conversation focused on a single topic.
Across 50 million conversations, that produced more than 150 million segments. Onboarding confusion and billing disputes can then become separate, countable behaviors, even when they appear in the same chat.
“Embed a 200-message transcript and the filler drowns out the intent,” Al-Rasheed said. “The signal is in the segments, so that is what we cluster. Most of the work was refusing to cut corners on that one decision.”
Structure without supervision
The scale also creates technical constraints.
The clustering approach many teams might use can run out of memory at large volumes. Moda was working with roughly 150 million segments, so Al-Rasheed built a compression step that reduces the full dataset to about 1 million representative points, runs the more expensive clustering process on those points and maps the rest back afterward.
A complete pass over the corpus finishes in about 90 minutes on one node, without human labels or predefined categories.
He documented the design in a technical post titled How We Clustered 150 Million Conversation Segments Without a Single Label.
The system groups conversation segments into a browsable hierarchy, then uses a language model to write plain-English labels for the resulting clusters.
On blind review, labels built that way were rated specific enough to act on 78% of the time, compared with 41% for random sampling.
“Every algorithm in there is standard. We invented none of them,” Al-Rasheed said. “The work was making them cooperate on 150 million segments from 30 companies that all talk to their users completely differently. Any threshold I hand-picked was wrong within a week of meeting real traffic.”
Watching behavior change
New segments arrive constantly, and each one has to be sorted quickly. Moda assigns incoming segments to existing clusters in about 7 milliseconds using two independent checks that both have to agree before an assignment holds.
Either check on its own mislabels around 5% of segments. Used together, the error rate falls to 2.3%.
Segments that fail both checks are set aside and re-examined every hour. A growing pile of unclassified examples can be an early sign of a new behavior, complaint or product issue that did not exist previously.
“The interesting things always surface in the noise first,” Al-Rasheed said. “A new complaint, a feature people started misusing, a change that quietly broke something. It is too small to move the top-line numbers for weeks. If you only watch the big clusters, you find out last.”
The data layer is the product
The broader AI industry is learning this problem the expensive way. A widely cited MIT study of enterprise deployments found that 95% of generative AI pilots produced no measurable impact on profit or loss.
In many cases, teams struggle to tell whether a system is working, where it is failing and which failure modes are happening often enough to matter.
Moda’s bet is that this understanding layer becomes a category of its own, similar to the role error monitoring came to play in traditional software.
Al-Rasheed’s clustering system is one version of that idea running at production scale.
“Everyone wants to talk about the model,” Al-Rasheed said. “But the model is close to a commodity now. What you actually own is your users’ data and whether you can read it. Tracing shows you what happened on one request. The real question is why it keeps happening, and that is a data problem, not a model problem.”
©2026 Cox Media Group
