How do you decide which business processes to automate with AI?
Score each workflow on four factors: Data volume, data consistency, verifiability and repeatability. According to Bumblebee CEO Hexi Xiao, AI automation works best when the AI handles a manageable amount of consistently structured data, people can check results quickly and the process runs the same way every time. Strong fits include ticket triage, QBR preparation, device monitoring and user provisioning. Deep domain knowledge helps teams score each factor accurately.
Businesses looking for places to deploy AI might be asking the wrong first question.
Instead of asking whether AI can automate a workflow, Hexi Xiao argued they should first ask whether the workflow was worth handing to AI at all. Xiao, CEO of Bumblebee, built his ChannelCon session, “AI Agents That Flopped: Hard Lessons from AI Automation Attempts,” around eight attempts to automate real business processes—including several that failed.
Those failures produced a four-part framework Xiao used to evaluate potential AI automations:
- How much data did the system need to process?
- How consistent was that data?
- How easily could a human verify the result?
- And how repeatable was the process?
The framework is less about finding everything AI could do than spotting the work it probably shouldn’t touch.
Sometimes Simple Work Hides a Lot of Complexity
One of Bumblebee’s early projects seemed almost embarrassingly straightforward: Calculate whether an MSP was making money on its clients. The math wasn’t the problem. The data was.
Determining profitability meant pulling time entries from potentially thousands of tickets. As the amount of information grew, the agent began running into the limits of how much context it could process. Attempts to patch the workflow helped only until the client asked for more data—a year rather than a month, for example. Bumblebee ultimately lost the client.
That failure changed the question Xiao asked before taking on an automation project.
“How much data do you actually need to chew through to understand the entire context of this problem?”
A second attempt revealed another trap. Bumblebee tried using AI to search tickets and comments for information relevant to cybersecurity alerts. This time, the team broke the workload into smaller pieces to avoid its earlier data-volume problem. The result still worked only about 40% of the time.
The information was there, but it wasn’t consistently where the AI expected to find it. For Xiao, that made consistency another warning sign: The more ambiguous the location or structure of the information, the greater the opportunity for hallucination and error.
Automation Can Work and Still Hit a Wall
Billing reconciliation fared much better. Bumblebee reduced a process that had taken five days to two days, then a day and a half, then one. And then progress stopped. The remaining day was largely spent having humans verify discrepancies identified by AI. Each round surfaced new edge cases requiring judgment.
“If you can scale up verifiability, you can achieve almost full autonomy or fully autonomous operation automation,” Xiao said. “But if you cannot make the verification process scalable, you will see gains, but your gains will be capped at whatever that verification process time is.”
That distinction matters. Automation doesn’t have to eliminate human involvement to create value. But its gains may ultimately be capped by the effort required to check its output.
“When some checks are off, we start to find challenges of automation,” Xiao said. “Even if we’re successful, we’re successful up to a limit.” Repeatability created a similar constraint. Bumblebee found that an accounting automation could save a client about four hours each month but adapting it to another client still required roughly 15 hours of setup. The books—and the businesses behind them—were simply different enough that the automation couldn’t be copied as easily as expected.
The Best AI Jobs Might Be the Boring Ones
When a workflow has manageable data volume, consistent information, easily verifiable results and repeatable processes, Xiao said, “magic happens.” His examples were notably unglamorous: Ticket summarization and triage, QBR preparation, device monitoring and user provisioning.
They worked precisely because their boundaries were clearer. The information was manageable and structured. Results could be checked. The same basic action happened repeatedly.
That offers a more useful way to think about AI than dividing projects into successes and failures. Some workflows can be partially automated. Others can produce durable time savings. Still others demand so much human checking, customization or context that automation simply moves the work around.
4 Questions to Ask Before Automating with AI
Xiao’s framework for evaluating a potential AI automation centers on four questions:
- Data Volume: How much information must the AI process to understand the problem?
- Consistency: Is the information structured consistently, and can the AI reliably find what it needs?
- Verifiability: Can a human quickly determine whether the AI produced the right result?
- Repeatability: Does the process work essentially the same way each time?
The ChannelCon audience surfaced a possible fifth consideration: Domain knowledge. Understanding a workflow deeply enough to score those four factors might itself require significant expertise in the business and its processes.
Before the Agent, Understand the Business
Then the audience complicated Xiao’s framework. A discussion about automating an auto-parts help desk quickly exposed problems his four criteria didn’t fully capture. Customers could describe the same part several ways. Different parts could share names. A customer might describe a symptom rather than the component needed. Junior employees might not possess enough expertise to recognize when the AI was wrong.
The conversation prompted an audience member to ask how much business knowledge was required before someone could even score a potential automation accurately.
“I would say the majority of the complexity lies around the discovery process,” Xiao said.
A few minutes later, another attendee offered what might have been the session’s most important revision: “So you need a fifth column in domain knowledge.”
“Scrap the presentation,” Xiao joked. But the exchange sharpened the lesson behind the failures. Choosing the right AI automation isn’t simply a technical exercise. Before deciding whether an agent could do the work, someone must understand the work well enough to know what can go wrong.
Access AI readiness resources on the GTIA Member Portal.
Not a member yet? Join now.