🧭 Dojo Compass
Module: Finance, Risk Management and Long-Term Resilience
Focus Area: Technology, AI and Future Readiness
Key Issue
Orion Services was an ambitious SME that had successfully introduced AI into a growing number of its internal workflows.
Employees used AI to prepare research, draft documents, analyze information, summarize meetings, develop customer materials, and support operational decision-making. To manage the risks associated with AI-generated outputs, the company adopted a human-in-the-loop approach.
On paper, the system appeared responsible.
AI generated the initial output, and a human employee reviewed it before the work was finalized or used.
However, management eventually identified an important weakness.
In many cases, the human role had become largely perfunctory.
Employees quickly reviewed AI-generated outputs, corrected obvious mistakes, and approved the work. Some were so accustomed to the quality of the AI output that they assumed it was broadly reliable. Others felt pressure to preserve the efficiency gains created by AI and therefore spent as little time as possible reviewing the results.
The human was technically “in the loop,” but was not always adding meaningful value.
More importantly, the company had not developed a systematic approach to determining where human intervention created the greatest value, what form that intervention should take, or how human involvement could help improve the workflow over time.
Orion’s management began asking a more sophisticated question:
Rather than simply asking whether a human should remain in the loop, how could the company design human intervention to create the greatest possible value at each stage of an AI-supported workflow?
Facts
Orion employed approximately 400 people across commercial, operational, financial, and professional service functions.
The company had adopted AI quickly. In many workflows, employees had developed an increasingly familiar pattern:
Human request → AI generation → Human review → Final output.
The process was simple and easy to understand.
However, an internal review revealed significant differences in the quality of human intervention.
In one team, employees carefully compared AI-generated analyses against underlying source material, identified recurring weaknesses, and adjusted prompts and workflow instructions to improve future outputs.
In another, employees primarily checked whether the output “looked reasonable.”
The difference was significant.
The first team was using human involvement to improve both the current output and the future workflow.
The second was using human involvement primarily as a final approval mechanism.
Management also discovered that human intervention was often poorly allocated.
Senior employees were spending time checking relatively routine outputs that could have been reviewed using automated validation or lower levels of human oversight.
At the same time, complex outputs involving significant judgment, customer relationships, or strategic implications sometimes received only a superficial review.
The company had treated human review as a relatively uniform control.
But not all workflows required the same type, intensity, or location of human involvement.
A simple factual summary might require occasional sampling and automated quality checks.
A complex strategic recommendation might require deep human analysis.
A customer communication might require human judgment regarding tone and relationships.
A recurring workflow with predictable outputs might initially require intensive human review but progressively less intervention as the process became more reliable.
Orion concluded that its existing approach was static.
It needed a more dynamic model of human value-add.
Solution
Orion redesigned its AI-supported workflows around a new principle:
Human involvement should not be measured by whether a person touched the workflow. It should be measured by the value that human intervention adds to the workflow.
The company began by examining where humans were involved in each major AI-supported process.
It identified several different forms of human value-add.
1. Human Framing
Before AI was used, employees could add value by defining the problem, establishing the objective, identifying relevant constraints, and determining what a useful output should actually look like.
Orion found that poor AI outputs often began with poorly defined tasks.
Rather than simply asking employees to “review the answer,” the company increasingly focused human attention on asking the right question in the first place.
2. Human Judgment
The company distinguished between checking an output for obvious errors and applying genuine professional judgment.
Human reviewers were encouraged to ask:
- Is the AI answering the right question?
- What assumptions is the output making?
- What important information may be missing?
- Does the output conflict with experience or business context?
- Are there alternative interpretations?
- What decision or action should follow?
This transformed review from proofreading into analysis.
3. Human Exception Management
Orion also recognized that humans did not need to review every output equally.
In more mature workflows, the company began designing systems to identify exceptions, uncertainty, unusual results, or risk indicators.
Human attention could then be directed toward the cases where it was most valuable.
Instead of asking people to review one hundred routine outputs, the workflow could identify the five that contained anomalies or fell outside established parameters.
4. Human Improvement
Perhaps the most important change involved using human intervention to improve the workflow itself.
Employees were asked to identify recurring weaknesses in AI outputs.
Were particular errors appearing repeatedly?
Was relevant context missing?
Could the workflow be redesigned?
Could prompts, source information, instructions, validation processes, or technological systems be improved?
Human review was therefore transformed into a feedback mechanism.
The purpose was not simply to correct today’s output.
It was to make tomorrow’s output better.
5. Human Escalation and Decision-Making
Finally, Orion identified the points where human responsibility should remain central.
Certain decisions involved strategic trade-offs, significant financial consequences, customer relationships, ethical considerations, or levels of uncertainty that made automated decision-making inappropriate.
In these areas, AI could support analysis, generate alternatives, or identify relevant information.
But the human role was not merely to approve the AI’s recommendation.
The human remained responsible for making the decision.
From Human-in-the-Loop to Human Value-in-the-Loop
Orion began describing its new approach as Human Value-in-the-Loop.
The distinction was subtle but important.
A traditional human-in-the-loop model asks:
Where should a person review or approve the AI?
A human value-in-the-loop model asks:
Where can human capabilities create the greatest improvement in the quality, reliability, and long-term performance of the workflow?
This required the company to treat human intervention as a dynamic variable.
A new AI-supported workflow might initially require extensive human involvement. Employees would review outputs carefully, identify weaknesses, and improve the system.
As the workflow matured, some forms of human intervention could be reduced.
Routine quality checks could become automated. Human review could shift toward sampling and exception management. Employees could focus on the most difficult or uncertain cases.
The goal was not necessarily to eliminate humans from the process.
It was to progressively move human attention away from routine checking and toward the activities where human judgment created the greatest marginal value.
Orion also began measuring the effectiveness of human intervention.
It examined factors such as:
- How often did human review identify meaningful errors or weaknesses?
- What types of errors were being identified?
- Were the same corrections being made repeatedly?
- Could recurring corrections be incorporated into the workflow?
- How much human time was required for each level of quality improvement?
- Were human reviewers identifying deeper issues or simply correcting formatting and obvious errors?
- As the workflow matured, could human involvement be reduced without reducing output quality?
This created a pathway for continuous improvement.
Key Takeaways
Orion’s experience illustrates several important principles for companies designing AI-supported workflows.
First, having a human in the loop does not automatically create value. A perfunctory review may provide the appearance of control without materially improving the quality or reliability of the output.
Second, human intervention should be designed around comparative advantage. Humans should focus increasingly on framing problems, applying judgment, managing exceptions, understanding context, making complex decisions, and improving the workflow itself.
Third, not every workflow requires the same level of human involvement. The appropriate level of intervention should depend on factors such as risk, complexity, output reliability, uncertainty, and the potential consequences of error.
Fourth, human review should improve future outputs as well as current outputs. If employees repeatedly correct the same type of AI mistake, that correction should become information for redesigning the workflow.
Fifth, the level of human involvement should evolve over time. New workflows may require intensive oversight. As reliability improves, human attention can shift toward sampling, exception management, and higher-value activities.
Finally, the strategic objective is not simply automation. The objective is to create a continuously improving relationship between human and artificial intelligence in which each performs the activities where it creates the greatest value.
For Orion, this represented an important evolution in its AI transformation.
The company stopped treating human involvement primarily as a safety mechanism positioned at the end of an AI workflow.
Instead, it began treating human capability as a strategic resource that could be deliberately positioned throughout the workflow.
The result was a more dynamic model:
Humans framed the problem. AI accelerated the work. Humans applied judgment where it mattered most. The workflow identified exceptions and uncertainty. Human feedback improved the system. And, as the workflow matured, routine human intervention could progressively decline while output quality improved.
The broader lesson was straightforward:
The competitive advantage of AI-supported work may not come simply from having humans in the loop. It may come from understanding exactly where human intelligence adds the greatest value—and continuously redesigning the workflow so that human intervention makes both today’s output and tomorrow’s system better.
Case Study Note
The case studies published by Business Warrior’s Dojo are intended primarily as tools for learning, discussion, and analysis.
They may be based on real business situations, publicly available case studies, professional experiences, or entirely hypothetical scenarios. In some cases, names and identifying details have been changed to preserve confidentiality. In others, facts, circumstances, timelines, or outcomes may have been substantially modified, combined, or simplified to better illustrate particular business issues or support discussion. Some case studies are entirely fictional and have been developed solely for educational purposes.
Leave a Reply