Interactive online edition: Open the article with region zoom controls.

jSwarm Case Studies · Article 01 · Quality

Your AI gave a great answer. Why is the customer still doing the work?

From recommendation to action: checking that AI changes the right thing without breaking the rest.

Mike Idengren
Mike IdengrenClaude Certified Architect–Professional badge

This installment of the jSwarm case-study series examines JarviSIM, a process-design product, to explore the gap between an AI recommendation and a completed product action.

Synopsis

A useful AI answer is only part of the job when a product promises to change a process. In this JarviSIM example, the user asks, “Make this process lower risk.” The chatbot recommends a specific control; the user chooses it and applies the change to the model. The reported structural check is one of one requested control added and seven of seven existing tasks retained. The three-run count is owner-supplied, not independently established by the screenshots. Whether the recommendation itself was good is a separate semantic evaluation.

A useful design principle spans all three priorities: reserve language-model judgment for interpreting and planning changes, and use deterministic functions wherever the operation can be specified precisely. This avoids unnecessary inference charges and waiting while making execution more predictable—benefits for cost, speed, and quality through reliability, rather than measured savings in this example.

Audience: AI Solution Product Managers · CIOs and CTOs · AI Developers

The business story

COST · PRIORITY

Optimize spend

Use the lowest-cost model that meets the product's quality bar.

SPEED · PRIORITY

Reduce time to outcome

Improve time to a useful answer and a completed change.

QUALITY · FOCUS

A process upgrade, not just a chatbot answer

Check the requested control, preserved tasks, and repeatability—without confusing structural results with advice quality.

STRUCTURAL · PASS · OWNER-REPORTED1/1 requested control added · 7/7 existing tasks retained
STABILITY · PASS · OWNER-REPORTEDStructural outcome reported across three runs
SEMANTIC · NOT ASSESSEDRecommendation quality needs its own rubric

But a good recommendation is not the same as a completed action. Imagine an airline assistant finding a better connection and explaining why it fits. If you choose that option, but the booking never changes, you still have to finish the job.

But your booking never changes. That gap—between advice and an outcome—is where a buyer should look closely.

Real product example: JarviSIM compliance quality

Part I

The request is broad. The recommendation is specific.

The user asks, “Make this process lower risk.” The chatbot describes the purchasing process, notes that the model alone cannot establish which risks occurred or how often, and recommends a requisition-coding completeness control for the existing “Validate requisition coding” task. Explore three separate regions below: the user’s intent, the chatbot’s explanation (semantic quality not assessed), and the recommendation with Use (the structural action starting point).

Advisor conversation: broad request, explanatory response, and actionable recommendation with Use.
From request to recommendation. Hover or focus a numbered region to highlight it. Select a region or its matching story label for an enlarged detail in this page.

The broad request starts the journey. It is an expression of intent, not evidence that risk has fallen.

The explanation describes the process and its limits. Gray means recommendation quality has not been evaluated in this example.

The specific proposal and Use button start the action path. Use selects a recommendation; it does not establish a successful change or stability across runs.

Part II

The user chooses. Tools carry out the change.

The user selects Use; the recommendation appears in Pending Model Changes. The separate Apply to Model action is the user's choice to proceed. The product then invokes its chatbot model-mutation operation (apply_chatbot_mutation) to make a candidate model change; the accepted revision is what the product checks next. This is an operation an AI product invokes—not just text for a person to copy into the process.

The LLM works out what should change. The heavy lifting happens while developing the recommendation: interpreting the request, considering what is already in the change queue, and working out how the Business Process Model and Notation (BPMN) model should change. The useful output is more than advice—it is a specification for the mutation.

Don’t over-use the LLM for what should be reliable, fast, free deterministic functions.

The tool carries out the specified change. The division of labor is simple: the LLM specifies the change; the mutation tool executes it. Deterministic model-editing functions apply precise instructions rather than asking the LLM to reason through the same decision again. That makes execution reliable, fast, and free of another LLM call. The product then checks the resulting model: was the requested control added, and were the existing tasks retained?

The design lesson. Use the LLM for judgment and planning. Use deterministic tools for execution. Structural quality checks whether the specified change was carried out correctly; stability checks whether that outcome repeats.

Selected recommendation in Pending Model Changes with the separate Apply to Model action.
Selection is not application. This overview shows the full sequence. Select “Pending Model Changes” or “Apply to Model” below to read that detail at full size.

The selected recommendation is queued as one pending change. The model has not yet been shown to change.

This separate action applies the selected proposal. The product’s apply_chatbot_mutation mechanism is described in the story; this screenshot alone does not prove a tool invocation.

Part III

One control added. Seven tasks kept.

The revision view shows one control reviewed and seven tasks. Its contribution summary describes the requisition-coding completeness check. The selected “Validate requisition coding” task carries both CHATBOT and COMPLIANCE contribution labels. The control checks that cost center, general-ledger account, and project code are populated and consistent before the requisition proceeds.

Applied revision with seven-task process canvas, robot contribution summary, task and control counters, and changes-applied message.
Inspect the resulting revision. Hover or focus a numbered region to highlight it. Select a region or its matching story label for an enlarged detail in this page.

The contribution trace says a requisition-coding completeness check was added to Validate requisition coding. This is the interface’s summary, not an independent tool-call trace.

Inspect the seven task nodes in the resulting canvas. Counting visible tasks is not proof that all seven original task identities were retained; that comparison is a supplied outcome claim.

The revision labels RE-ESTIMATED TASKS · 7 and CONTROLS REVIEWED · 1. Re-estimation is not measured chatbot latency or a speed improvement; reviewed is not proof of control effectiveness.

The advisor displays the applied-change message for this recommendation. One screenshot is not evidence of three separate runs or repeated stability.

Part IV

Inspect the control on the task.

Open the highlighted task details to see the attached control and the Chatbot and Compliance contributions.

Validate requisition coding selected above its task detail, attached control and displayed contributors.
Inspect the selected task. Hover or focus a numbered region to highlight it. Select a region or its matching story label for an enlarged detail in this page.

The selected node identifies where the change landed. The attached detail below lets the reader inspect the control rather than infer completion from chat alone.

Controls (1) identifies the requisition-coding completeness check. Its presence is a structural observation, not an evaluation of whether it reduces operational risk.

The task detail lists Compliance and Chatbot contributors. These interface labels describe attribution; they do not independently prove the underlying tool calls.

A successful example is a start. Repeatability is another question.

The reported structural check found the requested control added (1/1) and existing task identities retained (7/7), with the outcome reported across three runs. That is a useful repeatability question for a buyer to ask, but the run count comes from the owner; the four screenshots do not independently establish it. The screenshots also do not prove task-identity matching on their own.

The revision panel labels the process-metric count RE-ESTIMATED TASKS · 7. Those tasks were re-estimated under the test conditions; this is not measured chatbot latency, customer productivity, or a speed improvement. A structurally correct change does not prove risk fell, the control works in production, or the recommendation was semantically good. Advice quality and model outcome need separate checks.

About this example and its evidence

The four historical captures show the request and recommendation, the pending/apply action, the applied revision, and the selected task detail. They are interface illustrations—not a standalone test report. The 1/1 control, 7/7 task-identity, and three-run figures are supplied outcome claims; no underlying run-by-run records or task-identity comparison artifacts are presented here. The screenshots alone cannot independently verify those claims. The three-run basis is owner-supplied, not inferred from the four images.

This is development evidence, not a customer deployment or proof of production effectiveness. It makes no claim of risk reduction, cost savings, latency improvement, customer acceptance, or compliance effectiveness.

For Claude Certified Architect–Professional readers

Anthropic’s guidance on defining success criteria and building evaluations calls for criteria tied to the application’s purpose and user needs. Here, the structural checks are distinct from semantic response quality: add the requested control, retain the seven existing task identities, and ask whether the outcome repeats. This is an application of that evaluation principle, not an Anthropic-prescribed JarviSIM workflow. The recommendation rubric and any judgment of operational risk are outside this article’s evidence.