Guide · Agent orchestration
Measure Your First Multi-Agent Pilot with a Real Finish Line
Run a bounded pilot and compare accepted output, review time, cost, and failure recovery before expanding the agent team.
A useful pilot should end with a decision about the workflow. Choose one recurring job, define an acceptable result, and compare the agent arrangement with the way you handle that job now. Keep the scope small enough that you can inspect every output during the trial.
Choose a job with visible evidence
A hypothetical internal release-note workflow makes a manageable example. It reads a defined set of merged changes, drafts notes for a known audience, and checks each claim against the source. The accepted result is a reviewed draft. Publishing remains part of the team's existing process.
Pick a representative set of releases, including a small one, a confusing one, and one with an incomplete description. Do not select only clean inputs. The difficult cases reveal whether the workflow can ask for missing information and preserve uncertainty without inventing a finished story.
Write the acceptance criteria before running the pilot. For release notes, that could include correct scope, accurate customer-facing claims, no internal-only details, and links back to the changes that support each item.
Keep the first team small
Start with a coordinator and one clearly useful specialist, such as an evidence checker. Compare that arrangement with a single agent using the same source material and requirements. Add another worker only when you can explain the bottleneck it will address.
If you want to evaluate Astra and Fable 5.1, vary one role at a time where practical. Keep the configuration with each result. Treat a failed request setup separately from a valid run that produced weak work, and account for both when estimating the effort to operate the system.
Record the work after the response
Measure the time needed to inspect and repair each draft. Note unsupported claims, missed changes, duplicated sections, and useful findings from the reviewer. Record tool and model costs for the entire job, including retries.
A quick first response is encouraging, but a slow review can erase that benefit. Look at the complete path to acceptance before deciding that the workflow is faster.
- The proportion of drafts accepted after normal review.
- The kinds of corrections a person had to make.
- Time from assignment to an accepted artifact.
- Total operating cost for each completed job.
- How much useful work survived an interrupted run.
End with a specific decision
Set a review date and choose whether to keep, revise, expand, or stop the pilot. Tie the decision to the observed results. If source descriptions caused most failures, improve those inputs and rerun the same cases before changing the whole agent design.
Save the accepted examples, failures, and configuration as a small evaluation set. It becomes a practical check when you update prompts, add tools, or change models. That way, the pilot produces both a decision today and evidence you can reuse for the next improvement.