How to Evaluate an AI Agent Development Agency
What separates a stateful, resilience-tested system in production from a demo that worked once, priced at the task level instead of the slide level.
The gap you're evaluating for
Most of what gets pitched as an 'AI agent' today is a chatbot with a few tools attached, wrapped in a demo that ran once for the sales call. The gap between that and a system running unattended in production, on real inputs, is where an agency either earns its rate or doesn't. This page is what we'd want to know before hiring one: how to tell a stateful graph from a chain of prompts, what the work costs at the task level, and what to ask before signing anything.
Questions to ask before you sign
Ask to see the graph, not a slide
A production agent system is a state graph: typed nodes, conditional routing, checkpointing. If the only artifact an agency can show is a deck or a demo video, there is no graph behind it, just a sequence of prompts.
Ask what happens on a timeout
What does the system do when the model times out, returns malformed output, or a downstream API goes down? A vague answer means it has only ever been tested against the happy path.
Ask who approves the irreversible actions
Sending an email, writing to the ERP, issuing a payment: ask which of these the agent can trigger on its own, and where the human checkpoint sits before it does.
Ask what a run costs
Ask for cost per run or per task, not a monthly estimate. An agency that isn't tracking cost per node isn't optimizing it either, and the bill will surprise you later.
Ask for the rate, not a ballpark
Ask for the day rate and how the quote range is built. An agency that won't publish a rate is pricing the client, not the project.
Ask who gets paged
Ask for the runbook: what triggers an alert, what the rollback looks like, and who takes over when the pipeline fails in production.
What it costs
Our own rate, published rather than quoted case by case. It's the number every estimate on this site is built from.
What the work actually breaks down to
We quote from a task-level model instead of a feeling: 58 tasks, from a kickoff workshop to end-to-end tests and run monitoring, each carrying an estimate split into agent minutes and human minutes. Across those 58 tasks the model puts agent time at 1,250 minutes against 4,615 minutes of typical human time, so roughly a quarter. Worth being straight about what that is: these are our own estimates, not averages measured over delivered projects, and the calibration columns are still empty. What the split shows is the shape of the work, not a productivity multiplier. The kickoff workshop is 0 agent minutes by design, because no agent runs a discovery session. What compresses is the build, not the judgment.
Dead pilot, or system in production
Signs of a dead pilot
- Works once, in a scripted demo, and breaks on the first messy input.
- No state: every run starts from zero, with no memory of a prior turn or a prior failure.
- Success was measured once, at the sales meeting, and never again.
- No answer for what happens on an LLM timeout or a malformed output.
- No cost or latency number per run, because nobody is tracking either.
Signs of a system in production
- A stateful graph: typed state, conditional routing, checkpointing that resumes after a crash.
- Resilience tested: timeouts, malformed output and API outages simulated before launch, not discovered by a client.
- Monitored: traces per node, cost per run, an SLA with a number attached to it.
- A runbook: who gets paged, what the rollback is, when a human takes over.
- A stated boundary: which actions the agent takes alone, which ones need a human sign-off first.
Questions we get asked
What does an AI agent development agency actually build?
A graph of agents wired to a real process: nodes that call an LLM or a tool, typed state passed between them, conditional routing, and guardrails before any external action. Not a chatbot, and not a workflow that only handles the cases it was demoed on.
How much does it cost to hire an AI agent development agency?
Ours is €600 a day, or €2,500 for a booked week, and the same figures in dollars. A quote is estimated weeks times the weekly rate, then a ×1.4 range to cover scope drift. The exact figure depends on how many nodes, integrations, and resilience cases the process needs.
How long does it take to go from pilot to production?
It depends on the process, not a fixed template: a single-node automation with one integration is a matter of weeks; a multi-agent graph with several external systems and irreversible actions takes longer, because the resilience testing phase can't be skipped.
What's the difference between an AI agent and a chatbot with tools?
State and testing. A chatbot with tools calls a function and answers; an agent graph keeps typed state across steps, routes conditionally on what happened, and has been tested against timeouts, bad output, and outages before it runs unattended.
Can you take over an existing pilot instead of starting from scratch?
Yes. What a working pilot is usually missing is state, resilience tests and monitoring, and that is where the work goes. The mapping phase starts from what already exists rather than redoing it.
What happens after launch?
Langfuse traces show which nodes fail or run slowest, and each cycle produces an improved prompt version. Nodes with stable outputs move to a cheaper model. The pipeline is meant to keep improving after the first deployment, not to be handed over and forgotten.
Where this leads
Ready to stop guessing and start scoping?
Describe the process you want automated. We'll tell you what it actually takes.
Describe my project →