The market is noisy and the barrier to claiming expertise is low. Here are the questions that separate people who have deployed systems from people who have read about them.
Ask what broke
The single most useful question. Anyone who has run these systems at any scale has stories — the task that reported success having done half of it, the configuration limit nobody noticed, the integration that failed when a supplier changed a format.
A vendor who cannot tell you what went wrong has not run anything. Watch for a smooth answer about "challenges" that resolves into a capability pitch. You want a specific failure with a specific cause.
Ask what they would not automate
Anyone selling AI will tell you what it can do. Far fewer will tell you where they have deliberately kept it out.
A good answer is specific and reasoned — "not anything with a filing deadline," "not client-facing until it has run internally for months." A vendor who thinks everything is automatable is either inexperienced or selling.
Ask about month six
Not the build. What happens when the underlying system changes, when volume grows, when it starts behaving oddly. Who notices, who fixes it, what it costs.
A vendor without a clear answer has quoted you a project, not a system. Most of the total cost of ownership lives after go-live.
Ask how you will know it is working
If the answer is qualitative — it will save time, it will improve efficiency — push for a number and a baseline. A vendor who works this way will already have asked what your current error rate or cycle time is, because they cannot demonstrate improvement without it.
If nobody has asked you for a baseline, nobody is planning to prove the result.
Ask what they need from you
Every deployment requires access, process documentation, and someone internal who can answer questions. A vendor who claims to need almost nothing from you has either not thought it through or is planning to build for the process as documented rather than the process as it runs.
The correct answer includes some homework on your side. Be suspicious of one that does not.
Warning signs
Talking about the model rather than your process. Which model matters far less than whether the task is well-chosen.
Promising full autonomy immediately. Systems earn autonomy by demonstrating reliability.
No verification in the proposal. If nobody has costed the checking, nobody has thought about the failure modes.
Pricing that is all licence and no build. The software is the cheap part. A quote weighted toward licence fees is selling you a tool and leaving the hard part with you.
Case studies with no numbers. "Transformed their operations" is not a result.
The question behind all of them
You are not really assessing technical capability. You are assessing whether they will tell you something you do not want to hear — that your first idea is the wrong project, that the data needs cleaning first, that this will not pay back at your volume.
A vendor who has never talked a client out of a project is not being cautious. They are being paid.
Related: What Does It Cost to Build an AI Agent for a Small Business? and AI Agents and Client Data.