AI Vendor Evaluation: Twelve Questions to Ask Before You Sign

Every AI demo works. That is the problem. The vendor drives, the data is clean, the questions are the ones the product answers well, and forty minutes later someone wants a signature on a twelve-month agreement. AI vendor evaluation is the work of finding out beforehand whether a real system sits under that demo, or a thin layer over a model you could call yourself.

The odds justify caution. S&P Global Market Intelligence's 2025 survey of more than 1,000 respondents in North America and Europe found 42% had scrapped most of their AI initiatives, up from 17% a year earlier, with an average 46% of proof-of-concepts dropped before production, as reported by CIO Dive. Gartner, in a June 2025 forecast, predicted more than 40% of agentic AI projects will be canceled by the end of 2027 and named a culprit: "agent washing," the rebranding of existing chatbots and automation scripts as agents. Of the thousands of vendors selling agentic AI, Gartner estimated only about 130 have real agentic capability.

Those numbers do not mean AI purchases fail. They mean most buyers screen for the wrong things.

Table of Contents

1. What AI Vendor Evaluation Is Screening For
2. Twelve Questions to Ask Every AI Vendor
3. Scoring the Answers: A Rubric and a Worked Example
4. Designing a Pilot That Can Actually Fail
5. Contract Terms That Decide What Year Two Costs
6. When the Answer Is Not to Buy AI at All
7. Frequently Asked Questions
8. Ready to Evaluate an AI Vendor Without Guessing?

What AI Vendor Evaluation Is Screening For

AI vendor evaluation is not a features comparison. It tests three things a demo cannot show: whether the product does the job on your data, whether your data stays yours, and whether you can leave without losing the work.

A wrapper, in the sense buyers mean it, is a product whose whole capability is a prompt sent to someone else's model. That alone is not disqualifying; building on OpenAI or Anthropic is sensible engineering. The question is what the vendor added: integrations, review workflow, the testing that catches quality drift, domain rules. If the answer is nothing, you are paying a markup for an API call and a login screen.

The third test matters most and gets asked least. Six months in, the vendor holds your conversation history, lead scores, and tags. If none of it comes back usable, the renewal stops being a decision.

Twelve Questions to Ask Every AI Vendor

Ask these in one call, in this order, and write the answers down verbatim. The first three are gates: a bad answer to any one ends the evaluation.

Gates.

1. Do you train models on our data, and will that be in the contract? A baseline exists: OpenAI's enterprise privacy terms state that business and API data is not used to train its models by default and that the customer owns inputs and outputs. A vendor building on those APIs can inherit that and should say so in writing. Red flag: "your data is encrypted," which answers a different question.
2. If we cancel in month seven, what do we get back, and in what format? You want the raw records and the derived work, meaning scores, tags, summaries, and message history, as files you can load elsewhere. Red flag: "you can export to CSV," with nothing said about what the system generated.
3. Who approves output before it reaches a customer, a guest, or an owner, and can we change that threshold ourselves? AI should enhance human judgment, not replace the person accountable for what goes out. Red flag: that setting lives in the vendor's admin panel rather than yours.

System, or prompt?

4. Which model are you using, and what happens to our results when it changes? Naming it is normal. Refusing is a tell, as is having no plan for the day that model is updated.
5. Run this task on our data, on this call. Not the demo account. Red flag: four weeks to arrange, which tells you integration is the weak point.
6. What does it do when it does not know? A system that abstains or escalates was built by people who thought about being wrong. One that always answers was not.

Does it fit how you actually work?

7. Name every system you integrate with, and which fields cross over. PMS, CRM, accounting, phone. "We have an API" is homework assigned to you, not an integration.
8. Who else our size runs this, and what did their first ninety days look like? Pick the reference from a list yourself rather than taking the one offered.
9. What has to be true on our side before this works? The honest answer names your data cleanup. "Nothing, it works out of the box" is the red flag.

Economics.

10. What is the all-in year-one number? License, implementation, seats, usage overage, internal hours. Our breakdown of what a small business actually pays for AI covers what vendors leave out.
11. What happens to the price at twice our volume, and at renewal?
12. What would make you tell us not to buy this? A vendor with no answer either does not know where the product fails, or will not say.

Scoring the Answers: A Rubric and a Worked Example

Score the nine non-gate questions 0, 1, or 2, and treat the three gates as pass or fail. The point is not precision but forcing a decision before the follow-up call.

• 0 — evasive, or answered a different question.
• 1 — answered plausibly, nothing you can verify.
• 2 — answered with something checkable: a contract clause, a reference you chose, a live run, a written number.

Maximum is 18. Set the cutoff at 13, roughly 72%, and require all three gates to pass.

Consider two vendors pitching the same guest-messaging product. Vendor A passes the training and human-review gates but fails on export, because the derived data never comes back. Evaluation over at question two, before anyone compares features. Vendor B passes all three gates, then scores 2 on the live run, the model policy, the low-confidence behavior, the honest data-cleanup answer, the written all-in number, and the capped renewal; 1 on integrations (PMS but not accounting), 1 on references, and 1 for having no reason-not-to-buy. That is 15 of 18, so Vendor B earns a paid pilot. Nobody signs an annual contract yet.

Designing a Pilot That Can Actually Fail

A pilot is only evidence if it was possible to lose. Before day one, write down the metric, the threshold, and the date, and have the vendor sign the same page.

Start with the arithmetic, which often settles it by itself. Consider a hypothetical 140-property vacation rental manager looking at a guest-messaging tool at $1,400 a month plus a $6,000 implementation fee. Year one is $22,800. If a reservations coordinator costs $24 an hour fully loaded, break-even on labor alone is $22,800 ÷ $24 = 950 freed hours, about 18 hours a week.

Now test the claim. Say the portfolio takes 3,000 inbound guest messages a month and the vendor says it handles 60% without a human. That is 1,800 messages at three minutes each: 90 hours a month, 1,080 a year. Subtract review. If a quarter of its drafts need a one-minute edit, that is 450 minutes a month, 90 hours a year. Net saving, 990 hours, against a 950-hour break-even.

A 4% margin is not a business case. It is a rounding error on your own assumptions, and it means labor savings alone do not justify the purchase. The tool has to earn its keep on first-response time, escalation rate, or review scores, and that is what the pilot must measure. The same test applies to software you already own; we ran it for dynamic pricing.

Four rules keep it honest: instrument the baseline first, because you need today's median first-response time to compare against; use your messiest data, not the clean subset; run real volume for thirty days including a peak weekend, because sandboxes hide the failures that matter; and name the person who can end it.

Contract Terms That Decide What Year Two Costs

Marketing claims are not commitments. In March 2024 the SEC brought its first AI-washing cases against two investment advisers, settling for $400,000 combined over claims about AI they were not using. Then-chair Gary Gensler put it plainly: advisers "should not mislead the public by saying they are using an AI model when they are not." The website is not what you bought. The agreement is. Five clauses do most of the work:

• Training and ownership, as the vendor's obligation, not a link to a policy page they can revise.
• Renewal cap and overage pricing. A written ceiling on the year-two increase, and a number for what one busy month costs. Ask before you commit; your negotiating position only gets worse.
• Exit terms. Format, scope, and a delivery window in days.
• Subprocessors. You inherit every model provider and data store they use, and should get notice when that list changes.
• Support hours against your peak, not an annual uptime average. A July Saturday is when it matters.

This is also where governance gets written for a company that has no governance department. Deloitte's 2026 State of AI in the Enterprise, a late-2025 survey of 3,235 senior leaders across 24 countries, found only about one in five organizations has mature governance for autonomous AI agents. For a 40-person firm, the contract is the governance model.

When the Answer Is Not to Buy AI at All

Sometimes the right outcome of an evaluation is that nobody gets the order. Process first, technology second: software applied to an undefined process produces faster inconsistency, not efficiency. Five conditions where you should stop:

1. Two people do the task two different ways. Write the process down first; you cannot automate a judgment nobody has articulated.
2. The volume is too small. Under 200 events a month, a checklist and saved templates beat a subscription.
3. The data lives in three systems with no shared key. Build the reporting foundation first; every model inherits whatever mess it is fed.
4. A wrong answer is expensive and nobody will review it. Regulated communications, pricing commitments, and anything a customer can hold you to need a named human in the loop, or stay manual.
5. The real constraint is elsewhere. If you are losing deals on staffing, inventory, or price, faster drafting changes nothing.

The readiness check in Is Your Business Ready for AI? is worth running before you take a single demo.

Frequently Asked Questions

Ready to Evaluate an AI Vendor Without Guessing?

Most bad AI purchases are not bad products. They are good products bought for a process nobody defined, measured against a number nobody wrote down. ANtreliX is a strategy-first AI and business intelligence consultancy in North Myrtle Beach, South Carolina, serving local and remote clients, and we build the evaluation criteria before anyone books a demo. See our case studies, or Talk to ANtreliX.