Somewhere in your company, someone has said: "We should add AI to this."
Maybe it's a chatbot, a smart search, an automated summary, or a recommendation engine. Your developer says it's doable. A quote lands in your inbox. And you have to decide whether to approve it, usually without being able to judge the technical details.
You don't need to. You need to ask better questions.
I build AI products for a living, including Chatbotz, a SaaS platform where businesses deploy AI chatbots on their sites. These are the five questions I'd want every founder and marketing lead to ask before saying yes. They aren't a trap for your developer. A good developer will enjoy answering them, because they show you take the feature seriously.
Question 1: "What does one use of this feature cost us?"
Why ask: In regular software, serving one more user costs almost nothing. With AI, every request to a language model is a paid call. Your feature has a running cost that scales with how much people use it, which means success can raise your bill.
A healthy answer sounds like: "Roughly X per conversation, based on our expected message length. At 10 times today's traffic it would be about Y, and here's the cap we'll put in place."
Red flags:
- "It's basically free."
- "We'll see what the bill looks like after launch."
- No mention of limits on message count, conversation length, or request rate.
One detail worth knowing: language models don't remember conversations on their own. The whole history is typically sent again with every new message, so long conversations cost much more than short ones. A developer who knows this will bring up limits without being asked.
Question 2: "What happens when it's wrong?"
Why ask: AI will occasionally produce confident, plausible, incorrect answers. That isn't a bug you can fully remove. It's a property of the technology, so the real question is how much damage a wrong answer can do and what stops it.
A healthy answer sounds like: "For low-risk questions, it answers directly. For anything about pricing, refunds, legal, or health, it hands off to a human. We've listed what it must never say and we test for those cases."
Red flags:
- "It's very accurate."
- "The model is good enough that this won't happen."
- No plan for a human handoff, and no way to see conversations afterward.
This matters most when the people configuring the AI aren't experts. In a product like Chatbotz, a shop owner writes the instructions that shape the bot's behavior. Weak instructions produce weak answers, and the visitor blames the company, not the prompt. Good defaults and guardrails matter more than the model you pick.
Question 3: "What data does it see, and where does it go?"
Why ask: Most AI features send your users' text to a third-party provider. If that text contains names, emails, orders, or health or financial details, you've just made a privacy decision, whether you meant to or not.
A healthy answer sounds like: "It only sees X. We strip or avoid sending sensitive fields. We've checked the provider's terms on data retention and whether our data can be used for training, and here's what we chose."
Red flags:
- "I haven't looked at the provider's data policy."
- "It sees everything, it's easier."
- Nobody can say which provider is used or where the keys are stored.
You don't need to read the legal terms yourself. You need to confirm that someone has, and that the answer matches what you've promised your customers.
Question 4: "How will we know it's working?"
Why ask: AI features are easy to demo and hard to evaluate. Without a target, you'll end up with a feature that looks impressive and does nothing measurable.
A healthy answer sounds like: "The goal is to reduce first-response time on pricing questions by half. We'll log conversations and review the failures weekly."
Red flags:
- "It'll improve the customer experience."
- "We'll know it when we see it."
- No logging, so nobody can review what the AI actually said.
Agree on one metric before the build starts, and write it down. If the feature can't move a number you care about, it's a toy, not an investment.
Question 5: "Who owns this after launch?"
Why ask: Shipping is the start, not the finish. Providers change their models, update their pricing, and sometimes have outages. Instructions that worked well in March may behave differently in September. An AI feature without an owner slowly degrades.
A healthy answer sounds like: "I'll own it for the first three months. If a provider changes or goes down, we have a fallback. Here's what the monthly maintenance looks like."
Red flags:
- "It's done once it's deployed."
- The feature depends on one provider with no fallback and no one watching it.
- Nobody knows who gets alerted when something breaks.
The takeaway
Most AI projects don't fail because the technology doesn't work. They fail because nobody asked what it would cost, what happens when it's wrong, or who would look after it.
Ask those five questions, and you'll make better decisions than most companies that rush to "add AI."