Nobody sets out to build an irresponsible AI product. You’re moving fast, the backlog is a mile long, your standup is in ten minutes, and “governance” sounds like something the legal team handles in a conference room you’re not invited to. So you ship. And mostly it’s fine.
Until it isn’t.
AI incidents jumped 30% in recent years, and the teams that get burned aren’t the reckless ones. They’re the ones that were just… busy. The ones that meant to add the guardrail after the MVP. The ones that assumed someone else had thought about the thing nobody thought about. The gap between “we didn’t mean to” and “we’re on the front page of Hacker News for the wrong reasons” is usually one skipped conversation. This isn’t a governance framework written by lawyers. It’s a set of stories about real mistakes, the simple fixes that followed, and how those fixes made teams move faster, not slower. If you’re a product manager or an engineer who’s newer to thinking about AI governance, this is the version that was written for you.
Let’s start with data, because that’s where most teams first go wrong before they even write a line of AI code.
A team building a subscription service sat down to design the sign-up flow. They had a list of things they’d eventually need from users, and since they were going to need it all at some point, why not collect it upfront? More data is better data, right? The form got long. Not unreasonably long by the standards of the industry, but long enough that conversion dropped. More importantly, they were now sitting on a pile of user information they didn’t actually need for weeks, sometimes months, and every field they collected was a field they were now responsible for protecting.
When someone on the team finally asked “what’s the minimum we actually need to start the relationship?”, the answer was genuinely short. Email, payment method, and one preference. That’s it. Everything else could come later, through follow-up prompts at natural moments in the user journey, where it would also feel less like a form and more like a conversation. The fix wasn’t technically impressive. It was just a question that became a habit: “Do we actually need this now?” They called it the Minimum Necessary Data rule, and they asked it every single time a new data field was proposed. The form got shorter. Conversion went up. And the team was responsible for protecting significantly less sensitive information, which matters when you’re also running AI features that are touching that data.
Think of it like packing for a weekend trip. You can bring everything in your closet “just in case,” or you can ask what you realistically need for three days. One of those people misses the flight because they’re still arguing with airport security about their carry-on. The parallel to AI is direct. When you’re building a model, a feature, or an integration, the question isn’t what data could be useful. It’s what data you actually need to do the job. Anything beyond that isn’t an asset. It’s a liability.
A team deployed an AI receptionist to handle inbound questions. The bot was good. Genuinely helpful. It answered questions quickly, routed people correctly, and handled a volume of interactions that would’ve required several full-time employees. The team was proud of it, and they should have been.
What they didn’t do was tell anyone it was AI.
Not in a sneaky, malicious way. More in the way that it just didn’t come up as a required step. The bot had a name. It had a friendly tone. Nobody explicitly decided it was human, they just never explicitly said it wasn’t. And for a while, that was fine, because users got their questions answered and moved on. Then someone asked directly. And the answer wasn’t fast enough, or clear enough, and the experience fell apart. Trust, once broken that way, is expensive to rebuild. One person posted about it. Others had the same feeling. And the team spent weeks doing damage control over something that would’ve taken three seconds to prevent: “Hi, I’m an AI assistant. I can help with X, Y, and Z.”
That’s the fix. Disclosure at first contact. Not buried in the footer. Not in the terms of service that nobody reads. Right there at the start of the interaction, before the user invests any emotional energy in the conversation. Here’s why this is no longer just good manners: six of the seven major chatbot bills passed in 2025 include mandatory disclosure requirements. Washington State now requires disclosure at the start of every interaction, with reminders at least every three hours for certain chatbot types. California’s SB 243, signed into law in late 2025, requires “clear and conspicuous” notice any time a reasonable person could be misled into thinking they’re talking to a human . Teams that skipped the disclosure step aren’t just dealing with a trust problem. They’re retrofitting legal compliance into a product that was built without it, which is a much harder and more expensive problem.
The three-second disclosure isn’t a speed bump. It’s insurance.
The disclosure story gets more serious when you move from a receptionist bot to a customer service chatbot designed to handle emotionally sensitive situations. One team built exactly that. The bot had a name, a warm tone, a carefully crafted personality. It was good enough that users genuinely believed they were talking to a person. And because they believed that, some of them shared things they wouldn’t have shared with software: frustrations, personal circumstances, details about their situation that they were offering because they thought there was a human on the other end who cared.
When users eventually realized the truth, it wasn’t just disappointment. Some felt deceived in a meaningful way. They hadn’t consented to sharing personal information with a machine. The fact that the machine was helpful was almost beside the point. The fix the team landed on was what they started calling a “persona boundary” rule. The AI could have a name. It could have a personality. It could be warm and responsive and genuinely useful. But it couldn’t claim to be human when someone sincerely asked, and it had to proactively identify itself as AI at the start of every conversation. They added one line to the system prompt, and they assigned one person to spot-check conversations weekly, about thirty minutes of review to catch anything drifting outside those boundaries.
The broader context makes this more urgent than it might sound. AI-related security incidents more than doubled in certain categories in 2025, and chatbot deception is one of the fastest-growing sources of consumer complaints and regulatory scrutiny. California’s SB 243 specifically requires disclosure when a reasonable person could be misled, and creates a private right of action for injured users, meaning damages of at least $1,000 per violation . A system prompt and a weekly spot-check is a small investment against that exposure. The persona boundary rule also does something more subtle: it forces the team to be honest with themselves about what they’re building. If you can’t say “this AI will identify itself as AI when asked,” that’s worth examining before the product ships.
Here’s a failure pattern that’s less dramatic but arguably more common. A team builds an AI feature, ships it, celebrates the launch, and moves on to the next sprint. The feature works well in testing. It works well in the first few weeks. And then, gradually, it starts producing answers that are slightly off, occasionally wrong, and sometimes confidently incorrect in ways that are hard to catch unless you’re looking.
Nobody was looking.
The model wasn’t failing catastrophically. There was no alert, no crash, no spike in errors. The outputs just quietly drifted from useful to unreliable, the way a clock that loses ten seconds a day seems fine until you realize you’ve been late to every meeting for a month. A customer noticed before anyone on the team did. They took a screenshot. They posted it. It wasn’t a disaster, but it was the kind of embarrassing, avoidable incident that makes you think about what else you’re not watching. The fix was a weekly spot-check ritual. One team member, rotating each week, reviews twenty random AI outputs and flags anything that looks wrong, weird, or off-brand. No special tooling required. Thirty minutes, a spreadsheet, and basic judgment. The point isn’t to catch every error; automated testing handles the systematic stuff. The point is to maintain a human eye on what the model is actually doing in the wild, because models behave differently in production than they do in your test environment.
This matters more than most teams realize. Among companies that have AI in production, monitoring and incident response planning rates are “surprisingly low,” especially at smaller companies. Only 36% of small companies have a designated AI governance role at all. Which means the spot-check ritual isn’t just a good habit. For most teams, it’s the only regular mechanism catching real-world model behavior before it becomes a user-facing problem.
Twenty outputs. Thirty minutes. One person. Put it on the calendar.
The last story is the one teams are least likely to recognize in themselves, because it’s not a failure, it’s an assumption. A team needs an AI model for a new feature. They find a good third-party option, the benchmarks look solid, the integration is clean, and they’re on a deadline. They connect it to the product and ship.
Nobody asks where the model was trained. Nobody asks what it was trained to do well, and more importantly, what it was trained to do badly. Nobody asks whether the model has documented biases or known failure modes in contexts similar to theirs. Three months later, a user points out that the model’s responses about a particular topic have a consistent pattern that reads as biased. The team looks into it and discovers the training data had a known gap they would’ve found if they’d looked. They hadn’t looked because it hadn’t occurred to them to look. The model was just a vendor choice, not a governance decision.
The fix is a model intake checklist. Three questions, asked before any new model or AI tool gets connected to users:
Where was this model trained, and what kind of data was used? What failure modes or biases have been documented by the developer or in independent evaluations? What does it do poorly, specifically in contexts like ours?
You don’t need a formal audit to answer these. In most cases, you need fifteen minutes reading the model card, the developer documentation, and a few independent reviews. What you’re doing is making the assumption visible. A model that nobody questioned is a risk that nobody measured, and risks nobody measured have a way of showing up at the worst possible time.
If you’re a PM or an engineer earlier in your thinking about AI governance, here’s what the above actually simplifies to. Four habits. Not a framework document. Not a committee. Four habits that the teams who avoid incidents tend to share.
First, minimum necessary data. Before any new data collection, someone asks: do we actually need this now? If the answer requires a long justification, the answer is probably no.
Second, disclose at first contact. Every AI interaction tells the user it’s AI, right at the start, clearly. No footers. No terms. Right there, before the conversation begins.
Third, spot-check weekly. Twenty random outputs. Thirty minutes. One person on rotation. Put it on the calendar the same way you’d put a code review on the calendar.
Fourth, model intake questions. Before any new model touches users, answer the three questions: where was it trained, what are its documented failure modes, what does it do badly in contexts like yours.
None of these require slowing down. They require one person asking one question before the decision gets made. The teams that skip them don’t move faster. They move faster until they don’t, and then they move much slower for a while cleaning up something that didn’t have to happen. The honest reality for PMs and engineers new to governance thinking is that you’re not the legal team. You’re not expected to be compliance experts. But you’re the people closest to what gets built and how it gets built, which means you’re also the people with the most power to catch these things before they become problems. The checklist doesn’t have to be yours forever. It just has to be someone’s, right now, while the thing is being built.
The teams that build these habits early don’t just stay off the front page for the wrong reasons. They also build faster, because they stop spending sprints fixing things that a thirty-minute conversation would have prevented.
If any of these stories sound familiar, that’s the point. Share which rule your team broke, what the fix was, and how you kept moving. Tag @cesarmorenoai on social media. If you want more of this, you can find it at https://cesarmoreno.ai.