FIVE MISCONCEPTIONS THAT QUIETLY BREAK PRODUCTS, PARTNERSHIPS, AND TRUST
A few weeks ago, we found ourselves in one of those conversations that starts as a casual catch up and ends up feeling like a line in the sand.
We were reasoning with Andy Hutt, our friend at Morrama, about where AI safety is heading, and what “good” is going to mean when the users are children. Not “good” as in a policy document. Not “good” as in a moderation endpoint. “Good” as in: the kind of system design you can stand behind when a parent is watching over a shoulder, when a school is evaluating your product, and when a regulator asks you to explain what you built and why.
Around the same time, the FABA announcement of their NARRAI initiative landed. It was one of those moments that pushes you to think deeper, not just about what is possible, but about what is responsible. Building safe models and safe experiences for kids is quickly becoming a real category in its own right, and it deserves a higher bar than “we’ll patch it later” or “we added a filter”.
This post captures the patterns we keep seeing. These misconceptions are common, they’re understandable, and they’re avoidable. If you’re building AI for kids, we hope you manage these early, because the cost of learning them late is usually measured in trust.
MISCONCEPTION 1/ “WE’LL ADD SAFETY AFTER WE SHIP V1.”
WHY TEAMS BELIEVE IT
Because v1 needs to move fast, and “safety” feels like a layer you can add once you have usage and feedback.
WHAT ACTUALLY HAPPENS IN THE REAL WORLD
In kids’ AI, “later” is usually too late. The first unsafe interaction becomes a screenshot that lives forever, and the trust debt is brutal to repay. Parents, schools, and partners rarely give a second chance once a product is perceived as careless with children.
THE HIDDEN TECHNICAL TRAP
If your architecture is built around unconstrained generation and you try to patch it with “a filter”, you end up with fragile safety that breaks under pressure and breaks your product when it does work. You get false positives that ruin the experience and false negatives that ruin trust.
WHAT TO DO INSTEAD
Treat safety as a product requirement from day one, the same way you would treat payments, authentication, or uptime. Safety should be baked into architecture, tooling, and UX, not sprinkled at the end of a pipeline.
MISCONCEPTION 2/ “IF WE FILTER OUTPUTS, WE’RE SAFE.”
WHY TEAMS BELIEVE IT
Because output moderation is easy to integrate, easy to demo, and feels like a clear “safety checkbox”.
WHAT IT MISSES
Output filtering helps, but it is not the system. Real safety means controlling inputs, model behavior, retrieval, tools, logging, escalation, and UX. Kids don’t just get unsafe answers, they get unsafe journeys: rabbit holes, persuasion loops, oversharing, parasocial attachment, and “helpful” advice that crosses boundaries.
WHY IT CREATES A WORSE PRODUCT
Output-only safety often produces confusing refusals that feel arbitrary. Kids interpret this as a challenge, or they learn that the system is inconsistent. Both outcomes increase boundary testing.
WHAT TO DO INSTEAD
Shift safety left. Use pre-model routing and constraints. Decide what kind of response is appropriate before you generate. In many kid contexts, the safest answer is not a refusal, it’s a redirect or a constrained explanation that keeps the child moving in a safe direction.
MISCONCEPTION 3/ “A SINGLE LLM + MODERATION API IS A SAFETY STACK.”
WHY TEAMS BELIEVE IT
Because it looks complete in a diagram: model in, model out, moderation on the end
THE REAL ATTACK SURFACE
Kids do not interact like adults. They use voice, images, slang, misspellings, memes, and roleplay. They also use the product while tired, upset, bored, excited, or trying to impress friends. Your system needs to hold up under those real conditions.
WHAT A REAL STACK INCLUDES
A kid-safe system needs layered defense, for example:
A/ Age gating and age band modes
B/ rompt hygiene and input normalization
C/ Topic and intent classification
D/ Policy routing (explain vs refuse vs redirect)
E/ Retrieval constraints and safe sources
F/ Multimodal scanning for images and media
G/ Tool constraints (browsing, video, plugins)
H/ Parent controls and account-level governance
I/ Telemetry and human escalation paths
WHAT TO DO INSTEAD
Design your system like an aircraft, not a bicycle. Assume something will fail and make sure you have more than one way to prevent harm. That is what defense in depth looks like in kid AI.
MISCONCEPTION 4/ “WE’RE COPPA-COMPLIANT, SO WE’RE SAFE.”
WHY TEAMS BELIEVE IT
Because privacy compliance is measurable, and it feels like the core of “child safety”.
THE BIGGER RISK CATEGORY
Privacy compliance is necessary, but it is not sufficient. Most harm in kid AI is not “data theft”. It is developmental harm: manipulation, shame spirals, dependency, risky challenges, sexual content, self-harm content, and exposure to adult themes.
THE PART FOUNDERS UNDERESTIMATE
Founders often underestimate how quickly an AI experience can become emotionally authoritative. Kids can bond with systems, treat them like teachers, and follow suggestions with a level of trust that adults would not give.
WHAT TO DO INSTEAD
Build around a duty of care mindset. Ask not only “are we allowed to collect this?” but “what could a child reasonably misunderstand here, and what would that misunderstanding lead to?” Then design the system to reduce that risk.
MISCONCEPTION 5/ “KIDS ARE JUST SMALLER ADULTS.”
WHY TEAMS BELIEVE IT
Because the interface looks the same, the model is the same, and the content categories look similar.
WHAT ACTUALLY HAPPENS IN THE REAL WORLDKids interpret authority differently. They are more suggestible, more literal, and they test boundaries as play. They do not have adult judgment, context, or emotional regulation.
WHERE THIS BECOMES DANGEROUS FAST
A playful suggestion can become a dare. A “confidence” tone can become perceived authority. A roleplay character can become an emotional anchor. If you don’t design for kids specifically, you are effectively running an experiment on children with unknown outcomes.
WHAT TO DO INSTEAD
Treat ambiguity as risk, not cleverness. Use age-banded experiences. Use language that is supportive but not intimate, helpful but not directive, warm but not manipulative. Always design with the parent trust loop in mind.
THE PATTERN BEHIND ALL FIVE MISCONCEPTIONS
Every one of these misconceptions comes from the same root belief: that safety is a feature.
In kid AI, safety is not a feature. It is the product. It is the thing that makes everything else usable, scalable, and partnerable.
If you want to build something that schools will adopt, parents will trust, and partners will white-label, your safety posture cannot be a slide. It has to be a system.
WE'RE FINH
FINH is an innovation and product invention studio. We build end to end digital and physical products, and we spend a lot of our time in categories where trust is not a nice-to-have, but the entire game: fintech, kid-tech, education, and consumer platforms that need to operate safely at scale.
Alongside our product work, we run DIY.ORG, a premium creative platform for kids, and we have built AstroSafe, a child-first safety and governance stack that powers safer search, safer browsing, safer video discovery, and safer AI experiences for partners.
If you are building in this space and want to compare notes, pressure-test an approach, or simply sanity check what “good” looks like before you ship, we are always happy to talk.