Hard Lessons from Taking LLMs Out of the Lab and Into Production
Every engineering team I talk to today is building with AI, whether they say it out loud or not. Sometimes it’s a flashy copilot. Sometimes it’s a quiet LLM stitched into search, support, or internal tooling. The form factor changes, but the pattern doesn’t.
Teams move fast, ship value, and then stumble into risks they did not realize they had created.
After spending the last year reviewing real implementations, incidents, and production architectures, the same mistakes show up again and again. Not because teams are careless, but because LLMs break assumptions we’ve relied on for decades.
This blog outlines the most common AI application security mistakes teams make when moving LLM-powered features into production, based on real-world implementations and incidents.
Here are the seven mistakes I see most often.
The seven mistakes, up front
- Assuming the old application security threat model still applies
- Treating the model as trusted compute
- Encoding policy and business logic inside prompts
- Giving agents too much authority too quickly
- Treating RAG and vector stores as harmless infrastructure
- Dismissing misinformation as a “quality” problem
- Forgetting that tokens are a finite, attackable resource
Now let’s unpack why each of these shows up, and why they matter.
1. Assuming the old threat model still applies
This is the root mistake that feeds all the others.
Teams assume that securing an AI feature is just an extension of securing a web app or API. Input validation, authentication, authorization, …. These all get done like we’ve always done them. However, once an LLM enters the system, the application stops behaving like a deterministic service.
Models reason over untrusted input, retrieve external data, synthesize outputs, and increasingly act through tools. That creates failure modes that do not map cleanly to classic injection or access control flaws and the new class of issues like prompt injection, indirect injection through retrieved content, excessive agent behavior, and cost-based abuse are not edge cases, they are structural risks.
If your threat model does not change when a LLM is introduced, your system will surprise you later.
2. Treating the model as trusted compute
Many teams implicitly trust model output because it sounds reasonable.
That trust leaks into code and outputs are rendered directly into interfaces, they’re parsed into commands, they’re used to call internal services and they’re allowed to make commitments on behalf of the business.
This is backwards. An LLM is not a trusted execution environment, and it is closer to a probabilistic interpreter sitting on top of untrusted data. Every token that enters and leaves the model should be treated as tainted until validated.
Structure, schemas, sanitization, and sandboxing are not optional. If the model can influence behavior outside itself, guardrails must exist in code and infrastructure, not in natural language instructions.
3. Encoding policy and logic inside prompts
This mistake feels efficient right up until it fails.
Teams encode access rules, safety constraints, and business logic directly into system prompts. It works in early prototypes, but then prompts leak, or they get logged, extracted, copied into debugging tools, or surfaced through indirect injection.
Once policy lives in a prompt, it is no longer enforceable. Prompts are not versioned policy engines, and they should be thought of as suggestions. At DryRun, we have NLCPs (natural language code policies) that get enforced through our Custom Policy Agent. They are separate, enforceable policies.
The model should consume policy, not define it. If breaking a rule is possible because the model misunderstood or ignored instructions, that rule was never real.
4. Giving agents too much authority too quickly
Agentic systems compress time. They also compress blast radius.
As soon as an LLM can call tools, mutate state, or interact with production systems, mistakes become operational. A single crafted input can trigger actions that would normally require human intent, context, and approval.
Teams often start permissive and promise to tighten controls later. Later usually arrives as an incident as users and attackers exploit authority in novel ways using LLM intelligence and tools.
Least privilege still applies, and so do explicit allowlists, step limits, approvals for high-impact actions, and bounded execution. If an agent can spend money, move data, or change records, you should assume it will eventually be confused, manipulated, or both! Setting bounds on your agents is crucial.
5. Treating RAG and vector stores as harmless infrastructure
Retrieval systems feel safe because they look like search. They are not.
Vector stores are data systems with all the usual risks: misconfiguration, cross-tenant leakage, poisoning, and weak access controls. When they fail, they fail quietly. Answers degrade or skew long before anyone notices. Provenance, filtering, tenant isolation, and observability are table stakes, not advanced features.
RAG expands your data perimeter. If you don’t secure it like one, it could become an attack surface.
6. Dismissing misinformation as a quality problem
Hallucination is often treated as an embarrassment rather than a liability.
That framing is outdated now that you own the output. If your system presents information as authoritative, you own the consequences. This takes your control of your data and puts it in the hand of the courts, regulators, and customers who don’t care that the model “made it up.”
High-impact outputs require grounding, citations, or refusal paths. Some require humans in the loop and now correctness is now part of security and product design, not an afterthought.
7. Forgetting that tokens are a finite, attackable resource
LLMs introduce a new form of denial of service: resource exhaustion (e.g. money!).
Unbounded prompts, recursive agents, retries without backoff, and poorly designed loops can burn through budgets fast. Attackers are already exploring this and waiting for it to show up on provider dashboards will not save you in real time.
Tokens need limits and quotas and consider agents need timeboxed execution windows and circuit breakers.
The pattern behind the pattern
All seven mistakes trace back to the same assumption: that LLMs are magic instead of machinery.
Secure AI systems are built the same way secure systems have always been built: clear boundaries, least privilege, explicit policy, strong defaults, and controls outside the component you don’t fully trust.
If you are building with AI, the risks are already in your product. The only real decision left is whether you design for them intentionally or discover them the hard way.
We built a whitepaper that maps each of these risks to a reference architecture with controls, tools, and real incident examples. Get the full implementation guide here.
Want to see how DryRun Security catches these issues directly in your codebase before production? Let’s chat!
