Safety
Red teaming
Red teaming is deliberately attacking your own system — with injections, jailbreaks and edge cases — to find failures before someone else does.
For an LLM application it means trying to make the model ignore its instructions, reveal its system prompt, or misuse a tool. Anything an agent can do, an attacker will try to make it do.
It should be a repeated exercise rather than a launch checklist, because both the attacks and the models change.
In practice
The version that finds real problems targets your system, not the model. Providers have already tested whether the model will write malware; nobody but you has tested whether your retrieval index contains a document that instructs your agent. Attack the tools, the delimiters and the permissions, not the safety training.
Common questions
What should I red-team in my own application?
Your system, not the model. Providers have already tested whether the model will write malware; nobody but you has tested whether a document in your index can instruct your agent.
How often should red-teaming happen?
On every change that adds a tool, a data source or a permission. Those are the changes that move the attack surface, and each one individually looks too small to justify a review.