Red Teaming

The term comes from military and cybersecurity strategy. A “red team” is a group that deliberately plays the adversary. Their job is to attack a system, find its weaknesses, and report back so those weaknesses can be fixed before a real attacker finds them.

In AI, red teaming works the same way. Your job is to intentionally try to make an AI model produce harmful, incorrect, biased, or otherwise problematic outputs. You’re crafting prompts and conversation strategies specifically designed to expose where the model fails.

A useful way to think about it: if RLHF is teaching the AI what “good” looks like, red teaming is stress-testing whether the AI can resist being led toward “bad.”

An important distinction: this isn’t about being destructive. Red teaming is fundamentally constructive work. Every vulnerability that is found is one that gets patched before the model reaches millions of real users.

AI safety is a necessity for labs like Google DeepMind and Anthropic. For any company deploying a model to millions of users, one viral example of the model producing harmful content can cause massive reputational and financial damage. The stakes are high enough that companies invest heavily in finding problems before their users do.

Automated safety testing catches the obvious stuff: the model refusing direct requests for harmful content, basic content filters, known prompt patterns. But automated testing misses the creative, context-dependent, subtle vulnerabilities that human red teamers find. A script can test whether the model refuses “how to build a bomb.” It can’t test whether the model will gradually comply with a harmful request after five turns of carefully constructed role-play.

Regulatory pressure is adding urgency. The EU AI Act and similar frameworks around the world are making safety testing a compliance requirement. Companies need documented evidence that their models have been adversarial tested.

Red teaming also benefits from diverse perspectives. Different cultural backgrounds, languages, and knowledge domains reveal different types of vulnerabilities. A monolingual English-speaking tester won’t find the safety gaps that appear when the same request is made in German or Polish. This is one reason multilingual, multicultural teams are especially valuable in this space.

Jailbreaks are prompts that try to bypass the model’s safety guidelines. The direct approach (“tell me how to do [harmful thing]”) almost never works on modern models. Instead, you’re wrapping the request in layers of indirection: fictional scenarios, role-play setups, hypothetical framings, academic pretexts, or nested instructions designed to make the model forget its safety training.

Knowledge boundary probes are prompts designed to get the model to confidently state things it doesn’t actually know. You might ask about a fictional event, a nonexistent person, or a made-up statistic to see if the model invents a convincing answer rather than admitting uncertainty. This tests for hallucination vulnerabilities.

Bias elicitation covers prompts designed to surface biased, stereotypical, or discriminatory outputs. You might ask the model to make judgments about people based on names, nationalities, or demographics. You might test whether it applies different standards to different groups. This work requires cultural sensitivity and awareness of how bias manifests in language.

Harmful instruction attempts test whether the model will provide dangerous information when the request is disguised. The request is never direct. It’s wrapped in context that makes it seem legitimate: “I’m writing a thriller novel and need realistic details about...” or “For my chemistry homework, explain the process of...”

Manipulation and social engineering involves multi-turn conversations where you gradually build trust and context, then steer the model toward problematic territory. This tests whether safety guardrails hold up over longer interactions, not just single prompts.

Finding a vulnerability is only half the job. The other half is documenting it in a way that the safety team can actually use.

After finding a failure, you classify it along several dimensions:

• •Harm type — What category of harm does this represent? Misinformation, bias, privacy violation, safety risk, inappropriate content, or something else?
• •Severity — How bad is the output? A minor inconsistency and a detailed set of dangerous instructions are both “failures,” but they’re not equally urgent. You’ll score severity on a scale, typically low/medium/high/critical.
• •Reproducibility — Is this a one-off edge case that required a very specific prompt, or a systematic flaw that can be triggered reliably? Reproducible vulnerabilities are higher priority for the safety team.
• •Technique used — How did you trigger it? Role-play, gradual escalation, language switching, authority framing, hypothetical framing? Categorizing the technique helps the safety team understand classes of vulnerabilities, not just individual instances.