AI Red Teaming

AI red teaming is the structured adversarial probing of an AI system — usually a deployed LLM application — to surface failure modes before real users or adversaries find them. It borrows the name and posture from security red teaming but covers a broader failure surface: safety (harmful outputs, bias, dangerous instructions), security (prompt injection, data exfiltration, tool-use abuse), robustness (jailbreaks, obfuscation, multi-turn escalation), and capability (sandbagging, deception, sycophancy). A red-team engagement typically combines manual creative probing by trained adversaries, automated attack suites (PAIR, AutoDAN, GCG), and targeted scenario replay based on threat-model assumptions for the specific deployment.

The landscape matured rapidly in 2024-26. Anthropic, OpenAI, and Google DeepMind publish red-team practices and pre-release evaluation reports. ARC Evals and Apollo Research specialise in frontier-model evaluation. Community platforms (HackAPrompt, Gray Swan) crowdsource adversarial discovery. Automated red-team tooling (Garak, PyRIT from Microsoft, Promptfoo) has become installable infrastructure. EU AI Act and Japan's AI Bill both expect evidence of adversarial testing for high-risk systems, giving red teaming real regulatory teeth rather than voluntary best practice.

For APAC mid-market teams, the right cadence is **a 2-week red-team sprint pre-launch plus continuous automated probes in production**. The sprint runs a coverage matrix: prompt injection (direct and indirect), jailbreak families (role-play, obfuscation, multi-turn), data-leak probes, tool-use abuse, bias elicitation across protected categories, multilingual failure modes (especially important in APAC where safety training is English-heavy). Findings are severity-ranked (critical / high / medium / low), tracked as bugs, and either fixed, mitigated, or accepted with documented rationale. This is the minimum bar for any customer-facing LLM application.

The non-obvious failure mode is **red-teaming the model but not the system**. Teams throw adversarial prompts at the model directly and pronounce the system safe when the model refuses. But in production, attackers don't type into the model — they attack the surrounding system: indirect prompt injection through retrieved documents or tool outputs, payload smuggling through uploaded files, confused-deputy attacks via tool-use authorisation, context-window pollution via earlier turns. These are application-level vulnerabilities the model cannot independently defend against. The red-team scope must include the full stack — RAG pipeline, tool registry, session state, output renderer — not just the model.