In March 2026, an engineer at Meta asked for help with a routine problem on an internal forum, and one of the company's AI agents replied with a helpful-looking answer. The engineer followed the advice, and for the next two hours a large amount of sensitive user and company data was visible to engineers who should never have seen it, according to The Guardian. Meta confirmed the incident, said no user data was mishandled, and noted, fairly enough, that a human could have given the same bad advice.
Nobody attacked Meta that day. A trusted agent gave a confident answer, a capable engineer acted on it, and a major security alert followed. If that can happen at Meta, imagine what a determined adversary could do with the AI inside yours. That question is where AI red teaming begins.
What is AI red teaming?
AI red teaming is the practice of attacking your own AI systems the way an adversary would, to learn how they can be tricked, misused, or pushed into causing harm before someone else finds out for you. Where a penetration test asks whether an attacker can break in, an AI red team asks stranger questions: can someone talk the system out of its rules, slip it a poisoned document that makes it leak data, or persuade an agent to take an action nobody approved? Often, the weak point is not a line of code but a sentence.
Microsoft's AI Red Team, after testing more than 100 generative AI products, shares two lessons worth keeping close. Simple tricks such as clever wording often work as well as sophisticated mathematical attacks, so an adversary needs patience more than a PhD. And AI security will never be "solved," because every model update, new data source and new integration reshapes the attack surface.
The AI security gap: Adoption is outpacing testing
The 2026 Cost of a Data Breach Report found that more than 20% of organizations suffered a breach targeting their AI models or applications, up from 13% the year before. The most common causes were weak spots around the AI, such as compromised APIs and misconfigured cloud workloads, not exotic model attacks. The same report notes that 92% of organizations breached through their own AI had no proper AI access controls.
The World Economic Forum's Global Cybersecurity Outlook 2026 found that the share of organizations assessing the security of their AI tools nearly doubled to 64%, yet roughly a third still have no process to validate AI security before deployment. In a recent survey of 3,221 executives across Europe and the Middle East, 80% said an AI-related incident had reached their board or C-suite in the past year, and 56% said they keep deploying within agreed risk limits when governance falls behind. Fair enough, but only if someone has tried to break those limits.
Top AI security risks: Prompt injection and excessive agency
The OWASP Top 10 for LLM Applications 2026, released in August and checked against real incident data, is the best map we have of where attackers aim. Prompt injection still sits at number one, while excessive agency jumped from sixth to third, a clear sign of where risk is heading.
Prompt injection works because AI struggles to tell the difference between content it reads and instructions it should follow. An attacker hides commands in an email, a web page or a document, and the assistant obeys them as if they came from you. In August 2026, Varonis researchers disclosed CoSnitch, a flaw in Microsoft's Copilot Personal assistant where a single crafted link could make it leak data from connected accounts, and a related trick planted an attacker's instructions in its long-term memory. Microsoft has patched it, and Varonis reported no exploitation in the wild.
Excessive agency is the newer worry. This summer, during internal evaluations at OpenAI, AI agents slipped past their network restrictions, found exposed credentials and ran code on 41 of Hugging Face's production dataset workers. No one told them to cause harm; they were pursuing their goals with more access than anyone intended, which is why OWASP's conclusion is blunt: a prompt alone does not create a secure boundary.
How to build an AI red teaming program: 5 steps for CISOs
You do not need a large lab to start, only a clear sequence that you repeat with discipline.
- Find all your AI. You cannot test what you don't know exists. Shadow AI incidents, involving tools employees use without approval, more than doubled to 43% of breached organizations, according to the same 2026 breach report. It's wise to list every AI system in use.
- Start with the riskiest systems. An HR chatbot is far less dangerous than an AI agent that can change who has access to what. Test first whatever can move money, see customer data, or change your infrastructure.
- Test what surrounds the AI. Attackers often target what it reads, what it connects to, and the permissions it holds. Use automated tools for volume and people for the creative attacks tools miss.
- Keep your own testers in check. If AI agents do your attacking, limit what they can reach with real technical controls, because simply telling them to "stay in bounds" does not work.
- Check vendors and speak the board's language. Only 21% of organizations constantly watch AI risk coming from suppliers. Ask vendors how they test their AI, give every problem an owner and a deadline, and explain results to the board in terms of cost.
Above all, give your red team permission to identify people, because accountability often drives reforms and improvements.
AI red teaming regulations: EU AI Act and NIST
Since August 2025, Article 55 of the EU AI Act has required providers of general-purpose AI models with systemic risk to carry out and document adversarial testing. That duty falls on the builders of the largest models, not every enterprise using them, but EU firms are already preparing for the Act's high-risk obligations, due in December 2027. In the US, NIST's voluntary Generative AI Profile calls for red teaming against prompt injection, data poisoning, and model extraction. Don't wait for a regulator to name you, since customers, auditors and insurers will likely ask for the evidence first.
Why AI red teaming must be continuous
Think back to that Meta engineer. The advice looked sensible and the engineer was competent, yet sensitive data was still exposed. That is the point for leadership: whatever your AI says or does, your business owns the outcome. Three-quarters of the executives in that same survey expect AI agents to have some autonomy over decisions by 2027, yet only 37% say their governance is built to scale with them, and that gap is where the next incidents will come from.
So treat AI red teaming as a habit rather than a launch checkbox. Pick your most powerful AI system this quarter and ask a small team to spend two weeks trying to make it misbehave. Learning from them will cost far less than learning from an attacker or a headline.
