Generative AI security best practices: A guide to secure GenAI usage

Traditional security tools work on syntax, the form of the input. They catch things by their structure, such as a malformed file, a known malware signature, or a traffic pattern that looks abnormal. That works because conventional attacks look different from normal activity.

GenAI attacks arrive as ordinary, well-formed language, passing every structural check without tripping a rule. A prompt injection payload reads like a normal sentence. A jailbreak reads like a reasonable request. The model cannot distinguish between a trusted system instruction and a user-supplied override if both are embedded in the same token stream. The danger isn't in how the text is built, it's in what the text means, and meaning is exactly what syntax-based tools don't read.

Protecting GenAI takes controls that work at the level of meaning instead: content classifiers that judge intent, behavioral monitoring, output validation, and adversarial red-teaming, none of which come with a traditional application security stack.

Prompt injection hijacks the model's behavior within a session by embedding adversarial instructions that the model treats as authoritative. The attack can be direct (crafted user input) or indirect (malicious instructions hidden in external content the model retrieves and processes). The goal is typically to exfiltrate data, trigger unauthorized actions, or override application logic.

Jailbreaking is a specific form of prompt injection that targets the model's built-in safety guardrails—the restrictions its developers trained it to follow. A successful jailbreak makes the model disregard those restrictions entirely, enabling it to produce prohibited content or take actions outside its intended scope. The distinction matters for defense: Prompt injection defenses focus on input validation and trust boundaries; jailbreak defenses require adversarial training, model-level updates, and ongoing red-teaming against new bypass techniques.

The categories that carry the most compliance and competitive risk, such as customer PII, source code, login credentials, internal financial data, and confidential business plans, should never be entered into a GenAI tool. These are the off-limits categories that appear most often in leaked prompts. A clear acceptable use policy should name them explicitly, and prompt monitoring should enforce the list rather than relying on employees to remember it.

A standard GenAI chat interface generates text. An agentic AI system generates text and acts on it—calling APIs, executing code, sending emails, modifying files, querying databases, or browsing the web. This creates a qualitatively different risk profile:

• The blast radius of a successful prompt injection is no longer a harmful output. It is an unauthorized system action.
• Agents operating across multiple steps can compound errors or be manipulated to perform a sequence of individually low-risk actions that together constitute a serious breach.
• Indirect prompt injection becomes significantly more dangerous: Malicious instructions embedded in a document or web page the agent retrieves can cause it to exfiltrate data to attacker-controlled endpoints, delete records, or escalate permissions.
• Many agentic frameworks grant broad tool access by default, making excessive agency the rule rather than the exception unless explicitly constrained.

Standard incident response playbooks do not address the failure modes specific to GenAI systems. A GenAI incident response plan should cover:

• Prompt injection and jailbreak events: How to identify that the model has been manipulated, what data or actions were affected, and how to isolate the affected session or application.
• Model or training data compromise: Procedures for assessing whether a fine-tuned model or RAG knowledge base has been poisoned, and the criteria for rolling back to a known-clean model version.
• Data exposure via prompts: How to determine what sensitive content was submitted to which platform, whether that platform's data handling policies cover the exposure, and what notification obligations apply.
• Shadow GenAI incidents: How to respond when an unauthorized AI tool is found to have processed regulated data, including vendor communication and regulatory disclosure.
• Agentic AI failures: A containment procedure for autonomous agents that have taken unauthorized or unexpected actions, including how to revoke agent credentials and reverse affected system changes where possible.