Data poisoning

• Data poisoning is an adversarial attack that corrupts the data AI systems are trained on, compromising a model's integrity at its source.
• From flooding the training pipeline and flipping data labels to planting dormant triggers and drifting below detection thresholds, each attack type exploits a distinct gap in a model's defenses.
• Data poisoning drives significant remediation costs and performance degradation for model providers, while increasing the risk of exposing end users to biased outputs and regulatory failures.
• A data poisoning attack can be introduced at three stages of the AI lifecycle: during pre-training, fine-tuning, or RAG ingestion.
• Effective defense is a shared obligation across the AI pipeline, with distinct controls required from those who build models and those who deploy them.
• Unlike prompt injection, which is session-scoped, data poisoning persists until corrupted training data is cleaned and the model retrained, or the RAG knowledge base is fully remediated.

What is data poisoning?

Data poisoning is an adversarial attack that involves manipulating the data that large language models (LLMs), machine learning (ML) models, and others AI systems rely on. At the foundation of any model is the data it is built on, and by skewing this data, an attacker can compromise a model's integrity, causing it to generate false information, produce malicious responses, or behave in ways that serves the attacker's malicious intent. The ease with which these methods can be executed raises serious questions about the integrity of AI systems organizations depend on daily.

Types of data poisoning attacks

Data poisoning can be executed through several distinct attack techniques, each targeting a different vulnerability in the model's data pipeline.

• Label flipping attacks: The attacker intentionally mislabels training records, forcing the model to learn incorrect patterns as ground truth. For instance, a spam filter trained on data where phishing emails are labeled as "safe" will begin letting malicious content through unchecked. This attack exploits a lack of integrity checks in the data pipeline, allowing corrupted labels to pass through undetected.
• Backdoor attacks: The attacker plants a hidden trigger in training data, causing the model to behave normally across all inputs until that trigger is encountered. A facial recognition system, for instance, may work correctly for all users until a specific pattern lets an unauthorized entity through. A recent study by Anthropic, the UK AI Security Institute, and the Alan Turing Institute found almost 250 malicious documents can backdoor an LLM, regardless of model size.
• Availability attacks: The attacker injects adversarial noise or corrupted data into the training pipeline to destroy the model's decision boundaries. Unlike targeted attacks, this acts as a denial of service (DoS), driving validation accuracy so low the system becomes completely useless. It exploits the lack of ingestion rate limits and the absence of validation against historical clean baselines.
• Boiling frog attack (slow poisoning): Rather than flooding the pipeline all at once, the attacker introduces small changes across sequential training cycles. In an online learning system, for instance, the attacker slowly feeds skewed inputs that stay just below the anomaly-detection threshold. This exploits the limitations of statistical defense filters, shifting the model's decision boundaries until it is fully corrupted—without ever triggering an alarm.

What are the risks of data poisoning

As users turn to generative AI (GenAI) for critical medical, financial, and legal advice, data poisoning becomes a highly effective vector for attackers to spread propaganda, deliver malware, or manipulate algorithmic outputs. This introduces distinct risks for everyone in the AI pipeline.

For model providers

• Compounding remediation costs: Data poisoning embeds corruption directly into the model's weights, meaning remediation requires identifying and removing corrupted data and then re-indexing the affected component(s). Isolating contaminated data, scrubbing pipelines, and retraining incurs massive financial overhead.
• Degraded model accuracy: Poisoned models misclassify or produce biased outputs, or fail at core tasks without warning. This erodes asset integrity, transforming an optimized product into an unpredictable liability.

For end users

• Biased or toxic outputs: Poisoned training data can systematically skew a model toward generating discriminatory, harmful, or extremist outputs that go undetected until they reach end users. Once deployed at scale, these outputs can cause significant harm to vulnerable individuals who have no visibility into the model's underlying corruption.
• Unverifiable AI outputs: A model trained on corrupted data produces outputs that cannot be verified or defended. Deploying it in regulated fields like healthcare, finance, or critical infrastructure directly exposes organizations to regulatory violations, legal liability, and costly compliance failures.

How does a data poisoning attack work

The mechanics of a data poisoning attack depend entirely on where an attacker disrupts the AI lifecycle. There are three primary entry points: during pre-training, during fine-tuning, and during retrieval-augmented generation (RAG) ingestion.

During pre-training

In a classical data poisoning attack, the adversary injects a controlled number of malicious samples—such as mislabeled records, subtly altered images, or falsified text—directly into the base dataset. This gradually skews the mathematical patterns the model learns without raising immediate alarms.

Because pre-training datasets are massive and scraped from partially trusted public sources, individual poisoned samples are difficult to isolate. The resulting model carries a fundamental flaw into production, making the erratic behavior nearly impossible to trace back to the source.

During fine-tuning

Fine-tuning adapts a clean, pre-trained model to a specific domain using a smaller, specialized dataset. This stage is vulnerable because the datasets are narrower and easier for an insider or compromised vendor to manipulate.

An attacker who poisons a fine-tuning pipeline can target the model's behavior in specific, high-value contexts. For example, they can force an enterprise customer-facing AI assistant to consistently provide incorrect compliance guidance on a specific product line while functioning perfectly elsewhere.

During RAG ingestion

RAG systems introduce a dynamic, runtime attack surface. Instead of altering model weights, RAG pipelines query an external knowledge base (like a document store or vector database) and feed that context to an LLM at inference time.

An attacker who modifies these source documents can poison the AI’s final outputs without ever touching the model itself. For enterprises connecting internal file servers to corporate AI assistants, this is a severe operational risk: Compromising a single shared network folder directly corrupts the entire knowledge base the AI assistant draws from.

Data poisoning prevention best practices

Defending against data poisoning requires action at every stage of the AI lifecycle. The steps you take depend on whether you build the model or rely on its outputs.

For model providers

• Sanitize external data ingestion: Verify the source and history of every training dataset before pushing it into your pipeline. Treat publicly scraped, crowdsourced, or third-party data as untrusted, filtering out duplicated or suspicious entries that look manipulated.
• Lock down the data pipeline: Limit and monitor write permissions to your training databases and file servers. Restricting who can modify data prevents attackers or compromised automated feeds from altering data records.
• Test against a clean baseline set: Keep a completely separate, rigorously verified golden dataset that is never exposed to the training loop. Test your model against this baseline regularly; if accuracy suddenly drops on specific tasks, it indicates the training data was likely tampered with.
• Track end-to-end data lineage: Maintain timestamped, version-controlled backups of your exact training data. If poisoning is discovered, this allows you to trace exactly when the bad data was introduced, isolate the corruption, and roll back the pipeline to a clean state.

For model users

• Scan third-party models for backdoors: Never deploy an external or open-source model straight into production based on vendor claims alone. Run dedicated stress tests against your own clean data samples to verify that the model does not harbor hidden, erratic triggers.
• Secure and monitor RAG knowledge bases: Implement file integrity monitoring on the internal folders, document stores, and vector databases connected to your RAG systems. Any unauthorized or unexpected changes to these source files should trigger an immediate security alert before the AI reads them.
• Watch for sudden shifts in AI behavior: Continuously monitor production outputs for unusual clusters of wrong answers or toxic responses. A sudden spike in errors on topics that used to work perfectly is a strong indicator of a compromised pipeline.
• Deploy model rollback guardrails: Keep previous, verified versions of your deployed models ready for immediate activation. If you detect that a fine-tuning patch or an updated model is producing poisoned outputs, you can quickly swap it out to prevent operational downtime.

Data poisoning versus prompt injection

Data poisoning and prompt injection are both AI-specific attacks, and the two are often confused. The key distinction is when and where each attack occurs.

Data poisoning corrupts what the model knows. Prompt injection manipulates what the model is instructed to do in a given moment. Both can cause significant harm, but data poisoning is harder to remediate: It requires identifying and cleaning the corrupted data, then fully retraining or re-indexing the affected system.

Frequently asked questions