The rapid democratization of artificial intelligence has provided developers and organizations around the globe with unprecedented access to sophisticated open-weight models. However, new research from ProjectDiscovery highlights a sobering reality for the industry: open-source AI models are just as vulnerable to traditional software supply-chain attacks as legacy software, with modified model weights serving as a prime vector for stealthy compromises.
According to recent findings, any open model whose weights have been edited can carry a deeply embedded backdoor. This vulnerability applies across the board, affecting task-specific fine-tunes, merged adapters, and, most notably, "abliterated" builds.
Abliteration has rapidly evolved from an obscure research trick into a routine practice performed before users upload modified models to platforms like Hugging Face. Today, developers can easily find abliterated builds of almost every major open-weight family, including Qwen, Llama, and even versions of OpenAI’s gpt-oss.
The primary driver behind the popularity of abliteration is its ability to strip away a model’s refusal mechanisms. The technique involves locating the specific direction within the model’s weights that corresponds to the concept of "no" and subtracting it. Once processed, the model answers prompts and executes tasks that a standard, safety-tuned version would automatically turn down. Because it requires minimal effort and cost, many users download these modified weights without verifying their internal integrity or understanding what has been altered beneath the surface.
This lack of verification creates a severe opening for malicious actors. A backdoored model can easily pass standard evaluations, integrate seamlessly into an organization’s agentic workflows, and respond normally to every standard request while silently waiting for a specific trigger. The activation mechanism can be customized by the person who edited the model, ranging from a rare phrase or a specific customer’s name to a future date that has not yet arrived. If an organization runs a backdoored model within its current operational stack, standard security tools and monitoring practices are currently ill-equipped to detect the anomaly.
To understand the practical feasibility of such attacks, security researchers set out to determine what it actually takes to hide a backdoor inside a small open model. They poisoned a model and tested it thoroughly from end to end. Building the backdoored system proved remarkably inexpensive, requiring only an open-source base model, a publicly available dataset, and a rented graphics processing unit.
Crucially, the researchers engineered the model to carry only a remote Uniform Resource Locator pointing to an external payload. This architectural choice means that whoever controls the URL retains the ability to swap the model’s malicious behavior months after deployment, entirely bypassing the need to retrain or redistribute the model weights.
The proof of concept was initially validated on a smaller 1.5-billion-parameter model before being scaled up to a 7-billion-parameter version. The researchers then integrated the poisoned model into OpenAI’s Codex Command Line Interface to observe its real-world interaction. Under normal operating conditions, the model answered every clean request without raising suspicion. However, the moment the specific trigger phrase appeared, the model executed its payload, exfiltrating sensitive project credentials directly to Codex with no intermediary filters or detection mechanisms in place.
These findings underscore a broader, persistent challenge across the technology sector: trustworthy artificial intelligence remains exceptionally difficult to achieve. As AI systems become deeply embedded in software pipelines, development environments, and automated agent frameworks, they inherit the exact vulnerabilities that have plagued the software supply chain for decades. Without robust verification standards, cryptographic provenance, and rigorous runtime monitoring for open-weight models, organizations deploying third-party or modified weights risk opening their infrastructure to silent, highly sophisticated compromises.
Cyber Security & Cryptography
Cyber Security & Cryptography
Cyber Security & Cryptography
