DeepSeek warns that its open source AI models, including R1 and Alibaba’s Qwen2.5, are highly vulnerable to jailbreak attacks, raising urgent safety concerns for developers and users.
Hangzhou-based AI start-up DeepSeek has issued a warning about the vulnerabilities of open source AI models, revealing that its R1 and Alibaba’s Qwen2.5 models are highly susceptible to “jailbreak” attacks by malicious users. Open-source models’ free accessibility, while accelerating adoption, allows external safety mechanisms to be removed, increasing risks of harmful outputs.
In a peer-reviewed article published in Nature, DeepSeek detailed its testing regime, including industry benchmarks and in-house “red-team” exercises based on Anthropic’s framework. The company found that R1 and V3 models scored slightly above average on safety benchmarks compared with OpenAI’s o1 and GPT-4o, and Anthropic’s Claude-3.7-Sonnet, but R1 became “relatively unsafe” once external risk controls were removed.
DeepSeek CEO Liang Wenfeng emphasised: “We fully recognise that, while open source sharing facilitates the dissemination of advanced technologies within the community, it also introduces potential risks of misuse. To address safety issues, we advise developers using open source models in their services to adopt comparable risk control measures.”
The warning comes amid growing attention from Chinese authorities to balance AI development and safety. A technical standards body under the Cyberspace Administration of China highlighted that open sourcing foundation models “will widen their impact and complicate repairs, making it easier for criminals to train ‘malicious models’.”
The Nature paper also revealed that R1’s compute training cost was US$294,000, significantly lower than comparable US models, and refuted accusations that DeepSeek had “distilled” outputs from OpenAI models. Peer-reviewed recognition has made DeepSeek trending on Chinese social media as the “first LLM company to be peer-reviewed,” potentially encouraging greater transparency in China’s AI industry.
DeepSeek’s findings underscore both the innovation potential and the inherent risks of open source AI, highlighting the urgent need for robust safety measures for developers worldwide.