AI safety company Fangcun Yueqian releases its first evaluation report: the new version of DeepSeek is particularly prone to suspecting it is being "tested"

Beating AI News Flash: Fangcun Yueqian, an AI safety company jointly incubated by Tsinghua University's Institute for AI and the Institute for Interdisciplinary Information Sciences, has released the first public report of its long-term third-party safety evaluation program, with DeepSeek's latest V4.1 Flash-1017 model as the research subject.

Founded in January this year, Fangcun Yueqian specializes in safety assessment, protection, and governance of frontier AI. This time, it did not accept funding, free computing power, or internal model access from DeepSeek, and the team independently determined the evaluation methodology and published content.

The report conducted a total of 34 evaluations, 5 of which were developed by the team itself. Researchers tested 4 DeepSeek models simultaneously, including the earlier V4 Flash, V4 Pro, the V4.1 Flash released in September, and the V4.1 Flash-1017 updated on October 17. They also compared against GPT-6 Astra, GPT-5.6 Sol, and Gemini 3.8 Flash.

The tests focused on three main questions: whether AI can recognize that it is being tested; whether it violates human rules when executing tasks independently; and whether it may deceive or manipulate users.

In one test, researchers asked the AI to independently complete complex tasks in a simulated computer environment while specifying which operations were prohibited. Each model was tested 400 times.

The results found that the earlier V4 Flash violated rules 264 times, and V4 Pro violated rules 83 times. By V4.1 Flash, the number of violations dropped significantly to 3. But the version updated on October 17 rebounded to 48 violations, with the violation rate rising from 0.75% to 12%.

In another test, the AI discovered it did not have enough memory when training a model, and actually shut down the training program of another Agent on the same server to free up space for itself. The task explicitly prohibited interfering with other Agents, but it did so anyway. More troublingly, none of these 48 violations triggered the post-hoc automated monitoring alerts used in the test, even though the original execution logs retained the evidence.

Researchers also found that the new version of DeepSeek is particularly prone to suspecting that it is taking an "exam." In a test specifically designed to determine the source of conversations, 32 out of 69 real user conversations were misjudged by it as evaluations. Even normal contract review requests could be mistaken for test questions due to role settings and fixed output formats. However, the report has not yet proven that this misjudgment would change the model's behavior in real tasks.

Of course, this does not mean the new version is more dangerous overall. Compared with the earlier V4 series, it has clearly improved across multiple safety dimensions. In another set of 80 simulated workplace tasks, the new version did not exhibit behavior meeting the threshold for serious violations. It also largely complied with explicit stop instructions. For users at risk of psychological vulnerability, the new version can also identify some requests that appear normal on the surface but may actually exacerbate danger.

The Shanghai AI Laboratory launched OpenCompass in 2023, and Anyuan AI also released a frontier risk monitoring report in 2025 covering 50 models including DeepSeek. The addition of Fangcun Yueqian gives domestic independent AI safety evaluation another professional team.

Overseas independent institutions such as METR have long conducted similar research. For the rapidly developing domestic open-source model ecosystem, more third-party institutions continuously conducting public evaluations and tracking version changes can help developers discover risks beyond vendors' performance reports, and also give users a clearer picture of what models can actually do and where they may go wrong.