The model answered incorrectly, possibly because it didn't really look at it: Kimi gave the AI an eye test

According to 动察 Beating monitoring, the dark side of the moon has released PerceptionBench, which specifically examines whether a multimodal model has truly seen the image.

Existing evaluations often mix together vision, knowledge, and reasoning. After a model gives a wrong answer, it is difficult to determine whether it saw it wrong or thought about it wrong.

PerceptionBench first collects failure cases of cutting-edge models in 42 benchmarks, and then infers from these mistakes 10 basic visual abilities such as counting, localization, text recognition, and spatial reasoning.

Based on this, the team has created 3000 questions. Each question tests only one ability, can be answered just by looking at the image, without the need for reasoning or external knowledge.

The project's code and dataset have already been open-sourced.