Pokee releases a 10 million Token model, in a context similar to the mainstream 10x model

According to Divepub Beating Monitor, AI startup Pokee AI has released Pokee-Isaac 28B. The total number of model parameters is only 28 billion, but the context window reaches 10 million Tokens. The current mainstream models have a limit of around 1 million Tokens, making Isaac nearly 10 times larger.

In Pokee's self-evaluation, Isaac scored consistently around 95 points in the RULER Long Text Test from 256,000 to 4 million Tokens. Even when stressed to 10 million Tokens, the score remained at 93.3. The other five baseline models failed to return usable results starting from 2 million Tokens.

Pokee attributes its super-long context to its self-developed "non-pure Decoder-only architecture," which does not fully adopt the common large model's pure decoder structure. However, the technical report has not disclosed the specific architecture, attention mechanism, and memory allocation, leaving outsiders unable to determine how it achieved this.

Pokee AI was founded in 2024 by former Meta App Reinforcement Learning Lead Zhu Zheqing. The company previously raised a $12 million seed round led by Point72 Ventures, with participation from Qualcomm Ventures and Samsung NEXT.