Parallel tested 8 decision models: Jev takes early lead, OpenAI closes gap significantly by tweaking Prompt.

Beating AI News Flash: AI search company Parallel tested 8 decision models, including Jev, Perplexity, and OpenAI Decisions, using real cases from its own search business.

The test included 4 tasks: judging whether webpage changes are important, filtering low-quality content, selecting search indexes, and ranking search results. When using the initial prompts directly, Jev ranked first on average across the four tasks, Perplexity second, and OpenAI fifth.

Among them, the hardest was judging whether a webpage change is worth reprocessing. For example, if a job posting only adjusts the order of two requirements, it may not need to be updated; if a leaderboard ranking changes, it is worth recrawling. The test used a metric that comprehensively measures false positives and false negatives, with higher scores being better. Perplexity initially scored 93.3, Jev 87.9, and OpenAI only 79.0.

The team then had Claude help each model improve its prompts. Taking webpage change identification as an example, they first prepared 800 manually labeled cases, had Claude examine the model's incorrect judgments, and then added rules for situations prone to errors. They tried 5 consecutive rounds, keeping only the better-performing version in each round. Ultimately, on an independent test set, OpenAI rose from 79.0 to 88.8, Rune 26B from 74.1 to 83.9, and Nimble 9B from 65.1 to 75.3. All three models improved by about 10 points.

Combining the three tasks of webpage change identification, low-quality content filtering, and query routing to calculate the average score, Perplexity rose from 97.3 to 97.6, Jev from 95.5 to 96.5, and OpenAI jumped significantly from 88.7 to 95.1. The average scores of all 8 models improved, but the models already in the lead made limited progress, while OpenAI clearly narrowed the gap and ranked third after optimization.

The remaining search ranking task was calculated separately. After optimizing the prompts, the overlap rate between Jev's top search results and manually labeled results reached 71.45%, while Perplexity's was 70.68%. However, a small ranking model fine-tuned by Parallel itself reached 75.21%, about 3.8 percentage points higher than Jev, and its inference cost per token was only about one quarter of Jev's. The tradeoff is that it requires independently collecting labeled data, training, and deployment. For a large number of fixed tasks that need to be processed repeatedly, a specially trained small model may still be more cost-effective.