AI Begins to Challenge Real-World Engineering Design: GPT-6 Astra Achieves 37% Pass Rate, Five Models Score Zero

Beating AI News Flash: AI startup Xplore releases RWE-Bench to test large models' ability to solve real engineering problems.

11 models tackled 20 mechanical, electrical, and aerospace tasks. GPT-6 Astra ranked first with a 37% single-attempt average pass rate, Claude Opus 5.5 scored 29%, and GPT-6.1 Sol scored 25%. Another 5 models all failed entirely, including DeepSeek V4.1 Flash, GLM-5.3, and Grok 4.7.

The tasks covered engine crankshaft design, circuit board routing, and turbine blade thermal simulation. Designing a crankshaft requires balancing dimensions, weight, strength, and fatigue life; routing an eight-layer circuit board already mounted with nearly 300 components requires connecting 71 signal lines while controlling timing errors and impedance; simulating turbine blade heat dissipation also requires correctly calculating heat transfer between high-temperature airflow and the blades.

Each model used its own accompanying Agent toolchain to directly operate professional engineering software. Each problem allowed up to 24 hours of runtime and 5 independent attempts. Submitted designs were automatically validated by engineering software and physics solvers, and failure to meet any single hard requirement counted as a failure. Across all 1,100 tests, 4 problems were never solved by anyone. However, when each problem allowed 5 attempts, Claude Opus 5.5's task pass rate rose to 70%, overtaking Astra's 60%.

Xplore hopes to further turn such engineering tasks into reinforcement learning training environments, enabling AI to improve its engineering capabilities through repeated design, simulation, and error correction.