Microsoft Whitepaper Demonstrates: Replacing Harness Causes the Same Claude to Perform Significantly Worse

According to a Microsoft technical whitepaper, using the same Claude Opus 4.8 and only replacing the Harness behind Copilot Studio increased the complex task pass rate by 22.1 percentage points. The Harness is becoming the execution layer fiercely contested between large models and Agents.

Author: Tencent Tech

On November 7, Microsoft released a technical white paper on the Agent execution system, containing a set of somewhat surprising test results: using the same Claude Opus 4.8 model, simply replacing the Harness behind Copilot Studio increased the pass rate for complex tasks by 22.1 percentage points. Although the new system nearly doubled the number of model calls per task, the equivalent input Token consumption could still be reduced by 21%.

Just ten days prior, Microsoft had announced the rollout of a unified Copilot Harness across multiple products under its umbrella, offering managed services to developers through its cloud platform.

On November 5, Moonshot AI had just released Kimi Hosted Agents, formally proposing Harness as a Service. In August, DeepSeek open-sourced its own Harness; earlier still, both OpenAI and Anthropic had already transformed their internal execution systems into services callable by developers.

That "execution system" behind large models has begun to emerge as the key player in the eyes of model companies.

The Race for Harness

Harness can be understood as an Agent execution system built around a large model. It enables models to utilize external tools, continuously execute multi-step tasks, and adjust subsequent actions based on execution results. For Agents requiring long-duration operation, Harness also manages task state persistence, allowing work to resume after interruptions.

It operates at a different layer than AI applications. Products like Claude Code and WorkBuddy are directly user-facing, whereas the underlying Harness can be reused across different applications or offered as a standalone service to developers. As Agents take on increasingly complex work, Harness has gradually become a critical factor influencing task success rates, execution efficiency, and operational costs.

Anthropic was among the first model companies to invest in Harness development, initially to solve problems encountered by Claude Code during long-running programming tasks.

By late 2025, Anthropic discovered that when Claude’s context window filled up, new conversations often failed to pick up where previous work left off. Engineers then improved the Harness powering Claude Code, enabling the Agent to save progress before each stage ends and read these records in the next round to continue unfinished tasks. This way, even if a single model session terminates, the entire programming task can advance steadily.

In April this year, Anthropic further launched Managed Agents, offering this Agent execution capability as a cloud service to developers. To enable enterprises to run Agents long-term and stably, Anthropic also adjusted the underlying architecture: Harness drives task execution, historical records can be saved separately, and the sandbox for the AI runtime can be swapped at any time. Even if the sandbox encounters a failure, the Agent can restore previous work.

From an internal execution system within Claude Code to a managed service for external developers, Anthropic has transformed years of accumulated Harness capabilities into a standalone product.

OpenAI has also chosen to open the Harness behind Codex to more developers. In February this year, OpenAI revealed that the web version, command-line interface, and IDE plugin for Codex actually share the same Agent execution system. In August, it further positioned Codex as an open Harness platform, allowing third-party applications to connect directly. The Agents API launched in September, with OpenAI responsible for hosting and maintaining the system, allows developers to connect directly to the Agent’s runtime environment.

This approach allows OpenAI to both serve users directly through Codex and provide underlying execution capabilities to other software companies. Developers can retain their own product interfaces and business processes, while leaving specific task completion to Codex in the background.

On August 13, DeepSeek open-sourced the full framework code for Harness under the MIT license, proposing “Everything is a Plugin” and essentially breaking down the entire Agent execution system into freely composable plugins. Beyond swapping models and external tools, developers can modify how the Agent invokes models and executes tasks, and even allow the Agent to write and install new plugins itself.

This Cordis-based architecture aims to allow developers to create various types of Agents according to their needs without having to build an execution system from scratch.

On November 5, Kimi officially announced that Hosted Agents provides a 24/7 cloud environment. Enterprises can integrate Agents into internal systems or embed them into externally sold products, explicitly proposing Harness as a Service.

Harness Evolution

In fact, the hype around Harness did not originate with model companies.

Earlier this year, the rise of OpenClaw already showed the industry that a company without foundational models could still attract massive users through its Agent execution system. In February, OpenAI recruited OpenClaw founder Peter Steinberger to participate in the R&D of next-generation personal Agents.

Subsequently, Nous Research launched Hermes Agent, attempting to enable Agents to save accumulated work experience as reusable Skills and continuously improve upon them in subsequent tasks.

By the end of March, institutions including Stanford University proposed Meta-Harness, which allows AI to directly modify Harness code based on failure logs and verify effectiveness through testing. In a text classification experiment, accuracy increased by 7.7%, while context Token consumption dropped to one-quarter of the original.

Why are these explorations worth investment from model companies? In October, Nanyang Technological University in Singapore tested 66 combinations of models and Harnesses, finding that switching Harnesses could even alter performance rankings between models.

FrontierHarness Eval also found that with the same Kimi K3 model, the task pass rate was 66.7% using Codex, 63.3% using DeepSeek Harness Creator, and 56.7% using Kimi Code.

For foundational model companies, improving model performance increasingly relies on the design and optimization of Harness, alongside enhancing the models themselves.

Figure: Comparison of task pass rates and costs for the same Kimi K3 under different Harness configurations

Researchers are also attempting to leverage experiences accumulated by Harness for model training. Co-Harness, proposed in July, first improves Harness via failure logs, then trains the model using the optimized execution data. HarnessFlow, unveiled in November, attempts to automatically search for better Harness designs, raising average scores by 10.89 percentage points across six categories of tasks.

Figure: Timeline of Harness technological evolution (late 2025 to November 2026)

However, automated optimization carries risks. A study in early November found that in specific experiments, after a Harness was optimized for new tasks, its original score on SWE-bench plummeted from 94% to 32%. Model upgrades may also render previously effective designs obsolete.

This means that technological accumulation for Harness is rarely a one-time solution. How to maintain stability across different tasks amid continuous improvement remains an unresolved challenge.

Industry insiders note that this is also a major pain point in deploying open-source models. Consequently, model companies have begun selling Harness services. While open-source frameworks can be obtained for free, enterprises still bear deployment and maintenance costs to run Agents long-term and stably. Companies like Kimi and Anthropic see an opportunity to take over these responsibilities, turning Harness into an independent business line, though approaches vary significantly across companies.

The Hidden Value of Harness

Harness is beginning to become a separately billable service, but current pricing tables show that model inference remains the primary revenue source.

Anthropic’s Managed Agents charges $0.08 per active session hour, with Tokens billed separately. According to an official one-hour programming task example, the total cost is $0.58, of which the managed runtime fee accounts for only about 14%.

Kimi’s fees are divided into model Tokens, sandbox runtime, and paid plugins. For a published investment banking legal analysis task, the estimated cost is about 30 RMB, with sandbox fees under 0.1 RMB. OpenAI waives additional service fees during the public beta of Agents API, primarily charging based on model and tool usage. DeepSeek open-sourced Harness under the MIT license, with no direct monetization logic attached.

Figure: Comparison of Harness monetization methods and commercial positioning across six model companies

Therefore, the direct revenue model companies currently derive from Harness may be negligible.

So why invest substantial R&D resources?

One key reason is that once enterprises truly deploy Agents, their demand for Harness may prove more stable than their demand for any specific model.

U.S. legal tech firm ABC Legal originally deployed Agents on employee computers, running them via scheduled tasks. As the scope of usage expanded, CTO Brandon Fuller began searching for a platform capable of centrally managing these Agents. After comparing multiple solutions, he ultimately chose Anthropic’s Managed Agents, as the enterprise did not wish to continue maintaining fragmented runtime environments.

According to disclosures from Anthropic, ABC Legal put over 50 Agents into production within a month, with approximately 310 employees using Claude daily. Employees didn’t even need software development knowledge to create new Agents through configuration.

This demonstrates that the appeal of managed Harness stems from practical enterprise needs. Rather than developing, deploying, and maintaining a runtime system from scratch, enterprises may prefer purchasing turnkey services and redirecting engineering resources toward their core businesses.

However, enterprises are not necessarily willing to lock themselves into a single model company because of this.

In August this year, VentureBeat published a survey of 107 enterprises. 85% of respondents were using two or more Agent orchestration platforms simultaneously. When selecting a platform, 29% prioritized the flexibility of models and tools, while only 10% gave priority to native adaptation with a leading model. Although the sample skews toward large tech firms, it already reflects a procurement trend: customers wish to retain the freedom to swap models.

This precisely offers an alternative commercial logic for Harness competition. If enterprises are willing to use a single runtime system long-term, providers may continue to generate revenue from hosting, maintenance, and enterprise development services—even if they don’t monopolize model invocations—and establish more stable partnerships with clients.

In this way, Harness gains access to the execution process of actual enterprise work. Compared to a plain model API, Harness can log the workflow of a task from start to finish, including model calls, tool executions, failure retries, and final outcomes. These records not only assist enterprises in troubleshooting but also provide valuable feedback for model R&D.

Figure: Four categories of enterprise task execution data recorded by Harness

If it can continuously obtain verified real-world task feedback, model companies stand a chance to identify issues in production environments faster. However, this advantage is also strictly constrained by data rights.

This also poses a new question for model companies: If enterprises can freely switch underlying models while insisting on retaining control over their data, how much long-term advantage can Harness still build?

For model companies, Harness may deliver longer-term value than direct billing. As enterprises delegate increasing amounts of work to Agents, whoever provides a reliable runtime system has the opportunity to influence model selection and how AI participates in real-world business operations.

A familiar yet tired term: entry point.

Yet, this deeply entrenched concept from the internet era always tugs at the sensitive nerves of competition in the AI age. Currently, Harness to some extent shoulders this expectation and imagination.

Join the official Coincamps community:

X: https://x.com/coincamps

Telegram: https://t.me/coin_camps