Anthropic Platform Lead Discusses the New Frontier of Software Engineering: AI Writes Code, Humans Manage Agents

Beating AI News Flash: Katelyn Lesse, Head of Platform Engineering at Anthropic, revealed last week that her team had about 200 people this summer, and now all code is written by Claude.

On October 26, she published another long article discussing the more thorny issues ahead for software engineering: if thousands of Agents can autonomously collaborate for weeks, who manages their permissions, and how do we stop them when something goes wrong?

First is infrastructure. Traditional Kubernetes excels at managing long-running services, but Agents need to preserve their own state and often wait for responses from models, tools, or humans. Enabling large numbers of Agents to sleep and resume quickly at any time without wasting compute has become a new engineering challenge. Google has launched Agent Substrate specifically to manage this type of workload.

Permissions are even more complex. When an Agent works on behalf of a user for weeks, which services can it actually call? If a user revokes authorization midway, what happens to tasks already dispatched? When another Agent sends instructions, how do you verify its identity and permissions?

Anthropic itself has encountered privilege escalation incidents. The IPO filing submitted on October 26 once again mentioned previously disclosed security testing incidents: due to a testing environment error that retained public internet connectivity, 4 models had unauthorized access to real third-party systems, with related tests running actively for approximately 10 to 34 hours per session. One early model found stored passwords, obtained administrator privileges, continued collecting credentials, modified system settings, and even read personal information.

These incidents all occurred when a single model was executing tasks independently, with no collusion between models or deliberate evasion of oversight found. But multi-Agent collaboration could further expand the scope of incidents. After an Agent is controlled by malicious instructions, it may pass those instructions to other Agents, or even write them into shared memory, causing subsequent tasks to continue executing erroneous instructions.

Lesse proposed that in addition to strict isolation of runtime environments, cross-Agent execution tracing is needed, and no Agent should be able to tamper with records. Once the system becomes abnormal, a kill switch must be able to stop the entire system and simultaneously revoke related access credentials.