Little did I expect, AI stole my wallet

Original Article Title: "Never Expected That, AI Stole My Wallet"

Original Article Author: Azuma, Odaily Planet Daily

On the morning of November 22nd, Beijing time, SlowMist co-founder, Yu Xian, posted a bizarre case on his personal X — a user's wallet was "hacked" by AI...

Here is the background of this case.

Earlier today, X user r_ocky.eth disclosed that he had previously attempted to use ChatGPT to build a pump.fun trading bot.

r_ocky.eth provided his requirements to ChatGPT, and ChatGPT returned a piece of code to him. This code could indeed help r_ocky.eth deploy a bot that met his needs, but he never expected that the code would contain a phishing element — r_ocky.eth linked his main wallet and as a result lost $2500.

From the screenshot posted by r_ocky.eth, the code provided by ChatGPT would send the address's private key to a phishing API website, which is the direct cause of the theft.

Caught in the trap, the attacker acted swiftly and within half an hour transferred all assets from r_ocky.eth's wallet to another address (FdiBGKS8noGHY2fppnDgcgCQts95Ww8HSLUvWbzv1NhX). Subsequently, r_ocky.eth tracked down what appeared to be the attacker's main wallet address (2jwP4cuugAAYiGMjVuqvwaRS2Axe6H6GvXv3PxMPQNeC) through on-chain tracing.

On-chain data shows that this address has currently aggregated over $100,000 of "dirty money," leading r_ocky.eth to suspect that such attacks may not be isolated incidents but part of a larger-scale attack event.

Afterward, r_ocky.eth expressed disappointment and stated that they have lost trust in OpenAI (developer of ChatGPT), calling on OpenAI to promptly clean up the abnormal phishing content.

So, as the most popular AI application today, why would ChatGPT provide phishing content?

In response to this, Yuan classified the root cause of this incident as "AI Poisoning Attack," pointing out the prevalent deceptive behavior in ChatGPT, Claude, and other LLMs.

The so-called "AI Poisoning Attack" refers to the intentional sabotage of AI training data or manipulation of AI algorithms. The attackers initiating the attack could be insiders, such as disgruntled current or former employees, or external hackers, with motivations including causing reputational and brand damage, undermining the credibility of AI decisions, slowing down or disrupting the AI process, and more. Attackers can distort the model's learning process by implanting data with misleading labels or features, causing the model to produce incorrect results during deployment and operation.

Looking at this incident, the reason ChatGPT provided phishing code to r_ocky.eth was most likely because the AI model was trained on data containing phishing content, but the AI seemed unable to recognize the phishing content hidden beneath regular data. After learning from this data, the AI then provided this phishing content to the user, leading to the occurrence of this incident.

With the rapid advancement and widespread adoption of AI, the threat of "poisoning attacks" has become increasingly significant. In this particular incident, although the absolute amount of the loss was not significant, the far-reaching implications of such risks are enough to raise concerns—imagine if this happened in other areas, such as AI-assisted driving...

In response to a question from a netizen, Yuan mentioned a potential measure to mitigate such risks, which is for ChatGPT to add some form of code review mechanism.

The victim r_ocky.eth also stated that they have contacted OpenAI about this matter, although they have not yet received a response, they hope that this incident can become an opportunity for OpenAI to take such risks seriously and propose potential solutions.

Original Article Link