Skip to content

SBF speaks for the first time about regretting the sale of Anthropic: There is a 25% chance AI will destroy humanity, and young people need to thoroughly learn AI.

Nov 21, 12:31
SBF speaks for the first time about regretting the sale of Anthropic: There is a 25% chance AI will destroy humanity, and young people need to thoroughly learn AI.

Editor's note: Anthropic went public with a valuation of $1.5 trillion and at one point reached a market cap of nearly $2 trillion. Everyone is recalling that SBF invested in Anthropic at a tiny valuation back in the day—if held until now, it would be worth close to $40 billion, but unfortunately it was already sold.


This 14-year-old kid Eli Goldfine reached out to SBF's mother and had a phone call with SBF in prison. This is the best interview I've seen so far since the FTX blowup and SBF's imprisonment—it covers not only SBF's retrospective on the Anthropic investment and his views on AI, but also many tracks that FTX was already working on back in 2020 that are now being hyped as hot trends, as well as his current views on these hot tracks.


TL;DR


· The probability he assigns to AI causing human extinction or civilizational destruction is 25%: he initially said "below 50%, above 10%," ultimately landing around 25%; if split by criteria such as "whether mind uploading counts as a catastrophe," the range widens to 20%–60%.


· In the next 5–10 years, AI will reach or exceed human level on the vast majority of important tasks, just a small step away from "AI running most of the world," but the process will be more like a fuzzy gradual progression—there won't be a single year where it "snaps" across the line.


· The impact lands industry by industry, crossing the line one sector at a time: software development and mathematics are already happening, and it will spread to more domains next; the world as a whole isn't overturned, but locally there will be dramatic upheaval.


· The real question isn't "whether to slow down" but "whether everyone can slow down together": as long as there is one player who completely doesn't care about safety, unilateral slowdown just turns the race into a race to the bottom.


· There is no single high-level answer to alignment—the devil is entirely in the details: massive testing, massive debugging, massive caution, and assuming the model will eventually be used as an agent.


· Anthropic is a trillion-dollar-tier, top-two AI company: over the next decade AI will become a core industry, with cumulative market cap reaching tens of trillions or even over a hundred trillion dollars, and the companies that can hold this scale can be counted on one hand.


· The reason for only investing $500 million back then was that it only needed that much money: this was the result of the company not wanting to sell more equity and raising capital on an as-needed basis, not a misjudgment on his part.


· The alpha now isn't in "who has an AI story" but in distinguishing who is a real AI company; moreover, brand defends the moat better than technology (Coinbase and Robinhood are worth tens of billions, and it's the name that does it).


· That "51% doubling, 49% destruction" coin flip—on an abstract level he would take the bet, but in the real world he absolutely would not: because destruction is permanent, while growth is something others can eventually achieve anyway.


· When asked whether "given all the good things Future Fund did, was the whole arc of FTX from rise to collapse worth it," his answer was "no": he would love to believe it was worth it, but he cannot honestly say it was; if it hadn't ultimately blown up, it would have been worth it—and he doesn't think people have learned the lesson, because in a crisis people are extremely prone to losing their judgment about what truly matters.


I guess you've also heard that Anthropic is going public this week, with a valuation expected to exceed $1 trillion. What do you think of its current valuation?


I think they're worth a great deal of money.


Look at the biggest industries, the ones that truly drive certain parts of our economy—their market caps are in the trillions to tens of trillions of dollars. There's no reason to think AI won't become one of those core industries in the next decade, or even the most dominant one among them.


So if you expect the total market cap of AI companies to eventually reach tens of trillions, or even over $100 trillion, how does that money get divided? Maybe you leave two-thirds for companies that don't exist yet. Even so, that still leaves at least $10 trillion for existing companies. And there simply aren't that many companies that can support that kind of scale. You have OpenAI, you have Anthropic, you have Google and its various labs, probably DeepMind, xAI. The companies that truly count as core players can be counted on one hand. That basically means the top two or three are each worth at least a few trillion dollars.


And I think Anthropic, even if not number one, is clearly in the top tier.


Why do you think it's a top AI company?


One thing is looking at the capabilities of current models and how good the current products are—and they and OpenAI have clearly been going back and forth on that. Anthropic's consumer-facing products have received a lot of positive feedback, and that may not even be their most distinctive product. Their AI software development tools are the best in the world, and that's another enormous field. So at least as of today, you have to count them among the top two.


If we really enter an all-out race toward superintelligence, which now seems quite likely, how do you think the next decade will roughly unfold? What will the world look like?


Good question. Let's start with the next few years—absurd things could pop up at any time. This is just a median-case projection from me, and it could deviate at any moment.


But what we've seen so far is: AI had almost no impact on the world for a long time, even as its capabilities doubled again and again, because it was still in the "worse than humans" stage—and worse to the point that using humans to do things was still more cost-effective than using AI. Then, just in the last few years, it started crossing that line in more and more places and began to have an impact on the world.


It hasn't completely transformed the world yet. But as key industries successively hit that tipping point—where AI suddenly becomes far more productive than humans in a given field—you'll see fairly rapid explosions in those areas. So in the next year or two, I think a few more fields, possibly quite important ones, will go down the path of math and software development: most of the cutting-edge work will actually be done by AI models. The whole world won't be upended, but you'll start to see significant changes in certain places.


There are many problems. Let me step back. One is: could something go chaotic or catastrophically wrong? The most obvious example is an out-of-control AI getting access to weapons—say, an AI system designed to detect threats to a nuclear power, it goes rogue, and launches a thousand ballistic missiles in one go. That's one scenario. I think the probability of it happening in the next decade is low, but not zero—it could happen.


Setting doomsday scenarios aside for now, I think in the next decade we may find ourselves in a world where, suddenly, humans are no longer needed in most professions. Even if the world doesn't change that much in the short term, this would fundamentally distort society's culture. The impact would be enormous if we no longer define ourselves by work. Also, what happens when the output of certain industries becomes increasingly difficult for humans to understand? Does that count as AI quietly completing a takeover? And how would those AIs view humans?


I don't know if these questions will have answers within the next decade. But I think it's very likely they'll all come rushing out, or at least come out one after another, and in many respects, humanity's status as the central decider of its own future will already be written on the wall by then.


You had these ideas well before the mainstream. Back in 2021, before ChatGPT became a mainstream product, how did you see the AI curve playing out?


A lot of the credit here doesn't belong to me. Many important things were written by others—I was largely just emphasizing or restating what they said.


But a few threads. One is: look at what AI had accomplished by around 2020. The answer is, in most fields, nothing. In most fields, you could try to use AI, and it would be kind of dumb—impressively better than a decade ago, but still completely useless. But in a handful of things—things it was specifically trained on—it had already surpassed humans, like most board games, chess or Go; more complex areas like video games were starting to emerge too.


If AI only ever got good at chess, you wouldn't make much money, wouldn't upend the social order, and wouldn't change the future much. But there's no reason to think chess is all AI can do. It has been growing exponentially from a tiny starting point, and it doesn't need to double many more times before it starts surpassing humans in key areas of society.


So my question is: if you believe it's an exponential growth curve, why did you only invest $500 million in Anthropic back then?


Good question. The answer is basically: when you're founding a company and raising money, you usually don't want to raise an infinite amount. You have financial projections, you roughly know how much money you need to get the business going, and you don't want to sell more equity than that.


Anthropic was in that position at the time. It calculated it needed roughly $500 million in compute spending to get the model built, so that's what it raised. We basically invested all the money they were raising in that round.


But can't data center capacity be built anytime you spend money? If they raised more money, why couldn't they buy more GPUs, build more data centers, and push the model far beyond the level of other frontier labs at the time?


Maybe they should have. I guess part of the reason is they didn't want to sell more equity than necessary. But another part is that many companies are actually dragged down by their own success, by their own cash flow and cash on hand. Historically, many companies raised a big chunk of money and then blew it all on advertising because they felt they had to do something with it, and in the end got nothing in return—just stagnated.


So I guess they were also thinking at the time: how much money do we actually need to spend over the next few years, and try not to put the cart before the horse.


Can you talk about how FTX's venture arm operated at the time? Who was running it? What was your investment thesis?


The people involved rotated in batches, and later we hired directly to professionalize it. At first it was just me or others at the company making one-off investments as opportunities came up—and honestly, Anthropic was one of those.


But then we started doing very well. A lot of the early investments were crypto asset investments, and honestly they were basically doubling every year, even against the backdrop of the entire crypto market, so we decided to try to scale it up. We brought in Ramnik, Amy, and others to focus specifically on that.


Overall I was involved at a fairly high level across the whole portfolio. And for certain specific deals—Anthropic was obviously one, Solana was another—I would get more deeply involved, at least following on a weekly basis, especially when those investments interacted with Alameda and FTX, or when there were ways to help them.


Based on the information you can get in prison, where is the market alpha now? What is your current investment judgment?


I wish I had a better answer. Four years ago, I did have pretty good ideas about what to invest in. But the real answer to what to invest in today is: I shouldn't listen to my own advice. I need to do due diligence on the outside myself before I can give a decent answer.


Broadly speaking, I guess there will be many companies built on top of AI — small two-person teams that become $100 million or even $1 billion companies within a few years, and you can invest at much lower valuations. But the problem is that a lot of garbage will emerge at the same time. Now every company will stuff an AI line into their BP, even if it makes no sense — like a T-shirt company saying "we're going to use AI to design T-shirts." Fine, maybe it helps a bit, but they're repackaging themselves as essentially an AI company when essentially they're not.


So a big part of the alpha right now is distinguishing between these two: which companies are truly AI companies, which can truly achieve explosive growth through AI, and which are not — at least not yet. You can see it in the large-cap stocks too. Many early market participants actually didn't know how to distinguish Amazon, Google, Apple, and Netflix — they're all tech companies, so does that mean AI will affect them all the same way? The answer is of course not. A few of them are deeply tied to AI growth — Nvidia is the most obvious example; while others, like Apple, are indeed tech companies, but what has defined Apple's success over the past thirty years is essentially outstanding consumer product design: beautiful hardware, easy-to-use software, plus a powerful brand. At least as far as tech companies go, that's not AI-centric.


So are companies that aren't truly AI companies — like traditional software companies — doomed? Or is it that only companies that adopt AI can survive?


I don't think only AI companies can succeed. First, in some industries AI won't have a major impact for a long time. Take steel mining — the world still needs steel, and now AI has begun to enter physical mining, so it has suddenly become relevant overnight.


But another point I think is crucial is brand. You might have a software company with products and a sizable market cap, and then a competitor says: we can make almost exactly the same thing very cheaply. True, but that doesn't mean people can buy it. Having a brand standing there is extremely, extremely valuable.


Look at companies like Coinbase or Robinhood. What is their real differentiation? Why are they each worth tens of billions of dollars? It's not because of technology, and it's not because they have something unique in this space. Maybe it's regulation, but with them it's actually not entirely that either. It's brand — a name that American retail investors recognize and will use. That can be worth a lot of money, and for many companies, even if the core product becomes very easily replicable, that kind of thing will persist.


You said you typically spend very little time on large financial decisions relative to the size of the investment. How does that play out in venture capital, for example when a team brings you a project and you have to decide whether to approve it?


A big part of it is: if you're going to hire someone to do something, the premise is that you intend to trust them to do it well. If someone on the team has already spent 15 hours doing research and due diligence, that's fine—I'll look it over, add what I can add, and give them resources they don't have that I do.


But beyond that, if I'm not going to trust that they've made the right judgment and seen it correctly, then I'd have to spend those 15 hours myself redoing the due diligence. At that point, hiring them in the first place makes no sense.


You said the hardest part of being in prison was not being able to support the effective altruism movement, not being able to "earn to give." What do you plan to do now that you're out? Especially since many people in the movement have now distanced themselves from FTX.


Part of the answer will have to wait until that day comes, because there's a lot of information I don't have right now. The world is changing, and some information is much easier to get on the outside than on the inside.


But I will very seriously consider returning to "earn to give," supporting the movement, and supporting the projects I believe are doing good. That will probably be a default path.


Beyond that, other paths depend on what role I can still play in society. In the past I was positioned to have influence in many areas, but all of that will be much harder now because my reputation is extremely negative. So in terms of public discourse, politics, media—I'm not sure what role I can play in the future, maybe none at all. But "earn to give" is a default path.


So specifically, how will you earn? Back to crypto?


I will very seriously consider it. It depends on when I get out and how I get out. But crypto I will definitely consider, AI I will also consider, and finance more broadly as well.


You gave a lot of money to FTX Future Fund, and not simply by donating to established EA charities—you were looking for highly volatile projects with enormous upside if they succeeded. Can you review which were the best projects in Future Fund? On the whole, was it more effective than giving money to GiveWell's top charity funds?


I think it was indeed more effective.


A lot of what we did was "regranting": giving a sum of money to a group of people and letting them decide which small projects to fund, crowdsourcing opportunities that would otherwise have fallen outside our radar. There are many people with enormous potential in AI safety, biosecurity, politics, and other fields, but they need funding—maybe not much, perhaps just enough to finish college or to start a company. We were very happy to provide that kind of money.


Many of these projects look small individually, but are highly leveraged; taken together, the impact is significant. And we are happy to fund people who are just starting out: they have a philanthropic idea, or some other tool that benefits the world, which may or may not work, and they need money to give it a try. Our situation is that even if some of them ultimately go nowhere, we don't mind; we don't depend on any single project for a sense of fulfillment, or to prove that we have done good.


Ultimately I think this is a fairly effective approach, partly because at the time it was hard to find mature projects that could absorb tens of millions to hundreds of millions of dollars and still spend it efficiently. An organization with an annual budget of $8 million—if you suddenly give it $30 million more, it may have no idea what to do with the money.


Are there any particularly outstanding examples?


Investing in Anthropic was strictly speaking before the fund was established, and part of it was out of AI safety considerations. Setting that aside, we have funded a batch of biosecurity projects, some of which later developed new therapies for infectious diseases—the kind of diseases that kill many people globally—and some organizations are clearing the path to bring such therapies to market faster.


I think there are many things that would have had enormous potential impact but were halted when a crisis hit, and I would count quite a few political projects among them.


You said investing in Anthropic was not just for high returns, but also had a public-interest dimension of AI safety. Do you really think that if Anthropic leads, the AGI or race toward superintelligence will be much safer? Why?


Of course no one can be certain. But my guess is: as long as there is one company that particularly cares about safety, the development approach it brings will be more cautious than it would be without that company. That is not all you can hope for, because as long as there is still a participant that completely does not care about world safety, there will be a race, and possibly a race to the bottom, which would be very bad.


But having one more safety-conscious participant helps.


How does this manifest in practice? What specifically do they do to lower p(doom) (the term means the probability that AI causes human extinction or civilizational destruction)?


They can answer this question better than I can. But broadly speaking, part of it is slowing down the most frontier development and releases when models begin to have the potential to go out of control.


You can look at the Hugging Face hacking incident; the ultimate damage it caused was not large, but with a different target it could have been extremely destructive, and it was not far from a global catastrophe, from the perspective of "what an AI agent actually needs to do." So it is very important to make sure models cannot go out of control like that, to set as many checkpoints as possible, to take the model's ultimate goals, guiding principles, and alignment as seriously as possible, while maintaining real-time responsiveness. None of this is perfect, but it helps.


There's an argument: even if Anthropic believes the safest course is to slow down, it can't slow down, because China, or other companies that care less about safety, will get ahead, so it's actually not feasible. What do you think?


This is a real question. Perhaps the best outcome for the world is for everyone to slow down, but if others don't slow down, that doesn't necessarily mean that slowing down yourself is right, that it's rushing into a race to the bottom. Perhaps having one more safety-conscious competitor in this race is better, but this is obviously not the ideal situation—it's just that we may happen to be living in this situation.


The truly best approach is to influence other participants, to get all major players to reach a cross-camp consensus as much as possible, to all place greater emphasis on safety.


You are overall supportive of the second Trump administration. So many people in the Trump administration, including Trump himself, are very anti-effective altruism. What do you think?


I think they are responding to a specific event, and I don't know the details clearly, so I can't speak as precisely as I would like. Beyond that, I think the Biden administration at the time was very...


But don't you think that if the government is discouraging people from donating to effective charities, the net effect is extremely negative? Some people will stop being effective altruists because of this kind of public opinion.


That would be negative.


Compute is still expanding, and Anthropic's IPO this round will raise a lot of money. So, when will ASI arrive? What's your timeline?


I'm not sure it will be a black-and-white, discrete event. It could be, but it's equally possible that over the next five years we'll watch parts of society get swallowed by AI piece by piece. Software development is already happening, mathematical research as well, and chess and Go happened even earlier. This trend will continue until more and more parts of society are run by AI, with the degree of human oversight continuously declining over time.


Most people wouldn't consider current AI to already be "superhuman and comprehensively surpassing humans in a general sense," i.e., better than humans at all tasks. By this definition, do you think we're close to ASI? Will it happen?


I think it's already quite general. You can ask the top models almost any question and get an answer at least comparable to a smart person. What we see less of is AI breaking through to levels that humans fundamentally cannot break through.


As I understand it, the current frontier level is roughly equivalent to what a very smart, very rational person could ultimately achieve given plenty of time—except it's instantaneous, right at your fingertips. Whether that counts as superhuman, I don't know. I think it's a bit better than humans, but not much better, except that it can run many instances simultaneously and think much faster than humans.


So it may not necessarily be a clear line from human to superhuman. At least so far, what we've seen is a blurry line: it goes from "less than human" slowly—slowly, though actually just a matter of months or years—to an IQ of 100. If this trend continues, there may not be a moment where it obviously "snaps" across. But there could also be a year when it clearly leaps from human level to clearly above human.


You've never given a number for p(doom), the probability of a catastrophic event in which AI causes the extinction of most of humanity. Can you give a number?


I'd say below 50%, above 10%, roughly around 25%.


Let me clarify: the model in my head—I wouldn't say it's the best model—a large part of that probability comes from a complex scenario people will disagree strongly on: if in the future humans start uploading their brains, and those who don't upload get left behind, does that count as a catastrophic event? Different people have different answers. Depending on the answer, I could give any number between 20% and 60%.


What's your reasoning for the 20% to 60% range?


Start with the baseline: in the next 5 to 10 years, or even sooner, AI will likely be as good as, or better than, humans at most tasks—and it already is in many domains. Then it will get better, maybe much better, maybe only a little better, at not all but the vast majority of tasks that matter to the world.


Once it reaches that point, it's only a small step away from AI running most of the world and holding most of the power. The probability that a more powerful species would go a hundred years without ever using that power seems to me quite low. So I lean toward thinking that within the next decade we'll see AI at least "soft take over" a large part of the world's operations and infrastructure.


At that point, one possibility is peaceful coexistence, with humanity not undergoing massive change—but that doesn't look like the most likely guess. I think it's possible that humanity, with AI's help, grows into something beyond what we are now. It's also possible that AI simply gets bored with humans and wipes us out. There's also the possibility of AI-led but human-goal-oriented collaborative work bringing about massive technological expansion. My 20% to 60% depends on which of these scenarios you count as "doom."


If there were a superintelligent AI that could maximize the expected utility of your every decision, would you hand over decision-making authority to it?


If I were 100% sure it was doing this, not just sure that was its goal, but sure it wasn't stuck in some destructive failure mode, without a bunch of messy approximations, then, under all those assumptions, the answer is yes.


You mentioned a potential doomsday scenario related to "mind uploading." Would you be willing to upload yourself?


Probably. Again, it depends on the details, but I personally have no objection to it.


After uploading, do you still consider yourself the same person? How does that change your worldview?


Good question. I think the concept of "personhood" needs to be expanded. It's already a fact that no one is exactly the same person as they were yesterday. Some parts of us die, some parts grow; memories change, chemistry changes, bodies change, neurochemistry changes. But usually the differences from day to day are small enough that "the same person" is a good approximation.


And current AI is obviously not like that. I think that's the model we should refer to: a Claude Sonnet and a Claude Opus, are they the same person? Neither "yes" nor "no" is a good answer. I think if there were an "uploaded person," the situation would be the same.


When was the last time you could try an LLM? What was your impression?


I can have people copy and paste model outputs in here. My impression is that they're basically a meticulous, conscientious research assistant, except you can think as fast as you want, and a day's work can be done in a minute—that's very remarkable and very valuable.


But my current impression is that most of what they do, humans can also do, it just takes humans time. And you need reliable people guiding them, access to information, and the whole set of things that current models are already pre-connected to.


At FTX you launched some very interesting and innovative financial products. Brett Harrison said that one of your original visions for FTX was political prediction markets. Can you talk about that?


FTX started as a crypto exchange, doing Bitcoin, Ethereum, and their futures. But fundamentally, prediction markets are just futures contracts. From a financial perspective, both are just markets that settle to some value.


So from the very beginning we wanted to launch prediction markets, and we tested the waters on FTX International. We had a fairly active market for the 2020 presidential election—I remember the Trump vs. Biden market had volume of several hundred million dollars. This was something we wanted to grow long-term, but I couldn't grow it until we got regulatory clarity from the CFTC.


I don't know if you're aware, but prediction markets have been really hot lately—Kalshi is now raising at a $65 billion valuation. Why wasn't FTX able to scale prediction markets to that size back then? Was it just too early, before it hit the mainstream?


I think a big part of it was regulatory. When we were doing it, we didn't have the CFTC license needed to launch it in the US. So our markets were only open to the international site, and Americans weren't allowed to participate. Later, Kalshi, Polymarket, and others got a more formal US regulatory foothold, and that was exactly when they took off.


Polymarket also grew as an offshore exchange—especially during the 2024 election, a lot of US users got in via VPN.


Polymarket wasn't doing KYC back then, right? I don't think so. So it's very likely a large number of Americans came in via VPN like you said. Whereas we, as a centralized exchange, had to do KYC for all users—beyond IP checks, we required IDs, passports, and proof of address from outside the US.


Besides prediction markets, you also listed some derivatives that were rare at the time, like tokenized stocks. Those markets are much bigger now, but they were tiny on FTX. What products do you think could become huge markets today but just haven't hit the mainstream yet?


I think tokenized stocks still have a lot of room. They've definitely grown since 2021 and 2022, but fundamentally, their infrastructure is much better than the real stock market, so there's no reason they couldn't become the primary venue for securities trading.


Beyond that, prediction markets can cover a very wide range. I think political prediction markets haven't grown as much as I expected—part of that is still a regulatory issue. And stepping back further, margin trading, futures, and derivatives for US customers—that's another thing that basically doesn't exist, and that's exactly the form most international volume takes. That's also a regulatory barrier.


So I think there's still potential there. Of course, there will definitely be plenty of things I didn't anticipate that will emerge and surprise most people.


FTX also listed pre-IPO futures with contracts at different maturities, letting people bet on valuations at different points in time. Now a lot of crypto exchanges are doing perpetual contracts on these valuations. Why? My personal view is that a futures structure with maturities is actually better.


Well, customers like perpetuals. One thing is clear: regardless of whether it makes sense as an underlying financial product, "not having to deal with expiry" is convenient in itself. The actual mechanics are quite complex from a funding rate perspective, but I think the reason is basically that—people don't like worrying about expiry.


So they don't hate paying funding rates?


They don't really care that much. A lot of people just ignore it. Some people specifically take the favorable side to collect funding rates. I thought that would be a big financial disincentive for people, but it wasn't.


Why do you think that is?


Again, convenience is a huge factor. And I think a lot of people honestly just don't think about it and don't pay much attention to how much interest they're paying.


Then beyond that, liquidity attracts liquidity. Once those markets become consensus, customers are there, market makers are there, liquidity is there, big clients go there. You get a virtuous cycle, a feedback loop, and everyone converges on whichever market gained momentum first.


You once said something quite famous: "I'm very skeptical of books... I don't think there's any book worth reading, but I do believe it's roughly like this. I think if you wrote a book, you screwed up—it should have been a six-paragraph blog post." Now you're in prison, and reading is one of the few things you can do. How do you feel about reading now?


My view hasn't changed much. I'm reading a lot of fiction, a lot of sci-fi, it's okay, but honestly, I'd rather watch Netflix.


Is there any book that made you feel it was worth the time even after you get out?


Honestly, no. There are things I like, but the best sci-fi I've read—maybe I'd trade it for a mediocre Netflix show, but only maybe, and only barely. I might end up liking reading a bit more sci-fi than I expected, but not by much.


What exactly makes Netflix better than books? Watching shows also takes a lot of time.


It takes time. But part of it is an information throughput issue. Books are only visual, and only the visual of text; shows have sound, and they're visual too, and visuals can contain more than text—you can display arbitrary pixels. So from an information-theoretic perspective, you can do so much more, and I think that helps a lot.


The other part is that books have something shows usually don't: narration of characters' thoughts. I think that can be done away with; it's often redundant, the author's perspective indulging itself. And right now money and talent are flowing to shows, so that's naturally where the most development will happen.


Suppose there's a bet: a 51% chance of doubling Earth's utility somewhere else, and a 49% chance of everything going to zero. Would you take it? If you win, another round, double or nothing—would you keep playing? You told Tyler Cowen you'd accept a bet with "a 49% chance of destroying the world and a 51% chance of doubling utility," provided the universes are independent. Can you explain your reasoning? Has your view changed after FTX collapsed?


For that scenario to hold, many preconditions are needed. You have to assume the universes are independent, as you said; assume this is a one-time opportunity, and the future cannot replicate it at a lower cost; and many other unspoken assumptions.


But one way to simplify it is this: two worlds, roughly equal in population, with comparable expectations for the future, and completely independent of each other, never interacting, outside each other's light cones, whatever you want to call it. Then somehow, some strange alien god descends and says: each of these two worlds has a 50% chance of being destroyed; and for some reason, you believe this alien god is telling the truth. It says you can choose to make them correlated or anti-correlated. Option one: either both live or both are destroyed, 50% each. Option two: one lives and one is destroyed, randomly, and you don't know which is which.


In that scenario, I think it's clear these two choices are equivalent. And if one side offers you 51-to-49 odds, you should choose that over "live or die together," because each world doesn't care what happens to the other—they will never interact anyway. And this is almost exactly the original question: you have to choose between "exactly one world survives" and "a 50% chance both worlds survive."


Its expected value is positive, but not by much. But if you keep flipping, ten flips is over a thousand times. As the number of successes increases, will marginal utility diminish to the point where the expected value basically breaks even, making you less inclined to play this game?


This is an interesting question, and it depends on what exactly these "doublings" are doubling. You talk about increasing marginal utility and diminishing marginal value—utility of what?


If this refers to the long-term accumulated utility of the entire universe, then no—utility is linear in itself. But if it's actually some approximate metric, like if you're looking at GDP and asking "can GDP double," then there very likely is deep diminishing marginal return.


First, a doubling of GDP does not mean everyone's happiness doubles, and even less does it mean the world's long-term expected value doubles. That GDP doubling might have happened anyway thirty years later; or it might cause severe environmental damage and eventually collapse back. So in the real world, the metrics you use to measure may not be what you actually care about.


Is there any metric that can increase tenfold or a hundredfold without diminishing marginal returns? What would that look like?


There is one, but I have to add a caveat: I'm not sure which of these can really be achieved in the near term—all of these require technological progress. Colonizing other planets, colonizing the solar system, the galaxy—eventually you would truly have independent worlds, and you would just be enabling more human flourishing. That's one example.


What if we think differently: the population stays constant, forever maintained at one billion people—can their happiness continuously double? I think the answer is, we don't really know. This is almost a biological question, rather than anything else. Making people content and moderately happy is one thing; making them ecstatic is hard, and imagining what could make people twice or ten times as ecstatic is even harder—that's simply outside the range of emotions humans can typically experience. In theory it might be possible, but we have no good template, nor the technology or biology to achieve it.


Do you think the current development of AI is the real-world version of that coin toss? If you had to choose between: A, slowing down AI development to reduce p(doom) from 10% to 2%; B, continuing to sprint, accepting a higher p(doom) in exchange for greater upside. Which would you choose?


I think a clearer analogy is choosing between "continuing to sprint" and "permanently cutting off all AI or AI-like development." Because then you're basically choosing between "the world faces severe risks" and "limiting the world's capacity for prosperity," and the latter would very likely ultimately require some form of automation or digital intelligence. So in that case, permanently sealing off this path would be extremely costly for the future.


But if it's just "faster or slower," it's less clear, because there are two models you can use. One is the "perpetual exponential growth" model: once acceleration begins, utility doubles every year, endlessly. Under this model, starting one year earlier means permanently doubling the universe, but that's a bit illusory—it can't go on forever.


The other is to see it as a singularity-type event, after which some new plateau is reached, highly unknown. After the event happens, the universe may still have billions of years, so delaying it by one year, ten years, or even a hundred years, under this model, wouldn't reduce future utility by even 1% by much. Under this model, you'd be entirely willing to delay development by a hundred years to reduce p(doom) by 1%.


Of course, the real answer is probably somewhere in between. And I think "what the utility-over-time curve looks like in an AI explosion scenario" is a very interesting question.


Many people try to summarize your worldview based on your answer to that question: "always choose the bet with the highest expected value." Is that a fair summary?


That's part of it. But the first part of my answer is: there are many preconditions. You have to assume a lot of things about the situation to arrive at that answer, and those assumptions often don't hold in real life. In reality, all options are much messier than they're presented. And in the real world, accidentally destroying something is much easier than building something.


Let's use an example: Suppose there's a coin — 51% chance you get out of prison tomorrow, 49% chance your remaining sentence doubles. Would you take that bet?


Absolutely.


Because my sentence is life without parole. Realistically, my goal isn't to guarantee I get out at 60 and find a cabin to spend a few years in — that's not what I expect from life. I want to have a positive impact on the world, and that requires me to be out there. There's a high premium on that. If the cost is losing the final years of my life when I'm no longer productive, that's a price I'm completely willing to pay.


Can you give an example where, because of these premises, you actually wouldn't take the bet?


Yes. There's a scenario in reality that looks like a coin flip: heads, your model shows it could double the world's growth; tails, it could destroy the world. You absolutely must not take that bet.


Because in the real world, first of all, the "double the world's growth" side of that 50% — you're very likely building something others would have built sooner or later anyway. It's probably not a true, permanent doubling of world value. But if the world gets destroyed, that's permanent — gone forever.


So I think in real-world scenarios, once you add uncertainty and noise into every link of the chain, the weight of catastrophic destruction becomes far heavier, while moderate growth is still good but nowhere near enough to offset it.


Here's another coin-flip example. You've said you're completely non-religious. But Pascal's Wager is always there: Is believing in God positive expected value? If there really is an afterlife, not believing means hell, and the cost of believing today is limited — isn't that another example?


Same logic — if this logic has nothing to do with the specific content of the religion you're examining, then it applies equally to the "Lamp Post Religion": if you spend your entire life worshipping a lamp post, you go to heaven; otherwise, hell. Does that mean you should dedicate your life to worshipping a lamp post?


By the same logic, it applies equally to the "Anti-Lamp Post Religion": worshipping the lamp post sends you to hell, desecrating it sends you to heaven. Does that mean you should spend your life opposing lamp posts? The same logic gives you two completely opposite answers.


The answer is: for this world, you can imagine infinitely many bizarre hypotheses and possibilities, and you cannot assign significant positive probability to infinitely many unreasonable possibilities.


Another expected-value question related to effective altruism: Many EAs prioritize AI safety above all else, because it's positive expected value — we spend money, and maybe we can prevent human extinction, because we can align AI. Does this reasoning hold?


Partially true. The reasoning itself is absolutely correct: this is likely the decisive decade for AI, its potential impact is world-changing, possibly extremely good, possibly catastrophic, and that would drown out anything that will never change the world, or that you can't change the world by doing anything now. So it's a very compelling argument.


But what it doesn't answer is: what you can actually do. I think a lot of people end up spinning in circles: concluding that AI safety is the cause they should dedicate themselves to, yet unable to find anything that in practice actually makes AI safer. That means offering insight into "what really counts, what's really useful" is very valuable and very important.


You can look at those recent reports about Hugging Face being hacked, who knows, but it looks like those reports are quite beneficial to the world in expected value terms, and that's a great thing. It's just that finding that kind of clearly positive great thing isn't easy, and there's a lot of debate right now: whether EA's various actions are net positive or negative for AI safety.


Do you know what you would do to align AGI? What do you think is our best bet as a civilization right now?


My honest guess is that the devil is all in the details. Can there be a single, clear high-level answer that any company can follow and almost guarantee it proceeds safely? I think it would be a combination of a huge number of decisions, all stemming from the same mindset: caring deeply about the alignment, goals, and motivations of the models you create, and assuming they will be used in an agentic way, even if that wasn't the intent at design time; along with needing lots of testing, lots of debugging, lots of caution.


We all know your claim that you did a lot of good with the FTX Future Fund. Do you think the whole arc of FTX from rise to collapse, even with that ending, was worth it because of the good the Future Fund did?


No. I'd love to believe it was worth it, but I can't honestly say it was. It should have been worth it. If I hadn't screwed up in the end, I think it would have been worth it. If people had learned the right lessons from it, maybe it would have been worth it, but I don't think people did.


So what lessons should people have learned?


Several intertwined ones. A lot of them are about "what happens when things blow up," when you're in crisis mode. That's when it's easy to get bad advice. A lot of people quickly lose their judgment about what really matters. And things quickly get distorted through the lens of "who's the winner." It's really hard to step back and see clearly what's actually happening.


If the singularity really is approaching, we are on the threshold of superhuman AI. If you were me, 14 years old today, how would you proceed to create the greatest positive impact?


I think the first thing is to learn AI as thoroughly as possible, to understand it as deeply as you can, so that you can participate in this conversation, whether by building tools or by approaching it directly from an AI safety angle. I think that is the most important thing right now.


Before AI, you probably would have said "earn to give"—get good at trading, make a lot of money, and then donate it. Has that changed now?


Because the time between now and AI takeoff is compressing. The time you have to first build something else has gotten shorter; at the same time, the usefulness of understanding AI itself has become enormous. So even if your goal is to start a company and use it to make money to give away, for most people right now I would say: find an area where AI can automate something that is genuinely useful and genuinely important, some area of our infrastructure—that is the direction I would look at.


Join the official Coincamps community:

X: https://x.com/coincamps

Telegram: https://t.me/coin_camps

Recommended