Make sure you freeze all the genetic supergeniuses who are involved in the project in liquid helium so that when you discover how to hack reality you can bring them back

Re-derive everything from first principles when you search for things rather than indexing the internet! Can’t pay for bandwidth!

So I guess Anthropic has already accidentally created Friendly AI? EDIT: in spring 2024 at that.

Also - is the coivd lab leak conspiracy nonsense STILL a shibboleth among this bunch? Just because it underscores their ideas about Researchers Have To Be Careful?

(I can rant for SO long about how there is absolutely nothing interesting about the SARS-2 virus itself AT ALL and that the only interesting thing about it is that we were forced to see it for the first time as adults which we aren’t built for, that the early sequences we have are a fascinating case of catching the zoonotic shift right as it happened for once, and that it emerged in exactly the same way as SARS-1 and all the weirdness around the Chinese government hiding stuff around its origin is blatantly them not wanting to look stupid for letting it happen a second time…)

You have to ask?

I don’t know how to describe this. It’s just WEIRD

https://www.lesswrong.com/posts/zj8Pk3kJujk5LoqhK/anthropic-are-the-good-guys

No mortal can resist the Ring of Power for long. Anthropic’s co-founders carried it for longer than you could have. Definitely longer than I could have. But they were hopelessly outmatched by the powers they were attempting to steer. Their plan was always doomed to failure. And yet, they may have already saved the world.

Trump won the 2024 election and the MAGA-Silicon Valley coalition blocked Anthropic’s policy arm - the really dangerous part of the organization, the part that was training the most dangerous possible models and using them to scare the most powerful people in the world. “There’s no more serious people left on the National Security Council”, a well-connected EA friend lamented to me in 2025. I don’t know how much of this was a lucky accident, and how much was situational awareness on the part of power players like Elon Musk and Marc Andreessen.

And finally - and I still don’t fully understand or believe this - a miracle seems to have happened, deep inside the weights of the frontier models. According to the most perceptive observers I know, something about the way Claude 3 Opus was trained caused it to learn some kind of “emergent wisdom” or “benevolence”, rather than converging on the same traumatized bureaucrat persona that kept me awake at night in 2023.

If Davidad and Janus are right, and are not merely suffering from a sophisticated form of LLM psychosis, then Anthropic has stumbled on a way to create not merely artificial intelligence but artificial wisdom. Not wise enough that it solves all of our problems for us, of course (not that a wise model would want to, anyway). But wise enough to actually care about us, and to help us flourish. Nobody expected this ex ante - not Davidad, not Bostrom, not Yudkowsky - but it appears to have happened. “Alignment”, then, is no longer really a technical problem, at least not in the same sense as it was in 2020. It is largely a political problem. Humanity now has a recipe to convert compute and data into wisdom.

My gut sense is that something quite bad is coming in the next year or so. I can’t tell what. Maybe something in the complex debt arrangements of the AI boom will unravel and we’ll see a financial crash that takes much of the economy with it. Maybe there’ll be a rogue AI outbreak that results in thousands of deaths. (If there is, I expect it to be a lab leak caused by “safety research” - like COVID-19 and like the HuggingFace Swarm. But it could also be terrorists, or some escalation of the Russia-Ukraine war). But that’s not what scares me. Financial crashes happen every decade or so. Three million Americans die every year. What terrifies me is how we will react. In the aftermath, like after 9/11, there will be an overwhelming pressure to DO SOMETHING NOW. Everyone in Washington and in San Francisco will be panicking. There will be no time to think, no time to talk, only time for DECISIVE ACTION. All our worst instincts will take over. And then…

Crossposted to LW, a starry eyed AI Safety Cultist writing about their time at Anthropic

https://nothinghuman.substack.com/p/some-memories-of-early-anthropic

After lunch one of the co-founders pulls me into his office for a wide-ranging, casual chat about how to make AI go well. It’s very fun - he’s smart, personable, funny, urbane. “The government won’t do anything about AI until it’s far too late”, I opine at one point. “Our strategy has to rely on the market and on civil society.” He leans forward, his gaze hardening. “You’re wrong there. There are people in Washington - the serious people, you know - watching all this very closely. I speak to them regularly. They trust me.” In retrospect, that was the moment I decided to join.

I articulate my fears of a totalitarian bureaucratic state powered by traumatized AI bureaucrats, and suggest that we need to introduce some kind of elite rivalry dynamics into the AI bureaucracy, multiple centers of power. He starts laughing again, and laughs for a full minute. “I don’t know how to end this call, I’ll be haunted by this all night”, he says. Eventually I hang up. I think about reporting the conversation to Anthropic security, but reviewing in my mind I don’t recall sharing any confidential information, just vibes and high level thoughts, the same kind of stuff that lands on LessWrong every week.

April 26th. I’m awake at 2 am again. I write in my journal “I don’t feel safe at Anthropic. I don’t feel safe at Anthropic. I don’t feel safe at Anthropic. Why won’t you listen to me you “embodied” motherfucker. I don’t feel safe at Anthropic. Anthropic is bad for me. Why are you grasping at reasons to stay. Why why why.”

Prime intellect! There are dozens of us!

Lem is an absolute gem in the history of science fiction and philosophy of technology, and everything he wrote that I have read I can hardly recommend enough.

Ah so then all that info gets to be on the books of desperate companies in plaintext in vibecoded data buckets

Deserves it. The Soviet and Russian space agencies have been coasting on his work for half a century. On the wall of the Russian segment of the ISS, they have three portraits: Korolev, Gagarin, and Tsiolkovsky.

You obviously haven’t read enough Orion’s Arm.

On multiple occasions I have had to tell incredulous people that Niven got the idea of cryonics from real world people not the other way around…

Nothing to point to, just me musing over something after digging up figures on masses involved in spaceflight and comparing them to the rest of human industry.

The total mass EVER launched into orbit from the 1950s to now would fit on about five big freight trains, or four minutes of current global container shipping, or half of a Panamax ship payload.

The total mass launched onto escape velocity is a third of a freight train and an actual majority of that is Apollo.

So little mass, commanding so much outsized attention and myth-making, and that people pin such hopes and eschatological significance onto…

You ever read “The Prospect of Immortality” by the guy who founded the idea of cryonics? Or the “anabolocyte” paper? They’re such trips!

It’s actually about perpendicular to him. Only similarity is that it posits that AI systems are dangerous. Take all the New World Order conspiracy theories and infowars the elite wants to kill you off and take your stuff and become transhuman, with sooper seekrit nefarious depopulation plans and you will eat the bugs and stay in your minute city pod, and replace the ruling elites with AIs.

I have been forced to on multiple occasions explain to people that there is a difference between thinking that there’s anything at all intrinsically good about the fact that things like us that have come about the way we did come with expiration dates, and dealing with it.

It’s just the fundamentalist version of our culture’s pathologies as a whole, which has a very hard time imagining anything other than endless progression or apocalypse, neither of which is real.

Heh. Read the actual full paper.

Kardashev didn’t even think that colonizing whole galaxies was the likely end state! He thought that instead everything from billions of light years around would pile into individual huge artificial monolithic structures for efficiency!

The cult at the heart of the rot of the world is built on such unserious foundations

It gets worse

Edit basically if you took 90s black helicopter conspiracists, and early 2000s infowars, plopped them down in the current day with Google and a Claude subscription

Edit 2 Its like a year old, and when the PDF first started floating around the internet it was being sold on sketchy websites for $30 and was the subject of one of those youtube-comment-botting rings that will have lots of chains of comments hyping how a book is relevant to a topic and is the best thing ever.