Lesswrong has some funny objections to AI: 2040. They buy the overall premise, sure, but even that much still leaves lots of gaps in the “scenario”. And they have even more galaxy brain takes… My favorite complaint: https://www.lesswrong.com/posts/pFzctpJBat95SrCyC/ai-2040-plan-a?commentId=54dZes9uNqDdbNGBu

tldr; AI: 2040 is leaving out all the other sci-fi singularity tech just around the corner like genetic augmentations and cybernetic brain implants!

That and treating ARR as a reliable indicator of revenue. Which isn’t as bad as the valuations, but is still pretty bad. Ed Zitron has explained all the ways they game ARR, and, more importantly, it doesn’t matter how much your revenue grows if you are spending 2 dollars for every dollar you make (or spending 20 dollars for every dollar they make, as seems to be the case with their subscription plans).

Maybe the boosters will shut-up once OpenAI and Anthropic finally run out of venture capital to burn on subsidizing subscriptions, but actually, judging by the way the AI 2027 authors are still claiming credit for being right, they will probably just look for someone else to blame for the high costs.

Sometimes they also glue a bunch of “tools” to help the model reason. The model can call by extruding tokens with the right syntax and then get back information from the “tool” shoved in its context! That way, the model can at least handle stuff like basic arithmetic correctly! Except only sometimes, because the models frequently screw up calling the tools correctly or skip using the tool or any number of other completely dumb mistakes. Oh, and if the tool connects the model to the internet (or any insecure source of text) in any way, shape, or form, congratulations, you’ve now got a massive security vulnerability!

And for anyone skimming the comments instead of reading the articles, that is just one of two people Apple’s lawsuit references, there is another high level OpenAI person with a completely different set of egregious corporate espionage actions!

From the headlines I had been assuming it was a normal tech company patent troll slap fight but no… from the 9to5mac article:

When interviewing Apple employees for jobs at OpenAI, Mr. Tan uses Apple’s confidential information to gain access to even more insider knowledge. He has used an Apple internal project codename to ask, “What’s the plan[?]” for an unannounced Apple product.

He has directed job candidates still working for Apple to bring “Actual parts” from Apple to their interviews for “show and tell” sessions in which he and his team at OpenAI can elicit still more Apple confidential information. These directions to bring Apple’s parts to OpenAI job interviews surprised at least one of the candidates, who commented that he “didn’t even know we could take those from the office.”

OpenAI has been instructing Apple employees to bring “CAD/design artifacts” and “prototypes” to their interviews and to divulge details about their work such as “subsystem and component selection,” the “tools or methodologies you use for system integration, such as CAD software, simulation tools,” and “Vendor selection and communication/collaboration with vendors.”

That is absolutely batshit. Like I’ve heard the interview process has been getting worse and worse, but “steal shit from your current company” is on another level. I wonder if he lead in with any plausible deniability to it or just straight up jumped to that. Also, I have a hard time imagining being willing to steal from a company just to get through an interview at another company. Not out of company loyalty, but because I wouldn’t trust the interviewers to actually pay-up with a cushy job and salary.

Isn’t UK’s usually problem the other way around, they use "Protecting the Children"^tm as an excuse to ban sensible things or take away privacy from adults? This would be a first, taking away something from Children to protect the adults.

Lesswronger is mad the current pipeline to sanewash and legitimize lesswrong forum posts into academic content takes too long and thinks about ways to accelerate it: https://www.lesswrong.com/posts/wn5jTrtKkhspshA4c/michaeldickens-s-shortform?commentId=gyJWhLjq5fPh9Bv3b

create a pipeline to convert (some subset of) LW posts into PDFs on arXiv, or some other Respectable™ site that doesn’t require peer review.

The benefit of putting your predictions 15-30 years in the future is that people forget them by the time the deadline comes around and you can pivot more easily and also claim credit for the few successful predictions while all your failures are forgotten. The AI: 2027 could learn from Kurzweils’ example, except apparently you can just keep claiming credit for being right even when your wrong prediction is only a few years old!

Yeah, they’ve internalized their own fiction and fantasies so heavily they forget when they are talking about something they made up and not something real. And in turn, they forget to explain their terms for normies and ridicule commenters that can’t keep up with their lore.

Like in the comments and discussion for AI: 2040 on lesswrong, in response to a heavily downvoted comment, they treated “neuralese” like an obviously real thing just around the corner that will solve chain of thought. Or assuming continual learning is obviously just one or two inventions away and not a fundamentally missing feature of LLMs that no one has a good solution to.

Also, holy capitalist realism Batman

In between this, their cap-and-trade plan for AI regulation (which is a very market-centered attempt at regulation), their insistent terminology of “citizens dividend” (because even UBI is apparently too left wing for them, much less scary communist words like collective ownership), and some other word choices… it is like they are so capitalists brain rotted that even imagining non-capitalist systems is harder for them than believing in the coming AGI God.

My favorite part was the section where they worked out the logistics of China hiding a data center inside of a mountain.

So far reality is tracking closer to AI 2027 than even we expected. (2027 was our modal year at time of publication, not our median.)

They aren’t even trying to hide the fact that their predictions have already failed but instead claiming victory.

They just don’t call it model collapse, and that fixes the problem.

Reminds me of OpenAI’s blog post explaining the goblins. It described textbook model collapse, but somehow avoided saying the actual words “model collapse” even once.

edit: Nevermind… The Epstein Hysteria

That is pretty bad… all the standard rationalist thick-headed idiocy and quibbling details and choice of gross example. But even so if that is their worst, they still aren’t in the worst quartile of rationalist.

here’s how two robot gods who can perfectly emulate each other might work things out

Just gotta solve the halting problem, no biggie for a superintelligence or Eliezer.

Well you see, the acausalrobotgod actually hates dealing with assholes tying themselves in knots trying to predict its predictions, so actually you really are better off with CDT if you don’t want to get 3^^^3 dust specks in your eyes for all eternity.

More seriously, once you start positing beings that can perfectly predict you and reward/punish various decision theories you can basically rationalize any decision theory.

The whole algorithm predicting your algorithm thing feels like it is basically asking for/claiming the existence of a solution to the halting problem? But yeah, moot point, because the FDT stuff doesn’t get as far as ever talking about how to compute an answer and always treats the magic hyper computation as a given.

Some lesswrongers are happy about this news, talking up how their P(doom) is lower and that sort of thing. Of course some of them are sinophobic and worried now China will take the lead, which is obviously worse.

More the later than the former… they are better than purely marketing focused stuff pushed out by the LLM companies, and if you dig through them and read between the lines you can occasionally sift out useful details. Like here is a pretty solid sneer digging through Mythos’s ‘system card’ and pointing out all the ways it contradicts the hype and press headlines Anthropic was pushing.

But even so they have some big problems…

• the benchmarks the system cards reference are kind of useless and heavily gamed
• LLM companies want to keep lots of details secret from competitors, so fundamentals like number of model weights or parameters or size/quality of the training data set or other training specs are deliberately left out
• lots of the stuff they reference is booster garbage and/or doomer crit-hype
• they tend to be long, wastefully so, imitating the length of academic papers without having the corresponding amount of depth or information
• despite their length and wordiness they also neglect basic practical usage advice that isn’t even proprietary (or at least would be bound to leak if you poke around with the model at all and thus not worth keeping secret in the first place). Like not even big picture stuff I mentioned in my second bullet, but really simple stuff…

I think this would be funny if a cool, normal guy said it.

The word foid is a very clear alarm bell. But yeah, without that word and a different framing and it could be a guy humbly acknowledging that women’s health has a lot of extra complications.