Chiming in to agree your prediction write-ups aren’t particularly good. Sure they spark discussion, but the whole forecasting/prediction game is one we’ve seen the rationalists play many times, and it is very easy to overlook or at least undercount your misses and over hype your successes.

In general… I think your predictions are too specific and too optimistic…

Every time I see a rationalist bring up the term “Moloch” I get a little angrier at Scott Alexander.

I use the term “inspiring” loosely.

Depends what you mean by “steelman”. If you take their definition at it’s word, then they fail to try all the time, just look at any of their attempts at understanding leftist writing or thought. Of course, it often actually means “entirely rebuild the opposing argument into something different” (because they don’t have a basic humanities education or don’t want to actually properly read leftist thought) and they can’t resist doing that!

Putting this into the current context of LLMs… Given how Eliezer still repeats the “diamondoid bacteria” line in his AI-doom scenarios, even multiple decades after Drexler has both been thoroughly debunked and slightly contributed to inspiring real science, I bet memes of LLM-AGI doom and utopia will last long after the LLM bubble pops.

Lesswronger notices all of the rationalist’s attempts at making an “aligned” AI company keep failing: https://www.lesswrong.com/posts/PBd7xPAh22y66rbme/anthropic-s-leading-researchers-acted-as-moderate

Notably, the author doesn’t realize Capitalism is the root problem in misaligning the incentives, and it takes a comment directly point it out for them to get as far as noticing as link to the cycle of enshittification.

I brought this up right when it came out: https://awful.systems/post/5244605/8335074

(Not demanding credit on keeping better up to date on hate-reading the EA forums, just sharing the previous discussion)

Highlights from the previous discussion… I had thought Thiel was entirely making up his own wacky theology (because it was a distinctly different flavor of insanity from the typical right-wing Fundamentalist/Evangelical), but actually there is a “theologian” (I use that term loosely) who developed, René Girard, who developed the theology he is describing.

I keep seeing this sort of thinking on /r/singularity, people who are sure LLMs will be great once they have memory/ground-truth factual knowledge/some other feature that in fact the promptfarmers have already tried (and failed) to add via fancier prompting (i.e. RAG) or fine-tuning and would require a massive reinvention of the entire paradigm to actually fix. That, or they describe what basically amounts to a reinvention of the concept of expert systems like Cyc.

And we don’t want to introduce all the complexities of solving disagreements on Wikipedia.

What they actually mean is they don’t want them to be solved in favor of the dgerad type of people… like (reviewing the expose on lesswrong)… demanding quality sources that aren’t HBD pseudoscience journals or right wing rags.

Given that the USA has refused more comprehensive gun laws or better funding of public mental health services even after many many school shootings, I think you are far too optimistic about the LLM induced mental health crisis actually leading to a ban or even just tighter liability on LLMs. My expectation is age verification plus giant disclaimers, and the crisis continuing. The inference cost will force the LLMs to be more obviously dumb and unable to keep track of context, and the lack of a technological moat will lead to LLM chatbots becoming commoditized, but I’m overall not optimistic.

The LLM induced skill gap will be a thing yes… I predict companies trying to address it in the most hamfisted and belittling way possible. Like, they keep using code interviews (that are close to useless at evaluating the actual skills the employee needs), but now they want you to do the code interview with spyware installed to make sure you aren’t using an LLM to help you.

I don’t have a guaranteed way to recreate the issue, so I’m not absolutely sure, but that seems to have fixed it!

It’s a good post. A few minor quibbles:

The “nonprofit” company OpenAI was launched under the cynical message of building a “safe” artificial intelligence that would “benefit” humanity.

I think at least some of the people at launch were true believers, but strong financial incentives and some cynics present at the start meant the true believers didn’t really have a chance, culminating in the board trying but failing to fire Sam Altman and him successfully leveraging the threat of taking everyone with him to Microsoft. It figures one of the rare times rationalists recognize and try to mitigate the harmful incentives of capitalism they fall vastly short. OTOH… if failing to convert to a for-profit company is a decisive moment in popping the GenAI bubble, then at least it was good for something?

These tools definitely have positive uses. I personally use them frequently for web searches, coding, and oblique strategies. I find them helpful.

I wish people didn’t feel the need to add all these disclaimers, or at least put a disclaimer on their disclaimer. It is a slightly better autocomplete for coding that also introduces massive security and maintainability problems if people entirely rely on it. It is a better web search only relative to the ad-money-motivated compromises Google has made. It also breaks the implicit social contract of web searches (web sites allow themselves to be crawled so that human traffic will ultimately come to them) which could have pretty far reaching impacts.

One of the things I liked and didn’t know about before

Ask Claude any basic question about biology and it will abort.

That is hilarious! Kind of overkill to be honest, I think they’ve really overrated how much it can help with a bioweapons attack compared to radicalizing and recruiting a few good PhD students and cracking open the textbooks. But I like the author’s overall point that this shut-it-down approach could be used for a variety of topics.

One of the comments gets it:

Safety team/product team have conflicting goals

LLMs aren’t actually smart enough to make delicate judgements, even with all the fine-tuning and RLHF they’ve thrown at them, so you’re left with over-censoring everything or having the safeties overridden with just a bit of prompt-hacking (and sometimes both problems with one model)/1

Having seen the problem a few more times I can elaborate more clearly now. All the icons get temporarily massive oversized while the text actually stays the same size. I’ve only had it occur once on my computer browser, but it reoccurs regularly on my safari mobile browser, with a refresh fixing it each time.

Had me in the first few paragraphs…not gonna lie.

Yeah, the first few paragraphs actually felt like they would serve as a defense of Hamas: Israel engineered a situation were any form of resistance against them would need to be violent and brutal so Hamas is justified even if it killed 5 people to save 1.

The more I think about his metaphor the more frustrated I get. Israel holds disproportionate power in this entire situation, if anyone is contriving no-win situations to win temporary PR victories it is Israel (Netanyahu’s trial is literally getting stalled out by the conflict).

Lots of woo and mysticism already has a veneer of stolen Quantum terminology. It’s too far from respectable to get the quasi-expert endorsement or easy VC money that LLM hype has gotten, but quantum hucksters fusing quantum computing nonsense with quantum mysticism can probably still con lots of people out of their money.

I like how Zitron does a good job of distinguishing firm overall predictions from specific scenarios (his chaos bets) which are plausible but far from certain. AI 2027 specifically conflated and confused those things in a way that gave it’s proponents more rhetorical room to hide and dodge.

I like how he doesn’t even bother debunking it point by point, he just slams the very premise of it and moves on.

system memory

System memory is just the marketing label for “having an LLM summarize a bunch of old conversations and shoving it into a hidden prompt”. I agree that using that term is sneer-worthy.