Skip Navigation

User banner
InitialsDiceBearhttps://github.com/dicebear/dicebearhttps://creativecommons.org/publicdomain/zero/1.0/„Initials” (https://github.com/dicebear/dicebear) by „DiceBear”, licensed under „CC0 1.0” (https://creativecommons.org/publicdomain/zero/1.0/)S
scruiser @ scruiser @awful.systems
Posts
3
Comments
243
Joined
2 yr. ago

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 14th September 2025

    Jump

  • The Oracle deal seemed absurd, but I didn't realize how absurd until I saw Ed's compilation of the numbers. Notably, it means even if OpenAI meets its projected revenue numbers (which are absurdly optimistic, like bigger than Netflix and Spotify and several other services combined) paying Oracle (along with everyone else it has promised to buy compute from) will put it net negative on revenue until 2030, meaning it has to raise even more money.

    I've been assuming Sam Altman has absolutely no real belief that LLMs would lead to AGI and has instead been cynically cashing in on the sci-fi hype, but OpenAI's choices don't make any long term sense if AGI isn't coming. The obvious explanation is that at this point he simply plans to grift and hype (while staying technically within the bounds of legality) to buy few years of personal enrichment. And to even ask what his "real beliefs" are gives him too much credit.

    Just to remind everyone: the market can stay irrational longer than you can stay solvent!

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 14th September 2025

    Jump

  • This feels like a symptom of liberals having a diluted incomplete understanding of what made past movements that utilized protest succeed or fail.

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 14th September 2025

    Jump

  • It is pretty good as a source for science fiction ideas. I mean, lots of their ideas originate from science fiction, but their original ideas would make fun fantasy sci-fi concepts. Like looking off their current front page... https://www.lesswrong.com/posts/WLFRkm3PhJ3Ty27QH/the-cats-are-on-to-something cat's deliberately latching on to humans as the most lazy way of advancing their own value across the future seems like a solid point of fantasy worldworldbuilding...

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 14th September 2025

    Jump

  • To add to blakestacey's answer, his fictional worldbuilding concept, dath ilan (which he treats like rigorous academic work to the point of citing it in tweets), uses prediction markets in basically everything, from setting government policy to healthcare plans to deciding what restaurant to eat at.

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 14th September 2025

    Jump

  • Every tweet in that thread is sneerable. Either from failing to understand the current scientific process, vastly overestimating how easily cutting edge can be turned into cleanly resolvable predictions, or assuming prediction markets are magic.

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 14th September 2025

    Jump

  • He's the one that used the phrase "silent gentle rape"? Yeah, he's at least as bad as the worst evo-psych pseudoscience misogyny posted on lesswrong, with the added twist he has a position in academia to lend him more legitimacy.

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 14th September 2025

    Jump

  • He had me in the first half, I thought he was calling out rationalist's problems (even if dishonestly disassociating himself from then). But then his recommended solution was prediction markets (a concept which rationalists have in fact been trying to play around with, albeit at a toy model level with fake money).

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 7th September 2025

    Jump

  • Chiming in to agree your prediction write-ups aren't particularly good. Sure they spark discussion, but the whole forecasting/prediction game is one we've seen the rationalists play many times, and it is very easy to overlook or at least undercount your misses and over hype your successes.

    In general... I think your predictions are too specific and too optimistic...

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 7th September 2025

    Jump

  • Every time I see a rationalist bring up the term "Moloch" I get a little angrier at Scott Alexander.

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 7th September 2025

    Jump

  • I use the term "inspiring" loosely.

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 7th September 2025

    Jump

  • Putting this into the current context of LLMs... Given how Eliezer still repeats the "diamondoid bacteria" line in his AI-doom scenarios, even multiple decades after Drexler has both been thoroughly debunked and slightly contributed to inspiring real science, I bet memes of LLM-AGI doom and utopia will last long after the LLM bubble pops.

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 7th September 2025

    Jump

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 31st August 2025 - awful.systems

    Jump

  • It's a good post. A few minor quibbles:

    The “nonprofit” company OpenAI was launched under the cynical message of building a “safe” artificial intelligence that would “benefit” humanity.

    I think at least some of the people at launch were true believers, but strong financial incentives and some cynics present at the start meant the true believers didn't really have a chance, culminating in the board trying but failing to fire Sam Altman and him successfully leveraging the threat of taking everyone with him to Microsoft. It figures one of the rare times rationalists recognize and try to mitigate the harmful incentives of capitalism they fall vastly short. OTOH... if failing to convert to a for-profit company is a decisive moment in popping the GenAI bubble, then at least it was good for something?

    These tools definitely have positive uses. I personally use them frequently for web searches, coding, and oblique strategies. I find them helpful.

    I wish people didn't feel the need to add all these disclaimers, or at least put a disclaimer on their disclaimer. It is a slightly better autocomplete for coding that also introduces massive security and maintainability problems if people entirely rely on it. It is a better web search only relative to the ad-money-motivated compromises Google has made. It also breaks the implicit social contract of web searches (web sites allow themselves to be crawled so that human traffic will ultimately come to them) which could have pretty far reaching impacts.

    One of the things I liked and didn't know about before

    Ask Claude any basic question about biology and it will abort.

    That is hilarious! Kind of overkill to be honest, I think they've really overrated how much it can help with a bioweapons attack compared to radicalizing and recruiting a few good PhD students and cracking open the textbooks. But I like the author's overall point that this shut-it-down approach could be used for a variety of topics.

    One of the comments gets it:

    Safety team/product team have conflicting goals

    LLMs aren't actually smart enough to make delicate judgements, even with all the fine-tuning and RLHF they've thrown at them, so you're left with over-censoring everything or having the safeties overridden with just a bit of prompt-hacking (and sometimes both problems with one model)/1

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 31st August 2025 - awful.systems

    Jump

  • Lots of woo and mysticism already has a veneer of stolen Quantum terminology. It's too far from respectable to get the quasi-expert endorsement or easy VC money that LLM hype has gotten, but quantum hucksters fusing quantum computing nonsense with quantum mysticism can probably still con lots of people out of their money.

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 31st August 2025 - awful.systems

    Jump

  • I like how Zitron does a good job of distinguishing firm overall predictions from specific scenarios (his chaos bets) which are plausible but far from certain. AI 2027 specifically conflated and confused those things in a way that gave it's proponents more rhetorical room to hide and dodge.

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 31st August 2025 - awful.systems

    Jump

  • I like how he doesn't even bother debunking it point by point, he just slams the very premise of it and moves on.

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 31st August 2025 - awful.systems

    Jump

  • system memory

    System memory is just the marketing label for "having an LLM summarize a bunch of old conversations and shoving it into a hidden prompt". I agree that using that term is sneer-worthy.

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 31st August 2025 - awful.systems

    Jump

  • I have three more examples of sapient marine mammals!

    • whales warning the team of an impending solar flare in Stargate Atlantis via echolocation induced hallucinations
    • the dolphins in hitchhiker’s guide to the galaxy
    • whales showing up to help in one book of Animorphs while they are morphed into dolphins

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 31st August 2025 - awful.systems

    Jump

  • I was thinking this also, like it's the perfect parody of several lesswrong and EA memes: overly concerned with animal suffering/sapience, overly concerned with IQ stats, openly admitting to no expertise or even relevant domain knowledge but driven to pontificate anyway, and inspired by existing science fiction... I think the last one explains it and it isn't a parody. As cinnasverses points out, Cetacean intelligence shows up occasionally in sci-fi. to add to the examples... sapient whales warning the team of an impending solar flare in Stargate Atlantis via echolocation induced hallucinations, the dolphins in hitchhiker's guide to the galaxy, and the whales showing up to help in one book of Animorphs.