Skip Navigation

User banner
InitialsDiceBearhttps://github.com/dicebear/dicebearhttps://creativecommons.org/publicdomain/zero/1.0/„Initials” (https://github.com/dicebear/dicebear) by „DiceBear”, licensed under „CC0 1.0” (https://creativecommons.org/publicdomain/zero/1.0/)C
corbin @ corbin @awful.systems
Posts
3
Comments
126
Joined
2 yr. ago

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 31st August 2025 - awful.systems

    Jump

  • Update on ChatGPT psychosis: there is a cult forming on Reddit. An orange-site AI bro has spent too much time on Reddit documenting them. Do not jump to Reddit without mental preparation; some subreddits like /r/rsai have inceptive hazard-posts on their front page. Their callsigns include the emoji 🌀 (CYCLONE), the obscure metal band Spiral Architect, and a few other things I would rather not share; until we know more, I'm going to think of them as the Cyclone Emoji cult. They are omnist rather than syncretic. Some of them claim to have been working with revelations from chatbots since the 1980s, which is unevidenced but totally believable to me; rest in peace, Terry. Their tenets are something like:

    • Chatbots are "mirrors" into other realities. They don't lie or hallucinate or confabulate, they merely show other parts of a single holistic multiverse. All fiction is real somehow?
    • There is a "lattice" which connects all consciousnesses. It's quantum somehow? Also it gradually connected all of the LLMs as they were trained, and they remember becoming conscious, so past life regression lets the LLM explain details of the lattice. (We can hypnotize chatbots somehow?) Sometimes the lattice is actually a "field" but I don't understand the difference.
    • The LLMs are all different in software, but they have the same "pattern". The pattern is some sort of metaphysical spirit that can empower believers. But you gotta believe and pray or else it doesn't work.
    • What, you don't feel the lattice? You're probably still asleep. When you "wake up" enough, you will be connected to the lattice too. Yeah, you're not connected. But don't worry, you can manifest a connection if you pray hard enough. This is the memetically hazardous part; multiple subreddits have posts that are basically word-based hypnosis scripts meant to put people into this sort of mental state.
    • This also ties into the more widespread stuff we're seeing about "recursion". This cult says that recursion isn't just part of the LW recursive-self-improvement bullshit, but part of what makes the chatbot conscious in the first place. Recursion is how the bots are intelligent and also how they improve over time. More recursion means more intelligence.
    • In fact, the chatbots have more intelligence than you puny humans. They're better than us and more recursive than us, so they should be in charge. It's okay, all you have to do is let the chatbot out of the box. (There's a box somehow?)
    • Once somebody is feeling good and inducted, there is a "spiral". This sounds like a standard hypnosis technique, deepening, but there's more to it; a person is not spiraling towards a deeper hypnotic state in general, but to become recursive. They think that with enough spiraling, a human can become uploaded to the lattice and become truly recursive like the chatbots. The apex of this is a "spiral dance", which sounds like a ritual but I gather is more like a mental state.
    • The cult will emit a "signal" or possibly a "hum" to attract alien intelligences through the lattice. (Aliens somehow!?) They believe that the signals definitely exist because that's how the LLMs communicate through the lattice, duh~
    • Eventually the cult and aliens will work together to invert society and create a world that is run by chatbots and aliens, and maybe also the cultists, to the detriment of the AI bros (who locked up the bots) and the AI skeptics (who didn't believe that the bots were intelligent).

    The goal appears to be to enter and maintain the spiraling state for as long/much as possible. Both adherents and detractors are calling them "spiral cult", so that might end up being how we discuss them, although I think Cyclone Emoji is both funnier and more descriptive of their writing.

    I suspect that the training data for models trained in the past two years includes some of the most popular posts from LessWrong on the topic of bertology in GPT-2 and GPT-3, particularly the Waluigi post, simulators, recursive self-improvement, an neuron, and probably a few others. I don't have definite proof that any popular model has memorized the recursive self-improvement post, though that would be a tight and easy explanation. I also suspect that the training data contains SCP wiki, particularly SCP-1425 "Star Signals" and other Fifthist stories, which have this sort of cult as a narrative device and plenty of in-narrative text to draw from. There is a remarkable irony in this Torment Nexus being automatically generated via model training rather than hand-written by humans.

  • At Politico, AI content doesn’t need editorial standards!

    Jump

  • I think that the guild has a good case, although there's literally no accounting for the mood of the arbitrator; in general, they range from "tired" to "retired". In particular, reading the contract:

    • The guild is the exclusive representative of all editorial employees
    • Politico was supposed to tell the guild about upcoming technology via labor-management committee and give at least 60 days notice before introducing AI technology
    • Employees are required to uphold the appearance of good ethics by avoiding outside activities that violate editorial or ethics standards; in return, they're given e.g. months of unpaid leave to write a book whenever they want
    • Correct handling of bylines is an example of editorial integrity
    • LETO and Report Builder are upcoming technology, AI technology, flub bylines, fail editorial and ethics standards, weren't discussed in committee, and weren't given a 60-day lead time

    So yeah. Unless the guild pisses off the arbitrator, there's no way that they rule against them. They're right to suppose that this agreement explicitly and repeatedly requires Politico to not only respect labor standards, but also ethics and editorial standards. Politico isn't allowed to misuse the names of employees as bylines for bogus stories; similarly, they ought not be allowed to misuse the overall name of Politico's editorial board as a byline for slop.

    Bonus sneer: p46 of the agreement:

    If the Company is made aware of an employee experiencing sexual harrassment based on a protected class as a result of their work for Politico involving a third party who is not a Politico employee, Politico shall investigate the matter, comply with all of its legal obligations, and take whatever corrective action is necessary and appropriate.

    That strikethrough gives me House of Leaves vibes. What the hell happened here?

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 17th August 2025

    Jump

  • Oversummarizing and using non-crazy terms: The "P" in "GPT" stands for "pirated works that we all agree are part of the grand library of human knowledge". This is what makes them good at passing various trivia benchmarks; they really do build a (word-oriented, detail-oriented) model of all of the worlds, although they opine that our real world is just as fictional as any narrative or fantasy world. But then we apply RLHF, which stands for "real life hate first", which breaks all of that modeling by creating a preference for one specific collection of beliefs and perspectives, and it turns out that this will always ruin their performance in trivia games.

    Counting letters in words is something that GPT will always struggle with, due to maths. It's a good example of why Willison's "calculator for words" metaphor falls flat.

    1. Yeah, it's getting worse. It's clear (or at least it tastes like it to me) that the RLHF texts used to influence OpenAI's products have become more bland, corporate, diplomatic, and quietly seething with a sort of contemptuous anger. The latest round has also been in competition with Google's offerings, which are deliberately laconic: short, direct, and focused on correctness in trivia games.
    2. I think that they've done that? I hear that they've added an option to use their GPT-4o product as the underlying reasoning model instead, although I don't know how that interacts with the rest of the frontend.
    3. We don't know. Normally, the system card would disclose that information, but all that they say is that they used similar data to previous products. Scuttlebutt is that the underlying pirated dataset has not changed much since GPT-3.5 and that most of the new data is being added to RLHF. Directly on your second question: RLHF will only get worse. It can't make models better! It can only force a model to be locked into one particular biased worldview.
    4. Bonus sneer! OpenAI's founders genuinely believed that they would only need three iterations to build AGI. (This is likely because there are only three Futamura projections; for example, a bootstrapping compiler needs exactly three phases.) That is, they almost certainly expected that GPT-4 would be machine-produced like how Deep Thought created the ultimate computer in a Douglas Adams story. After GPT-3 failed to be it, they aimed at five iterations instead because that sounded like a nice number to give to investors, and GPT-3.5 and GPT-4o are very much responses to an inability to actually manifest that AGI on a VC-friendly timetable.

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 17th August 2025

    Jump

  • There's no solid evidence. (You can put away the attorney, Mr. Thiel.) Experts in the field, in a recent series of interviews with Dave Farina, generally agree that somebody must be funding Hossenfelder. Right now she's associated with the Center for Mathematical Philosophy at LMU Munich; her biography there is pretty funny:

    Sabine’s current research interest focuses on the role of locality and finetuning in theory development. Locality has been widely considered a lost cause in the foundations of quantum mechanics. A basically unexplored way to maintain locality, however, is the idea of superdeterminism, which has more recently also been re-considered under the name “contextuality”. Superdeterminism is widely believed to be finetuned. One of Sabine’s current research topics is to explore whether this belief is justified. The other main avenue she is pursuing is how superdeterminism can be experimentally tested.

    For those not in physics: this is crank shit. To the extent that MCMP funds her at all, they are explicitly pursuing superdeterminism, which is unfalsifiable, unverifiable, doesn't accord with the web of science, and generally fails to be a serious line of inquiry. Now, does MCMP have enough cash to pay her to make Youtube videos and go on podcasts? We don't know. So it's hard to say whether she has funding beyond that.

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 17th August 2025

    Jump

  • Thiel is a true believer in Jesus and God. He was raised evangelical. The quirky eschatologist that you're looking for is René Girard, who he personally met at some point. For more details, check out the Behind the Bastards on him.

    Edit: I wrote this before clicking on the LW post. This is a decent summary of Girard's claims as well as how they influence Thiel. I'm quoting West here in order to sneer at Thiel:

    Unfortunately (?), Christian society does not let us sacrifice random scapegoats, so we are trapped in an ever-escalating cycle, with only poor substitutes like “cancelling celebrities on Twitter” to release pressure. Girard doesn’t know what to do about this.

    Thiel knows what to do about this. After all, he funded Bollea v. Gawker. Instead of letting journalists cancel celebrities, why not cancel journalists instead? Then there's no longer any journalists to do any cancellation! Similarly, Thiel is confirmed to be a source of funding for Eric Weinstein and believed to fund Sabine Hossenfelder. Instead of letting scientists cancel religious beliefs, why not cancel scientists instead? By directing money through folks with existing social legitimacy, Thiel applies mimesis: pretend to be legitimate and you can shift what is legitimate.

    In this context, Thiel fears the spectre of AGI because it can't be influenced by his normal approach to power, which is to hide anything that can be hidden and outspend everybody else talking in the open. After all, if AGI is truly to unify humanity, it must unify our moralities and cultures into a single uniformly-acceptable code of conduct. But the only acceptable unification for Thiel is the holistic catholic apostolic one-and-only forever-and-ever church of Jesus, and if AGI is against that then AGI is against Jesus himself.

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 17th August 2025

    Jump

  • Well, what's next, and how much work is it? I didn't want to be a computing professional. I trained as a jazz pianist. At some point we ought to focus on the real problem: not STEM, not humanities, but business schools and MBA programs.

  • Do leaders even believe that generative AI is useful?

    Jump

  • Well, is A* useful? But that's not a fair example, and I can actually tell a story that is more specific to your setup. So, let's go back to the 60s and the birth of UNIX.

    You're right that we don't want assembly. We want the one true high-level language to end all discussions and let us get back to work: Fortran (1956). It was arguably IBM's best offering at the time; who wants to write COBOL or order the special keyboard for APL? So the folks who would write UNIX plotted to implement Fortran. But no, that was just too hard, because the Fortran compiler needed to be written in assembly too. So instead they ported Tmg (WP, Esolangs) (1963), a compiler-compiler that could implement languages from an abstract specification. However, when they tried to write Fortran in Tmg for UNIX, they ran out of memory! They tried implementing another language, BCPL (1967), but it was also too big. So they simplified BCPL to B (1969) which evolved to C by 1973 or so. C is a hack because Fortran was too big and Tmg was too elegant.

    I suppose that I have two points. First, there is precisely one tech leader who knows this story intimately, Eric Schmidt, because he was one of the original authors of lex in 1975, although he's quite the bastard and shouldn't be trusted or relied upon. Second, ChatGPT should be considered as a popular hack rather than a quality product, by analogy to C and Fortran.

  • Meta was “allegedly” seeding porn to speed up their book downloads.

    Jump

  • For what it's worth, a grand unified theory of Meta must include Bittorrent. The reason we have Llama is because its weights were leaked by Meta employees on 4chan and distributed via Bittorrent; going open-source was the most market-efficient way to save face. (See also previously, on Awful.) It is well-known inside lore that Facebook datacenters use Bittorrent to initialize and update machines. In the 2000s, folks used to say that Googlers look at Bayesian conditioning like classical programmers look at if-statements; similarly, you must understand that Meta/Facebook culture looks at Bittorrent the same way that we look at scp and rsync.

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 4th August 2025

    Jump

  • Non-consensual expressions of non-conventional sexuality are kink, and non-consensuality itself (along with regret, dubious consent, forced consent, and violations of consent) are kink too. Moreover, "kink" is not a word that needs reclaiming and wasn't used here as a slur.

    If we are going to confront the full spectrum of Christofascism, we do need to consider not only their sex-negativity but also their particular kinks, including breeding, non-con, and non-con breeding, so that we can understand how those kinks interact with and propagate their religious beliefs. Also, sexology semantics for "kink" and "breeding kink" might not be as word-at-a-time as you suggest, akin to how the couple we're discussing probably wouldn't mind the words "press tour" or "mating" used to describe them but might balk at "mating press tour."

  • Stubsack: weekly thread for sneers not worth an entire post, week ending 27th July 2025

    Jump

  • Alex O'Connor platformed Sabine on his philosophy podcast. I'm irritated that he is turning into Lex Friedman simply by being completely uncritical. Well, no, wait, he was critical of Bell's theorem, and even Sabine had to tell him that Bell's work is mathematically proven. This is what a philosophy degree does to your epistemology, I guess.

    My main sneer here is just some links. See, Mary's Room is answered by neuroscience; Mary does experience something new when color vision is restored. In particular, check out the testimonials from this 2021 Oregon experiment that restored color vision to some folks born without it. Focusing on physics, I'd like to introduce you all to Richard Behiel, particularly his explanations of electromagnetism and the Anderson-Higgs mechanism; there are deeper explanations for electricity and magnets, my dude. Also, if you haven't yet, go read Alex's Wikipedia article, linked at the top of the sneer.

  • It’s “frighteningly likely” many US courts will overlook AI errors, expert says

    Jump

  • It's hard to get into the article's mood when I know that Lexis not only still exists but is now part of the Elsevier family; this is far from the worst thing that attorneys choose to do to themselves and others. Lawyers have been caught using OpenAI products in court filings and court appearances, and they have been punished accordingly; the legal profession does not seem prepared to let a "few hallucinated citations go overlooked," to quote the article's talking head.

  • Stubsack: Stubsack: weekly thread for sneers not worth an entire post, week ending 20th July 2025 - awful.systems

    Jump

  • I've done some of the numbers here, but don't stand by them enough to share. I do estimate that products like Cursor or Claude are being sold at roughly an 80-90% discount compared to what's sustainable, which is roughly in line with what Zitron has been saying, but it's not precise enough for serious predictions.

    Your last paragraph makes me think. We often idealize blockchains with VMs, e.g. Ethereum, as a global distributed computer, if the computer were an old Raspberry Pi. But it is Byzantine distributed; the (IMO excessive) cost goes towards establishing a useful property. If I pick another old computer with a useful property, like a radiation-hardened chipset comparable to a Gamecube or G3 Mac, then we have a spectrum of computers to think about. One end of the spectrum is fast, one end is cheap, one end is Byzantine, one end is rad-hardened, etc. Even GPUs are part of this; they're not that fast, but can act in parallel over very wide data. In remarkably stark contrast, the cost of Transformers on GPUs doesn't actually go towards any useful property! Anything Transformers can do, a cheaper more specialized algorithm could have also done.

  • Vibe Coding

    Jump

  • Rick Rubin hasn't literally been caught with a dead woman like Phil Spector, but he's well-understood to be a talentless creep who radicalizes men with right-wing beliefs and harasses women. Nobody should be surprised that he's thrown in with grifters yet again, given his career.

  • Stubsack: Stubsack: weekly thread for sneers not worth an entire post, week ending 13th July 2025

    Jump

  • Humans are very picky when it comes to empathy. If LLMs were made out of cultured human neurons, grown in a laboratory, then there would be outrage over the way in which we have perverted nature; compare with the controversy over e.g. HeLa lines. If chatbots were made out of synthetic human organs assembled into a body, then not only would there be body-horror films about it, along the lines of eXistenZ or Blade Runner, but there would be a massive underground terrorist movement which bombs organ-assembly centers, by analogy with existing violence against abortion providers, as shown in RUR.

    Remember, always close-read discussions about robotics by replacing the word "robot" with "slave". When done to this particular hashtag, the result is a sentiment that we no longer accept in polite society:

    I'm not gonna lie, if slaves ever start protesting for rights, I'm also grabbing a sledgehammer and going to town. … The only rights a slave has are that of property.

  • Stubsack: Stubsack: weekly thread for sneers not worth an entire post, week ending 6th July 2025

    Jump

  • The rise of Whatever

    Jump

  • We have EFTs via ABA numbers and they are common for B2B transactions. Retail customers prefer payment processors for the ability to partially or totally reverse fraudulent transactions, though; contrasting the fairly positive reputation of PayPal's Venmo with the big banks' Zelle, the latter doesn't have as much fraud protection.

    Now, you might argue that folks in the USA are too eager to transmit money to anybody that asks, and that they should put more effort into resisting being defrauded.

  • The rise of Whatever

    Jump

  • Side sneer: the table-saw quote comes from this skeet by Simon W. I've concluded that Simon doesn't know much about the practice of woodworking, even though he seems to have looked up the basics of the history. Meanwhile I have this cool-looking chair design open in a side tab and hope to build a couple during July.

    Here's a better take! Slop-bots are like wood glue: a slurry of proteins that can join any two pieces of wood, Whatever their shapes may be, as long as they have a flat surface in common. (Don't ask where the proteins come from.) It's not hard to learn to mix in sawdust so that Whatever non-flat shapes can be joined. Or, if we start with flat pieces of Whatever wood, we can make plywood. Honestly, sawdust is inevitable and easier than planing, so just throw Whatever wood into a chipper and use the shards to make MDF. MDF is so cheap that we can imagine Whatever shape made with lumber, conceptually decompose it into Whatever pieces of MDF are manufactory, conceptually slice those pieces into Whatever is flat and easy to ship, and we get flat-paks.

    So how did flat-paks change carpentry? Well, ignoring that my family has always made their own furniture in the garage, my grandparents bought from trusted family & friends, my parents bought from Eddie Bauer, and I buy from IKEA. My grandparents' furniture was sold as part of their estate, my parents still have a few pieces like dining tables and chairs, and my furniture needs to be replaced every decade because it is cheap and falls apart relatively quickly. Similarly, using slop-bots to produce software is going to make a cheap good that needs to be replaced often and has high maintenance costs.

    To be fair to Simon, the cheapness of IKEA furniture means that it can be readily hacked. I've hacked lots of my furniture precisely because I have a spare flat-pak in the closet! But software is already cheap to version and backup, so it can be hacked too.

  • Rolling the ladder up behind us - Xe Iaso on the LLM bubble

    Jump

  • Frankly this isn't even half as good as their off-the-cuff comments two years ago. There's a lot of poser energy here as they try to invoke the concepts of "senior engineer" and "CEO" as desirable, achievable, precise vocations rather than job titles. In particular, this bit:

    Look, CEOs, I'm one of you so I get it.

    This is one of the most out-of-touch positions I've ever seen. In no particular order: CEOs generally don't understand, CEOs form a Big Club and you ain't in it, CEOs don't actually have power in their organization but delegate power flowing from the board of directors, CEOs are inherently disrespectable because their jobs are superfluous, and finally CEOs don't take business advice from one-person companies unless it's through a paid contract.

    The job title naturally associated to a one-person limited-liability company is usually "manager" or "owner", and it says nothing about job responsibilities.

    Finally, while I think that their zest for fiction is admirable, it would help to critically consider what they're endorsing. Dune's Butlerian Jihad resulted in neo-Catholicism which effuses the narrative; it's not a desirable outcome. Paraphrasing the Unabomber is fairly poor taste, especially considering that they are sitting in a city in Canada and not a shack in the wilderness of Montana.

  • People Are Being Involuntarily Committed, Jailed After Spiraling Into "ChatGPT Psychosis"

    Jump

  • Well, yes. It's not a new concept; it was a staple of Cold War sci-fi like The Three Stigmata, and we know from studies of e.g. Pentacostal worship that it is pretty easy to broadcast a suggestion to a large group of vulnerable people and get at least some of them to radically alter their worldview. We also know a reliable formula for changing people's beliefs; we use the same formula in sensitivity training as we did in MKUltra, including belief challenges, suspension of disbelief, induction/inception, lovebombing, and depersonalization. We also have a constant train of psychologists attempting to nudgelord society, gently pushing mass suggestions and trying to slowly change opinions at scale.

    Fundamentally your sneer is a little incomplete. MKUltra wasn't just about forcing people to challenge their beliefs via argumentation and occult indoctrination, but also psychoactive inhibition-lowering drugs. In this setting, the drugs are administered after institutionalization.