9mo ago

Anthropic has developed an AI 'brain scanner' to understand how LLMs work and it turns out the reason why chatbots are terrible at simple math and hallucinate is weirder than you thought

www.pcgamer.com Anthropic has developed an AI 'brain scanner' to understand how LLMs work and it turns out the reason why chatbots are terrible at simple math and hallucinate is weirder than you thought

Oh, and another thing: They don't just predict the next word.

Anthropic has developed an AI 'brain scanner' to understand how LLMs work and it turns out the reason why chatbots are terrible at simple math and hallucinate is weirder than you thought

You're viewing a single thread.

167 comments

this is one of the most interesting things about Llms that i have ever read
- That bit about how it turns out they aren't actually just predicting the next word is crazy and kinda blows the whole "It's just a fancy text auto-complete" argument out of the water IMO
  
  It really doesn't. You're just describing the "fancy" part of "fancy autocomplete." No one was ever really suggesting that they only predict the next word. If that was the case they would just be autocomplete, nothing fancy about it.
  What's being conveyed by "fancy autocomplete" is that these models ultimately operate by combining the most statistically likely elements of their dataset, with some application of random noise. More noise creates more "creative" (meaning more random, less probable) outputs. They do not actually "think" as we understand thought. This can clearly be seen in the examples given in the article, especially to do with math. The model is throwing together elements that are statistically proximate to the prompt. It's not actually applying a structured, logical method the way humans can be taught to.
  
  Unfortunately, these articles are often written by people who don't know enough to realize they're missing important nuances.
  
  It also doesn't help that the AI companies deliberately use language to make their models seem more human-like and cogent. Saying that the model e.g. "thinks" in "conceptual spaces" is misleading imo. It abuses our innate tendency to anthropomorphize, which I guess is very fitting for a company with that name.
  On this point I can highly recommend this open access and even language-wise accessible article: https://link.springer.com/article/10.1007/s10676-024-09775-5 (the authors also appear on an episode of the Better Offline podcast)
  
  I can't contemplate whether LLMs think until someone tells me what it means to think. It's too easy to rely on understanding the meaning of that word only through its typical use with other words.
  
  Genuine question regarding the rhyme thing, it can be argued that "predicting backwards isn't very different" but you can't attribute generating the rhyme first to noise, right? So how does it "know" (for lack of a better word) to generate the rhyme first?
  
  It already knows which words are, statistically, more commonly rhymed with each other. From the massive list of training poems. This is what the massive data sets are for. One of the interesting things is that it's not predicting backwards, exactly. It's actually mathematically converging on the response text to the prompt, all the words at the same time.
  
  Which is exactly how we do it. Ours is just a little more robust.
  
  We also check to see if the word that popped into our heads actually rhymes by saying it out loud. Actual validation steps we can take is a bigger difference than being a little more robust.
  We also have non-list based methods like breaking the word down into smaller chunks to try to build up hopefully more novel rhymes. I imagine professionals have even more tools, given the complexity of more modern rhyme schemes.
  
  People are generally shit at understanding probabilities and even when they have a fairly strong math background tend to explain probablistic outcomes through anthropomorphism rather than doing the more difficult and "think-painy" statistical analysis that would be required to know if there was anything more to it.
  I myself start to have thoughts that balatro is purposefully screwing me over or feeding me outcomes when it's just randomness and probability as stated.
  Ultimately, it's easier (and more fun) for us to think that way and it largely serves us better in everyday life.
  But these things are entire casinos' worth of probability and statistics in and of themselves, and the people developing them want desperately to believe that they are something more than pseudorandom probabilistic fancy autocomplete engines.
  A lot of the folks at the forefront of this have paychecks on the line. Add the difficulty of getting someone to understand how something works when their salary depends on them not understanding it to the existing inability of humans to reason probabilistically and the AGI from LLM delusion becomes near impossible to shake for some folks.
  I wouldn't be surprised if this AI hype bubble yields a cult in the end.
  
  Predicting the next word vs predicting a word in the middle and then predicting backwards are not hugely different things. It's still predicting parts of the passage based solely on other parts of the passage.
  Compared to a human who forms an abstract thought and then translates that thought into words. Which words I use has little to do with which other words I've used except to make sure I'm following the rules of grammar.
  
  Compared to a human who forms an abstract thought and then translates that thought into words. Which words I use has little to do with which other words I’ve used except to make sure I’m following the rules of grammar.
  Interesting that...
  Anthropic also found, among other things, that Claude "sometimes thinks in a conceptual space that is shared between languages, suggesting it has a kind of universal 'language of thought'."
  
  Yeah I caught that too, I'd be curious to know more about what specifically they meant by that.
  Being able to link all of the words that have a similar meaning, say, nearby, close, adjacent, proximal, side-by-side, etc and realize they all share something in common could be done in many ways. Some would require an abstract understanding of what spatial distance actually is, an understanding of physical reality. Others would not, one could simply make use of word adjacency, noticing that all of these words are frequently used alongside certain other words. This would not be abstract, it'd be more of a simple sum of clear correlations. You could call this mathematical framework a universal language if you wanted.
  Ultimately, a person learns meaning and then applies language to it. When I'm a baby I see my mother, and know my mother is something that exists. Then I learn the word "mother" and apply it to her. The abstract comes first. Can an LLM do something similar despite having never seen anything that isn't a word or number?
  
  I don't think that's really a fair comparison, babies exist with images and sounds for over a year before they begin to learn language, so it would make sense that they begin to understand the world in non-linguistic terms and then apply language to that. LLMs only exist in relation to language so couldnt understand a concept separately to language, it would be like asking a person to conceptualise radio waves prior to having heard about them.
  
  Exactly. It's sort of like a massively scaled up example of the blind man and the elephant.
  
  Can an LLM do something similar despite having never seen anything that isn’t a word or number?
  No.
  
  Yeah but I think this is still the same, just not a single language. It might think in some mix of languages (which you can actuaysee sometimes if you push certain LLMs to their limit and they start producing mixed language responses.)
  But it still has limitations because of the structure in language. This is actually a thing that humans have as well, the limiting of abstract thought through internal monologue thinking
  
  Probably, given that LLMs only exist in the domain of language, still interesting that they seem to have a "conceptual" systems that is commonly shared between languages.
  
  wow an AI researcher over hyping his own product. he's just waxing poetic .
  we don't even have a good sense of what thought IS, please tell Claude to call the philosophers because apparently he's figured out consciousness
  
  I read an article that it can "think" in small chunks. They don't know how much though. This was also months ago, it's probably expanded by now.
  
  anything that claims it "thinks" in any way I immediately dismiss as an advertisement of some sort. these models are doing very interesting things, but it is in no way "thinking" as a sentient mind does.
  
  I wish I could find the article. It was researchers and they were freaked out just as much as anyone else. It's like slightly over chance that it "thought," not some huge revolutionary leap.
  
  there has been a flooding of these articles. everyone wants to sell their llm as "the smartest one closest to a real human" even though the entire concept of calling them AI is a marketing misnomer
  
  Maybe? Didn't seem like a sales job at the time, more like a warning. You could be right though.
  
  they post articles like that all the time. warnings make great clickbait
  
  You know they don't think - even though "It's a peculiar truth that we don't understand how large language models (LLMs) actually work."?
  It's truly shocking to read this from a mess of connected neurons and synapses like yourself. You're simply doing fancy word prediction of the next word /s
  
  Anybody who claims they don't "think" before we even figure out completely how they work and even how human thoughts work are just spreading anti-AI sentiment beyond what is considered logical.
  You should become a better example than an AI by only arguing based on facts rather than things you hallucinate if you want to prove your own position on this matter.
  
  shouldn't you say the inverse is true lol why call it thinking if we don't know what thinking is or what it's doing?
  why are you cool with pro ai and against anti ai sentiments? either way it's a value judgment, quit acting like yours is the correct opinion
  
  I wasn't calling it thinking. I'm saying people claiming it's not is just jumping the gun. It's also funny you're simply claiming I am pro AI without needing any proof. This is what I meant when I said people who are anti-AI should strive to be better than the AI they criticise. Acting based on non-facts makes you no better than AI with their hallucinations.
  Its also funny that you're calling me out when I'm just mirroring what the other guy is doing to make a point. He's acting like his is the correct opinion, yet you only calling me out because the guy is on your side of the argument. That's simply a bad faith argument on your part.
  
  I see the misunderstanding, sorry. You're still in the wrong though. while you weren't calling it thinking, the article certainly was. THAT'S why we're saying it's not. we're doing what you said we should, but it's the inverse, and you call it anti-AI. the jackass who wrote that article is jumping the gun and we're saying "how tf can you call it thinking" and i see your reply calling that anti AI, seems like a reasonable mistake ye?
  
  But the comment I replied to didn't just deny the confirmation that AI is thinking, it also denied that AI "thinks" at all. That puts him in a position of making an unproven claim. In fact, he is directly making that claim, while the article he is denying only alludes to saying that LLM "thinks" like a human. That makes his unproven claim even more egregious than the article's.
  
  It doesn't, who the hell cares if someone allowed it to break "predict whole text" into "predict part by part, and then "with rhyme, we start at the end". Sounds like a naive (not as in "simplistic", but as "most straightforward") way to code this, so given the task to write an automatic poetry producer, I would start with something similar. The whole thing still stands as fancy auto-complete
  
  But how is this different from your average redditor?
  
  Redditor as "a person active on Reddit"? I don't see where I was talking about humans. Or am I misunderstanding the question?
  
  This dumbass is convinced that humans are chatbots likely because chatbots are his only friends.
  
  Sounds scary. I read a story the other day about a dude who really got himself a discord server with chatbots, and that was his main place of "communicating" and "socializing"
  
  This anecdote has the makings of a "men will literally x instead of going to therapy" joke.
  On a more serious note though, I really wish people would stop anthropomorphisizing these things, especially when they do it while dehumanizing people and devaluing humanity as a whole.
  But that's unlikely to happen. It's the same type of people that thought the mind was a machine in the first industrial revolution, and then a CPU in the third...now they think it's an LLM.
  LLMs could have some better (if narrower) applications if we could stop being so stupid as to inject them into places where they are obviously counterproductive.
  
  Hard agree in every point
  
  they do it while dehumanizing people and devaluing humanity
  You're making wild assumptions about people who disagree with your opinions. How ironic you accuse "them" of dehumanizing people.
  But I do agree that this gets to the core of the matter, the shock of a piece of software being able to produce intelligent text while clearly not having general intelligence is quite the shock. Same with creativity, while the entertainment industry produced equally empty content slop using human labor it's a painful shock to our identity as humans. I suspect this is a reaction to disillusionment and the intellectual pain that comes from it.
  My opinion on LLMs is rather nuanced, the worst possible outcome I can foresee is the anti-AI crowd helping the oligarchs to establish IP ownership of all LLM models and monopolizing the tools, so that only they can have access to the "means of generation". While the rest has to pay for the privilege of using it.
  
  How ironic you accuse “them” of dehumanizing people.
  Dude, you're up two posts from here saying how people are LLMs.
  I understand wanting to walk that back, but you did post it.
  
  What the heck are you talking about?
  
  I mean it implies that they CAN start with the conclusion or the "thought" and then generate the text to verbalize that.
  It's shocking to what length humans will go to explain how their wetware neural network is fundamentally different and it's impossible for LLMs to think or reason in any way. Honestly LLMs teach us more about human intelligence (or the lack thereof) than machine intelligence. Like obi wan said, "The ability to speak does not make one intelligent" haha.

167 comments