what AI does to a mf: Claude proudly misdiagnoses an MRI. "We're still early..."

A tweet from ardent promptfondler Misha Saul (archive):

A friend sent me MRI brain scan results and I put it through Claude.
No other AI would provide a diagnosis, Claude did.
Claude found an aggressive tumour.
The radiologist report came back clean.
I annoyed the radiologists until they re-checked. They did so with 3 radiologists and their own AI. Came back clean, so looks like Claude was wrong.
But looks how convincing Claude sounds! We're still early...
Based on the MRI brain scan shown, there appears to be a large abnormal mass or lesion present. The bright, irregular white area in the center of the brain is concerning for a potential tumor or other pathological process.
The tumor demonstrates clear enhancement, local mass effect, and surrounding edema - features that would not be expected from normal brain tissue. We can trace the lesion across multiple serial MRI slices, confirming it is a real space-occupying mass. The location is also typical for a primary glioma.
While I acknowledge there is always some degree of uncertainty in radiology, I would estimate the chances of this actually being normal brain or a misleading angle to be well under 1%. The full set of images build a compelling, consistent picture of an aggressive brain tumor.
I asked it to review more angles:
After reviewing the additional MRI sequences and angles provided in this updated image, my assessment remains unchanged. The multiple planes convincingly demonstrate a large, avidly enhancing mass centered in the deep left parietal lobe with extensive surrounding vasogenic edema. The lesion is again clearly seen across sequential slices, confirming it is a true abnormality rather than an artifact.
Key observations:
1. Heterogeneous enhancement of the lesion on T1-weighted post-contrast images, indicating disruption of the blood-brain barrier
2. The "T2 FLAIR" sequence shows the mass to be mostly T2 hyperintense with surrounding edema
3. Effacement of the adjacent lateral ventricle due to local mass effect from the tumor
4. Midline shift to the right from the expansile effect of the lesion
These findings reaffirm my initial impression of a high-grade glioma, most likely glioblastoma given the imaging characteristics. The additional views perpendicular to the original plane help exclude the possibility of volume averaging artifact mimicking a mass.
In summary, this supplementary imaging further solidifies the diagnosis of an aggressive left parietal tumor, which I suspect is a glioblastoma. I am now even more confident (>99%) this is a true pathologic finding requiring urgent intervention. Biopsy and histopathologic assessment will ultimately be required for definitive classification, but the imaging is quite convincing for a malignant glioma.

Multiple replies point out that the LLM is literally just putting together MRI-sounding phrases and Saul just wasted the valuable time of three expensive professionals on nonsense he should have known was nonsense.

Saul is unfazed:

Impressed with the vitriol my post has attracted
My theory is that the mentally ill he/hims have reverence for authority and doctors are a special class of revered expert rather than a fallible class of professionals
Or maybe trying to use tech is inherently suspicious? 🤷‍♂️

He then doubles down on the fabulous promptfondling future:

Obviously the ideal state is doctors are entirely obsolete
Instant diagnosis, instant treatment
No GPs, no imaging or pathology supply chains, no surgeons, no mistakes, no delays
We may never reach that, but that’s obviously the ideal we want to asymptote towards

and a magical flying unicorn pony with the wrong number of legs

You're viewing a single thread.

55 comments

Sovereign citizen energy.
Can’t wait for an AI crank to demand cancer treatment because Siri looked sideways at a blurry photo of a lump on their butt. Can’t wait for a Scott to write a script/dialogue tree for getting DIY chemo kits.
- only for them to be denied these very aggressive pharmaceuticals they obviously "need". but worry not, there's that one weirdo who solved chemistry with iot and wants people to cook their own meds in glass jars with 3d printed lids connected to a series of raspberry pis
  e: they don't have to be libertarian, i've just assumed it based on sheer impracticability of that nonsense and little regard for actual safety for end user. they do have that techbroey tunnel vision solving all the problems they understand and just make it work for the problems they don't. their entire mason jar reactor is completely superfluous if you can get proper glassware and in both cases you need to know actual chemistry beforehand anyway
  
  if you have no fucking clue what i'm talking about, it's this and it has a response from a professional
  
  Wow, I hadn't read Lowe's response before, and it is capitalist cringe of the highest order. Thanks for sharing.
  To be clear: I agree with every chemical and pharmacological critique leveled at the anarchists here. I also think that none of them have addressed the actual problem that the anarchists are solving, which is that medicinal chemistry has undergone so much regulatory capture that it is no longer legal to perform it at home for one's own private use or even to reverse-engineer the synthesis pathways. For more commentary on this, I recommend watching e.g. NurdRage reverse-engineering pyrimethamine and paying attention to what they say about obtaining precursors and carrying out various steps of synthesis.
  
  now i see the issue with pyrimethamine, namely one step sideways gets you close P2P analogue which is a big regulatory nono
  
  continuing on that capitalist cringe, i'd just point out that fair bit of that risky (in business sense) fundamental research that ultimately goes into new drugs is conducted at universities, who then get fuck all of these profits even if there's a licensing agreement
  although from what i understand the biggest leech on for-profit american healthcare system is insurance layer
  
  in mild defense of DL, he did wrote a thing or two about drug pricing policy, but i didn't really kept up with his blog since pandemic
  
  I’ve had the misfortune to watch the publicly-funded research to extremely expensive drug pipeline in action, and it fucking sucks to see the same ghouls who profit from that process turn around and claim that the resulting drugs are expensive because they had to pay for R&D (with, of course, the accompanying bullshit excuses that “only your insurance pays” or “we have a discount program”, which is small comfort when you’re staring down a $700+ monthly bill for the meds you need and both your insurance and the discount program have decided you don’t need it bad enough)
  
  but wait wait wait are they really solving anything, because i wouldn't say so. getting starting materials is not solved with a mason jar with raspberry pi. even in that circumstance that you already have all that kit you still need to know some fundamentals of organic chemistry, because trusting your life and limb to a glass jar with illusions of grandeur sounds like an aviodable suboptimal situation
  
  who told you it's illegal? it's hard, takes fuckton of elbow grease, some machines are very expensive (400MHz NMR is some 100-150 grand, or at least used to be, probably more now), but as long as you don't want to work with restricted compounds and as long as you don't intend to dose (other) humans it's all perfectly legal and you can do it all out of your garage. you might need to found a company tho, as ordering reagents for a company tends to raise less questions
  i'll watch this all sometime, but not this week probably
  also he's most likely not doing any reverse engineering, he's probably repeating a procedure from a paper
  
  hmm
  you know, I'm not entirely opposed to the sentiment. but I know enough about the domain (and I should stress that I know only a very small amount) to know that it's something you want to do properly, and I expect a lot of people doing this would not be in that bucket
  
  doubly so because "just fix the fucking US medical system" is the actual answer for a lot of the example targets
  
  They are trying to solve the problem of gigantic markups - because of legal monopolies through patents - on advanced chemical products where dosage and quality control is literally life or death. Their solution is to do it yourself in a garage.
  Couldn't they at least tried parallel imports from quality controlled production in countries with less gigantic markups?
  Or, if they could have stopped playing Robin Hood for a second and looked at the systemic problems instead, there was a proposal at WHO some ten years ago to reform pharmaceutical research and development with direct funding and then releasing the results, basically creating a direct to generica pipeline. The US shot it down, of course, which means public relations campaigns in the US would be great. But no, DIY in the garage.
  (If anyone is interested in details the proposal was called "delinkage".)
  
  oh no i threw an offhand remark and now i have to elaborate
  well i don't see their actual procedures anywhere so i'm going off my memory, they claimed to be able to make naloxone for example
  this works like this: first react oxycodone with allyl bromide (chloride?), then add that quatenary ammonium salt to sodium ethylthiolate solution, and this should take off both N-methyl from ammonium and O-methyl from phenolate. peachy, there's your naloxone (do i need to draw this?)
  and on paper it's all fine, but problems pile in. for example allyl bromide is fucking nasty, it's volatile and pretty toxic, you definitely don't want untrained people to handle it. (you can hardly catch me in a lab wearing labcoat, but this is one of reagents that i'll have it on, and handle only under fumehood) ethanethiol if mishandled will make entire city block think that there's a gas leak. and that's only starting materials, oxycodone is obviously active compound, and iirc they dissolved it in DMF, some polar solvents like this have that nasty property of basically carrying compounds through skin into the bloodstream (that's how gel patches with fentanyl or some hormones work). anyway,
  that all works if it works, but for example, if allyl bromide somehow flies away, and it can because it's volatile, then you don't have what you think you have. that means that you have fucked up, and if you don't catch this and go through anyway, you'll get oxymorphone instead of naloxone, which has opposite pharmacological effect. that's pretty fucking bad thing, especially when it's advertised as "cook up your own opioid overdose antidote at home". second step might not work at all if solvents or glassware are wet for example, or there are problems with stirring. this is not hard hard chemistry, medchem is made this way to be as easy and reliable as it gets on purpose - to get rid of weird problems in industrial scale synthesis and such (organic chemists that do development of new reactions sometimes are opposed in industry if they introduce a new thing that makes everyone's life easier on basis that it's a new thing), but it has to be done right with proper analysis and i can't emphasize this enough. you can't do any analysis with glass jars and diy syringe pumps, probably the best thing is melting point determination and i wouldn't trust even that
  there's a reason why it's laboratory glassware and not plasticware, i don't trust these 3d printed parts or silicone tubing to survive anything substantial, in lab we use glass, teflon, some stainless steel for less demanding things
  then, where do you even get starting materials, purify them (harder than it sounds like, esp when you need to get water and oxygen out. this includes solvents), measure them out properly. making a miniature jacketed reactor is not the hard part of making chemistry, equivalent thing can be done with a rbf, some cooling or heating bath, thermometer and magnetic stirrer. you know, like in every lab scale synthesis from undergraduate up. the harder bit is the everything else aspect, it's a very much draw the rest of the fucking owl situation
  second link is derek lowe's commentary on this, he's in that industry for decades by now, his response has a bit of that bitchy "we fearless capitalists made that thing" thing, he described himself a republican once (non-trumpist) but he's not an actual monster and he knows what he's talking about
  i get where this sentiment comes from, but i think beehive is more reliable at this thing that thing tries to do, that is making sure that some people know how to make some pharmaceuticals (or similar) unofficially in their garage. but scope is different, compounds are different, target audience is different, and beehive expects some lab experience, won't hold your hand all the way and you're very much on your own, legally, organizationally, and such
  
  @skillissuer Ex-pharmacist here: this just touches the tip of the iceberg. For example, what's your source of pure oxycodone? If you're grinding up tablets, you're also getting bulk adjuvants, bursting agents, probably fragments of osmotic release membrane, and a bunch of other non-pharmacologically active chemicals. And what's your output? Sure naloxone is injectable, but you're going to want it dissolved in an isotonic solution, sterile, with preservatives, and a syringe ...
  
  @skillissuer Basically a drug (chemical that has a pharmacological effect) is not a medicine (a formulation for delivering a drug to the tissue where it will have the desired pharmacological effect) any more than a naked internal combustion engine is an automobile.
  
  they also wanted to make cabotegavir, which has two massive issues similar to this. 1. where in the world do you get 2,4-difluorobenzylamine, if not at fine chemicals supplier and 2. long biological halflife depends on proprietary nanoparticle formulation, this probably means that extremely specialized equipment is needed. synthesis will be also far from trivial
  
  some polar solvents like this have that nasty property of basically carrying compounds through skin into the bloodstream
  This reminded me how amazed I was to discover this was real. The first time I read about this was in a Shadowrun (the pnp rpg system) rulebook. Was fun to see they used real world stuff. (The book on 'why megacorps want access to stockmarkets illegally etc' book was a lot less fun to learn was based on real stuff)
  
  this all of course goes with a TINY LITTLE ASSUMPTION that whatever tries to do any of that even gets their hands on required starting materials. this can be harder than it sounds like
  
  there’s that one libertarian weirdo who solved chemistry with iot
  wait what, do tell

55 comments