ChatGPT generates cancer treatment plans that are full of errors — Study finds that ChatGPT provided false information when asked to design cancer treatment plans

I'm trying really hard for the latter.

This is true and well-stated. Mainly what I wish people would understand is there are current appropriate uses, like 'rewrite my marketing email', but generating information that could result in great harm if inaccurate is an inappropriate use. It's all about the specific model, though - if you had a ChatGPT system trained extensively on medical information, it would result in greater accuracy, but still the information would need expert human review before any decision were made. Mainly I wish the media had been more responsible and accurate in portraying these systems to the public.

I don’t think it’s good enough to have a blanket conception to not trust them completely.

On the other hand, I actually think we should, as a rule, not trust the output of an LLM.

They’re great for generative purposes, but I don’t think there’s a single valid case where the accuracy of their response should be outright trusted. Any information you get from an AI model should be validated outright.

There are many cases where a simple once-over from a human is good enough, but any time it tells you something you didn’t already know you should not trust it and, if you want to rely on that information, you should validate that it’s accurate.

Another related confusion in academia recently is the 'AI detector'. It could easily be defeated with minor rewrites, if they were even accurate in the first place. My favorite misconception is there was a story of a professor who told students "I asked ChatGPT if it wrote this, and it said yes" which is just really not how it works.

I can understand the plagiarism argument, though you have to extend the definition of it. If I am expected to write an essay, but I use ChatGPT instead, then I am fraudulently presenting the work as my own. Plagiarism might not be the right word, or maybe it's a case where language is going to evolve so that plagiarism includes passing off AI generated work as your own. Either way it's cheating unless I was specifically allowed to use AI.

It's not merely a preconception. It's a rather obvious and well-known limitation of these systems. What I am decrying is that some people, from apparent ignorance, think things like "ChatGPT can give a reliable cancer treatment plan!" or "here, I'll have it write a legal brief and not even check it for accuracy". But sure, I agree with you, minus the needless sarcasm. It's useful to prove or disprove even absurd hypotheses. And clearly people need to be definitely told that ChatGPT is not always factual, so hopefully this helps.

Why the hell are people down voting you?

This is absolutely correct. We need to do the science. Always. Doesn't matter what the theory says. Doesn't matter that our guess is probably correct.

Plus, all these studies tell us much more than just the conclusion.

"After an extensive three-year study, I have discovered that touching a hot element with one's bare hand does, in fact, hurt."

"That seems like it was unnecessary..."

"Do U even science bro?!"

Not everything automatically deserves a study. Were there any non-rando people out there claiming that ChatGPT could totally generate legit cancer treatment plans that people could then follow?

It's not even a preconception, it's willful ignorance, the website itself tells you multiple times that it is not accurate.

The bottom of every chat has this text: "Free Research Preview. ChatGPT may produce inaccurate information about people, places, or facts. ChatGPT August 3 Version"

And when you first use it, a modal pops up explaining the same thing.

ChatGPT isn't some newly discovered sentient species.

It's a machine designed and built by human engineers.

This is like suggesting that we study fortune cookies to see if they can accurately forecast the future. The manufacturer can simply tell you the limitation of their product... Being that they can not divine the future.

It doesn't check the stuff it generates other than on grammatical and orthographical errors. It's not intelligent or has knowledge outside of how to create text. The text looks useful, but it doesn't know what it contains in a way something intelligent would.

If you want an AI that can create cancer treatment, you need to train it on creating cancer treatment, and not just use one that is trained on general knowledge. Even if you train it on science publications, all it can now reliably do is mimic a science journal since it has not been trained on how to parse the knowledge in the journal itself.

It did pass it didn't it. But who said it can replace doctors?

Good old "garbage in, garbage out".

Just a couple days ago it continually told me it was possible to re-tile part of my shower that is broken without cutting tiles, but none of the math added up. (18.5H x 21.5w area) "Place a 9" tile vertically. Place another 9“ tile vertically on top on the same side. Place another 9" tile on top vertically to cover the remainder of the area."

I told chatgpt it was wrong, which it admitted, and spit out another wrong answer. I tried specifying a few more times before I started a new chat and dumbed it down to just a simple math algorithm problem. The first part of the chat said it was possible, layed out the steps, and then said it wasn't possible in the last sentence.

I surely wouldn't trust chatgpt to advise my healthcare, but after seeing it spit out very wrong answers to a basic math question, I'm just wondering why anyone would try to have it advise anyone's health are.

They've been trying this shit for decades already with established AI like Big Blue. This isn't a new pattern. Those in charge need to keep driving costs down and profit up.

Race to the bottom.

If that were legal, I'd absolutely be worried, you make a good point.

Even Doctor need special additional qualifications to do things like diagnose illnesses via radiographic imagery, etc. Specialised AI is making good progress in aiding these sorts of things, but a generalised and very poor AI like ChatGPT will never be legally certified to do this sort of thing.

Once we have a much more effective generalised AI, things will get more interesting. It'll have to prove itself thoroughly though, before being certified, so it'll still be a few years after it appears before we see it used in clinical applications.

I'm specifically referring to ChatGPT. GPT-4 is a different beast that I'm sure is quite adept.

Its fantastic at that. I had it help me with a Dark Heresy session. Its not bad at generating names, places, and even personalities for jobbers.

It can leave the bullets if you tell it.

It's possible to build intelligent AI

What does intelligent AI that we can currently build look like?

For me the issue isn't the tool. It's people. The tool is used just as it is. A tool.

I always like to compare these things to other physical tools. If you take a philips screw driver to a flathead screw you don't blame the tool you blame yourself for bringing the improper tool because as a human you can make mistakes. As a human you should have figured out prior, "do I need a flathead or philips?" There are tools capable of doing the job and doing it properly.

Same if you are an operator on a piece of machinery. If you take a forklift to destroy a house you probably aren't going to get very far.

All of these tools were designed to make life easier and provide a positive to life when doing something but it is how you use the tool that matters.

The same with a gun. I am not a gun ownership kind of guy because of all the shit human beings that just can't use one properly or claim to use it properly. Guns get more complicated and so do their use cases but the truth is a gun was designed to kill or defend from being killed (this is not a topic about gun rights just using it as an example.) However, in the hands of the wrong person a gun can kill unintentionally. That isn't the guns fault after all its design was to kill.

ChatGPT wasn't designed to kill, inherently. It wasn't designed to do anything other than take databases of information and provide what it thinks is correct. If you as a person don't know how to use it or what to do with it probably and you aren't seeking actual medical attention or advice from a professional then I think that is the person's fault.

ChatGPT can't make a disclaimer for every little thing. A car on the other hand having a recall issue can. If you want to compare to a faulty part in a car then sure. Modify ChatGPT to just not provide medical advice.

See tools can be changed midway through. The tool isn't the problem how the person uses the tool is the issue. Access to that tool and what that tool has access to can be an issue but the great thing about tools is laws can change and tools can change.

It isn't the A.I.s fault if your legislature doesn't care to enforce that change or law. The same legislature that half of Lemmy is opposed to literally all the time. Tools are only good in ways they can be used as well.

So let's say for arguments sake the tool is dangerous and in your defense it absolutely can be used dangerously. Do you call upon the government to shut it down just like you would call upon the government to regulate or change gun laws?

Do you also ignore the positive impacts ChatGPT can have because it is doing something else terribly? Imagine a system that medical professionals do create and they modify a version that does provide good medical advice, accurate, and professional? What then? Is ChatGPT still bad? It's not out of the realm of possibility. A.I isn't the enemy because someone's leadership decided to fire you. Leadership is the enemy. Tools are only as bad as the people using them.

Or for the sake of a recalled care that can kill they are as bad as the user manufacturing them. I don't deny you can get a bad car, a bad screwdriver. My point is if you let the bad outweigh the good then you are missing the point. The bad should be handled by people who understand it better and can design laws and tools to enforce better usage to make something less bad. So again don't blame the tool blame the people that aren't protecting you with said tool.

I mean, people are slightly more complicated than that. But sure, at their most basic, people simply communicate with statistical models.

IMO for AI to reach a useful point it needs to be able to learn. Now I’m no expert on neural networks, but if it can’t learn anything new once it’s been trained, it’s never really going to reach its true potential. It can imitate a human, but that’s about it. Once AI can really learn, it’ll become an order of magnitude more useful. Don’t get me wrong: all this AI work is a step in the right direction, but we’ll only be able to go so far with pre-trained models.

Might be true for you but most people do have a concept of true and false and don't just dream up stuff to say.

Hah! That's the response I always give! I'm not saying our brains work the exact same way because they don't and there's still a lot missing from current AI but I've definitely noticed that at least for myself, I do just predict the next word when I'm talking or writing (with some extra constraints). But even with LLMs there's more going on then that since the attention mechanism allows it to consider parts of the prompt and what it's already written as it's trying to come up with the next word. On the other hand, I can go back and correct mistakes I make while writing and LLMs can't do that..it's just a linear stream.

I unfortunately know someone that literally owns every tool he could need. He's so damn impatient that when he needs a tool, whatever is in reach becomes that tool whether it's the tool he needs or not. I do not allow him near my tools

I thought it was 4 before looking into it. It seems they ran the experiment in the same month GPT-4 was released, that's unfortunate.

That wouldn't guarantee correct answers.

It's arguably more dangerous if ChatGPT gives mostly sane specific medical advice because it makes people put more trust in it than they should.

Can you give some recent use examples of this? I don't use CHATGPT anymore but I used to get my code written by it and many of my classmates still use it