Skip Navigation
42 comments

  • This is hardly surprising. It's immediately noticeable in images, but we'll have to be very careful with other forms of output as the decline could be subtle enough to go unnoticed at first. There's a very real risk of poisoning our sources of data by allowing AI to write back to them without oversight. And given that the sources of data seem to be things like Reddit and twitter this is a real concern.

    • Not really.

      The problem is with pushing out the edge of a normal distribution curve by the output regression to the mean.

      Merely mixing what's fed back in between AI generated and human generated would avoid this outcome, and arguably as long as the AI output was generally rated as better than the mean human output would even lead to recursive mixed training iterations improving over the original models.

      This and the Stanford paper were problematic exclusively training on new AI generated output over and over, which increased median tokens or diffusions and dropped edges until you ended up with output that had overfitted lackluster discrimination.

      The real takeaway here isn't "oh noz, we can't feed AI output back into AI training" but rather "humans in both generator and discriminator roles will be critical in future AI training."

      There's been a recent troubling trend of binaryisms in the ML field as hype and attention has increased, and it's important to be careful not to improperly extrapolate a finding of a narrow scope to an overly broad interpretation.

      So yes, don't go training recursively on only synthetic data over and over. But even something as simple as using humans upvoting or downvoting the generations to decide if you feed them back in or don't (i.e. human discriminator and AI generator) would largely avoid the outcomes here.

      Which means that human selection of the 'best' output from several samples for initial sharing and human rating of shared outputs for broader distribution is already cleaning up AI generations online enough that fears of 'poisoning' the data as suggested here and in the Stanford study are almost certainly overblown.

      Edit: From section 5 of the paper it even addresses some of this.

      One might suspect that a complimentary perspective to the previous observation—that fresh new data mitigates the MAD generative process—is that synthetic data hurts a fresh data loop generative process. However, the truth appears to be more nuanced. What we find instead is that when we mix synthetic data trained on previous generations and fresh new data, there is a regime where modest amounts of synthetic data actually boost performance, but when synthetic data exceeds some critical threshold, the models suffer.

  • I think people may be confused about what this is saying, so an example might help.

    Remember when Stable Diffusion first came out and you could spot AI generated images as if they killed your father and should be prepared to die?

    Had those six+ digit monstrosities been fed back into training the model, you'd have quickly ended up with models generating images with even worse hands from hell.

    But looking at a study like this and worrying about AI generated content poisoning the Internet for future training is probably overblown.

    Because AI content doesn't just find its way onto the web directly the way it is in this (and the Stanford) study. Often a human is selecting from multiple outputs to decide what to post, or even if it is directly posted, humans are voting content up or down based on perceived quality.

    So practically, if models were being trained recursively on popular content online that had been generated by AI, it wouldn't be content that overfits spider hands or striped faces or misshapen pupils or repetitive text or broken links or any other number of issues generative AI currently has.

    Because of the expense in human review of generated content this and the previous paper aren't replicating the circumstances that real world recursive training of a mixed human and AI Internet would represent, and the issues which arose will likely be significantly muted in real world circumstances outside the lab.

    TL;DR: Humans filtering out six fingered outputs (and similar) as time goes on is a critical piece of the practical infrastructure which isn't being represented, and this is simply a cautionary tale against directly piping too much synthetic data back into training.

  • From the article:

    Knowing means that the search for a watermark that identifies AI-generated content (and that's infallible) has now become a much more important - and lucrative - endeavor, and that the responsibility for labeling AI-generated data has now become a much more serious requirement.

    Simply wanting such a thing to exist isn't going to magically make it happen. I seriously doubt that any such "watermark" (I think they meant "fingerprint" since it'd need to work even if not deliberately added) can be found.

    I suspect the actual solution is to curate the quality of the input data, regardless of whether it's AI-generated or not. The problem of autophagy is the loss of rare inputs, so try to ensure those inputs are found and included in the input data. It's probably fine to have some AI generated content in the training data in addition to the real stuff. Indeed, as long as the AI-generated content is subject to the same sort of selective pressure as the real content it's probably good to have.

    • We'd need to test and see if AI-generated content that is curated by human quality assurance still causes MADness.

      My suspicion is that would only slow down the degradation of the outputs, rather than stop it completely.

      • I wasn't proposing only using curated AI-generated content. If the problem is the loss of "rare data" from the edges, then adding some AI-generated data to a data set that still includes that rare data shouldn't be a problem.

        The article doesn't say that AI-generated data is somehow "infectious", just that the data set becomes more and more limited with each cycle since rare information gets lost each time.

  • I'd go mad too if someone tried to train me on AI created data all the time...

  • Considering that training is extracting the main features of a dataset, there is always some data that is discarded as "noise" in the process, then when data is generated, that discarded information is filled back with actual random noise to partially replicate the original data.

    Iterate and you're going to end up with progressively less meaningful features. I just didn't expect it to take only 5 iterations, that's a lot of feature loss in training even with so many parameters.

  • I don't know. If you locked someone in a room. Showed them a single news broadcast, and from then on just replayed back at them anything they say.. They'd go mad too.

42 comments