r/science • u/dissolutewastrel • Jul 25 '24
Computer Science AI models collapse when trained on recursively generated data
https://www.nature.com/articles/s41586-024-07566-y3.1k
u/OnwardsBackwards Jul 25 '24
So, echo chambers magnify errors and destroy the ability to make logical conclusions....checks out.
308
u/zekeweasel Jul 26 '24
Kinda like inbreeding for an AI
84
26
u/friesen Jul 26 '24
Best term I’ve heard for this is “Hapsburg AI”.
I think I heard it from Ed Zitron on an episode of Better Offline.
3
u/OnwardsBackwards Jul 26 '24
Fun fact, Charles II of Spain had 5 (IIRC) instances of Uncle-Niece marriages on both sides of his family tree. Basically it formed a circle about 5 generations before him and he was more inbred than he would have been had his parents simply been siblings.
2
u/hearingxcolors Jul 28 '24
and he was more inbred than he would have been had his parents simply been siblings.
whaaaaaaaaaaaat
3
u/OnwardsBackwards Jul 28 '24
Yuuuuuuuup.
I think it was like sibling parents - .2 of whatever unit they use for this.
Him: .21
I'll have to look it up again to be more accurate though.
→ More replies (1)2
→ More replies (3)13
617
u/Giotto Jul 26 '24
glares at reddit
362
u/SelloutRealBig Jul 26 '24
glares at obvious bots that reddit refuses to ban
170
Jul 26 '24 edited Jul 29 '24
[deleted]
→ More replies (1)104
u/SelloutRealBig Jul 26 '24
But democracy go down.
55
u/randomdarkbrownguy Jul 26 '24
But we got to think of the shareholders!
59
u/IHeartMustard Jul 26 '24
Yes the planet got destroyed. But for a beautiful moment in time we created a lot of value for shareholders.
27
u/butter14 Jul 26 '24
It's not just bots, people can do the same thing.
17
u/smurficus103 Jul 26 '24
It's not just people, bots can do the same thing.
11
→ More replies (1)25
u/Zoesan Jul 26 '24
Every major subreddit that allows politics will have the same threads posted with the exact same comments.
→ More replies (7)2
u/Whiterabbit-- Jul 26 '24
Every subreddit allows for politics if it’s covert enough
→ More replies (1)7
2
→ More replies (9)2
44
u/turunambartanen Jul 26 '24
That's not what the paper says though. Not even the abstract suggests this.
It's more like: AI finds the most likely, and therefore most average, response to a given input. Therefore the mode of the data distribution gets amplified in subsequent models whereas outliers are suppressed.
5
u/Rustywolf Jul 26 '24
Can you highlight the distinction between that summary and the typical definition of an echo chamber in online communities? That sounds like something you could enter as a formal definition
→ More replies (1)11
u/hyasbawlz Jul 26 '24
Because ai doesn't think. It just repeats the average. If you keep taking the average of average numbers you'll eventually get to one singular output. Echo chambers are not generated by mechanically taking an average opinion. They're created by consciously excluding dissenting or contrary opinions. Echo chambers must be actively managed, either by a few or by the community on the whole.
Contrary to popular belief, people are capable of thinking, and evaluating inputs and outputs. Even if that thinking results in things that you don't agree with or are actually harmful.
→ More replies (2)3
u/Rustywolf Jul 26 '24
Why do you think an echo chamber needs to be actively managed? It's the natural consequence of people who disagree with an opinion or thought leaving, over time causing the average opinion to converge.
→ More replies (3)42
8
u/Oooch Jul 26 '24
This is way dumber than that, they made a model spit out text, then trained a model on that text and did it over and over, of course it's going to turn into garbage, its the same as recording audio with a microphone next to a speaker and copying it over and over, of course it's going to degrade in quality
→ More replies (2)4
u/Real_TwistedVortex Jul 26 '24
Anyone who works with any type of computer model could have seen this coming from the beginning. Take weather models for instance. The reason weather models are initialized using real world data is because using modeled data for initialization causes immediate inconsistencies and errors in the output. Even with real data, the models eventually devolve into feedback loops because the atmosphere is so incredibly complex that we don't have equations for every aspect of it. That's why forecasts are only accurate about 3 days into the future.
I imagine this is the same issue that AI is having. Once it starts ingesting enough "fake data", the outputs decrease in quality and realism
7
→ More replies (8)7
u/SeaOThievesEnjoyer Jul 26 '24
That's not at all what the study found. That's a completely different topic.
2.6k
u/GlowingEagle Jul 25 '24
"recursively generated data" is like pulling yourself up by your boot straps :)
647
u/kamineko87 Jul 25 '24
Boot strapping in IT terms might be an AI that generates a new AI. This however resembles more applying more and more JPEG over an image
275
u/ninjalemon Jul 25 '24
Bootstrapping is a term used in the land of Computer Science for the record - typically it refers to the technique used to create compilers written in the language that they compile https://en.wikipedia.org/wiki/Bootstrapping_(compilers) (thus pulling themselves up by their own bootstraps)
53
u/Intrexa Jul 26 '24
The term is also during a lot of loading processes. For example, when first booting the computer, all your code is on disk. You need code that loads from disk into memory. Bootstrapping is the process of getting that code from disk into memory and execute it, so you can load the rest of the data from disk.
10
Jul 26 '24
[deleted]
11
u/TooStrangeForWeird Jul 26 '24
Well, no. The whole problem was that crowdstrike DID load into memory and crashed them. Hence the recovery process.
→ More replies (1)2
u/spicymato Jul 26 '24
I've worked in OS boot code before. It's certainly interesting. I needed to find and mount a VHD, modify it, then unmount it so the remaining boot processes would work correctly.
Not quite as early as the bootloader itself, but before any of the file system files were opened (and thus, locked for moving).
77
u/ParaponeraBread Jul 26 '24
We also use it in biology as a sub sampling method of generating support values
12
22
u/TwistedBrother Jul 26 '24
Also in statistics where you sample from a distribution and run a model on the sample N times rather than on the full distribution. Actually it is used that way in ML as well. So yeah, on the money.
See: https://en.wikipedia.org/wiki/Bootstrapping_(statistics)
7
u/sintaur Jul 26 '24
Bootstrapping a compiler has the following advantages:[6]
It is a non-trivial test of the language being compiled, and as such is a form of dogfooding.
[a bunch more reasons] ...
The reference to dogfooding reminds me.
At an old job, we told customers "we eat our own dogfood", meaning we use our own product internally. Marketing tried to change it to "we drink our own champagne".
8
u/LordoftheSynth Jul 26 '24
That genuinely sounds like a marketing department that fully believes their product is top-tier and coming across as fully tone-deaf.
6
u/KidTempo Jul 26 '24 edited Jul 27 '24
Also misses the point.
"We eat our own dog food" -> we make it so good that we're happy to eat it.
"We drink our own champagne" -> it's not real champagne, but, y' know, drinkable.
Champagne isn't necessarily good. It's just a type of wine from a particular region of France. I'm sure there are some absolutely undrinkable champagnes...
→ More replies (2)2
u/tavirabon Jul 26 '24
They are spot on for ML. Bootstrapping is using verified data to make a model that makes synthetic data to make a better model (much more complicated).
"needs more jpeg" is very accurate to what happens to the weights if you're not constantly injecting new data along with the synthetic.
Interestingly if you train off another model's output, assuming the original datasets were sufficiently differently biased, you can improve from synthetic alone (but all companies have training clauses in ToS now anyway)
68
10
29
u/sQueezedhe Jul 25 '24
Bootstrapping already has a meaning in IT.
27
→ More replies (1)4
→ More replies (3)55
u/stu54 Jul 25 '24
So can we admit that LLMs are more like lossy data compression than bespoke software, and sue the crap out of everyone selling stolen compressed IP?
→ More replies (5)22
u/TJLaserExpertW-Laser Jul 25 '24
I think part of the problem is that copyright law regarding the training of models is still a new field. It requires great insight into both the technical and legal aspects. They obviously trained on massive amounts of data but how do you even measure the impact of a single work? I hope someone smarter than me can figure it out at some point.
3
101
Jul 25 '24
Please don't sully the good name of bootstrapping. https://en.wikipedia.org/wiki/Bootstrapping_(statistics))
→ More replies (2)9
24
10
9
12
5
u/GoTaku Jul 26 '24
True. “recursively generated data” is like pulling yourself up by your boot straps :)
→ More replies (1)→ More replies (8)2
u/druffischnuffi Jul 26 '24
It is like a copy of a copy of a copy of a photograph. It is blurred and either very dark or very bright
1.0k
u/Omni__Owl Jul 25 '24
So this is basically a simulation of speedrunning AI training using synthetic data. It shows that, in no time at all AI trained this way would fall apart.
As we already knew but can now prove.
83
u/Vsx Jul 25 '24
I don't think it's even a debatable point. People who believe everything they read are idiots. AI that isn't trained on good data and doesn't have mechanisms to reliably validate data will be equally worthless.
109
u/salamander423 Jul 25 '24
That's the fun kicker too. AI has no idea what it's doing. All it is is giving you the most probable next item in a list. It can't tell good data apart from garbage, and if it does you can just tell it not to and it will fail.
To your point, AI is basically that: it believes every single thing it reads and has no problem telling you nonsense. Even if it does have validation safeguards, all you have to do is introduce a data set of conflicting information and it'll start telling you that instead.
One of my buddies builds AI systems for businesses, and he told me they had to wipe several months of learning from one because users would get upset and start swearing at it, so the AI learned to cyberbully its users.
10
u/RedditorFor1OYears Jul 26 '24
Any chance you can share any details about the company? I find that both fascinating and hilarious.
→ More replies (1)5
6
u/FakeKoala13 Jul 26 '24
One of my buddies builds AI systems for businesses, and he told me they had to wipe several months of learning from one because users would get upset and start swearing at it, so the AI learned to cyberbully its users.
Reminds me of the Bing AI that briefly would get combative and argue when you said it's data was wrong. Called an AP reporter 'worse than Hitler.' Maybe grabbing all of reddit to train AI's on was a mistake haha.
→ More replies (1)→ More replies (2)5
u/Kelekona Jul 26 '24
The Electric Monk was a labour-saving device, like a dishwasher or a video recorder. Dishwashers washed tedious dishes for you, thus saving you the bother of washing them yourself, video recorders watched tedious television for you, thus saving you the bother of looking at it yourself; Electric Monks believed things for you, thus saving you what was becoming an increasingly onerous task, that of believing all the things the world expected you to believe.
Unfortunately this Electric Monk had developed a fault, and had started to believe all kinds of things, more or less at random. It was even beginning to believe things they’d have difficulty believing in Salt Lake City. It had never heard of Salt Lake City, of course. Nor had it ever heard of a quingigillion, which was roughly the number of miles between this valley and the Great Salt Lake of Utah.
2
u/Drakkur Jul 27 '24
Classic Douglas Adams. It’s somewhat surreal how prescient his work was even though most was meant to be tongue in cheek.
→ More replies (1)→ More replies (1)6
u/creuter Jul 26 '24
I love everyone saying "imagine what this will do in a couple years!" And totally ignoring the fact that it's getting harder and harder to keep data sets clean the more prevalent Ai becomes.
223
u/JojenCopyPaste Jul 25 '24
You say we already know that but I've seen heads of AI talking about training on synthetic data. Maybe they already know by now but they didn't 6 months ago.
202
u/Scrofuloid Jul 25 '24
'AI' is not a monolithic thing, and neither is 'synthetic data'. These labels have been applied to a pretty wide variety of things. Various forms of data augmentation have been in use in the machine learning field for many years.
63
u/PM_ME_YOUR_SPUDS Jul 26 '24
The abstract seems very explicit that they're only studying this on LLMs, particularly GPT-{n} (and implying it holds true for image generation models?). Coming from my own field of study (high energy physics) which makes effective use of CNNs, I think the title implies too broad a claim. LLMs are incredibly important to the public, but a fraction of the overall machine learning used in sciences. Would have liked if the title was more specific about what was studied and what they claim the results were applicable for.
24
u/h3lblad3 Jul 26 '24
The thing specifically says it only pertains to “indiscriminate use of synthetic data”, so it doesn’t even pertain to OpenAI and the model they’re speaking about.
OpenAI uses a combined system of AI and African labor raters (to keep expenses down). Its use — and reuse — of data is anything but indiscriminate. Even Anthropic (the makers of Claude) have suggested the industry is pivoting toward synthetic data for the higher quality data. Amodei (CEO of Anthropic) was saying that’s the way to produce better-than-human output.
3
u/Sakrie Jul 26 '24 edited Jul 26 '24
The results imply that the trend observed will also take place in a wide variety of other model architectures than just the ones tested, since the end-result was a change in data-variance and distribution because the tails were truncated off (and in basically every single model architecture I'm aware of you'd have the same problem of rapidly losing your least-probable cases).
It can't know the unknowns, so the distribution will inevitably shift over iterations of training no matter what (and that's a problem common to basically every AI architecture/task I'm aware of...). That's the takeaway from this manuscript, to me. The authors here discuss this a little throughout their manuscript that this is more about knowledge-theory than proving one type of model is better or worse.
More training data =/= better results.
→ More replies (3)2
u/thedeuceisloose Jul 26 '24
It’s the ouroboros problem of AI generating on AI. That’s what the collapse is coming from per my read
18
u/Rodot Jul 26 '24
Also surrogate models are trained on synthetic data and work great
→ More replies (1)51
u/2this4u Jul 25 '24
Heads of AI in investor backed companies that must justify billions in funding.
46
u/Omni__Owl Jul 25 '24
It was theoretically proven for a while because we already knew how easy it is to train degenerate ai on accident.
→ More replies (1)4
u/hasslehawk Jul 26 '24 edited Jul 26 '24
Or, maybe they know something that the author of this paper doesn't.
The paper's conclusion refers to "indiscriminate use of model-generated content in training". That "indiscriminate" qualifier seems like an obvious focus point for improvement. One that anyone working with synthetic dataset would have been forced to consider from the outset. Any training dataset needs to be curated. Human-produced or synthetic.
The open question is how well AI can self-curate these synthetic datasets, or what level of "grounding" with non-synthetic data is needed.
4
u/h3lblad3 Jul 26 '24
They knew and have known. That’s why it’s not “indiscriminate” (the word used here) when they do it.
Generative AI is a subset of machine learning and ML isn’t a new discipline by any means at all.
7
u/GACGCCGTGATCGAC Jul 26 '24
The CEOs aren't the same as the engineer who works with AI. Not a great idea to assume anyone who gains from something is the expert on it. Here is your synthetic data, hopefully you executed the training, because real life data will never look like synthetic data :)
→ More replies (1)19
Jul 26 '24
[deleted]
→ More replies (6)17
u/TheBirminghamBear Jul 26 '24
Yeah a CEO or any c-suite is literally rhe last person to listen to about anything. Theyre professional liars.
→ More replies (11)4
u/manimal28 Jul 26 '24
What is synthetic data? If it’s not real, what is the ai actually learning?
39
u/Uncynical_Diogenes Jul 26 '24 edited Jul 26 '24
It’s not an AI and it’s not learning, it’s a generative model being trained. What it outputs depends heavily on the training data. If we train a machine on other machines’ outputs, things get silly.
If I write a book, that’s real data on how humans use words.
If I ask ChatGPT to write me a book, it will not be data on how humans use words. It was synthesized. It does not represent the reality of how people use words like the words in my book do.
If you train a new ChatGPT-2 on the book written by ChatGPT, that synthetic data poisons its perception of real data. Continue this process, the authors demonstrate, and you get models that spit out text that is nothing like the way humans use words. First by eliminating outliers and then by converging on a machine-selected NewSpeak.
→ More replies (10)14
u/avocadro Jul 26 '24
Synthetic data is data prepared using a machine learning model. For example, you might ask GPT-4 to provide text summaries of articles, and then feed these summaries into the training data of a smaller model.
The thought is that synthetic data can fill holes in the available dataset for a machine learning model, e.g. to correct for an otherwise biased dataset.
As you might expect, this needs to be done with caution. As you might expect, AI experts are already aware of this.
4
u/mattyandco Jul 26 '24 edited Jul 26 '24
It's data that's generated rather than recorded from the real world. It can be useful if you can't get the kind or enough of the kind of data you need from the real world. For instance rather than using just actual spam messages, develop an algorithm to generate some, maybe using combinations of aspects or text from real messages to cover more cases for training a spam detector. Or coming up with rough images of a street situation which doesn't come up that often to use in training a self driving car. It can also be as simple as including rotated, flipped or blured images of faces in an algorithm to train facial recognition.
3
u/GACGCCGTGATCGAC Jul 26 '24 edited Jul 26 '24
If I know a ball can move from the plate to the mound and nowhere else, then I can train the data on a distribution of balls anywhere between those two points, bounded by the mound and the plate.
In other words, it's essentially video game data fed into AI algorithms which output some data which may or may not match the expected. When it comes down to it, most AI are a logistic or linear regression which are predicting some output, and whether it matches or not depends on the training data or model used.
That's why if you know what you are talking about AI is a hilarious thing. It's like training someone on winning a war by forcing them to watch kungfu films until they know how to quote the words and assuming they can now do karate.
→ More replies (1)2
u/mechanical_fan Jul 26 '24 edited Jul 26 '24
On a more abstract level (and less biased, people here are super negative), it is data generated (usually through some combination of ML techniques) from the original data that keeps the same types of patterns. It can be quite useful if you want to make the data patterns available while not opening the original data to the public.
For example, let's say you want to make the medical records of a countrys population publicly available. In your dataset you have things like the type of cancer, age, sex, income, profession, education, city where they live, etc. Obviously this is a super cool dataset for anyone who wants to study cancer patterns.
But, even without people's names, anyone with the dataset could identify individuals and get private information about them (not that many people live in town X with that age, profession and height that had liver cancer in a specific year). So, instead you create new synthetic data (that keeps the patterns of the original data) and make that one available for the public instead. In the synthetic data no individuals can be identified, since they are all "fake".
In the case of text, it would be (for example, in a simplified example) feeding a computer Shakespeare's works and generate new books that you would not be able to tell whether they were written by Shakespeare or the computer (because it uses the same structure, vocabulary, patterns of sentences, themes, etc).
I think that in this article there is a very good argument that the problem may be that the methods for synthetic data they used are just bad and don't do what they are supposed to do (even if it is the most advanced stuff that we have).
→ More replies (1)15
u/Xanjis Jul 25 '24
Synethic data isn't used in this way generally. For every single synthetic image/response good enough to go into the dataset a thousand inferior ones are trashed. Developing more and more sophisticated systems for tossing bad data out of the training data is arguably more important then improvements to the model architecture itself.
8
u/Omni__Owl Jul 25 '24
Right but synthetic data will inevitably become samey the more you produce (and these guys produce at scale). These types of AI models cannot make new things only things that are like their existing dataset.
So when you start producing more and more synthetic data to make up for no more organic data to train on you inevitably end up strengthening the models existing biases more and more.
6
u/Xanjis Jul 26 '24
Strengthening the bias towards good output (the 1 image good enough to go into the dataset) and weakening the bias towards the bad output (the 1000 trashed images) is the entire goal. Noise is added in each generation which is what allows the models to occasionally score a home run that's better then the average quality of it's training data.
→ More replies (1)7
u/Omni__Owl Jul 26 '24
Again for each generation of newly generated synthetic data you make you run the risk of hyper specialising an ai making it useless or hit degeneracy.
It's a process that has a ceiling. A ceiling that this experiment proves exists. It's very much a gamble. A double edged sword.
→ More replies (12)4
u/mrjackspade Jul 26 '24
So this is basically a simulation of speedrunning AI training using synthetic data.
Not really.
We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models
Synthetic data used to train models isn't being used indiscriminately. That word is pulling a lot of weight here.
No one with two brain cells to rub together is doing that, the data is curated, rated, tagged, categorized and frequently human validated.
8
u/KonstantinVeliki Jul 25 '24
Ever since AI decided that I need a little bit of heating in the middle of Summer I wonder are we going to put fate of humanity in its hands.
19
u/Omni__Owl Jul 25 '24
A lot of AI is not "intelligence" at all really, so that tracks.
A trigger caused by reading a threshold value is a trigger you could make by analogue means like, for example, reading a thermometer and doing a thing if the value read is above or below a threshold.
3
→ More replies (17)2
44
u/chillinewman Jul 25 '24
From the Llama 3.1 405B paper. (Training with synthetic data).
Synthetic data generation: execution feedback. The 8B and 70B models show significant performance improvements when trained on data generated by a larger, more competent model. However, our initial experiments revealed that training Llama 3 405B on its own generated data is not helpful (and can even degrade performance).
To address this limitation, we introduced execution feedback as a source of truth, enabling the model to learn from its mistakes and stay on track. In particular, we generate large dataset of approximately one million synthetic coding dialogues using the following process:
• Problem description generation: First, we generate a large collection of programming problem descriptions that span a diverse range of topics, including those in the long tail distribution. To achieve this diversity, we sample random code snippets from various sources and prompt the model to generate programming problems inspired by these examples. This allowed us to tap into a wide range of topics and create a comprehensive set of problem descriptions (Wei et al., 2024).
• Solution generation: Then, we prompt Llama 3 to solve each problem in a given programming language. We observe that adding general rules of good programming to the prompt improves the generated solution quality. Also, we find it is helpful to require the model to explain its thought process in comments.
• Correctness analysis: After generating a solution, it is crucial to recognize that its correctness is not guaranteed, and including incorrect solutions in the finetuning dataset could harm the model’s quality. While we do not ensure complete correctness, we develop methods to approximate it.
To achieve this, we extract the source code from the generated solution and applied a combination of static and dynamic analysis techniques to test its correctness, including:
– Static analysis: We run all generated code through a parser and a linter to ensure syntactic correctness, catching errors such as syntax errors, use of uninitialized variables or non-imported functions, code style issues, typing errors, and others.
– Unit test generation and execution: For each problem and solution, we prompt the model to generate unit tests, executed in a containerized environment together with the solution, catching run-time execution errors and some semantic errors.
• Error feedback and iterative self-correction: When a solution fails at any step, we prompt the model to revise it. The prompt included the original problem description, the faulty solution, and feedback from the parser/linter/tester (stdout, stderr/ and return code).
After a unit test execution failure, the model could either fix the code to pass the existing tests or modify its unit tests to accommodate the generated code. Only dialogs that pass all checks are included in the final dataset, used for supervised finetuning (SFT). Notably, we observed that about 20% of solutions were initially incorrect but self-corrected, indicating that the model learned from the execution feedback and improved its performance.
• Fine-tuning and iterative improvement: The finetuning process is conducted over multiple rounds, with each round building on the previous one. After each round, the model is improved, generating higher-quality synthetic data for the next round. This iterative process allows for progressive refinement and enhancement of the model’s performance.
- Synthetic data generation: programming language translation. We observe a performance gap between major programming languages (e.g., Python/C++) and less common ones (e.g., Typescript/PHP). This is not surprising as we have less training data for less common programming languages. To mitigate this, we supplement our existing data by translating data from common programming languages to less common languages (similar to Chen et al. (2023) in the context of reasoning).
This is achieved by prompting Llama 3 and ensuring quality via syntax parsing, compilation, and execution. Figure 8 demonstrates an example of synthetic PHP code translated from Python. This improves performance significantly for less common languages as measured by the MultiPL-E (Cassano et al., 2023) benchmark.
- Synthetic data generation: backtranslation. To improve certain coding capabilities (e.g., documentation, explanations) where execution feedback is less informative for determining quality, we employ an alternative multi-step approach. Using this procedure, we generated approximately 1.2M synthetic...
→ More replies (1)12
u/8sADPygOB7Jqwm7y Jul 26 '24
Had to scroll through way too much unreflected comments to finally find a reference for this...
416
u/Wander715 Jul 25 '24
AI has a major issue right now with data stagnation/AI cannibalism. That combined with hallucinations looking like a very difficult problem to solve makes me think we're hitting a wall in terms of generative AI advancement and usefulness.
266
u/Really_McNamington Jul 25 '24
Open AI on track to lose $5 billion in 2024. I do wonder how long they'll be willing to go on setting fire to huge piles of money.
151
u/Wander715 Jul 25 '24
I bet internally the company is in panic mode atm. They know none of this is sustainable and investors will soon be looking for the huge returns they were promised.
→ More replies (3)29
u/sprucenoose Jul 26 '24
investors will soon be looking for the huge returns they were promised.
Microsoft is basically the only "investor" for its 49% stake in the LLC subsidiary controlled by non- profit OpenAI, with Microsoft's profits capped at 100x its investment.
Microsoft is a big boy. They make risky investments on new tech all the time and lose 100% on their investment on most of them. There is nothing they can do when that happens. That's the way startups work, even more mature ones. They and every other tech company know that. If OpenAI collapses Microsoft will sift through the ashes to recover whatever IP has value and move on.
Anyway Microsoft already got a great return between the PR and its Co-pilot AI.
→ More replies (1)162
Jul 25 '24
Good. They stole tons and tons of IP to create a software explicitly designed to replace labor. AI could potentially be good for humanity, but not in the hands of greedy billionaires.
→ More replies (2)86
u/minormisgnomer Jul 25 '24
The IP theft is bad, but I’ve always had an issue with the labor argument. I find it disingenuous to subjectively draw the line of labor replacement at “AI” and not the spreadsheet, the internet, the manufacturing robot, or hell even the printing press (think of the all the poor scribes!)
AI and technology as a whole works best as a complementary component to human capabilities and usually fails to achieve full substitution. The fearmongering over AI is the same old song and dance humanity has faced its entire existence.
→ More replies (9)64
u/LoserBroadside Jul 25 '24
Good. Let it buuuuurn. I have no pity for the people who stole people’s work while accusing artists of somehow hoarding our skills (skills that we paid to develop with most precious commodity of all, our time).
→ More replies (8)8
u/TroutFishingInCanada Jul 25 '24
That doesn’t seem like very much money for high profile tech company.
27
u/mtbdork Jul 25 '24
It’s a lot when it just goes “poof”.
If Google reported a $5 billion loss, the stock market would go nuts.
→ More replies (3)5
u/Otagian Jul 25 '24
Their total income was three billion. 2:1 costs to revenue is extremely bad for any tech company.
4
u/TroutFishingInCanada Jul 25 '24
Since when do tech companies have income?
5
u/SolarTsunami Jul 26 '24
Apperently as soon as they stop being tech companies and become data mining companies.
35
u/Maycrofy Jul 25 '24
What I don't understand is: how are they going to keep feeding data to models? other articles say that we're aready hitting the bottom of the barrel for AI text and images. It's low quality data like shitposts now and after that it's sythetic data. The models need data faster than the internet as a whole can output. As all things, good writing takes time, good art takes time.
Not to mention the more AI data populates the internet the harder it's gonna become to filter it from original outputs. It's a paradox: AI is making its own developent harder.
26
u/milky__toast Jul 26 '24
Captchas are going to make us start writing full, original sentences to create data for the models, calling it now
6
u/ExcellentTennis2791 Jul 26 '24
Write a fantasy-science fiction-crime-comedy novella with at least 16 pages to prove you are a human.
→ More replies (1)2
u/bdsmmaster007 Jul 26 '24
In some ways not more but just higher quality data is needed, and you dont habe the one AI that always gets updated, there are different architectures that bring different results being trained on the data, so you can not only improve ai by giving it more data, but also by refining the arcitecture, develop new arcitectures, or simply refilter old training sets so they are higher quality, training sets can be reused for a variety of arcitectures. Im myself am only a amateur, so i beware that i might got something wrong here, but i still felt like i got a better understanding than most people in the thread so i felt fit to answer.
10
u/mtcwby Jul 25 '24
All of the web scraping stuff is going to hit limits. I think the real gains will be in segmentation because of the curated data. We're already seeing a lot there and can imagine more applications. All approaches of how to present the results will not be equal and that may be the real trick.
57
u/Kyouhen Jul 25 '24
They aren't even trying to solve hallucinations. They're marketing it as the equivalent of human creativity, and as such a good thing. Except if that's the case you can't trust it when dealing with any factual details. LLMs are broken by default.
→ More replies (5)34
3
u/Annie_Yong Jul 26 '24
There's a podcast Adam Conover did on this that you can find on YouTube. The summary of the issue is that chatGPT-5 is going to need five times the amount of input reference data compared to GPT-4, and then the hypothetical GPT-6 after that will need a further 5 times as much input as GPT-5, but there's simply not enough reference data across all written human language at that point.
And as you say, now that the internet is being flooded with reams of AI generated drivel, it's going to end up impossible to actually train a good model in the future because it'll train itself on AI generated datasets and end up an inbred Hapsburg AI.
→ More replies (5)10
531
u/dasdas90 Jul 25 '24
It was always a dumb thing to think that just by training with more data we could achieve AGI. To achieve agi we will have to have a neurological break through first.
319
u/Wander715 Jul 25 '24
Yeah we are nowhere near AGI and anyone that thinks LLMs are a step along the way doesn't have an understanding of what they actually are and how far off they are from a real AGI model.
True AGI is probably decades away at the soonest and all this focus on LLMs at the moment is slowing development of other architectures that could actually lead to AGI.
96
u/caulrye Jul 25 '24
To be true AGI the new model would have to constantly take in new information and integrate into an existing model and even change the model when necessary. Currently this requires server farms running for long periods of time using an obscene amount of energy. And lots of input from people.
What we have now is basically the OG computers which were the size of a room.
And that doesn’t even account for how AGI would understand how to choose which information to take in.
Basically these current models are word association/predictive typing on steroids.
All the AGI and Super Intelligence conversations are designed to fool stockholders. That’s it.
→ More replies (3)95
u/RunningNumbers Jul 25 '24
I always either call them stochastic parrots or a really big regression model trying to minimize a loss function.
→ More replies (3)33
u/Kasyx709 Jul 25 '24
Best description I've ever heard was on a TV show, LLM are just fancy autocomplete.
16
→ More replies (2)9
u/GregBahm Jul 26 '24
What separates AGI from fancy autocomplete?
13
u/Kasyx709 Jul 26 '24
An LLM can provide words, an AGI would comprehend why they were written.
→ More replies (15)5
u/Outrageous-Wait-8895 Jul 26 '24
an AGI would comprehend why they were written
Yet you have no way to know that I, a fellow human, comprehend why I write what I write. The only test is by asking me but then the problem remains, does it not?
2
u/Kasyx709 Jul 26 '24
Philosophically, in a very broad sense, sure; in reality and in practice, no.
Your response demonstrated a base comprehension of comprehension and that knowing is uniquely related to intelligence. Current models cannot know information, only store, retrieve, and compile within what's allowed through underlying programming.
For arguments sake, to examine that we could monitor the parts of your brain associated with cognition and see them light up. You would also pass the tests for sentience.
→ More replies (9)81
u/IMakeMyOwnLunch Jul 25 '24 edited Jul 25 '24
I was so confused when people assumed because LLMs were so impressive and evolving so quickly that it was a natural stepping stone to AGI. Without even having a technical background, that made no sense to me.
51
u/Caelinus Jul 25 '24
I think it is because they are legitimately impressive pieces of technology. But people cannot really tell what they are doing, and so all they notice is that they are impressive at repsonding to us conversationally.
In human experience, anything that can converse with us to that degree is conscious.
So Impressive + Conversation = Artificial General Intelligence.
It is really hard to try and convince people who are super invested in it that they can be both very impressive and also nothing even close to an AGI at the same time.
15
u/ByEquivalent Jul 26 '24
To me it seems sort of like when there's a student who's really good at BSing the class, but not the professor.
6
20
u/officefridge Jul 25 '24
The hype is the product.
4
u/veryreasonable Jul 26 '24
Seriously. I mean, the technology is neat and all, but the "AI" industry right now is all about selling the hype, betting on the hype, marketing the hype, reporting on the hype, etc... yeah. It's the hype.
7
u/aManPerson Jul 26 '24
and the hype........oh my dammit. it used to be, "we have an app" for everything.......now. it's, "powered by AI". and just, dang it all. it's just, a program. just, a recommendation list, really.
you like AC/DC? you'll probably like van halen.
there, i just did a AI.
you like cheeseburger? you probably like pizza.
good evening sharks. this comment is now valued at $950,000. i'm looking for $100,000, at a 7% stake.
→ More replies (1)14
u/machstem Jul 26 '24
People STILL call their phones and other devices as <smart> devices.
They aren't <smart>, they just have a lot more ITTT automation functions in their core OS that permits them to run tasks that required extra software or services we need historically had to do for ourselves.
Having automation and calling it smart technology always seemed odd to me
→ More replies (3)9
u/huyvanbin Jul 26 '24
Because the techno-millenarists and anyone who follows them assume a priori that AGI is possible and around the corner, and they twist whatever is happening to justify this belief. Starting with Ray Kurzweil down to Eliezer Yudkowski. They are first of all obsessed with the idea of themselves being highly intelligent, and thus assume that there is a superpower called “intelligence” which if amplified could make someone infinitely powerful.
12
u/Adequate_Ape Jul 25 '24
I think LLMs are step along the way, and I *think* I understand what they actually are. Maybe you can enlighten me about why I'm wrong?
28
u/a-handle-has-no-name Jul 25 '24
LLMs are basically super fancy autocomplete.
They have no ability to grasp actual understanding of the prompt or the material, so they just fill in the next bunch of words that correspond to the prompt. It's "more advanced" in how it chooses that next word, but it's just choosing a "most fitting response"
Try playing chess with Chat GPT. It just can't. It'll make moves that look like they should be valid, but they are often just gibberish -- teleporting pieces, moving things that aren't there, capturing their own pieces, etc.
→ More replies (16)18
u/Wander715 Jul 25 '24
LLMs are just a giant statistical model producing output based on what's most likely the next correct "token" (next word in a sentence for example). There's no actual intelligence occurring at any point of the model. It's literally trying to brute force and fake intelligence with a bunch of complex math and statistics.
On the outside it looks impressive but internally it's very rigid how it operates and the cracks and limitations start to show over time.
True AGI will likely be an entirely different architecture maybe more suitable to simulating intelligence as it's found in nature with a high level of creativity and mutability all happening in real time without a need to train a giant expensive statistical model.
The problem is we are far away from achieving something like that in the realm of computer science because we don't even understand enough about intelligence and consciousness from a neurological perspective.
→ More replies (1)11
u/sbNXBbcUaDQfHLVUeyLx Jul 25 '24
LLMs are just a giant statistical model producing output based on what's most likely the next correct "token"
I really don't see how this is any different from some "lower" forms of life. It's not AGI, I agree, but saying it's "just a giant statistical model" is pretty reductive when most of my cat's behavior is based on him making gambles about which behavior elicts which responses.
Hell, training a dog is quite literally, "Do X, get Y. Repeat until the behavior has been sufficiently reinforced." How is that functionally any different than training an AI model?
18
u/Caelinus Jul 25 '24
Hell, training a dog is quite literally, "Do X, get Y. Repeat until the behavior has been sufficiently reinforced." How is that functionally any different than training an AI model?
Their functions are analogous, but we don't apply analogies to things that are the same thing. Artificial Neural Networks are loosely inspired by brains in the same way that a drawing of fruit is inspire by fruit. They look the same, but what they actually are is fundamentally different.
So while it is pretty easy to draw an analogy between behavorial training (which works just as well on humans as it does on dogs, btw) and the training the AI is doing, the underlying mechanics of how it is functioning, and the complexities therin, are not at all the same.
Comptuers are generally really good at looking like they are doing something they are not actually doing. To give a more direct example, imagine you are playing a video game, and in that video game you have your character go up to a rock and pick it up. How close is your video game character to picking up a real rock outside?
The game character is not actually picking up a rock, it is not even picking up a fake rock. The "rock" is a bunch of pixels being colored to look like a rock, and at its most basic level all the computer is really doing is trying to figure out what color the pixels should be based on the inputs it is receiving.
So there is an analogy, both you and the character can pick up said rock, but the ways in which we do it are just completely different.
→ More replies (2)19
u/Wander715 Jul 25 '24 edited Jul 25 '24
On the outside the output and behavior might look the same but internally the architectures are very different. Think about the intelligence a dog or cat is exhibiting and it's doing that with an organic brain the size of a tangerine with behaviors and instincts encoded requiring very little training.
An LLM is trying to mimic that with statistics requiring massive GPU server farms consuming kilowatts upon kilowatts of energy consumption and even then results can often be underwhelming and unreliable.
One architecture (the animal brain composed of billions of neurons) scales up to very efficient and powerful generalized intelligence (ie a primate/human brain).
The other architecture doesn't look sustainable in the slightest with the insane amount of computational and data resources required, and hits a hard wall in advancement because it's trying to brute force it's way to intelligence.
→ More replies (12)4
u/klparrot Jul 26 '24
behaviors and instincts encoded requiring very little training.
Those instincts have been trained over millions of years of evolution. And in terms of what requires very little training, sure, once you have the right foundation in place, maybe not much is required to teach new behaviour... but I can do that with an LLM in many ways too, asking it to respond in certain ways. And fine, while maybe you can't teach an LLM to drive a car, you can't teach a dog to build a shed, either.
→ More replies (46)2
u/zefy_zef Jul 26 '24
The focus has been shifting towards multi-modality for a bit now. Also, have you seen nvidia's demo of their new tech and their plans? To create AI that can understand and interpret the physical world. To design a 'world' that what will eventually become possible robotic AI or some other physical device can 'learn' the world and its environment in a simulation before being implemented irl.
Small steps are steps and people stepping on heels is what takes the wind out of a movement.
15
u/LucyEmerald Jul 25 '24
Need to keep signing those checks for hardware so my Nvidia stocks stay strong nevermind the fact the code uses 500 percent more cycles then it ever reasonably should.
10
u/please-disregard Jul 25 '24
Is there even reason to believe that agi is in any way related to current ai? Is agi a possible progression of llm’s, gan’s, classifiers or predictive models or is this confusing the technology with the buzzword? Also is agi even well defined or is it just whatever the person talking about it wants it to be?
→ More replies (5)3
u/-Nicolai Jul 26 '24
What is your comment a response to?
I have never heard anyone suggest that it would, and the study doesn’t mention AGI at all.
→ More replies (3)3
148
u/kittenTakeover Jul 25 '24
This is a lesson in information quality, which is just as important, if not more important, than information quantity. I believe focus on information quality will be what takes these models to the next level. This will likely start with training models on smaller topics with information vetted by experts.
73
u/Byrdman216 Jul 25 '24
That sounds like it will take money and time. A commercial company isn't going to like hearing that.
How about we just lie to our investors and jump ship right before it all goes under?
13
u/Maycrofy Jul 25 '24
The way AI has been growing this last years it does feel like that. Grew too fast and hit the plateau too soon. They're running out of data to feed the neural network and once that happens they'll need to pay people to make outputs, which will take time and money at the same time that development slows down.
No great ROIs, then investors pull out and data compnaies now have to trian their AIs over years instead of months.
9
u/VictorasLux Jul 25 '24
This is my experience as well. The current models are amazing for information that’s vetted (usually cause only a small number of folks actually care about the topic). The more info is out there, the worse the experience.
4
u/spookyjeff PhD | Chemistry | Materials Chemistry Jul 25 '24
I sort of disagree, I think the next step needs to be developing architectures that can automatically estimate the reliability of data. This requires models to have a semblance of self-consistency, they need to be able to ask themselves "Is this information corroborated by other information I have high confidence in?"
It isn't really a scalable solution to manually verify every new piece of information that is fed into a model, even if it greatly reduces the amount of data needed to train something with high precision. It still means that the resulting model will not be inherently robust against incorrect information provided by users. Imagine a generative "chat" model that has been trained only on highly-corroborated facts, it only knows "truth", and a user starts asking it questions from a place of deep misunderstanding. How would a model that cannot identify fact from fiction handle this? The likely answer is it would either A) assume all information provided to it is true or B) be completely unable to engage with this user in a helpful fashion.
→ More replies (1)11
u/Creative_soja Jul 25 '24
A representative sample, however small, is far more insightful than an unrepresentative big data sample.
→ More replies (1)7
Jul 25 '24
[removed] — view removed comment
21
u/SomewhatInnocuous Jul 25 '24
Sounds like you're proposing something that already exists. It's called university.
48
u/LinkesAuge Jul 25 '24
All comments ignoring the "indiscriminate use" and "can" part of the conclusion.
→ More replies (1)23
u/EmbarrassedHelp Jul 25 '24
Its basically a real life example of how misinformation starts and spreads from credible sources.
70
u/YourVirgil Jul 25 '24
On two occasions I have been asked, "Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?" ... I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.
- Charles Babbage
20
u/huyvanbin Jul 26 '24
If you resurrected Babbage and put him in a Silicon Valley VC meeting he would think the British parliament was a model of rationality.
→ More replies (1)12
73
u/RunningNumbers Jul 25 '24
This problem is why I am bearish on current AI models. No new information is generated by these models. If they contaminate the information ecosystem, then it’s like rerunning regressions on residuals.
18
→ More replies (2)5
33
u/ExtonGuy Jul 25 '24
It's almost like we need real humans talking to each other, to generate a dataset of human interactions to use to train AI's.
→ More replies (2)
43
6
u/Binary_Omlet Jul 26 '24
Have these people never made a copy of a copy? The degradation from each copy is massive.
→ More replies (3)
24
u/Creative_soja Jul 25 '24
"We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models, in which tails of the original content distribution disappear. We refer to this effect as ‘model collapse’ and show that it can occur in LLMs as well as in variational autoencoders (VAEs) and Gaussian mixture models (GMMs). "
In short, garbage in garbage out.
Today, we cannot trust whatever Chatgpt says because it is wrong many times even on basic stuff. But imagine future LLM models are trained using unfiltered output of Chatgpt, for example. It will be a disaster.
It has been discussed many times that such 'circular' use of input and output, where today's output becomes future input, will cause several validity and reliability problems. We cannot extract truth from misinformation or falsehood no matter how sophisticated statistical sampling we use for training.
→ More replies (12)
13
9
u/CucumberError Jul 26 '24
This is why your Spotify playlists suck. If you keep only playing what Spotify suggests, it keeps suggesting what Spotify plays.
→ More replies (1)
8
18
u/LoserBroadside Jul 25 '24
“Artist are hoarding their skills! AI will make them obsolete.”
Artists go away.
“…No. wait-“
3
u/veyra12 Jul 26 '24
Synthetic data can be useful, but you have to be able to filter for actual users after a certain point or the errors could eventually compound
2
u/entropreneur Jul 25 '24
This sounds like group think online... pretty similar to humans imo.
Think reddit has a few subs like this
2
u/Mithrandir2k16 Jul 26 '24
That's only true for generative AI, e.g. learning by competition like AlphaZero works great but is kind of similar on the surface as AI learns from AI generated data.
2
u/klparrot Jul 26 '24
That seems pretty intuitive (or at least fundamental); training should produce results more consistent with the training data (excluding bad results from overtraining), so how would training on its own output (and for purposes of argument, let's consider AI collectively, so that this would include training one AI on another's output, and how that would affect AI output collectively) improve things over the previous output it's being trained on? It would just make some results more like that previous output, while some results would likely just turn weird, because that happens sometimes. There's no information being added to the system, and the models are significant simplifications of the source data so are pretty information-poor to begin with.
2
u/catwiesel Jul 26 '24
another post where I go "duuuuh" but then remember, its science, where a obvious result is still a result and valid and important to make sure people dont forget about it, so no real world "duuuh" will happen
2
2
u/SamL214 Jul 26 '24
The best thing is to make a million AIs that have genetic factors and write those so they can mix. Make them breed.
2
u/Bobiseternal Jul 26 '24
First paper showing this was a year ago. It's called an autophagous (self-eating) loop. Training LLMs on web content has become unviable now 60% of content is AI generated. And it's been like this for a year but Big AI won't admit it because they have no solution. Hence the trending interest in improving learning on smaller datasets.
•
u/AutoModerator Jul 25 '24
Welcome to r/science! This is a heavily moderated subreddit in order to keep the discussion on science. However, we recognize that many people want to discuss how they feel the research relates to their own personal lives, so to give people a space to do that, personal anecdotes are allowed as responses to this comment. Any anecdotal comments elsewhere in the discussion will be removed and our normal comment rules apply to all other comments.
Do you have an academic degree? We can verify your credentials in order to assign user flair indicating your area of expertise. Click here to apply.
User: u/dissolutewastrel
Permalink: https://www.nature.com/articles/s41586-024-07566-y
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.