One way to explain this is to realise that LLMs don’t learn about relationships between facts, but between tokens, the linguistic forms that Bender described. The tokens “Tom Cruise’s mother” are linked to the tokens “Mary Lee Pfeiffer”, but the reverse is not necessarily true. The model isn’t reasoning, it’s playing with words, and the fact…

…this kind of short-sighted innovation, just the kind Silicon Valley adores, is untested and untrustworthy and will cause a societal meltdown that will make the results of Hurricane Katrina in New Orleans look like Sunday school, possibly ending in a near-famine if adopted. 

Researchers have found that the buildup of AI-generated content on the web is set to “collapse” machine learning models unless the industry can mitigate the risks. The University of Oxford team found that using AI-generated datasets to train future models may generate gibberish, a concept known as model collapse. In one example, a model started…