UNIST researchers used EchoTrace to examine LLM-generated recommendations, user profiles and training data across movie, book and local-business datasets. Direct recommendation models produced out-of-catalog items at rates of 4.07%, 7.40% and 13.14%, respectively; occupation inference from movie histories had a 93.16% error rate. Errors persisted through five rounds of feeding recommendations back into training, with stronger popularity concentration in some systems. These are results for specific datasets and systems. Operators should distinguish observed user behavior from AI-generated data and test distortion across repeated cycles.
Back to Tech
UNIST's EchoTrace tracks hallucinations and bias in recursive recommendations
Beyond accuracy, check item existence, user-profile validity and how generated recommendations re-enter training.