‘Completely off the rails’: TikTok is scaling back its AI summaries feature after it creates bizarre and inaccurate captions — as if TikTok wasn’t bad enough for misinformation already

TikTok has been testing AI summaries for its videosThe feature is throwing up wildly inaccurate text captionsTikTok says it will now pull back on the technology

In the age of AI deepfakes, it’s a good idea to treat everything you see on social media with a certain degree of skepticism, but the misinformation problem on TikTok has been made worse with some wildly inaccurate AI captions — and it’s bad enough that the video platform is now scaling back this captioning technology.

As reported by Business Insider, TikTok had been testing AI-powered text summaries for videos with a limited number of users. However, after numerous mistakes and hallucinations, the technology is going to be limited to identifying products in videos, rather than fully describing the video’s contents.

Those mistakes and hallucinations included describing a video of celebrity Charli D’Amelio talking to the camera as showing a “collection of various blueberries with different toppings”, and labeling a dog-training video as “a captivating display of intricate origami art, meticulously folded from a single sheet”.

You don’t have to look far on social media to find further examples: there’s what seems to be an image of two cats with the caption “a person demonstrating an impressive new robot arm with multiple dexterous fingers”, for example.

‘Garbage that has nothing to do with the video’

It’s not clear exactly what’s been going wrong that’s causing TikTok’s AI summaries to get the wrong idea so regularly (though presumably the feature did work at least some of the time). Recognizing the contents of images and videos is usually something AI can do pretty reliably.

That clearly hasn’t been the experience of many TikTok users, however. One Redditor described the captions as “completely off the rails”, while another said they were seeing “garbage that has nothing to do with the video” — with the AI summary also serving to distract from the actual caption on the video.

Other examples online show a Kentucky Derby horse race video described as “showcasing an intricate piece of calligraphy”, and a cookery video with an overhead shot of a gray pan getting the label “a single ball bouncing and rolling on a green surface” — although these screenshots could also be faked, of course.

Even as AI is pushed into more and more of our apps and devices, hallucinations and errors remain a significant problem, which AI companies don’t like admitting to. Whether it’s a TikTok video or a legal document, if you’re getting AI to summarize something, you’d be wise to run additional checks.

Source: Latest from TechRadar US in Internet News