Researchers at KAIST, in South Korea, have built a lip-reading AI that makes fewer mistakes by doing something every crossword fan knows: fill in the words you are sure about first, then use them to crack the hard ones. The system turns silent video of a speaker’s mouth into text, the university reported on 13 November.
Why it matters: lip reading by machine could help people who are deaf or hard of hearing follow a conversation, and it could let workers “talk” to devices in places where machines drown out every voice. The catch has always been accuracy. This work shows that changing the order in which the AI writes its answer is enough to move the needle.
Why lips are so hard to read
Say “pack”, “back” and “mack” in front of a mirror. Your lips do almost exactly the same thing each time. The sounds /p/, /b/ and /m/ are nearly impossible to tell apart by sight alone, and spoken language is full of look-alikes like these.
Most speech AI writes a sentence the way we write on paper: left to right, one word after another, never going back. That is risky for lip reading. If the second word of a sentence is a coin toss and the AI picks wrong, the mistake stays, and it can drag the following words down with it.
The team’s own example makes it clear. The speaker says “I pack my bags for the trip”. A left-to-right model reaches the second word, finds “back” slightly more likely than “pack”, writes “back” and moves on. By the time “bags” and “trip” arrive, words that would have settled the question, it is too late.
Certain words first, doubts later
The KAIST system, called DLLM-VSR, works differently. It is built on a diffusion large language model: a language AI that starts with a row of blanks and fills them in over several rounds, in any order it likes, instead of writing strictly from left to right.
In each round, the model commits only to the words it is confident about. In the example, “bags” and “trip” go in early. Those words then act as clues on both sides of the doubtful blank, and with bags and a trip in the sentence, “pack” wins easily. According to the team’s paper, confident words are locked in first and then used as context, before and after, to refine the ambiguous ones.
There is one extra puzzle. A row of blanks needs a length, and silent video does not say how many words are coming. So the system uses the duration of the clip to guess several plausible sentence lengths, writes a candidate sentence for each one, and keeps the candidate it trusts most.
How good is it?
The team tested the model on LRS3, a standard benchmark with 433 hours of labelled video of people speaking English. It reached a word error rate of 19.4%, which means it gets roughly four words out of five right from lip movements alone. The authors report this as the best result among methods that use only LRS3 as labelled training data. In the paper’s comparison table, similar AI systems trained the same way, without extra hints, sit at around 25%.
Two honest notes. First, lip reading is still far from perfect: one word in five is wrong. Second, the paper itself lists a system trained on about eight times more labelled video that scores slightly better, at 19.1%. The KAIST result is impressive because it gets close with far less data, not because it beats everything.
The work was posted as a preprint in May and has been accepted at EMNLP 2026, one of the main conferences on language AI. The first author is postdoctoral researcher Jeong Hun Yeo, with PhD candidates Chae Won Kim and Hyeongseop Rha, in the group of Professor Yong Man Ro at KAIST’s School of Electrical Engineering. The code is public.
Where it could help
- Noisy workplaces: on a factory floor or next to heavy machinery, a microphone hears mostly noise. A camera can still see lips.
- Accessibility: live text from lip movements could support people who are deaf or hard of hearing.
- Robots and wearables: the team wants machines that understand people reliably in everyday life and on industrial sites.
The same “decide the sure things first” idea shows up elsewhere in AI this year. We recently covered AI agents that ask for help only when they are unsure, and a robot that fixes its own fuzzy memory. Knowing what you don’t know turns out to be a very useful skill, for machines too.
What happens next
The model is not ready for your glasses yet. Trying several sentence lengths costs extra computing time, the authors note, and so far it has only been tested in English. The team’s next step is to make it smaller and lighter, so it can run in real time on wearables and robots without an internet connection. If that works, the first place you meet it may be somewhere very loud.
AI courses on Udemy, from the basics to building your own apps, at your own pace.See courses with the discount →Best AI certifications →
Read next
- AI agents that ask for help only when unsure work 3x better
- Robot thinks a chimney is a squirrel, then fixes its own memory
- Sonar in your smartwatch can now track your whole body
The week's best tech stories, free
One email a week with the stories that matter, explained simply. No spam.
Sources
- Lip-reading AI fills gaps in sentences by recognizing most certain words first, KAIST via Tech Xplore, 13 November 2026
- Diffusion Large Language Models for Visual Speech Recognition, Yeo, Kim, Rha and Ro, arXiv preprint, 27 May 2026 (revised 1 September 2026)
- Diffusion Large Language Models for Visual Speech Recognition, Proceedings of EMNLP 2026, ACL Anthology, October 2026
Disclosure: this article contains affiliate links
If you buy through them, the editorial company may earn a commission at no extra cost to you. Our picks are based on specifications, prices and published reviews, not on any payments.
