Smartwatch body tracking with sonar: Cornell's EchoMotion

Your smartwatch already has a tiny speaker and a microphone. Researchers at Cornell University have taught an ordinary one to use them like a bat: it sends out sound too high for us to hear, listens to the echoes bouncing off your arms and legs, and works out the pose of your whole body. No cameras, no extra gadgets. The system is called EchoMotion, the Cornell Chronicle reported on 11 November.

Why it matters: tracking your whole body today usually means cameras in the room, a VR headset or several sensors strapped to you. A watch you already wear is a much easier ask, and sound gives away far less about you and the people around you than video does.

Sonar on your wrist

The idea is active acoustic sensing, which is a fancy name for sonar. The watch plays ultrasound (sound above the range of human hearing) that spreads around you. Your limbs reflect it, and the microphone picks up the echoes. As you move, the echo pattern changes.

A small deep learning model (an AI that learns from examples) then turns those echo patterns into body positions. It estimates where 12 joints are, from your head down to your ankles. Normal fitness watches can’t do this: their motion sensors only feel how the watch itself moves, not what your legs are doing.

Researcher Saif Mahmud sits at a desk wearing a smartwatch, next to a laptop and a monitor showing a figure of a body pose. — Lead author Saif Mahmud demonstrates EchoMotion with an off-the-shelf smartwatch · Image: Cornell University (provided).

How well does it work?

The team, led by doctoral candidate Saif Mahmud in the SciFi Lab of associate professor Cheng Zhang, first trained the model with 10 volunteers wearing the watch on their non-dominant wrist while doing 18 exercises. Cameras recorded the real body poses so the AI could learn which echoes match which pose, Tech Xplore explains.

On a normal watch: on an unmodified Samsung Galaxy Watch 6, tested with 15 new people, each joint was placed on average within about 10 cm (4 inches) of where it really was.
With a quick setup: a one-time, 12-minute calibration cut that error by more than a third, to about 6.1 cm (2.4 inches).
On other watches: the same model ran without retraining on a Galaxy Watch 7 and a Google Pixel Watch 3 (the reports give no accuracy numbers for those two).

It is also light on the battery. The sensing draws about 12 milliwatts, roughly 40 times less than a wearable camera, and a 30-minute session used 5 to 6 % of a Galaxy Watch 6 battery, Cornell says.

Real rooms, real noise

Ten more volunteers tried it in a small room, in a narrow hallway and next to a noisy road. Furniture and traffic noise didn’t upset it. The hallway was a little harder, because echoes from other people walking by got mixed in. Crowded places such as an underground train are still a challenge; the researchers expect that training with thousands of users instead of dozens will help.

What it could be used for

Workouts: the team’s main focus is fitness, such as a coach in your watch that counts reps and corrects your form during a split squat.
VR and games: body tracking without cameras around the room.
Rehabilitation: following a patient’s progress at home through everyday movement.
Smart glasses: the lab is also exploring pairing EchoMotion with glasses that see what your hands are holding.

Cornell researchers have form with sound tricks: earlier this year they used the same kind of watch sonar to track fingers. And if you like the idea of tech that blends into what you already wear, read about the touchscreen woven into a clothing sleeve.

What happens next

EchoMotion is a research prototype, not an app you can download. The paper was presented at the UbiComp/ISWC conference in Shanghai on 13 October and published in the Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. The big question now is whether watchmakers pick it up: the hardware is already on millions of wrists, so this could arrive as a software update rather than a new gadget.