Tech media outlet 9to5Mac reported yesterday (November 21) that a newly published Apple research paper demonstrates that large language models (LLMs) can accurately recognise user activities by analysing textual descriptions of audio and motion data — a technique that could eventually be implemented on the Apple Watch.

The technology, dubbed “Late Multimodal Sensor Fusion”, combines the reasoning power of LLMs with traditional sensor data to precisely identify what a user is doing, even when raw sensor information is limited or incomplete.

The core innovation is that the LLM never directly processes raw audio recordings or motion data. Instead, it works exclusively with text descriptions generated by smaller, specialised on-device models.

• An audio model produces textual descriptions of the sound environment (e.g., “running water,” “clanging dishes”).
• An IMU-based motion model outputs textual predictions of movement type (e.g., “repetitive up-and-down arm motion”).

This approach both protects user privacy (raw audio never leaves the device) and showcases LLMs’ impressive ability to fuse and reason over multi-source textual information.

To validate the method, researchers utilised the Ego4D dataset, which comprises thousands of hours of first-person video footage. They selected 12 everyday activities — including vacuuming, cooking, washing dishes, playing basketball, and weightlifting — with 20-second clips for each.

The textual descriptions generated by the smaller models were then fed to several leading LLMs, including Google’s Gemini 2.5 Pro and Alibaba’s Qwen-32B, and tested under:

• Zero-shot conditions (no examples provided)
• One-shot conditions (a single example given)

Results showed that even without task-specific training, LLMs significantly outperformed random guessing, achieving strong F1 scores (a balanced measure of precision and recall). Performance improved further when a single reference example was provided.

The study concludes that late fusion using LLMs enables powerful multimodal applications without the need to develop and store specialised models for every scenario, thereby saving both memory and computational resources. Apple has open-sourced the experimental data and code, allowing other researchers to reproduce and build upon the work.

The above content is compiled by ModeZone, a fashion and entertainment magazine.