Sensor-enhanced LLM for Smart Health Systems

22^nd July 2025, 13:00 Ashton Lecture Theatre
Guoliang Xing
The Chinese University of Hong Kong

Abstract

The widespread adoption of Large Language Models (LLMs) has fundamentally transformed modern AI development. However, current deployments face critical challenges in bridging the gap between digital intelligence and physical environments, particularly in processing heterogeneous multi-modal sensory data, operating within the computational constraints of edge devices, and safeguarding user privacy.

In this talk, I will introduce SensorLLM, a novel framework that seamlessly integrates real-world sensing capabilities with LLMs for smart health applications. SensorLLM features three core innovations: (1) an edge-cloud collaborative architecture that enables open-class classification on resource-constrained devices, (2) cost-efficient encoding schemes ensuring secure and privacy-preserving Transformer inference across distributed systems, and (3) an intelligent orchestration system that coordinates multi-modal sensors to address complex user queries.

I will demonstrate SensorLLM's transformative impact through three smart health applications. DrHouse functions as an LLM-based virtual consultation system that synthesizes wearable sensor data with medical expertise to deliver personalized clinical recommendations. Nuna, an LLM-powered smart necklace, enables intuitive emotional tracking and continuous health monitoring for everyday users. KoalaFM represents the first LLM-enabled platform for early diagnosis, personalized intervention, and complex cross-disease analysis of aging-related degenerative conditions, currently undergoing validation through a comprehensive five-year clinical trial with 1,000 participants.

To conclude, I will briefly highlight other research directions in my group that leverage similar principles of sensor-AI integration, including infrastructure-assisted autonomous driving systems and innovative sensing technologies for dark environments.

Biography

Guoliang Xing is a Professor of Information Engineering at The Chinese University of Hong Kong (CUHK). He received his D.Sc. from Washington University in St. Louis in 2006 and served as faculty at Michigan State University from 2008 to 2017.

Prof. Xing‘s research lies at the intersection of systems and Embedded AI, with transformative applications in healthcare, autonomous driving, and sustainability. Highlights of his work include leading the development and field deployment of large-scale systems for roadside infrastructure-assisted autonomous driving, early clinical diagnosis and treatment of Alzheimer’s Disease, and real-time volcano monitoring. He has received several prestigious awards, including the US NSF CAREER Award (2010), the Withrow Rising Scholar Award from Michigan State University (2014), and the Research Excellence Award from CUHK (2024). His work has received 6 Best Paper Awards, 5 Best Demo/Poster/Artifact Awards, and 7 Best Paper Finalist distinctions at top-tier international conferences such as MobiCom, MobiSys, SenSys, ICNP, and IPSN. He is a Fellow of both the ACM and IEEE.

Additional Materials

• https://staff.ie.cuhk.edu.hk/~glxing/