OpenTSLM: Time-Series Language Models for Reasoning over Multivariate Medical Text- and Time-Series Data
Patrick Langer, Thomas Kaar, Max Rosenblattl, Maxwell A. Xu, Winnie Chow, Martin Maritsch, Robert Jakob, Ning Wang, Juncheng Liu, Aradhana Verma, Brian Han, Daniel Kim, Henry Chubb, Scott Ceresnak, Aydin Zahedivash, Alexander Sandhu, Fatima Rodriguez, Daniel McDuff, Elgar Fleisch, Oliver Aalami, Filipe Barata, Paul Schmiedmayer
Abstract
Large Language Models have shown strong capabilities in interpreting multimodal data but remain limited in handling time-series data. Addressing this gap could help to translate longitudinal and wearable data into actionable insights and patient-facing digital health applications. We propose OpenTSLM, an open-source family of Time Series Language Models integrating time-series as a native modality into pretrained LLMs, enabling natural-language prompting and reasoning over multiple time-series via either soft prompting (OpenTSLM-SoftPrompt) or cross-attention (OpenTSLM-Flamingo). To enable training models for time-series reasoning, we introduce three datasets: HAR-CoT (human activity recognition), Sleep-CoT (sleep staging), and ECG-QA-CoT (electrocardiogram question answering). Across tasks, OpenTSLM models outperform baselines, reaching F1 scores of 69.88% in sleep staging and 67.64% in HAR; OpenTSLM-Flamingo also scales more efficiently in memory as the number and length of time series increase. Expert evaluations with cardiologists show that OpenTSLMs exhibit strong reasoning capabilities on raw ECG data.