Benchmark to Bedside

Building Modular Multimodal Machine Learning Infrastructure for Healthcare at Scale

Slides Dissertation

Counting Exercise

Eighty-four (84) lives were saved by one AI-enabled warning system in one trial.¹

• 84 people, 84 lives are pictured on this slide.

• In a recent trial of an AI early warning system in New York City in two health systems, it was estimated that in one year, an estimated 84 people who might have died, didn’t.

• This is how important AI in healthcare can be: it has the potential to save lives by predicting and preventing adverse outcomes.

• This is the promise of machine learning in healthcare: to use data to predict and prevent adverse outcomes, and ultimately save lives.

• Across 60,893 hospital encounters, the intervention group experienced a 35.6% reduction in instantaneous in-hospital mortality risk (AHR  0.64; 95%CI 0.53–0.78; P < 0.0001) and an 11.2% shorter length of stay (IRR  0.91; 95%CI 0.90–0.93; P < 0.0001) compared to usual care.

• Patients generate digital footprints of their health journey when they visit care providers.

• These footprints are often stored in electronic health records (EHRs), which contain a wealth of information about patients’ medical history, diagnoses, treatments, and outcomes.

• It’s a complex, heterogeneous, and often messy collection of structured codes (diagnoses, labs, medications) and unstructured text (clinical notes).

• This complexity is both a challenge and an opportunity for machine learning in healthcare.

• The richness of EHR data allows for powerful predictive models, but also requires careful handling to ensure reproducibility and clinical relevance.

1. [1], Rossetti et al., 2025, “Real-time surveillance system for patient deterioration: A pragmatic cluster-randomized controlled trial,” Nature Medicine

ML/AI for Health in a Nutshell

• So let’s first look at the basics of machine learning in healthcare.
• Hospitals, clinics, and healthcare systems generate vast amounts of data, including electronic health records (EHRs), medical imaging, and physiological signals.
• Machine learning can be used on parts of this data to predict patient outcomes, assist in diagnosis, and optimize treatment plans.
• The essence is often predicting or diagnosing something about a patient at time T so that we can intervene at time T+1 to improve outcomes.

The Promise and Reality of AI in Healthcare

ArXiv papers with AI and health keywords in title/abstract and FDA-approved AI medical devices ¹:

ML_PATTERN = r"(?ix)\b(machine[ _]?learning|deep[ _]?learning|neural[ _]?networks?|artificial[ _]?intelligence|\bAI\b|random[ _]?forest|support[ _]?vector[ _]?(machine|regression|classif\w+)?|\bSVM\b|gradient[ _]?boost\w*|\bXGBoost\b|\bLightGBM\b|\bCatBoost\b|\btransformer\b|\bBERT\b|\bGPT\b|\bLLM\b|\bVLM\b|large[ _]?language[ _]?models?|foundation[ _]?models?|(convolutional|recurrent|graph)[ _]?neural|\bLSTM\b|\bGRU\b|generative[ _]?adversarial|\bGAN\b|reinforcement[ _]?learning|(semi|self|un)[ _-]?supervised[ _]?learning|transfer[ _]?learning|federated[ _]?learning|natural[ _]?language[ _]?processing|\bNLP\b|computer[ _]?vision|(image|object|text)[ _]?(classification|detection|segmentation)|\bautoencoder\b|\bdiffusion[ _]?model\b)\b"
HEALTH_PATTERN = r"(?ix)\b(health(care)?|medical|clinical|patients?|hospital|diseases?|disorders?|syndrome|diagnosis|diagnostics?|prognosis|prognostic|treatments?|therapy|therapeutic|\bEHR\b|\bEMR\b|epidemiol\w*|radiol\w*|pathol\w*|genomics?|proteomics?|transcriptomics?|biomarkers?|\bdrug\b|pharmaceutical|pharmacolog\w*|cancer|tumor|tumour|oncol\w*|cardiac|cardiovascular|coronary|diabetes|diabetic|\bCOVID\b|\bSARS\b|pandemic|epidemic|biomedical|mortality|morbidity|\bICU\b|intensive[ _]?care|surgical|surgery|symptoms?|comorbid\w*|screening|triage|mental[ _]?health|psychiatr\w*|neurolog\w*|\bMRI\b|\bCT[ _]?scan\b)\b"

• The amount of research on machine learning in health has been growing rapidly, reflecting the increasing interest and potential of this field as shown here from the ArXiv preprint server with ML and health keywords in the title or abstract. Total amount of papers: 3,008,410.
• However, despite this growth in research, the translation of ML4H research into clinical practice is limited as can be seen from the relatively small number of FDA-approved AI/ML-enabled medical devices.
• 76% of the 331 FDA devices approved in 2025 is in radiology, which is has a long history of using machine learning for image analysis, but other areas of healthcare are still underrepresented in terms of AI-enabled devices.

1. [2], Health, Tue, 03/10/2026 - 09:05, “Artificial Intelligence-Enabled Medical Devices,” FDA

The Translation Gap in Healthcare

Few AI/ML models have been successfully deployed in clinical practice.¹

Main barriers to translation:

⚡ \(P_1\): Incomplete patient state²

⚡ \(P_2\): Replication crisis³

⚡ \(P_3\): Lack of robust infrastructure⁴

Out of scope: ​⚡​ \(P_4\): Ethical and regulatory hurdles⁵ and ​⚡​\(P_5\) Financial incentives⁶

• There is something of a translation gap in ML in healthcare: despite the promise of machine learning research, new datasets, and advances in methodology few models have been successfully deployed in clinical practice.
• \(P_1\): Incomplete patient state: Retrospective data may not capture the real-time information that is needed for prediction.
• \(P_2\): Replication crisis: Different papers report different performance for the same task due to variations in datasets, preprocessing, and evaluation.
• \(P_3\): Lack of robust infrastructure: Each institution has to build their health AI from scratch, making it hard to share research results and build on each other’s work.
• \(P_4\): Ethical and regulatory hurdles: I will not go into detail today.

1. [3], Apell et al., 2026, “Explaining the Slow Adoption of AI Innovations in Health Care: Network Analysis Approach,” JMIR AI [4], Henzler et al., 2025, “Healthcare professionals’ perspectives on artificial intelligence in patient care: A systematic review of hindering and facilitating factors on different levels,” BMC Health Services Research

2. [5], Agniel et al., 2018, “Biases in electronic health record data due to processes within the healthcare system: Retrospective observational study,” BMJ

3. [6], Futoma et al., 2020, “The myth of generalisability in clinical research and machine learning in health care,” The Lancet Digital Health

4. [7], McDermott, 2025, “The (lack of?) Science of Machine Learning for Healthcare,” PMLR; [8], Donoho, 2024, “Data Science at the Singularity,” Harvard Data Science Review

5. [9], Van Calster et al., 2023, “There is no such thing as a validated prediction model,” BMC Medicine

6. [10], Abramoff et al., 2024, “Scaling Adoption of Medical AI — Reimbursement from Value-Based Care and Fee-for-Service Perspectives,” NEJM AI

Research Contribution per Challenge

⚡ \(P_1\): Incomplete patient state → \(C_1\): Multi-ward Multi-modal Warning Systems 🛏️

⚡ \(P_2\): Replication crisis → \(C_2\): Enabling Reproducible Prediction Experiments 🧪

⚡ \(P_3\): Lack of robust infrastructure → \(C_3\): Framework for Foundational AI Research 🏗️

Thesis Statement: Efficient clinical translation of ML requires prospectively collected multimodal data, reproducibility, and an ecosystem for foundational research.

UN WHO Global Strategy on Digital Health Objectives (1/4):

“To promote digital health collaborations and partnership models within and across organizations on the use of software global goods, open-standards, and common digital health architecture.” ¹

My thesis is structured around three main contributions:

• We conducted a prospective cohort study, which collects data from surgical patients in the general ward and integrates wearable device data to improve early warning systems for postoperative complications.
• We developed the YAIB and Recipies framework to lower the barrier for reproducible research and enable better validation of ML for health studies.
• I contributed to the development of the MEDS framework, which provides a standardized data format and tools for scalable machine learning research across healthcare institutions.
• Thesis Statement: The overarching thesis is that healthcare machine learning requires rich data, reusable model components, and an extensive validation ecosystem to bridge the gap between research and clinical practice.
• This aligns well with the WHO’s global strategy on digital health, which emphasizes the importance of open standards and collaboration in this field.

1. [11], Mariano, 2020, “Towards a global strategy on digital health,” Bulletin of the World Health Organization

Research Contributions in Pillars

\(C_1\): Multi-ward Multi-modal Warning Systems 🛏️

Robin P. van de Water, Axel Winter, Daniela Zuluaga Lotero, Bjarne Pfitzner, Lara Faraj, Bert Arnrich, Patrick Rockenschaub, Wenzel Schoning, Thomas Malinka, Christian Denecke, Johann Pratschke, Igor M. Sauer, and Max M. Maurer. Continuous Multimodal AI with Wearable Vital Signs Predicts Postoperative Complications in the General Ward. Nov. 2025. doi: 10.1101/2025.11.25.25340950. [12], Van de Water et al., 2025, “Continuous Multimodal AI with Wearable Vital Signs Predicts Postoperative Complications in the General Ward,” Medrxiv preprint; Submitted to Lancet Digital Health (Submitted to Lancet Digital Health)

Christoph Riepe, Robin van de Water, Axel Winter, Bjarne Pfitzner, Lara Faraj, Robert Ahlborn, Maximilian Schulze, Daniela Zuluaga, Christian Schineis, Katharina Beyer, Johann Pratschke, Bert Arnrich, Igor M. Sauer, and Max M. Maurer. [13], Riepe et al., 2025, “90-day mortality prediction in elective visceral surgery using machine learning: A retrospective multicenter development, validation, and comparison study,” International Journal of Surgery

Robin van de Water, Axel Winter, Max M. Maurer, Felix August Treykorn, Daniela Zuluaga, Bjarne Pfitzner, Igor M. Sauer, and Bert Arnrich. “Combining Hospital-grade Clinical Data and Wearable Vital Sign Monitoring to Predict Surgical Complications”. Mar. 2024. [14], Van de Water et al., 2024, “Combining Hospital-grade Clinical Data and Wearable Vital Sign Monitoring to Predict Surgical Complications,” ICLR 2024 Workshop on Learning from Time Series For Health

Robin van de Water, Axel Winter, Max M. Maurer, Felix August Treykorn, Bjarne Pfitzner, Igor M. Sauer, and Bert Arnrich. “Combining Time Series Modalities to Create Endpoint-driven Patient Records”. [15], Van de Water et al., 2024, “Combining Time Series Modalities to Create Endpoint-driven Patient Records,” ICLR 2024 Workshop on Data-centric Machine Learning Research (DMLR): Harnessing Momentum for Science

\(C_2\): Enabling Reproducible Prediction Experiments 🧪

Robin P. van de Water, Hendrik Schmidt, and Patrick Rockenschaub. “ReciPies: A Lightweight Data Transformation Pipeline for Reproducible ML”., p. 9261. doi: 10.21105/joss.09261 [16], Van de Water et al., 2026, “ReciPies: A lightweight data transformation pipeline for reproducible ML,” Journal of Open Source Software

Robin van de Water, Hendrik Nils Aurel Schmidt, Paul Elbers, Patrick Thoral, Bert Arnrich, and Patrick Rockenschaub. “Yet Another ICU Benchmark: A Flexible Multi-Center Framework for Clinical ML”. doi: 10.48550/arXiv.2306.05109. [17], Van de Water et al., 2024, “Yet Another ICU Benchmark: A Flexible Multi-Center Framework for Clinical ML,” The Twelfth International Conference on Learning Representations

\(C_3\): Framework for Foundational AI Research 🏗️

Matthew McDermott, Ethan Steinberg, Jason Fries, Robin P. van de Water, Chao Pang, Patrick Rockenschaub, Pawel Renc, Jungwoo Oh, Kamilė Stankevičiūtė, Justin Xu, Tom Joseph Pollard, Nassim Oufattole, Michael Wornow, Teya Bergamaschi, Hyewon Jeong, Simon Lee, Vincent Jeanselme, Kiril Klein, Mikkel Odgaard, Maria Elkjær Montgomery, Arkadiusz Sitek, Mads Nielsen, Jeffrey Chiang, Noa Dagan, Isaac Kohane, Shalmali Joshi, Edward Choi, and Nigam Shah. [18], McDermott et al., 2025, “MEDS: An Emerging Data Standard and Ecosystem for Health AI Research,” Accepted to New England Journal Of Medicine (NEJM) AI (2026) vol: 3, issue 6

MEDS Working Group: (alphabetized) Bert Arnrich, Edward Choi, Jason A. Fries, Matthew B. A. McDermott, Jungwoo Oh, Tom J. Pollard, Nigam Shah, Ethan Steinberg, Michael Wornow, Robin van de Water. [19], van de Water et al., 2024, “Medical Event Data Standard (MEDS): Facilitating Machine Learning for Health,” ICLR 2024 workshop on learning from time series for health

Matthew B. A. McDermott, Aleksia Kolo, Chao Pang, Edward Choi, Ethan Steinberg, Hyewon Jeong, Jack Gallifant, Jason Alan Fries, Jeffrey N. Chiang, Jungwoo Oh, Justin Xu, Kamilė Stankevičiūtė, Kiril Vadimovic Klein, Mikkel Fruelund Odgaard, Nassim Oufattole, Patrick Rockenschaub, Pawel Renc, Robin van de Water, Shalmali Joshi, Simon Austin Lee, Teya Bergamaschi, Tom Pollard, Vincent Jeanselme, Nigam Shah, Michael Wornow, Aparajita Kashyap, Xinzhou Jiang, Yanwei Li, Young Sang Choi, Yuta Kobayashi, and Ryan King. [20], McDermott et al., 2024, “MEDS Decentralized, Extensible Validation (MEDS-DEV) Benchmark: Establishing Reproducibility and Comparability in ML for Health,” Machine Learning for Health (ML4H), 15-16 December 2024, Vancouver, Canada

Matthew B. A. McDermott, Justin Xu, Teya S. Bergamaschi, Hyewon Jeong, Simon A. Lee, Nassim Oufattole, Patrick Rockenschaub, Kamilė Stankevičiūtė, Ethan Steinberg, Jimeng Sun, Robin P. Van De Water, Michael Wornow, John Wu, and Zhenbang Wu. “MEDS: Building Models and Tools in a Reproducible Health AI Ecosystem”. SIGKDD Tutorial [21], McDermott et al., 2025, “MEDS: Building Models and Tools in a Reproducible Health AI Ecosystem,” ACM

Robin P. Van de Water. “Scaling Up Clinical ML from Datasets to Entire Health Systems through the MEDS Ecosystem”. In: Berlin, June 2025 [22], Van de Water, 2025, “Scaling Up Clinical ML from Datasets to Entire Health Systems through the MEDS Ecosystem,” SBHD 2025 International Conference on Systems Biology of Human Diseases

• Here you can find the publications that correspond to each of the three main contributions I will talk about today organized by research pillar.

All Publications Chronologically

Matthew McDermott, Ethan Steinberg, Jason Fries, Robin P. van de Water, Chao Pang, Patrick Rockenschaub, Pawel Renc, Jungwoo Oh, Kamilė Stankevičiūtė, Justin Xu, Tom Joseph Pollard, Nassim Oufattole, Michael Wornow, Teya Bergamaschi, Hyewon Jeong, Simon Lee, Vincent Jeanselme, Kiril Klein, Mikkel Odgaard, Maria Elkjær Montgomery, Arkadiusz Sitek, Mads Nielsen, Jeffrey Chiang, Noa Dagan, Isaac Kohane, Shalmali Joshi, Edward Choi, and Nigam Shah. [18], McDermott et al., 2025, “MEDS: An Emerging Data Standard and Ecosystem for Health AI Research,” Accepted to New England Journal Of Medicine (NEJM) AI (2026) vol: 3, issue 6

Robin P. van de Water, Hendrik Schmidt, and Patrick Rockenschaub. “ReciPies: A Lightweight Data Transformation Pipeline for Reproducible ML”., p. 9261. doi: 10.21105/joss.09261 [16], Van de Water et al., 2026, “ReciPies: A lightweight data transformation pipeline for reproducible ML,” Journal of Open Source Software

Matthias Kirchler, Matteo Ferro, Veronica Lorenzini, Robin P. van de Water, Christoph Lippert, and Andrea Ganna. [23], Kirchler et al., 2025, “Large language models improve transferability of electronic health record-based predictions across countries and coding systems,” Accepted to npj Digital Medicine

Katharina Alefs, Susanne Ibing, Pia Francesca Rissom, Jan Carlo Schmid, Arkadiusz Kwasigroch, Robin van de Water, Bernhard Y. Renard, and Eugenia Alleva. [24], Alefs et al., 2025, “Towards Foundation Model-Based Propensity Score Matching from Electronic Health Records,” Machine Learning for Health Symposium 2025

Christoph Riepe, Robin van de Water, Axel Winter, Bjarne Pfitzner, Lara Faraj, Robert Ahlborn, Maximilian Schulze, Daniela Zuluaga, Christian Schineis, Katharina Beyer, Johann Pratschke, Bert Arnrich, Igor M. Sauer, and Max M. Maurer. [13], Riepe et al., 2025, “90-day mortality prediction in elective visceral surgery using machine learning: A retrospective multicenter development, validation, and comparison study,” International Journal of Surgery

Bjarne Pfitzner, Max M. Maurer, Axel Winter, Christoph Riepe, Igor M. Sauer, Robin van de Water, Christian Denecke, Johann Pratschke, and Bert Arnrich. [25], Pfitzner et al., 2025, “Differentially-Private Federated Learning with Non-IID Data For Surgical Risk Prediction.” International Journal of Semantic Computing

Axel Winter, Bjarne Pfitzner, Robin P. van de Water, Lara Faraj, Christoph Riepe, Wolf-Heinrich Hahn, Felix Krenzien, Christian Schineis, Thomas Malinka, and Wenzel Schöning. [26], Winter et al., 2025, “Overcoming the data barrier: Transfer learning for 90-day mortality prediction in general surgery–a retrospective multicenter development and comparison study,” International Journal of Surgery

Robin Philippus van de Water, Ethan Steinberg, Michael Wornow, Patrick Rockenschaub, and Matthew McDermott. MIMIC-IV Demo Data in the Medical Event Data Standard (MEDS). 2025. doi: 10.13026/T2Y8-EA41. [27], Van de Water et al., 2025, “MIMIC-IV demo data in the Medical Event Data Standard (MEDS),” PhysioNet

Robin P. Van de Water. “Scaling Up Clinical ML from Datasets to Entire Health Systems through the MEDS Ecosystem”. In: SBHD 2025 International Conference on Systems Biology of Human Diseases. Berlin, June 2025 [22], Van de Water, 2025, “Scaling Up Clinical ML from Datasets to Entire Health Systems through the MEDS Ecosystem,” SBHD 2025 International Conference on Systems Biology of Human Diseases

Robin van de Water, Hendrik Nils Aurel Schmidt, Paul Elbers, Patrick Thoral, Bert Arnrich, and Patrick Rockenschaub. “Yet Another ICU Benchmark: A Flexible Multi-Center Framework for Clinical ML”. doi: 10.48550/arXiv.2306.05109. [17], Van de Water et al., 2024, “Yet Another ICU Benchmark: A Flexible Multi-Center Framework for Clinical ML,” The Twelfth International Conference on Learning Representations

MEDS Working Group: (alphabetized) Bert Arnrich, Edward Choi, Jason A. Fries, Matthew B. A. McDermott, Jungwoo Oh, Tom J. Pollard, Nigam Shah, Ethan Steinberg, Michael Wornow, Robin van de Water. “Medical Event Data Standard (MEDS): Facilitating Machine Learning for Health”. 2024. [19], van de Water et al., 2024, “Medical Event Data Standard (MEDS): Facilitating Machine Learning for Health,” ICLR 2024 workshop on learning from time series for health

Matthew B. A. McDermott, Aleksia Kolo, Chao Pang, Edward Choi, Ethan Steinberg, Hyewon Jeong, Jack Gallifant, Jason Alan Fries, Jeffrey N. Chiang, Jungwoo Oh, Justin Xu, Kamilė Stankevičiūtė, Kiril Vadimovic Klein, Mikkel Fruelund Odgaard, Nassim Oufattole, Patrick Rockenschaub, Pawel Renc, Robin van de Water, Shalmali Joshi, Simon Austin Lee, Teya Bergamaschi, Tom Pollard, Vincent Jeanselme, Nigam Shah, Michael Wornow, Aparajita Kashyap, Xinzhou Jiang, Yanwei Li, Young Sang Choi, Yuta Kobayashi, and Ryan King. [20], McDermott et al., 2024, “MEDS Decentralized, Extensible Validation (MEDS-DEV) Benchmark: Establishing Reproducibility and Comparability in ML for Health,” Machine Learning for Health (ML4H), 15-16 December 2024, Vancouver, Canada

Matthew B. A. McDermott, Justin Xu, Teya S. Bergamaschi, Hyewon Jeong, Simon A. Lee, Nassim Oufattole, Patrick Rockenschaub, Kamilė Stankevičiūtė, Ethan Steinberg, Jimeng Sun, Robin P. Van De Water, Michael Wornow, John Wu, and Zhenbang Wu. “MEDS: Building Models and Tools in a Reproducible Health AI Ecosystem”. In: Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2. Toronto ON Canada: ACM, Aug. 2025, pp. 6243-6244. isbn: 979-8-4007-1454-2. doi: 10.1145/3711896.3737608. [21], McDermott et al., 2025, “MEDS: Building Models and Tools in a Reproducible Health AI Ecosystem,” ACM

Max M. Maurer, Bjarne Pfitzner, Robin P. van de Water, Lara Faraj, Christoph Riepe, Daniela Zuluaga, Felix Krenzien, Nathanael Raschzok, Robert Siegel, Christian Schineis, Bert Arnrich, Katharina Beyer, Johann Pratschke, Igor M. Sauer, and Axel Winter. [28], Maurer et al., 2025, “Privacy preserving federated learning for 90-day mortality prediction in colorectal surgery: A multicenter retrospective development and comparison study,” International Journal of Surgery (London, England)

Robin van de Water, Axel Winter, Max M. Maurer, Felix August Treykorn, Daniela Zuluaga, Bjarne Pfitzner, Igor M. Sauer, and Bert Arnrich. [14], Van de Water et al., 2024, “Combining Hospital-grade Clinical Data and Wearable Vital Sign Monitoring to Predict Surgical Complications,” ICLR 2024 Workshop on Learning from Time Series For Health

Robin van de Water, Axel Winter, Max M. Maurer, Felix August Treykorn, Bjarne Pfitzner, Igor M. Sauer, and Bert Arnrich. [15], Van de Water et al., 2024, “Combining Time Series Modalities to Create Endpoint-driven Patient Records,” ICLR 2024 Workshop on Data-centric Machine Learning Research (DMLR): Harnessing Momentum for Science

Axel Winter, Robin P. van de Water, Bjarne Pfitzner, Marius Ibach, Christoph Riepe, Robert Ahlborn, Lara Faraj, Felix Krenzien, Eva M. Dobrindt, and Jonas Raakow. [29], Winter et al., 2024, “Enhancing Preoperative Outcome Prediction: A Comparative Retrospective Case–Control Study on Machine Learning versus the International Esodata Study Group Risk Model for Predicting 90-Day Mortality in Oncologic Esophagectomy,” Cancers

Bjarne Pfitzner, Max M. Maurer, Axel Winter, Christoph Riepe, Igor M. Sauer, Robin van de Water, and Bert Arnrich. [30], Pfitzner et al., 2024, “Differentially-Private Federated Learning with Non-IID Data for Surgical Risk Prediction,” IEEE

Orhan Konak, Robin van de Water, Valentin Doring, Tobias Fiedler, Lucas Liebe, Leander Masopust, Kirill Postnov, Franz Sauerwald, Felix Treykorn, and Alexander Wischmann. [31], Konak et al., 2023, “HARE: Unifying the Human Activity Recognition Engineering Workflow,” Sensors

Orhan Konak, Alexander Wischmann, Robin van de Water, and Bert Arnrich. [32], Konak et al., 2023, “A Real-time Human Pose Estimation Approach for Optimal Sensor Placement in Sensor-based Human Activity Recognition,” ACM

Robin van de Water, Francesco Ventura, Zoi Kaoudi, Jorge-Arnulfo Quiane-Ruiz, and Volker Markl. [33], Van de Water et al., 2022, “Farming Your ML-based Query Optimizer’s Food,” IEEE [Best demonstration paper award]

Robin van de Water, Francesco Ventura, Z Kaoudi, J Quiane-Ruiz, and Volker Markl. “Farm Your ML-based Query Optimizer’s Food!-Human-Guided Training Data Generation-”. In: Conference on Innovative Data Systems Research (CIDR). 2022 [34], Van de Water et al., 2022, “Farm your ML-based query optimizer’s Food!–Human-Guided training data generation–,” Conference on Innovative Data Systems Research (CIDR)

Lientje Maas, Mathan Geurtsen, Florian Nouwt, Stefan F. Schouten, Robin van de Water, Sandra Van Dulmen, Fabiano Dalpiaz, Kees Van Deemter, and Sjaak Brinkkemper. [35], Maas et al., 2020, “The Care2Report System: Automated Medical Reporting as an Integrated Solution to Reduce Administrative Burden in Healthcare.” HICSS

Under submission:

Jan Carlo Schmid, Susanne Ibing, Stefan Kalabakov, Katharina Alefs, Robin P. van de Water, Arkadiusz Kwasigroch, Eugenia Alleva, Bert Arnrich, Bernhard Y. Renard, and Maia Kayal. [36], Schmid et al., 2025, “Expert-defined, chart-reviewed features outperform EHR foundation models for predicting crohn’s-like disease of the pouch,” Under review at The American Journal of Gastroenterology

Preprints (preparing for submission):

Robin P. van de Water, Axel Winter, Daniela Zuluaga Lotero, Bjarne Pfitzner, Lara Faraj, Bert Arnrich, Patrick Rockenschaub, Wenzel Schoning, Thomas Malinka, Christian Denecke, Johann Pratschke, Igor M. Sauer, and Max M. Maurer. [12], Van de Water et al., 2025, “Continuous Multimodal AI with Wearable Vital Signs Predicts Postoperative Complications in the General Ward,” Medrxiv preprint; Submitted to Lancet Digital Health (Submitted to Lancet Digital Health)

Alisher Turubayev, Anna Shopova, Fabian Lange, Mahmut Kamalak, Paul Mattes, Victoria Ayvasky, Bert Arnrich, Bjarne Pfitzner, and Robin P. van de Water. [37], Turubayev et al., 2025, “Closing Gaps: An Imputation Analysis of ICU Vital Signs,” http://arxiv.org/abs/2510.24217

Manuscripts in preparation:

Robin P. van de Water, Katharina Marie Alefs, Susanne Ibing, Stefan Kalabakov, Aadil Rasheed, Pia Francesca Rissom, Jan Carlo Schmid, Arkadiusz Kwasigroch, Eugenia Alleva, and Christoph Lippert. “FLAMES: EHR Foundation Models as a Resource for Health Systems”. 2026

Robin P. van de Water, Christoph Riepe, Bjarne Pfitzner, Lara Faraj, Daniela Zuluaga, Christian Schineis, Katharina Beyer, Johann Pratschke, Igor Sauer, Axel Winter, and Max Maurer. “Machine Learning for 30-Day Mortality Prediction for High-Risk General Emergency Surgery”. In preparation (2025)

Wouter van Amsterdam, Michael Kamp, Rajesh Ranganath, Robin P. van de Water, Florian Markowetz, Evangelia Christodoulou, Yamuna Krishnamurthy, Christoph Lippert, Jeff Clark, Julia E. Vogt, Raul Santos-Rodriguez, Thomas Gartner, Gilbert Koch, and Brett Beaulieu-Jones. “Missing Incentives, Missing Oversight: The Challenges of AI Monitoring in Clinical Practice”. 2025

\(C_1\): Multi-ward Multi-modal Warning Systems 🛏️

\(C_2\): Enabling Reproducible Prediction Experiments 🧪

\(C_3\): Framework for Foundational AI Research 🏗️

• These are all the publications that I have contributed to until now. You can find them on the slides in the footer and they should link to the corresponding work.

⚡ \(P_1\): Incomplete patient state

Multi-ward Multi-modal Early Warning Systems 🛏️

​⚡​ \(P_1\): Incomplete patient state: Retrospective, observational, data does not accurately capture the state of the patient.

300M+ surgeries annually¹.

~1/3 of hospital costs are surgery-related².

Up to 25% experience life-threatening adverse events, often on the ward³.

Ward monitoring insufficient for subtle condition changes⁴.

• The first contribution is related to the limitation of retrospective data in capturing real-time state information. This leads to models that don’t capture the full complexity of the patient journey. This is reflected in the case of surgery rehabilitation.
• Surgery is a major part of healthcare, with more than 300 million surgical procedures performed annually worldwide
• Surgery is also a major driver of healthcare costs, with approximately one-third of hospital costs being surgery-related.
• Postoperative complications are common, with up to 25% of patients experiencing adverse events after surgery which can lead to death and significantly impact quality of life.
• But, preceding physiological patterns are often subtle, nonspecific, and time-varying, with substantial heterogeneity between patients → hard to detect on ward, where most postoperative patients rehabilitate.
• Why not just monitor patients in the ICU? → expensive and can lead to complications like infections, delirium, and deconditioning.

1. [38], Dobson, 2020, “Trauma of major surgery: A global problem that is not going away,” International Journal of Surgery (London, England)

2. [39], Shrime et al., 2015, “Global burden of surgical disease: An estimation from the provider perspective,” The Lancet Global Health

3. [40], Downey et al., 2023, “Impact of in-hospital postoperative complications on quality of life up to 12 months after major abdominal surgery,” The British Journal of Surgery; [41], Mualla et al., 2021, “The Impact of Postoperative Complications on Long-Term Oncological Outcomes Following Curative Resection of Colorectal Cancer (Stage I-III): A Systematic Review and Meta-Analysis,” Cureus; [42], Baum et al., 2019, “Mortality and Complications Following Visceral Surgery,” Deutsches Ärzteblatt international

4. [43], Roppelt et al., 2024, “Artificial intelligence in healthcare institutions: A systematic literature review on influencing factors,” Technology in Society; [44], Stam et al., 2022, “The prediction of surgical complications using artificial intelligence in patients undergoing major abdominal surgery: A systematic review.” Surgery

A Prospective Cohort Study: “Hybrid Ward”

Can we leverage continuous data from wearable devices to improve warning systems for postoperative complications (surgical site infections) in the general ward? ¹

• What could be a solution? Hybrid ward with minimally intrusive wearables
• With the experimental surgery group at Charite, we conducted a prospective cohort study in a hybrid ward setting, where regular care was provided and continuous, data from wearable devices was collected through informed consent.
• Corsano band: Photoplethysmogram (PPG) Sensor (HR/BP), Acceleration, oxygen saturation, Respiration Rate
• Temperature sensor patch for core body temperature
• Feed data into a Bluetooth mesh network, which allows for real-time monitoring and data collection without the need for constant manual intervention.
• Left: Timeline of the perioperative patient journey in surgical patients.
• This is largest reported cohort to investigate continuous, telemetric vital sign analysis in the field of surgical care.
• Right: Prospective data curation by endpoint labeling and data extraction from hospital data information systems.

1. Van de Water et al. [45], 2024, “Combining Hospital-grade Clinical Data and Wearable Vital Sign Monitoring to Predict Surgical Complications,” ICLR 2024 Workshop on Data-centric Machine Learning Research (DMLR); Van de Water et al. [14], 2024, “Combining Hospital-grade Clinical Data and Wearable Vital Sign Monitoring to Predict Surgical Complications,” ICLR 2024 Workshop on Learning from Time Series For Health

¹

• This figure shows our end-to-end multimodal prediction pipeline, from raw data to dynamic risk predictions.
• We start with multiple data sources: wearable and physiologic signals, body temperature, medications, clinical notes, labs, ICU vitals and observations, plus baseline patient characteristics.
• Each modality is preprocessed with methods matched to its type, including MiniRocket embeddings, HRV feature extraction, multilingual text embeddings, and clinical harmonization and aggregation.
• Next, we apply temporal processing: imputation, statistical feature extraction, and retrospective windows from 24 hours down to 1 hour before the prediction time.
• These modality-specific representations are fused into one multimodal model input.
• The model then predicts endpoint risk over multiple horizons, from 12 hours to 0 hours, updating dynamically every 30 minutes. In total, this produces 7,280 features over 371,813 segmented trajectories.

1. Van de Water et al. [12], 2025, “Continuous Multimodal AI with Wearable Vital Signs Predicts Postoperative Complications in the General Ward,” Medrxiv preprint; Submitted to Lancet Digital Health

Clinical ASsist Heuristic Early Warning System+

• Use open-source framework for training and model selection.
  

• Use optimal model outputs with alarm mechanics to reduce alarm burden (alarms/patient/day) while maintaining precision.¹
• Decision curve analysis: harm-to-benefit ratio drives alarm selection.
  

• Multiple model architectures, encompassing classical machine learning (ML) and Deep Learning (DL), were trained and evaluated using cross-validation and hyperparameter optimization.
• Model selection was based on predictive performance and computational efficiency, with LightGBM identified as the most favorable trade-off.
• The optimal model was evaluated by performing domain ablation and feature importance analysis with Shapley explanations.
• Based upon continuous model outputs, an actional alarm system including specific alarming frameworks was evaluated. The slope coefficient and persistence quantify the increase in the probability of sounding an alarm.
• An 8-hour silencing period is implemented to prevent alarm fatigue.

1. Work with master thesis student Daniela Zuluaga Lotero

How well does it work?

→ CASHEWS(+) significantly improves discrimination performance.

→ Lead time performance has little drop-off.

💡 Monitoring at the right time extends AI-assisted surveillance to resource-constrained wards.

• Metrics: areas under the receiver operating characteristic (AUROC) and (b) boxplots of patient-level areas under the precision–recall curve (AUPRC)
• First: the static baseline model that aggregate the data until the start of the surgery, then directly after the surgery and directly after the ICU stay. Static models using only preoperative and ICU data performed near chance level.
• Then the dynamic, longitudinal model that aggregates the data in 30-minute time windows. Where we have CASHEWS without wearable data and CASHEWS+ with wearable data. These models had much more predictive power for identifying surgical site infections.
• We also compare this with a baseline of the Remote Early Warning Score (REWS), which is a threshold-based wearable score.
• Looking at the lead time performance, we see that there is little drop-off in performance as we move further away from the clinical detection of the complication, which suggests that there are indicators of deterioration that can be detected well in advance of clinical recognition.
• Boxplots of AUROC and (d) boxplots of AUPRC of CASHEWS+ for varying lead times prior to clinical detection. The last hour was set to be positive, while all subsequent time windows were removed from training and evaluation.
• The key implication is that due to the filling this missing data gap, we can do predictive surveillance can be extended beyond the ICU into general wards, where many actionable postoperative complications actually emerge.

⚡ \(P_2\): Replication crisis

• Now that we’ve looked at a specific clinical use case of ML lets see how easy it is for others can reproduce these results and build on them.

Enabling Reproducible Prediction Experiments 🧪

​⚡ \(P_2\): Replication crisis¹:​ Comparing approaches across papers is difficult because of:

Data/Splits

Cohort

Preprocessing

Evaluation

Code/Models²

Can we create a framework for reliably reproducing scientific results of clinical prediction models across datasets and preprocessing choices?

• That leads me to the second contribution, which is about enabling reproducible research in clinical machine learning.
• The reproducibility crisis in clinical machine learning is a significant barrier to scientific progress and clinical translation.
• Datasets (private datasets or different public datasets) and splits
• Cohort definitions (patient inclusion/exclusion varies)
• Preprocessing (normalization, imputation, features)
• Evaluation metrics
• Code not published, incomplete, not reusable or adaptable.

1. [46], Moor et al., 2021, “Early Prediction of Sepsis in the ICU Using Machine Learning: A Systematic Review,” Frontiers in Medicine; [47], Fleuren et al., 2020, “Machine learning for the prediction of sepsis: A systematic review and meta-analysis of diagnostic test accuracy,” Intensive Care Medicine

2. [48], Shillan et al., 2019, “Use of machine learning to analyse routinely collected intensive care unit data: A systematic review,” Critical Care

Considerations for Reproducible Clinical ML Research

Problems:

• Medical research is inherently complex.
• One size fits-all always impossible.
• Hardcoded ≠ reproducible.

Goals:

• Modular setup maintaining reproducibility.
• Extensibility across experiment lifecycle.
• Out-of-the-box support: datasets, tasks, models.

Outcome:

• ReciPies¹: Declare, execute, and share ML preprocessing pipelines.
• Yet Another ICU Benchmark (YAIB)²: Reproducible prediction experiments across (open-access) datasets.

• So what are the considerations that we need to take into account when designing a framework for reproducible clinical machine learning research?
• First, we need to acknowledge that medical research is inherently complex, and that one size fits all solutions are often impossible. This means that we need to design a framework that is modular and extensible, allowing researchers to adapt it to their specific needs while maintaining reproducibility.
• Second, we need to ensure that the framework provides customization and extension and that there is out-of-the-box support for common datasets, tasks, and models.
• To address these considerations, we developed two tools: ReciPies, a modular ML preprocessing framework, that allows you to declare preprocessing steps
• Yet Another ICU Benchmark (YAIB), a framework for reproducible ICU prediction experiments across 5 existing datasets and 5 common tasks.
• Open-access Intensive Care Unit (ICU) data has become popular for applications, models, and benchmarks in ML for health, but comparing approaches often remains difficult

1. Van de Water et al. [16], 2026, “ReciPies: A lightweight data transformation pipeline for reproducible ML,” Journal of Open Source Software

2. Van de Water et al. [17], 2024, “Yet Another ICU Benchmark: A Flexible Multi-Center Framework for Clinical ML,” The Twelfth International Conference on Learning Representations

Different Definitions, Different Conclusions

Sepsis: Life-threatening response to infection that causes injury to its own tissues and organs.

→ ⚡ Problem: no single, universally accepted definition.

Prediction on MIMIC-IV¹ using Gated Recurrent Unit (GRU) and Light Gradient Boosting Machine (LGBM) models with different cohort definitions from literature²:

• To demonstrate the need of a common reference point for reproducible research, we can look at the task of sepsis prediction, which is a common task in clinical machine learning.
• Sepsis is a body’s extreme response to an infection, which can lead to tissue damage, organ failure, and death.
• However, there are different definitions of sepsis that have been proposed in the literature, and these definitions can significantly alter the resulting performance metrics of models trained to predict sepsis.
• This then leads to varying model performance and clinical implications.
• For clinicians this means that selecting the right model for their setting is difficult without a common reference point.
• This underlines the need for a framework that can test the effect of different definitions, preprocessing choices, and datasets on model performance in a reproducible way.

1. [49], Johnson et al., 2023, “MIMIC-IV, a freely accessible electronic health record dataset,” Scientific Data

2. [50], Seymour et al., 2016, “Assessment of Clinical Criteria for Sepsis: For the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3),” JAMA; [51], Moor et al., 2021, “Predicting sepsis in multi-site, multi-national intensive care cohorts using deep learning,” http://arxiv.org/abs/2107.05230; [52], Calvert et al., 2016, “A computational approach to early sepsis detection,” Computers in Biology and Medicine

YAIB: Mortality Prediction Across Datasets

External validation: training on row dataset, evaluation on column dataset using GRU:

Transfer learning: Train a model on “generalizable” eICU and finetune on HiRID:

💡 Reproducible, harmonized benchmarking enables wide-ranging prediction experiments.

• Using YAIB, we can easily compare generalization across datasets and transfer learning approaches for mortality prediction. Each row corresponds to a model trained on a dataset, and the columns correspond to the evaluation performance.
• Some datasets are more similar to others. The European datasets are different from the US datasets. However, eICU, which consists of data from 208 US hospitals is more generalizable than the other datasets, which are from single hospitals.
• Finetuning on the “difficult” HiRID dataset can improve performance with only a fraction of the data, suggesting that pretraining on a large, generalizable dataset can help overcome some of the challenges of dataset shift and improve generalization to new settings.
• The key point is that without a framework like YAIB, it would be difficult to enable this important reproducible research across datasets.

⚡ \(P_3\): Lack of robust infrastructure

Now that we’ve looked at structured open-access ICU data let’s see how we can enable this reproducible research across healthcare systems with more complex data and models.

Framework for Foundational Health AI 🏗️

​⚡ \(P_3\): Lack of robust infrastructure: Training high-parameter AI models for heterogeneous data is difficult.

Event-based Foundation Models (FMs) show promise in Health AI.¹

Progress is hindered by lack of reusable, scalable tooling.²

How can we enable reproducible AI research across healthcare systems?

MEDS: Medical Event Data Standard ³

• Unfortunately,health data is not always like the structured, clean, and well-labeled open-access ICU datasets that we just discussed.
• Luckily, with the advent of Large Language Models, we now have high-parameter AI “foundation models”, that are trained on partially structured large-scale EHR data, such as entire health systems, with the objective to predict the next token.
• However, with the increasing complexity of models and data, comparing results across settings becomes increasingly important, but more difficult.
• Applying these high-parameter AI models to health data still lacks robust, standardized infrastructure, which is a significant barrier to reproducible research.
• This leads us to the question of how can we enable reproducible AI research across healthcare systems?
• Well, with a new data standard and ecosystem for health AI research, which we call the Medical Event Data Standard (MEDS).
• Donoho highlights the term of frictionless reproducibility for data science which highlights the growing need for robust infrastructure to support reproducible research across healthcare systems.

1. [53], Bommasani et al., 2021, “On the Opportunities and Risks of Foundation Models,” http://arxiv.org/abs/2108.07258; [54], Steinberg et al., 2021, “Language models are an effective representation learning technique for electronic health record data,” Journal of Biomedical Informatics

2. [7], McDermott, 2025, “The (lack of?) Science of Machine Learning for Healthcare,” PMLR; [8], Donoho, 2024, “Data Science at the Singularity,” Harvard Data Science Review

3. [19], van de Water et al., 2024, “Medical Event Data Standard (MEDS): Facilitating Machine Learning for Health,” ICLR 2024 workshop on learning from time series for health (Founding MEDS working Group)

Data Standards in Healthcare

Source: https://xkcd.com/927/

• OMOP ¹, PCORnet ², FHIR³, and i2b2⁴.
• Have enabled observational studies and health data exchange ⁵.
• But, they are not designed for high-parameter AI research.

• I know that researchers are eager to create new data standards and frameworks, that might already exist as evident in this comic.
• There are existing data standards in healthcare, such as OMOP, PCORnet, FHIR, and i2b2, have been instrumental in enabling observational studies and health data exchange.
• They are Disease-specific cohorts and specific manual featurization of engineered features
• But, they are generally not suitable to train over a general-purpose representation of the entire EHR

1. [55], Hripcsak et al., 2015, “Observational Health Data Sciences and Informatics (OHDSI): Opportunities for Observational Researchers,” Studies in health technology and informatics

2. [56], Collins et al., 2014, “PCORnet: Turning a dream into reality,” Journal of the American Medical Informatics Association

3. [57], Bender and Sartipi, 2013, “HL7 FHIR: An Agile and RESTful approach to healthcare information exchange,” The 26th IEEE International Symposium on Computer-Based Medical Systems

4. [58], Murphy et al., 2010, “Serving the enterprise and beyond with informatics for integrating biology and the bedside (I2b2),” Journal of the American Medical Informatics Association

5. [59], Callahan et al., 2021, “Using aggregate patient data at the bedside via an on-demand consultation service,” NEJM Catalyst Innovations in Care Delivery; [60], Suchard et al., 2019, “Comprehensive comparative effectiveness and safety of first-line antihypertensive drug classes: A systematic, multinational, large-scale analysis,” The Lancet

The Medical Event Data Standard (MEDS)

• ML-First: not observational research.
• Flexible: adapts to data source and research needs.
• Event-Based: longitudinal healthcare data.
• Interoperable: tools composable across settings.

Adapted from McDermott et al. “MEDS: An Emerging Data Standard and Ecosystem for Health AI Research” Accepted to NEJM AI (2026), to appear in vol:3, iss:6 (2026).

We wanted to create a framework that is

1. ML-first: Optimized for ML tasks, not observational research.
2. Flexible - Works with OMOP, FHIR, open-access datasets, proprietary (private) health systems, and is quickly adaptable to new data sources.
3. Event-Based - Natural representation for longitudinal healthcare data
4. Interoperable - Tools compose together across datasets and institutions

• We created the Medical Event Data Standard (MEDS), a flexible, event-based data standard optimized for machine learning research in healthcare.

MEDS-DEV: Decentralized External Validation

• Reproducible benchmarking is critical for scientific progress.
• Sharing raw patient data across institutions is often not possible due to privacy concerns.

Define standardized predicates for tasks and have drop-in definitions for each dataset¹.

9 Datasets (1 private, 8 open-access)

11 Tasks

3 Models²

• We can use MEDS to enable decentralized external validation across datasets and institutions.
• Reproducible benchmarking is critical for scientific progress, but sharing raw patient data across institutions is often not possible due to privacy concerns.
• The solution is to define standardized predicates for tasks and have drop-in definitions for each dataset, which allows for benchmarking across datasets without sharing raw patient data.
• MEDS-DEV is a collective effort that I expanded to 8 other datasets, including health systems open-access datasets
• Each site independently trains and evaluates models using the same standardized definitions, allowing for meaningful comparisons while respecting data privacy.
• The configurations can be shared across sites, enabling researchers to identify factors that contribute to performance variability.
• 3 Complete Blood Count, 5 blood chemistry, 2 Vital Sign, Mortality, Readmission
• 3 models: MEDS-Tab, CEHR-BERT, GenHPF: a tabular transformer, a BERT-based model for EHR data, and a general healthcare foundation model.

1. [McDermott et al. [20], 2024, “MEDS Decentralized, Extensible Validation (MEDS-DEV) Benchmark: Establishing Reproducibility and Comparability in ML for Health,” Machine Learning for Health (ML4H), 15-16 December 2024, Vancouver, Canada][61], Xu et al., 2024, “ACES: Automatic Cohort Extraction System for Event-Stream Datasets,” The Thirteenth International Conference on Learning Representations

2. Oufattole et al. [62], 2024, “MEDS-Tab: Automated tabularization and baseline methods for MEDS datasets,” http://arxiv.org/abs/2411.00200;Pang et al. [63], 2021, “CEHR-BERT: Incorporating temporal information from structured EHR data to improve prediction tasks,” http://arxiv.org/abs/2111.08585; Hur et al. [64], 2024, “GenHPF: General Healthcare Predictive Framework for Multi-Task Multi-Source Learning,” IEEE Journal of Biomedical and Health Informatics

Grouping by dataset and model:

Grouping by task and model:

No domination of one model across datasets and tasks.

• If we then benchmark each task across sites and models, we get an enormous amount of results.
• Each point on the left represents a model trained and evaluated at a specific site, with the same task definition
• Some models incompatible with some datasets/tasks.
• The spread of points across sites for the same model and task highlights the variability in performance due to differences in data and implementation, underscoring the need for standardized benchmarking.
• The overall performance trends across models and tasks can be compared, but the variability emphasizes that results from one site may not generalize to others without careful validation.
• This highlights once again the importance of easy reproducibility and standardized benchmarking frameworks to understand what actually drives model performance across different settings.

Foundation Models for Health Systems

¹

• So let’s look at doing this on a health system level, where we use the entire Mount Sinai Health System dataset of 12 million patients across different medical centers around New York City.
• This dataset is already in OMOP format, and we use the OMOP ETL I developed to convert it to MEDS. Leaving behind the legacy imported patients that have not correctly been imported into OMOP, we have 6.8 million patients and a total of 3.5 billion events in the dataset.
• The MEDS ecosystem then enables the use of different tools and of foundation model training for this dataset and the open-access datasets and other health systems
• Here, we will be looking at foundation model training and using the frozen embeddings for linear probing as a case study of how the MEDS ecosystem
• This can enable a wide range of applications, such as downstream clinical prediction systems to serve specialties in the health systems.

1. Van de Water [22], 2025, “Scaling Up Clinical ML from Datasets to Entire Health Systems through the MEDS Ecosystem,” SBHD 2025 International Conference on Systems Biology of Human Diseases; Van de Water et al. [65], 2025, “FLAMES: EHR foundation models as a resource for health systems”

What Can We Do With Foundation Models?

• Train SOTA FMs¹, CEHR-XGPT² and MOTOR³, on entire MSHS (5.5M patients).
• Embed a variety of patient representations until prediction time.
• Linear probing: train a linear classifier on top of frozen FM representations.
• Effect of sample size on performance:

→ For small sample sizes, FM representations can outperform existing machine learning models.

→ Cheaper than finetuning, and proxy for quality of the foundation model representations for downstream tasks.

→ FM-generated representations can provide a useful starting point for specialty AI-applications.

• I trained state-of-the-art foundation models on the entire MSHS dataset, and then we used the frozen embeddings to train a linear classifier on top of the foundation model representations for a range of downstream clinical prediction tasks.

• The performance of the linear classifier on top of the frozen foundation model representations can be compared to traditional machine learning models trained on the same tasks, and this can help us understand the value of the foundation model representations for downstream tasks.

• Cheaper than finetuning, and can be used as a quick benchmark for the quality of the foundation model representations for downstream tasks.

• For lower sample set sizes, we see that the linear classifier on top of the foundation model representations can outperform traditional machine learning models, suggesting that the foundation model representations capture useful information for downstream tasks even with limited data.

• The performance gap decreases as we get more samples as can be expected, but there is still a slight boost compared to the baselines

• This suggests that the foundation model representations can provide a useful starting point for downstream tasks, even when finetuning is not feasible due to computational constraints or limited data. Extra:

• This may reflect differences in pretraining strategies, as MOTOR is explicitly optimized for representation learning (e.g., time-to-event prediction), whereas CEHR-XGPT is trained using nexttoken prediction.

1. [66], Pang et al., 2025, “FoMoH: A clinically meaningful foundation model evaluation for structured electronic health records,” http://arxiv.org/abs/2505.16941

2. [67], Pang et al., 2025, “CEHR-XGPT: A Scalable Multi-Task Foundation Model for Electronic Health Records,” http://arxiv.org/abs/2509.03643

3. [68], Steinberg et al., 2023, “MOTOR: A Time-to-Event Foundation Model For Structured Medical Records,” The Twelfth International Conference on Learning Representations

MEDS: Community Adoption

Adapted from McDermott et al. “MEDS: An Emerging Data Standard and Ecosystem for Health AI Research” Accepted to NEJM AI (2026), to appear in vol:3, iss:6 (2026).

💡 MEDS seems to meet Health AI community’s needs.

• So what is the impact of the MEDS ecosystem?
• It is used in at least 21 institutions, has supported the development of state-of-the-art models for EHR data, and has been used in over 27 academic papers and preprints across 17 datasets and 12 AI algorithms.
• It has also enabled a reduction of lines of code up to 70% for some projects
• I co-designed the schema and ecosystem and created 3 tools and 4 ETLs.

Discussion and Impact

Let’s summarize the key takeaways from my contributions:

Discussion and Impact

→ Continuous wearable data nearly doubles AUPRC.

→ Wearables provides physiological insights.

→ Predictive surveillance can be extended to the ward. → First author for largest surgical wearable-enabled prospective cohort.

→ Experimental setup often matters more than model choice.

→ Harmonizing data provides external validation and fine-tunable models.

→ Multi-center framework for reproducible ML4H (44 cit., GH 97 ⭐).

→ Co-founder of MEDS: enables AI research across health systems.

→ Created 4 Dataset, OMOP, FHIR ETLs and 3 tools.

→ MEDS-DEV: Scaled cross-institutional benchmarks.

→ Adoption across the community (87 agg. cit.; 113 GH followers).

• Let’s summarize the key takeaways from this thesis:
• For the first pillar, we found that continuous wearable data can nearly double the AUPRC for surgical site infection prediction, and that predictive surveillance can be used beyond the ICU into the ward. We also found that alarm strategies are key to clinical utility.
• For the second pillar, we found that the experimental setup often matters more than the choice of model, and we developed a multi-center framework for reproducible ML4H benchmarking that has been widely adopted in the community.
• For the third pillar, we developed the MEDS ecosystem, which enables reproducible AI research across health systems, and I, personally, created 4 dataset, OMOP, and FHIR ETLs and 3 tools, and benchmarked 9 datasets across 11 tasks.
• Overall, we see different scales of impact here, and we need to recognize that these scales all influence each other. For example, improvements in foundation modelling can inform the development of better early warning systems. And, simultaneously, collecting prospective data can inform the development of better multi-modal foundation models.

Limitations

Amplification of existing racial, gender, and socioeconomic biases.¹

Need validation strategies and randomized trials before deployment.²

Need for causal approaches to improve theoretical validity. ³

(Large) AI models have environmental impact and are “black boxes”.⁴

• Models may amplify existing (historical) racial, gender, and socioeconomic biases if not carefully validated because they are trained on historical data that reflects these biases.
• Validation strategies and randomized controlled trials are needed to demonstrate clinical utility and safety before deployment. We might not be able to take shortcuts here, even though rcts are expensive and time-consuming, because of the potential for harm if models are deployed without proper validation.
• Working backward from the end goal of positively impacting clinically relevant outcomes using AI, leads to considerations of causality in model development and validation, and subsequently a better development pipeline. Healthcare AI should be “actionable,” and the change in actions induced by AI should improve outcomes. Quantifying the effect of changes in actions on outcomes is causal inference.
• Randomized Controlled Trial (RCT) emulation / Target Trials Algorithmically emulates randomized studies from observational data Drug efficacy estimation
• Propensity Score Matching Balances treatment/control groups on observed confounders Comparing surgical vs. medical interventions
• (Large) AI models can have sizeable environmental impact. We see this in the training of large language models, and this is something that we need to be mindful of as we develop and deploy AI models in healthcare. We should strive to develop simpler models where possible as this can even lead to an improved physiological understanding as we saw in the first contribution.

1. [69], Borges do Nascimento et al., 2023, “Barriers and facilitators to utilizing digital health technologies by healthcare professionals,” npj Digital Medicine

2. [Wong et al. [70], 2021, “External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients,” JAMA internal medicine; ]

3. [71], Joshi et al., 2025, “AI as an intervention: Improving clinical outcomes relies on a causal approach to AI development and validation,” Journal of the American Medical Informatics Association

4. [72], Hosseini et al., 2025, “A social-environmental impact perspective of generative artificial intelligence,” Environmental Science and Ecotechnology

Future Research Directions

Advances in multi-modal health Foundation Models.

Generalizability of Foundation Models in health.¹

Scaling up: datasets, model size, benchmarking.²

Predictive surveillance and transferable patient representations.³

• Multi-modal: Clinical Notes, Pathology, Radiology, Genomics, Wearables, etc. I am currently working on expanding my FMs to pathology as this is widely available in MSHS.
• Health data is different from text used in LLMs, because different health systems have different data generating processes and different patient populations. So, understanding the generalizability of foundation models in health is an important area of research.
• Scaling up: how does performance improve with more data, compute, and model size? Understanding scaling laws in health AI can inform the development of more efficient and effective models.
• Predictive surveillance: We can schedule mammograms in advance if a patient is at high risk of developing breast cancer.
• Finally, we can develop representations that combat fragmentation across different healthcare providers (GP and specialists), which can lead to better continuity of care and better patient outcomes.

1. [73], Burkhart et al., 2025, “Foundation models for electronic health records: Representation dynamics and transferability,” http://arxiv.org/abs/2504.10422; [74], Futoma et al., 2020, “The myth of generalisability in clinical research and machine learning in health care,” The Lancet Digital Health; [9], Van Calster et al., 2023, “There is no such thing as a validated prediction model,” BMC Medicine

2. [75], Waxler et al., 2025, “Generative Medical Event Models Improve with Scale,” http://arxiv.org/abs/2508.12104; [76], Zhang et al., 2025, “Exploring Scaling Laws for EHR Foundation Models,” http://arxiv.org/abs/2505.22964; [77], Huang et al., 2025, “Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling,” Forty-second International Conference on Machine Learning

3. [78], Kern et al., 2024, “Care Fragmentation, Care Continuity, and Care Coordination—How They Differ and Why It Matters,” JAMA internal medicine

Coming back to the those 84 patients that we started with, we can see that this could just be the beginning of AI in healthcare. By closing the reproducibility gap, we can enable scientist to build upon each others work rather than each starting from scratch. This should lead to prolonged patient lives through trustworthy and equitable healthcare AI for the many.

Acknowledgements

All friends, colleagues, mentors, collaborators, reviewers, and family who supported me.

HPI:

• Comittee: Christoph Lippert, Lothar Wieler, Felix Naumann, Gerard de Melo, Bernhard Renard
• Executive assistants: Edit Tatar, Cornelia Philippson, Katharina Lorenz-Schroeder, Lena Kaese.
• Students: Hendrik Schmidt, Daniela Zuluaga Lotero, Youssef Mecky

HPI:

• All people in the Digital Health - Connected Healthcare/Machine Learning groups and the Digital Health Cluster

Charité:

• Max M. Maurer
• Axel Winter
• Daniela Zuluaga Lotero

Mentors and Collaborators:

• Patrick Rockenschaub (Medical University of Vienna)
• Matthew McDermott (Columbia University)
• Eugenia Alleva (Mount Sinai)
• Thomas Sutter (ETH Zurich)

Reviewers:

• Padhraic Smyth (UC Irvine)
• Mykola Pechenizkiy (TU Eindhoven)

I am sure I forgot people here..

View the slides.

Source: https://xkcd.com/1403/

Read the thesis.

• With that, I want to thank all the people who supported me during this journey and all people that are present here today.

References

[1]. Rossetti, Sarah C., Patricia C. Dykes, Chris Knaplund, Sandy Cho, Jennifer Withall, Graham Lowenthal, David Albers, Rachel Y. Lee, Haomiao Jia, et al. (2025). “Real-time surveillance system for patient deterioration: A pragmatic cluster-randomized controlled trial.” Nature Medicine, 31(6), 1895–1902. Nature Publishing Group.

[2]. Health, Center for Devices and Radiological. (Tue, 03/10/2026 - 09:05). “Artificial Intelligence-Enabled Medical Devices.” FDA. FDA.

[3]. Apell, Petra, Sara Locher, Annie Milde, and Henrik Eriksson. (2026). “Explaining the Slow Adoption of AI Innovations in Health Care: Network Analysis Approach.” JMIR AI, 5(1), e60458. JMIR Publications Inc., Toronto, Canada.

[4]. Henzler, Dennis, Sebastian Schmidt, Ayca Koçar, Sophie Herdegen, Georg L. Lindinger, Menno T. Maris, Marieke A. R. Bak, Dick L. Willems, Hanno L. Tan, et al. (2025). “Healthcare professionals’ perspectives on artificial intelligence in patient care: A systematic review of hindering and facilitating factors on different levels.” BMC Health Services Research, 25(1), 633.

[5]. Agniel, Denis, Isaac S. Kohane, and Griffin M. Weber. (2018). “Biases in electronic health record data due to processes within the healthcare system: Retrospective observational study.” BMJ, 361), k1479. British Medical Journal Publishing Group.

[6]. Futoma, Joseph, Morgan Simons, Trishan Panch, Finale Doshi-Velez, and Leo Anthony Celi. (2020). “The myth of generalisability in clinical research and machine learning in health care.” The Lancet Digital Health, 2(9), e489–e492. Elsevier.

[7]. McDermott, Matthew. (2025). “The (lack of?) Science of Machine Learning for Healthcare.” Proceedings of the 4th Machine Learning for Health Symposium, 19–29. PMLR.

[8]. Donoho, David. (2024). “Data Science at the Singularity.” Harvard Data Science Review, 6(1). The MIT Press.

[9]. Van Calster, Ben, Ewout W. Steyerberg, Laure Wynants, and Maarten van Smeden. (2023). “There is no such thing as a validated prediction model.” BMC Medicine, 21(1), 70.

[10]. Abramoff, Michael D., Tinglong Dai, and James Zou. (2024). “Scaling Adoption of Medical AI — Reimbursement from Value-Based Care and Fee-for-Service Perspectives.” NEJM AI, 1(5), AIpc2400083. Massachusetts Medical Society.

[11]. Mariano, Bernardo. (2020). “Towards a global strategy on digital health.” Bulletin of the World Health Organization, 98(4), 231–231A.

[12]. Van de Water, Robin P., Axel Winter, Daniela Zuluaga Lotero, Bjarne Pfitzner, Lara Faraj, Bert Arnrich, Patrick Rockenschaub, Wenzel Schoning, Thomas Malinka, et al. (2025). “Continuous Multimodal AI with Wearable Vital Signs Predicts Postoperative Complications in the General Ward.” Medrxiv preprint; Submitted to Lancet Digital Health, 2025.11.25.25340950.

[13]. Riepe, Christoph, Robin van de Water, Axel Winter, Bjarne Pfitzner, Lara Faraj, Robert Ahlborn, Maximilian Schulze, Daniela Zuluaga, Christian Schineis, et al. (2025). “90-day mortality prediction in elective visceral surgery using machine learning: A retrospective multicenter development, validation, and comparison study.” International Journal of Surgery, 111(6), 3742.

[14]. Van de Water, Robin, Axel Winter, Max M. Maurer, Felix August Treykorn, Daniela Zuluaga, Bjarne Pfitzner, Igor M. Sauer, and Bert Arnrich. (2024). “Combining Hospital-grade Clinical Data and Wearable Vital Sign Monitoring to Predict Surgical Complications.”

[15]. Van de Water, Robin, Axel Winter, Max M. Maurer, Felix August Treykorn, Bjarne Pfitzner, Igor M. Sauer, and Bert Arnrich. (2024). “Combining Time Series Modalities to Create Endpoint-driven Patient Records.”

[16]. Van de Water, Robin P., Hendrik Schmidt, and Patrick Rockenschaub. (2026). “ReciPies: A lightweight data transformation pipeline for reproducible ML.” Journal of Open Source Software, 11(117), 9261. The Open Journal.

[17]. Van de Water, Robin, Hendrik Nils Aurel Schmidt, Paul Elbers, Patrick Thoral, Bert Arnrich, and Patrick Rockenschaub. (2024). “Yet Another ICU Benchmark: A Flexible Multi-Center Framework for Clinical ML.” Proceedings of The Twelfth International Conference on Learning Representations.

[18]. McDermott, Matthew, Ethan Steinberg, Jason Fries, Robin van de Water, Chao Pang, Patrick Rockenschaub, Pawel Renc, Jungwoo Oh, Kamilė Stankevičiūtė, et al. (2025). “MEDS: An Emerging Data Standard and Ecosystem for Health AI Research.” Accepted to New England Journal Of Medicine (NEJM) AI (2026) vol: 3, issue 6.

[19]. van de Water, Robin (MEDS Working Group), Matthew B. A. McDermott, and Jason A. Fries. (2024). “Medical Event Data Standard (MEDS): Facilitating Machine Learning for Health.” ICLR 2024 workshop on learning from time series for health.

[20]. McDermott, Matthew B. A., Aleksia Kolo, Chao Pang, Edward Choi, Ethan Steinberg, Hyewon Jeong, Jack Gallifant, Jason Alan Fries, Jeffrey N. Chiang, et al. (2024). “MEDS Decentralized, Extensible Validation (MEDS-DEV) Benchmark: Establishing Reproducibility and Comparability in ML for Health.” Machine Learning for Health (ML4H), 15-16 December 2024, Vancouver, Canada.

[21]. McDermott, Matthew B. A., Justin Xu, Teya S. Bergamaschi, Hyewon Jeong, Simon A. Lee, Nassim Oufattole, Patrick Rockenschaub, Kamilė Stankevičiūtė, Ethan Steinberg, et al. (2025). “MEDS: Building Models and Tools in a Reproducible Health AI Ecosystem.” Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, 6243–6244. ACM.

[22]. Van de Water, Robin P. (2025). “Scaling Up Clinical ML from Datasets to Entire Health Systems through the MEDS Ecosystem.”

[23]. Kirchler, Matthias, Matteo Ferro, Veronica Lorenzini, Robin van de Water, Christoph Lippert, and Andrea Ganna. (2025). “Large language models improve transferability of electronic health record-based predictions across countries and coding systems.” Accepted to npj Digital Medicine.

[24]. Alefs, Katharina, Susanne Ibing, Pia Francesca Rissom, Jan Carlo Schmid, Arkadiusz Kwasigroch, Robin van de Water, Bernhard Y. Renard, and Eugenia Alleva. (2025). “Towards Foundation Model-Based Propensity Score Matching from Electronic Health Records.”

[25]. Pfitzner, Bjarne, Max M. Maurer, Axel Winter, Christoph Riepe, Igor M. Sauer, Robin van de Water, Christian Denecke, Johann Pratschke, and Bert Arnrich. (2025). “Differentially-Private Federated Learning with Non-IID Data For Surgical Risk Prediction.” International Journal of Semantic Computing, 19(3).

[26]. Winter, Axel, Bjarne Pfitzner, Robin P. van de Water, Lara Faraj, Christoph Riepe, Wolf-Heinrich Hahn, Felix Krenzien, Christian Schineis, Thomas Malinka, et al. (2025). “Overcoming the data barrier: Transfer learning for 90-day mortality prediction in general surgery–a retrospective multicenter development and comparison study.” International Journal of Surgery, 10–1097. LWW.

[27]. Van de Water, Robin Philippus, Ethan Steinberg, Michael Wornow, Patrick Rockenschaub, and Matthew McDermott. (2025). “MIMIC-IV demo data in the Medical Event Data Standard (MEDS).” PhysioNet.

[28]. Maurer, Max M., Bjarne Pfitzner, Robin P. van de Water, Lara Faraj, Christoph Riepe, Daniela Zuluaga, Felix Krenzien, Nathanael Raschzok, Robert Siegel, et al. (2025). “Privacy preserving federated learning for 90-day mortality prediction in colorectal surgery: A multicenter retrospective development and comparison study.” International Journal of Surgery (London, England).

[29]. Winter, Axel, Robin P. van de Water, Bjarne Pfitzner, Marius Ibach, Christoph Riepe, Robert Ahlborn, Lara Faraj, Felix Krenzien, Eva M. Dobrindt, et al. (2024). “Enhancing Preoperative Outcome Prediction: A Comparative Retrospective Case–Control Study on Machine Learning versus the International Esodata Study Group Risk Model for Predicting 90-Day Mortality in Oncologic Esophagectomy.” Cancers, 16(17), 3000. MDPI.

[30]. Pfitzner, Bjarne, Max M. Maurer, Axel Winter, Christoph Riepe, Igor M. Sauer, Robin van de Water, and Bert Arnrich. (2024). “Differentially-Private Federated Learning with Non-IID Data for Surgical Risk Prediction.” 2024 IEEE First International Conference on Artificial Intelligence for Medicine, Health and Care (AIMHC), 120–129. IEEE.

[31]. Konak, Orhan, Robin van de Water, Valentin Döring, Tobias Fiedler, Lucas Liebe, Leander Masopust, Kirill Postnov, Franz Sauerwald, Felix Treykorn, et al. (2023). “HARE: Unifying the Human Activity Recognition Engineering Workflow.” Sensors, 23(23), 9571. MDPI.

[32]. Konak, Orhan, Alexander Wischmann, Robin van de Water, and Bert Arnrich. (2023). “A Real-time Human Pose Estimation Approach for Optimal Sensor Placement in Sensor-based Human Activity Recognition.” Proceedings of the 8th international Workshop on Sensor-Based Activity Recognition and Artificial Intelligence, 1–6. ACM.

[33]. Van de Water, Robin, Francesco Ventura, Zoi Kaoudi, Jorge-Arnulfo Quiane-Ruiz, and Volker Markl. (2022). “Farming Your ML-based Query Optimizer’s Food.” 2022 IEEE 38th International Conference on Data Engineering (ICDE), 3186–3189. IEEE.

[34]. Van de Water, Robin, Francesco Ventura, Z Kaoudi, J Quiane-Ruiz, and Volker Markl. (2022). “Farm your ML-based query optimizer’s Food!–Human-Guided training data generation–.” Conference on Innovative Data Systems Research (CIDR).

[35]. Maas, Lientje, Mathan Geurtsen, Florian Nouwt, Stefan F. Schouten, Robin van de Water, Sandra Van Dulmen, Fabiano Dalpiaz, Kees Van Deemter, and Sjaak Brinkkemper. (2020). “The Care2Report System: Automated Medical Reporting as an Integrated Solution to Reduce Administrative Burden in Healthcare.” HICSS, 1–10.

[36]. Schmid, Jan Carlo, Susanne Ibing, Stefan Kalabakov, Katharina Alefs, Robin P. van de Water, Arkadiusz Kwasigroch, Eugenia Alleva, Bert Arnrich, Bernhard Y. Renard, et al. (2025). “Expert-defined, chart-reviewed features outperform EHR foundation models for predicting crohn’s-like disease of the pouch.” Under review at The American Journal of Gastroenterology.

[37]. Turubayev, Alisher, Anna Shopova, Fabian Lange, Mahmut Kamalak, Paul Mattes, Victoria Ayvasky, Bert Arnrich, Bjarne Pfitzner, and Robin P. van de Water. (2025). “Closing Gaps: An Imputation Analysis of ICU Vital Signs.”

[38]. Dobson, Geoffrey P. (2020). “Trauma of major surgery: A global problem that is not going away.” International Journal of Surgery (London, England), 81), 47–54.

[39]. Shrime, Mark G, Stephen W Bickler, Blake C Alkire, and Charlie Mock. (2015). “Global burden of surgical disease: An estimation from the provider perspective.” The Lancet Global Health, 3), S8–S9.

[40]. Downey, Candice L., Jamie Bainbridge, David G. Jayne, and David M. Meads. (2023). “Impact of in-hospital postoperative complications on quality of life up to 12 months after major abdominal surgery.” The British Journal of Surgery, 110(9), 1206–1212.

[41]. Mualla, Noor M., Maryam R. Hussain, Muhammad Akrmah, Preeti Malik, Sadia Bashir, and Jenny J. Lin. (2021). “The Impact of Postoperative Complications on Long-Term Oncological Outcomes Following Curative Resection of Colorectal Cancer (Stage I-III): A Systematic Review and Meta-Analysis.” Cureus, 13(1), e12837.

[42]. Baum, Philip, Johannes Diers, Sven Lichthardt, Carolin Kastner, Nicolas Schlegel, Christoph-Thomas Germer, and Armin Wiegering. (2019). “Mortality and Complications Following Visceral Surgery.” Deutsches Ärzteblatt international.

[43]. Roppelt, Julia Stefanie, Dominik K. Kanbach, and Sascha Kraus. (2024). “Artificial intelligence in healthcare institutions: A systematic literature review on influencing factors.” Technology in Society, 76), 102443.

[44]. Stam, Wessel T, Lotte K Goedknegt, Erik W Ingwersen, Linda J Schoonmade, Emma R J Bruns, and Freek Daams. (2022). “The prediction of surgical complications using artificial intelligence in patients undergoing major abdominal surgery: A systematic review.” Surgery, 171(4), 1014–1021.

[45]. Van de Water, Robin, Axel Winter, Max M. Maurer, Felix August Treykorn, Daniela Zuluaga, Bjarne Pfitzner, Igor M. Sauer, and Bert Arnrich. (2024). “Combining Hospital-grade Clinical Data and Wearable Vital Sign Monitoring to Predict Surgical Complications.”

[46]. Moor, Michael, Bastian Rieck, Max Horn, Catherine R. Jutzeler, and Karsten Borgwardt. (2021). “Early Prediction of Sepsis in the ICU Using Machine Learning: A Systematic Review.” Frontiers in Medicine, 8).

[47]. Fleuren, Lucas M., Thomas L. T. Klausch, Charlotte L. Zwager, Linda J. Schoonmade, Tingjie Guo, Luca F. Roggeveen, Eleonora L. Swart, Armand R. J. Girbes, Patrick Thoral, et al. (2020). “Machine learning for the prediction of sepsis: A systematic review and meta-analysis of diagnostic test accuracy.” Intensive Care Medicine, 46(3), 383–400.

[48]. Shillan, Duncan, Jonathan A. C. Sterne, Alan Champneys, and Ben Gibbison. (2019). “Use of machine learning to analyse routinely collected intensive care unit data: A systematic review.” Critical Care, 23(1), 284.

[49]. Johnson, Alistair E. W., Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J. Pollard, Benjamin Moody, Brian Gow, et al. (2023). “MIMIC-IV, a freely accessible electronic health record dataset.” Scientific Data, 10(1), 1.

[50]. Seymour, Christopher W., Vincent X. Liu, Theodore J. Iwashyna, Frank M. Brunkhorst, Thomas D. Rea, André Scherag, Gordon Rubenfeld, Jeremy M. Kahn, Manu Shankar-Hari, et al. (2016). “Assessment of Clinical Criteria for Sepsis: For the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3).” JAMA, 315(8), 762.

[51]. Moor, Michael, Nicolas Bennet, Drago Plecko, Max Horn, Bastian Rieck, Nicolai Meinshausen, Peter Bühlmann, and Karsten Borgwardt. (2021). “Predicting sepsis in multi-site, multi-national intensive care cohorts using deep learning.”

[52]. Calvert, Jacob S., Daniel A. Price, Uli K. Chettipally, Christopher W. Barton, Mitchell D. Feldman, Jana L. Hoffman, Melissa Jay, and Ritankar Das. (2016). “A computational approach to early sepsis detection.” Computers in Biology and Medicine, 74), 69–73.

[53]. Bommasani, Rishi, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, et al. (2021). “On the Opportunities and Risks of Foundation Models.”

[54]. Steinberg, Ethan, Ken Jung, Jason A. Fries, Conor K. Corbin, Stephen R. Pfohl, and Nigam H. Shah. (2021). “Language models are an effective representation learning technique for electronic health record data.” Journal of Biomedical Informatics, 113), 103637.

[55]. Hripcsak, George, Jon D Duke, Nigam H Shah, Christian G Reich, Vojtech Huser, Martijn J Schuemie, Marc A Suchard, Rae Woong Park, Ian Chi Kei Wong, et al. (2015). “Observational Health Data Sciences and Informatics (OHDSI): Opportunities for Observational Researchers.” Studies in health technology and informatics, 216), 574–578.

[56]. Collins, Francis S, Kathy L Hudson, Josephine P Briggs, and Michael S Lauer. (2014). “PCORnet: Turning a dream into reality.” Journal of the American Medical Informatics Association, 21(4), 576–577.

[57]. Bender, Duane, and Kamran Sartipi. (2013). “HL7 FHIR: An Agile and RESTful approach to healthcare information exchange.” Proceedings of the 26th IEEE International Symposium on Computer-Based Medical Systems, 326–331.

[58]. Murphy, Shawn N, Griffin Weber, Michael Mendis, Vivian Gainer, Henry C Chueh, Susanne Churchill, and Isaac Kohane. (2010). “Serving the enterprise and beyond with informatics for integrating biology and the bedside (I2b2).” Journal of the American Medical Informatics Association, 17(2), 124–130.

[59]. Callahan, Alison, Saurabh Gombar, Eli M Cahan, Kenneth Jung, Ethan Steinberg, Vladimir Polony, Keith Morse, Robert Tibshirani, Trevor Hastie, et al. (2021). “Using aggregate patient data at the bedside via an on-demand consultation service.” NEJM Catalyst Innovations in Care Delivery, 2(10). Massachusetts Medical Society.

[60]. Suchard, Marc A, Martijn J Schuemie, Harlan M Krumholz, Seng Chan You, RuiJun Chen, Nicole Pratt, Christian G Reich, Jon Duke, David Madigan, et al. (2019). “Comprehensive comparative effectiveness and safety of first-line antihypertensive drug classes: A systematic, multinational, large-scale analysis.” The Lancet, 394(10211), 1816–1826.

[61]. Xu, Justin, Jack Gallifant, Alistair Johnson, and Matthew B. A. McDermott. (2024). “ACES: Automatic Cohort Extraction System for Event-Stream Datasets.”

[62]. Oufattole, Nassim, Teya Bergamaschi, Aleksia Kolo, Hyewon Jeong, Hanna Gaggin, Collin M. Stultz, and Matthew B. A. McDermott. (2024). “MEDS-Tab: Automated tabularization and baseline methods for MEDS datasets.”

[63]. Pang, Chao, Xinzhuo Jiang, Krishna S. Kalluri, Matthew Spotnitz, RuiJun Chen, Adler Perotte, and Karthik Natarajan. (2021). “CEHR-BERT: Incorporating temporal information from structured EHR data to improve prediction tasks.”

[64]. Hur, Kyunghoon, Jungwoo Oh, Junu Kim, Jiyoun Kim, Min Jae Lee, Eunbyeol Cho, Seong-Eun Moon, Young-Hak Kim, Louis Atallah, et al. (2024). “GenHPF: General Healthcare Predictive Framework for Multi-Task Multi-Source Learning.” IEEE Journal of Biomedical and Health Informatics, 28(1), 502–513.

[65]. Van de Water, Robin P., Katharina Marie Alefs, Susanne Ibing, Stefan Kalabakov, Aadil Rasheed, Pia Francesca Rissom, Jan Carlo Schmid, Arkadiusz Kwasigroch, Eugenia Alleva, et al. (2025). “FLAMES: EHR foundation models as a resource for health systems.”

[66]. Pang, Chao, Vincent Jeanselme, Young Sang Choi, Xinzhuo Jiang, Zilin Jing, Aparajita Kashyap, Yuta Kobayashi, Yanwei Li, Florent Pollet, et al. (2025). “FoMoH: A clinically meaningful foundation model evaluation for structured electronic health records.”

[67]. Pang, Chao, Jiheum Park, Xinzhuo Jiang, Nishanth Parameshwar Pavinkurve, Krishna S. Kalluri, Shalmali Joshi, Noémie Elhadad, and Karthik Natarajan. (2025). “CEHR-XGPT: A Scalable Multi-Task Foundation Model for Electronic Health Records.”

[68]. Steinberg, Ethan, Jason Alan Fries, Yizhe Xu, and Nigam Shah. (2023). “MOTOR: A Time-to-Event Foundation Model For Structured Medical Records.”

[69]. Borges do Nascimento, Israel Júnior, Hebatullah Abdulazeem, Lenny Thinagaran Vasanthan, Edson Zangiacomi Martinez, Miriane Lucindo Zucoloto, Lasse Østengaard, Natasha Azzopardi-Muscat, Tomas Zapata, and David Novillo-Ortiz. (2023). “Barriers and facilitators to utilizing digital health technologies by healthcare professionals.” npj Digital Medicine, 6(1), 161. Nature Publishing Group.

[70]. Wong, Andrew, Erkin Otles, John P. Donnelly, Andrew Krumm, Jeffrey McCullough, Olivia DeTroyer-Cooley, Justin Pestrue, Marie Phillips, Judy Konye, et al. (2021). “External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients.” JAMA internal medicine, 181(8), 1065–1070.

[71]. Joshi, Shalmali, Iñigo Urteaga, Wouter A C van Amsterdam, George Hripcsak, Pierre Elias, Benjamin Recht, Noémie Elhadad, James Fackler, Mark P Sendak, et al. (2025). “AI as an intervention: Improving clinical outcomes relies on a causal approach to AI development and validation.” Journal of the American Medical Informatics Association, 32(3), 589–594.

[72]. Hosseini, Mohammad, Peng Gao, and Carolina Vivas-Valencia. (2025). “A social-environmental impact perspective of generative artificial intelligence.” Environmental Science and Ecotechnology, 23), 100520.

[73]. Burkhart, Michael C., Bashar Ramadan, Zewei Liao, Kaveri Chhikara, Juan C. Rojas, William F. Parker, and Brett K. Beaulieu-Jones. (2025). “Foundation models for electronic health records: Representation dynamics and transferability.”

[74]. Futoma, Joseph, Morgan Simons, Trishan Panch, Finale Doshi-Velez, and Leo Anthony Celi. (2020). “The myth of generalisability in clinical research and machine learning in health care.” The Lancet Digital Health, 2(9), e489–e492. Elsevier.

[75]. Waxler, Shane, Paul Blazek, Davis White, Daniel Sneider, Kevin Chung, Mani Nagarathnam, Patrick Williams, Hank Voeller, Karen Wong, et al. (2025). “Generative Medical Event Models Improve with Scale.”

[76]. Zhang, Sheng, Qin Liu, Naoto Usuyama, Cliff Wong, Tristan Naumann, and Hoifung Poon. (2025). “Exploring Scaling Laws for EHR Foundation Models.”

[77]. Huang, Hongzhi, Defa Zhu, Banggu Wu, Yutao Zeng, Ya Wang, Qiyang Min, and Zhou Xun. (2025). “Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling.”

[78]. Kern, Lisa M., Julie P. W. Bynum, and Harold Alan Pincus. (2024). “Care Fragmentation, Care Continuity, and Care Coordination—How They Differ and Why It Matters.” JAMA internal medicine, 184(3), 236–237.

[79]. Gaffney, Adam, Steffie Woolhandler, David U. Himmelstein, and Danny McCormick. (2025). “Health care in the USA: Money has become the mission.” The Lancet, 406(10519), 2588–2600. Elsevier.

[80]. Jang, Eugene. (2025). “Oops? Interdisciplinary Stories of Sociotechnical Error| When Faulty AI Falls Into the Wrong Hands: The Risks of Erroneous AI-Driven Healthcare Decisions.” International Journal of Communication, 19), 7–7.

[81]. Chouffani El Fassi, Sammy, Adonis Abdullah, Ying Fang, Sarabesh Natarajan, Awab Bin Masroor, Naya Kayali, Simran Prakash, and Gail E. Henderson. (2024). “Not all AI health tools with regulatory authorization are clinically validated.” Nature Medicine, 30(10), 2718–2720. Nature Publishing Group.

[82]. van Amsterdam, Wouter, Michael Kamp, Rajesh Ranganath, Robin P. van de Water, Florian Markowetz, Evangelia Christodoulou, Yamuna Krishnamurthy, Christoph Lippert, Jeff Clark, et al. (2025). “Missing incentives, missing oversight: The challenges of AI monitoring in clinical practice.”

[83]. Van Calster, Ben, Laure Wynants, Jan F. M. Verbeek, Jan Y. Verbakel, Evangelia Christodoulou, Andrew J. Vickers, Monique J. Roobol, and Ewout W. Steyerberg. (2018). “Reporting and Interpreting Decision Curve Analysis: A Guide for Investigators.” European urology, 74(6), 796–804.

[84]. Lundberg, Scott M, and Su-In Lee. (2017). “A unified approach to interpreting model predictions.” Advances in neural information processing systems, 30). Curran Associates, Inc.

[85]. Johnson, Alistair E. W., Tom J. Pollard, Lu Shen, Li-wei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, et al. (2016). “MIMIC-III, a freely accessible critical care database.” Scientific Data, 3(1), 160035.

[86]. Pollard, Tom J., Alistair E. W. Johnson, Jesse D. Raffa, Leo A. Celi, Roger G. Mark, and Omar Badawi. (2018). “The eICU Collaborative Research Database, a freely available multi-center database for critical care research.” Scientific Data, 5(1), 180178.

[87]. Hyland, Stephanie L. (2020). “Early prediction of circulatory failure in the intensive care unit using machine learning.” Nature MedIcIne, 26), 28.

[88]. Thoral, Patrick J., Jan M. Peppink, Ronald H. Driessen, Eric J. G. Sijbrands, Erwin J. O. Kompanje, Lewis Kaplan, Heatherlee Bailey, Jozef Kesecioglu, Maurizio Cecconi, et al. (2021). “Sharing ICU Patient Data Responsibly Under the Society of Critical Care Medicine/European Society of Intensive Care Medicine Joint Data Science Collaboration: The Amsterdam University Medical Centers Database (AmsterdamUMCdb) Example*.” Critical Care Medicine, 49(6), e563–e577.

[89]. Baker, Stephanie, Wei Xiang, and Ian Atkinson. (2020). “Continuous and automatic mortality risk prediction using vital signs in the intensive care unit: A hybrid neural network approach.” Scientific Reports, 10(1), 21282.

[90]. Lu, Sheng-Chieh, Cai Xu, Chandler H. Nguyen, Yimin Geng, André Pfob, and Chris Sidey-Gibbons. (2022). “Machine Learning–Based Short-Term Mortality Prediction Models for Patients With Cancer Using Electronic Health Record Data: Systematic Review and Critical Appraisal.” JMIR Medical Informatics, 10(3), e33182. JMIR Publications Inc., Toronto, Canada.

[91]. Medic, Goran, Melodi Kosaner Kließ, Louis Atallah, Jochen Weichert, Saswat Panda, Maarten Postma, and Amer El-Kerdi. (2019). “Evidence-based Clinical Decision Support Systems for the prediction and detection of three disease states in critical care: A systematic literature review.” F1000Research, 8), 1728.

[92]. Sharma, Alok, Anupam Shukla, Ritu Tiwari, and Apoorva Mishra. (2017). “Mortality Prediction of ICU patients using Machine Leaning: A survey.” Proceedings of the International Conference on Compute and Data Analysis - ICCDA ’17, 49–53. ACM Press.

[93]. Syed, Mahanazuddin, Shorabuddin Syed, Kevin Sexton, Hafsa Bareen Syeda, Maryam Garza, Meredith Zozus, Farhanuddin Syed, Salma Begum, Abdullah Usama Syed, et al. (2021). “Application of Machine Learning in Intensive Care Unit (ICU) Settings Using MIMIC Dataset: Systematic Review.” Informatics, 8(1), 16.

[94]. Huang, Wei, Yuwen Chen, Peng Wang, Xiang Liu, and Shuguang Liu. (2021). “An Interpretable Temporal Convolutional Network Model for Acute Kidney Injury Prediction in the Intensive Care Unit.” 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), 3021–3028.

[95]. Nikkinen, Okke, Timo Kolehmainen, Toni Aaltonen, Elias Jämsä, Seppo Alahuhta, and Merja Vakkala. (2022). “Developing a supervised machine learning model for predicting perioperative acute kidney injury in arthroplasty patients.” Computers in Biology and Medicine, 144), 105351.

[96]. Pan, Ziyuan, Hao Du, Kee Yuan Ngiam, Fei Wang, Ping Shum, and Mengling Feng. (2019). “A Self-Correcting Deep Learning Approach to Predict Acute Conditions in Critical Care.”

[97]. Rank, Nina, Boris Pfahringer, Jörg Kempfert, Christof Stamm, Titus Kühne, Felix Schoenrath, Volkmar Falk, Carsten Eickhoff, and Alexander Meyer. (2020). “Deep-learning-based real-time prediction of acute kidney injury outperforms human predictive performance.” npj Digital Medicine, 3(1), 139.

[98]. Shamout, Farah, Tingting Zhu, and David A. Clifton. (2021). “Machine Learning for Clinical Outcome Prediction.” IEEE Reviews in Biomedical Engineering, 14), 116–126.

[99]. Wang, Hui, Fuxing Deng, Buyao Zhang, and Shuangping Zhao. (2020). “Real-Time Prediction of AKI Among Middle-Aged and Older in ICU: A Retrospective and Machine Learning Study.” In Review.

[100]. Koyner, Jay L., Kyle A. Carey, Dana P. Edelson, and Matthew M. Churpek. (2018). “The Development of a Machine Learning Inpatient Acute Kidney Injury Prediction Model*:” Critical Care Medicine, 46(7), 1070–1077.

[101]. Kok, Christopher, V. Jahmunah, Shu Lih Oh, Xujuan Zhou, Raj Gururajan, Xiaohui Tao, Kang Hao Cheong, Rashmi Gururajan, Filippo Molinari, et al. (2020). “Automated prediction of sepsis using temporal convolutional network.” Computers in Biology and Medicine, 127), 103957.

[102]. Lauritsen, Simon Meyer, Mads Ellersgaard Kalør, Emil Lund Kongsgaard, Katrine Meyer Lauritsen, Marianne Johansson Jørgensen, Jeppe Lange, and Bo Thiesson. (2020). “Early detection of sepsis utilizing deep learning on electronic health record event sequences.” Artificial Intelligence in Medicine, 104), 101820.

[103]. Merath, Katiuscha, J. Madison Hyer, Rittal Mehta, Ayesha Farooq, Fabio Bagante, Kota Sahara, Diamantis I. Tsilimigras, Eliza Beal, Anghela Z. Paredes, et al. (2020). “Use of Machine Learning for Prediction of Patient Risk of Postoperative Complications After Liver, Pancreatic, and Colorectal Surgery.” Journal of Gastrointestinal Surgery, 24(8), 1843–1851.

[104]. Fleuren, Lucas M., Patrick Thoral, Duncan Shillan, Ari Ercole, Paul W. G. Elbers, Mark Hoogendoorn, Ben Gibbison, Thomas L. T. Klausch, Tingjie Guo, et al. (2020). “Machine learning in intensive care medicine: Ready for take-off?” Intensive Care Medicine, 46(7), 1486–1488.

[105]. Moor, Michael, Max Horn, Bastian Rieck, Damian Roqueiro, and Karsten Borgwardt. (2019). “Early Recognition of Sepsis with Gaussian Process Temporal Convolutional Networks and Dynamic Time Warping.” Proceedings of the 4th Machine Learning for Healthcare Conference, 2–26. PMLR.

[106]. Muralitharan, Sankavi, Walter Nelson, Shuang Di, Michael McGillion, Pj Devereaux, Neil Grant Barr, and Jeremy Petch. (2021). “Machine Learning–Based Early Warning Systems for Clinical Deterioration: Systematic Scoping Review.” Journal of Medical Internet Research, 23(2), e25187.

[107]. Reyna, Matthew A, Chris Josef, Salman Seyedi, Russell Jeter, Supreeth P Shashikumar, M Brandon Westover, Ashish Sharma, Shamim Nemati, and Gari D Clifford. (2019). “Early Prediction of Sepsis from Clinical Data: The PhysioNet/Computing in Cardiology Challenge 2019.” 2019 Computing in Cardiology (CinC), Page 1–Page 4.

[108]. Wang, Yuqing, Yun Zhao, Rachael Callcut, and Linda Petzold. (2022). “Integrating Physiological Time Series and Clinical Notes with Transformer for Early Prediction of Sepsis.”

[109]. Tomašev, Nenad, Xavier Glorot, Jack W. Rae, Michal Zielinski, Harry Askham, Andre Saraiva, Anne Mottram, Clemens Meyer, Suman Ravuri, et al. (2019). “A clinically applicable approach to continuous prediction of future acute kidney injury.” Nature, 572(7767, 7767), 116–119. Nature Publishing Group.

[110]. Futoma, Joseph, Mark Sendak, Blake Cameron, and Katherine Heller. (2016). “Predicting Disease Progression with a Model for Multivariate Longitudinal Clinical Data.” Proceedings of the 1st Machine Learning for Healthcare Conference, 42–54. PMLR.

[111]. Perotte, Adler, Rajesh Ranganath, Jamie S Hirsch, David Blei, and Noémie Elhadad. (2015). “Risk prediction for chronic kidney disease progression using heterogeneous electronic health record data and time series analysis.” Journal of the American Medical Informatics Association : JAMIA, 22(4), 872–880.

[112]. Cheng, Peng, Lemuel R. Waitman, Yong Hu, and Mei Liu. (2018). “Predicting Inpatient Acute Kidney Injury over Different Time Horizons: How Early and Accurate?” AMIA Annual Symposium Proceedings, 2017), 565–574.

[113]. Shillan, Duncan, Jonathan A. C. Sterne, Alan Champneys, and Ben Gibbison. (2019). “Use of machine learning to analyse routinely collected intensive care unit data: A systematic review.” Critical Care, 23(1), 284.

[114]. Guo, Chonghui, Menglin Lu, and Jingfeng Chen. (2020). “An evaluation of time series summary statistics as features for clinical prediction tasks.” BMC Medical Informatics and Decision Making, 20(1), 48.

[115]. Maaten, Laurens van der, and Geoffrey Hinton. (2008). “Visualizing Data using t-SNE.” Journal of Machine Learning Research, 9(86), 2579–2605.

Extra Slides: Motivation

V’s of Big Data in Health

V	Description	Health Data
Volume	Amount of data	Millions of patient records, lab results, notes, and images
Variety	Different types and sources of data	Structured (labs, meds), unstructured (notes), time series (vitals), images, genomics
Velocity	Speed at which data is generated and processed	Real-time vital signs, continuous monitoring, frequent updates
Veracity	Data quality, accuracy, and trustworthiness	Missing values, errors in coding, inconsistent documentation
Value	Usefulness of data for insights and decision-making	Predicting patient deterioration, supporting clinical decisions
Variability	Changes in meaning/context, heterogeneity across sources	Different EHR systems, varying clinical workflows, evolving coding standards

AI Mixed Success

- Need for oversight (e.g., insurance discrimination, surveillance).^{1 2}

• Can be used for harmful purposes (e.g., insurance discrimination, surveillance, etc.). There are already examples of this happening, and this is why legislation should push for transparency and accountability in the development and deployment of AI models in healthcare.

1. [79], Gaffney et al., 2025, “Health care in the USA: Money has become the mission,” The Lancet; [80], Jang, 2025, “Oops? Interdisciplinary Stories of Sociotechnical Error| When Faulty AI Falls Into the Wrong Hands: The Risks of Erroneous AI-Driven Healthcare Decisions,” International Journal of Communication

2. Adapted from slide deck by Benjamin Glicksberg (Mount Sinai)

Not All AI Tools are Clinically Validated

¹

1. [81], Chouffani El Fassi et al., 2024, “Not all AI health tools with regulatory authorization are clinically validated,” Nature Medicine

Causal Effects of AI Interventions

¹

1. [71], Joshi et al., 2025, “AI as an intervention: Improving clinical outcomes relies on a causal approach to AI development and validation,” Journal of the American Medical Informatics Association

Stakeholder Incentives and Monitoring Gaps

• AI model vendors: Driven to show ROI and avoid bad press; monitoring resources are inconsistent.
• Hospitals/health systems: Value efficiency, cost, and safety; ROI is hard to quantify and infra is uneven.
• Clinicians: Care about outcomes but have limited time and support for monitoring.
• AI researchers/architects: Interested in long-term performance, but funding and incentives favor publications.
• Patients: Most affected, yet lack information and avenues to act. ¹

1. [82], van Amsterdam et al., 2025, “Missing incentives, missing oversight: The challenges of AI monitoring in clinical practice”

Extra Slides: C1: 🛏️ Multi-modal Early Warning Systems

Cohort Selection

Cohort Characteristics

Pipeline

Alarm Strategy and Decision Curve Analysis

• CASHEWS+ reduces alarm burden maintaining precision using alarm strategies¹.

• Decision curve analysis²: how much harm am I willing to accept?

• So how can we integrate this model into the clinical routine? We have to look at alarm systems.

• We can regulate the alarm frequency with a combination of a grace period, persistence, and silencing to prevent alarm fatigue. With this approach, we can achieve around 2 alarms per day on a 100-bed ward.

• But which recall do we want?

• Decision curve analysis including harm-to-benefit ratios for optimal alarm strategies per recall bin using CASHEWS+. This hinges on how much harm (intervention/diagnostics/probes) am I willing to accept. The top alarm reflecting the threshold probability interval is shown above the dashed red line.

• This is a way in which clinicians can make informed decisions about the optimal alarm strategy based on their willingness to accept false positives and available resources.

• Alert persistence, defined as repeated threshold crossings within a sliding time window, was set to three alerts within the preceding 6 h. Following activation, each alarm was automatically silenced for 8 hours, corresponding to one nursing shift13. A grace period of 48 h — defined as the interval during which alerts preceding a complication were considered true positives — was selected based on the observed lead time in the cohort (51 hours; IQR 9.5–103.0).

• Net benefit is calculated as a weighted combination of true and false positives

1. Work with master student Daniela Zuluaga Lotero

2. [83], Van Calster et al., 2018, “Reporting and Interpreting Decision Curve Analysis: A Guide for Investigators,” European urology

Warning system example

Model Comparison

Model Interpretation Grouped by Modality

Fold-averaged, normalized, grouped model attributions according to Shapley values¹:

• Clinician: subtle physiological changes indicate early autonomic stress and neurocardiovascular dysregulation.
• Wearables: core temperature, HRV features, and PPG embeddings are top contributors.

• Taking the common Shapley value approach for model interpretation, we can look at the importance ranking of modalities and their specific sub-features after normalization to laboratory values as the modality of highest importance.
• Importance ranking of modalities and their specific sub-features after normalization to laboratory values as the modality of highest importance
• A clinical interpretation of this is that subtle physiological changes are indicators of early autonomic stress, loss of physiological complexity, and impending decompensation or precursors to the surgical site infection.
• However, wearable modalities are a close second with core temperature, HRV features, and RR interval features being the top single contributors and the PPG embeddings accounting for a large portion of the overall contribution.

1. [84], Lundberg and Lee, 2017, “A unified approach to interpreting model predictions,” Curran Associates, Inc.

Model Explainability Subgrouped by Modality

Alarm Strategy and Decision Curve Analysis

Alarm burden and precision-recall

• Alarm strategy¹: CASHEWS+ balances sensitivity and alarm burden

• Decision curve analysis ²: how much harm am I willing to accept?

  

• So how can we integrate this model into the clinical routine? We have to look at alarm systems.
• On the left we can see the Alarm burden quantified as alarms per patient per day (y-axis) across all recall levels (x-axis) for REWS-based alerts, raw ML predictions using a static threshold
• Here we can see that with a significant decrease in alarm burden, the CASHEWS+ strategy maintains the high precision of the raw ML predictions.
• But which recall do we want?
• Decision curve analysis including harm-to-benefit ratios for optimal alarm strategies per recall bin using CASHEWS+. This hinges on how much harm (intervention/diagnostics/probes) am I willing to accept. The top alarm reflecting the threshold probability interval is shown above the dashed red line.
• This is a way in which clinicians can make informed decisions about the optimal alarm strategy based on their willingness to accept false positives and available resources.
• Now that we’ve discussed a specific clinical use case of ML lets see if others can reproduce these results and build on them.
• Alert persistence, defined as repeated threshold crossings within a sliding time window, was set to three alerts within the preceding 6 h. Following activation, each alarm was automatically silenced for 8 hours, corresponding to one nursing shift13. A grace period of 48 h — defined as the interval during which alerts preceding a complication were considered true positives — was selected based on the observed lead time in the cohort (51 hours; IQR 9.5–103.0).

1. Work with master student Daniela Zuluaga Lotero

2. [83], Van Calster et al., 2018, “Reporting and Interpreting Decision Curve Analysis: A Guide for Investigators,” European urology

Why This Matters: The Gap Between Research and Practice

Lesson Learned: Prospective studies reveal what retrospective studies miss

Aspect	Retrospective	Prospective
Data quality	Pre-existing, variable	Controlled, standardized
Timing	Fixed (events already occurred)	Real-time (can intervene)
Causal inference	Confounded	Can design for causality
Clinical relevance	Varies by hospital	Context-specific
Generalization	Hard to assess	Direct validation possible

Result: Our prospective CASSANDRA study found that hybrid monitoring (ICU + GW wearables) significantly improves early detection of postoperative complications, which retrospective analyses failed to capture.

The Clinical Deployment Gap

	ICU	General Ward (GW)
Monitoring Frequency	High (vital signs every 1-5 minutes)	Sporadic (every shift/8 hours)
Intervention Capability	Immediate	Delayed
Staffing & Resources	High	Lower, different patterns
Anomaly Detection	Early, rapid response possible	Often delayed, higher advanced complication rates

Feature Count Ablation

Model Comparison

Embeddings Notes

Embeddings Medications

Intake distribution

Extra Slides: C2: 🧪 Experiment Reproducibility

The YAIB Experimental Pipeline

Creation of harmonized cohorts

Benchmarking Framework

• Ideally, clinical experts inject their clinical domain knowledge of the respective dataset into the dataset and task definitions
• ML researchers can then focus on the modeling and evaluation, while being able to easily reproduce results across datasets and preprocessing choices.

ReciPies: A Simple Modular Preprocessing Framework for ML Research

• ReciPies is a modular preprocessing framework that allows researchers to define and execute data transformation
• UML type diagrams of the main components and their interactions.

ReciPies Example

Dataset Characteristics

Related Work

Datasets and Tasks

Four publicly available ICU datasets:

Dataset	MIMIC-III / IV	eICU	HiRID	AUMCdb
Stays	40k (0.1k) / 73k	201k (2k)	34k	23k
Version	v1.4 / v2.2	v2.0	v1.1.1	v1.0.2
Frequency (time-series)	1 hour	5 minutes	2 / 5 minutes	up to 1 minute
Origin	USA	USA	Switzerland	Netherlands
Originally published	20151 / 20202	20173	20204	20195

Five common ICU prediction tasks: Mortality, Acute Kidney Injury, Sepsis, Kidney Function, Length of Stay.

1. [85], Johnson et al., 2016, “MIMIC-III, a freely accessible critical care database,” Scientific Data

2. [49], Johnson et al., 2023, “MIMIC-IV, a freely accessible electronic health record dataset,” Scientific Data

3. [86], Pollard et al., 2018, “The eICU Collaborative Research Database, a freely available multi-center database for critical care research,” Scientific Data

4. [87], Hyland, 2020, “Early prediction of circulatory failure in the intensive care unit using machine learning,” Nature MedIcIne

5. [88], Thoral et al., 2021, “Sharing ICU Patient Data Responsibly Under the Society of Critical Care Medicine/European Society of Intensive Care Medicine Joint Data Science Collaboration: The Amsterdam University Medical Centers Database (AmsterdamUMCdb) Example*,” Critical Care Medicine

Prediction Task Overview

Note: The related work is non-exhaustive.

No	Task	Frequency	Type	Related Work
1	Mortality	Once per stay*	C	[89], Baker et al., 2020, “Continuous and automatic mortality risk prediction using vital signs in the intensive care unit: A hybrid neural network approach,” Scientific Reports; [90], Lu et al., 2022, “Machine Learning–Based Short-Term Mortality Prediction Models for Patients With Cancer Using Electronic Health Record Data: Systematic Review and Critical Appraisal,” JMIR Medical Informatics; [91], Medic et al., 2019, “Evidence-based Clinical Decision Support Systems for the prediction and detection of three disease states in critical care: A systematic literature review.” F1000Research; [92], Sharma et al., 2017, “Mortality Prediction of ICU patients using Machine Leaning: A survey,” ACM Press; [93], Syed et al., 2021, “Application of Machine Learning in Intensive Care Unit (ICU) Settings Using MIMIC Dataset: Systematic Review,” Informatics
2	Acute Kidney Injury (AKI)	Hourly	C	[94], Huang et al., 2021, “An Interpretable Temporal Convolutional Network Model for Acute Kidney Injury Prediction in the Intensive Care Unit,” 2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM); [95], Nikkinen et al., 2022, “Developing a supervised machine learning model for predicting perioperative acute kidney injury in arthroplasty patients,” Computers in Biology and Medicine; [96], Pan et al., 2019, “A Self-Correcting Deep Learning Approach to Predict Acute Conditions in Critical Care,” http://arxiv.org/abs/1901.04364; [97], Rank et al., 2020, “Deep-learning-based real-time prediction of acute kidney injury outperforms human predictive performance,” npj Digital Medicine; [98], Shamout et al., 2021, “Machine Learning for Clinical Outcome Prediction,” IEEE Reviews in Biomedical Engineering; [99], Wang et al., 2020, “Real-Time Prediction of AKI Among Middle-Aged and Older in ICU: A Retrospective and Machine Learning Study,” In Review; [100], Koyner et al., 2018, “The Development of a Machine Learning Inpatient Acute Kidney Injury Prediction Model*:” Critical Care Medicine
3	Sepsis	Hourly	C	[101], Kok et al., 2020, “Automated prediction of sepsis using temporal convolutional network,” Computers in Biology and Medicine; [102], Lauritsen et al., 2020, “Early detection of sepsis utilizing deep learning on electronic health record event sequences,” Artificial Intelligence in Medicine; [103], Merath et al., 2020, “Use of Machine Learning for Prediction of Patient Risk of Postoperative Complications After Liver, Pancreatic, and Colorectal Surgery,” Journal of Gastrointestinal Surgery; [104], Fleuren et al., 2020, “Machine learning in intensive care medicine: Ready for take-off?” Intensive Care Medicine; [51], Moor et al., 2021, “Predicting sepsis in multi-site, multi-national intensive care cohorts using deep learning,” http://arxiv.org/abs/2107.05230; [105], Moor et al., 2019, “Early Recognition of Sepsis with Gaussian Process Temporal Convolutional Networks and Dynamic Time Warping,” PMLR; [106], Muralitharan et al., 2021, “Machine Learning–Based Early Warning Systems for Clinical Deterioration: Systematic Scoping Review,” Journal of Medical Internet Research; [107], Reyna et al., 2019, “Early Prediction of Sepsis from Clinical Data: The PhysioNet/Computing in Cardiology Challenge 2019,” 2019 Computing in Cardiology (CinC); [98], Shamout et al., 2021, “Machine Learning for Clinical Outcome Prediction,” IEEE Reviews in Biomedical Engineering; [108], Wang et al., 2022, “Integrating Physiological Time Series and Clinical Notes with Transformer for Early Prediction of Sepsis,” http://arxiv.org/abs/2203.14469
4	Kidney Function (KF)	Once per stay*	R	[109], Tomašev et al., 2019, “A clinically applicable approach to continuous prediction of future acute kidney injury,” Nature; [110], Futoma et al., 2016, “Predicting Disease Progression with a Model for Multivariate Longitudinal Clinical Data,” PMLR; [111], Perotte et al., 2015, “Risk prediction for chronic kidney disease progression using heterogeneous electronic health record data and time series analysis,” Journal of the American Medical Informatics Association : JAMIA; [112], Cheng et al., 2018, “Predicting Inpatient Acute Kidney Injury over Different Time Horizons: How Early and Accurate?” AMIA Annual Symposium Proceedings
5	Length of Stay (LoS)	Hourly	R	[113], Shillan et al., 2019, “Use of machine learning to analyse routinely collected intensive care unit data: A systematic review,” Critical Care; [114], Guo et al., 2020, “An evaluation of time series summary statistics as features for clinical prediction tasks,” BMC Medical Informatics and Decision Making

• C: Classification task
  
• R: Regression task
  
• Once per stay: Indicates predictions are made once during the ICU stay.

Transfer Learning

• What if we train a generalized model on “most generalizable” eICU?
• Finetune on “difficult” HiRID

AUROC:

AUPRC:

External Validation with Mortality

AUROC:

AUPRC:

• Column: Model trained on dataset
• Row: Model evaluated on dataset
• Diagonal: Internal validation (same dataset)

Classification performance

Regression performance

Mortality Leakage Experiment

Static Experiment

Sepsis Cohort

Extra Slides: C3: 🏗️ Framework for Foundational AI

How Do We Create Research that Translates to Clinical Practice?

In Data science at the singularity, Donoho defines three characteristics that enable frictionless reproducibility¹ :

• Data: Datafication of everything, with a culture of research data sharing.
• Re-execution: Research code sharing including the ability to exactly re-execute the same complete workflow by different researchers.
  
• Challenges: Adopting challenge problems as a new paradigm powering methodological research.

The translation gap in healthcare ML can be reframed as a failure to achieve the “data science singularity” as defined by Donoho. To reach this state, we need to ensure that data is widely available and standardized, that code is shared and can be re-executed by others, and that we adopt challenge problems to drive methodological research. My thesis addresses these needs through the development of tools and frameworks that promote reproducibility, clinical relevance, and scalable infrastructure for healthcare machine learning research.

1. [8], Donoho, 2024, “Data Science at the Singularity,” Harvard Data Science Review

MEDS FM Timeline

MEDS Modelling

Sequence Lengths of Events in EHR Data

Tokenizer Comparison

Zipf’s Law in EHR Data

{fig-cap=“Zipf’s Law in EHR Data” width=80% .no-dark}

Representations

• Analyze the learned representations of foundation models to understand what they capture and how they differ across tasks and datasets.
• Use dimensionality reduction techniques (e.g., t-SNE ¹, UMAP) to visualize the embeddings and identify clusters or patterns related to specific clinical conditions or outcomes.
• Leukocytis embeddings t-SNE colored by task labels (positive/negative):

1. [115], Maaten and Hinton, 2008, “Visualizing Data using t-SNE,” Journal of Machine Learning Research

HarmonizEHR Embeddings

MEDS-Inspect

MEDS-Inspect Subject Timeline

Differences Between MOTOR and CEHR‑X‑GPT

Aspect	MOTOR	CEHR‑X‑GPT
Primary training objective	Time‑to‑event forecasting over intervals	Next‑token prediction + TD + TTE on time tokens
Main use in paper	Representation learning (linear probing)	Representation, zero‑shot prediction, synthetic EHR generation
Fine‑tuning	Reported to add little over linear probing	Fine‑tuning gives large gains over linear probing on several tasks
Input domains	Includes measurements, observations	Excludes measurements/observations so far
Zero‑shot prediction	Not supported in this comparison	Strong zero‑shot performance on multiple risk tasks
Synthetic data	Not evaluated	Generates large‑scale synthetic OMOP EHR with good utility and low risk

\n" + "\n" + "\n" + "\n" + ""