‹ back to 2026-08-11

Towards Expert-level Medical AI for Real-time Video Consultations

arXiv cs.AI Medical/Healthcare AI Mahvish Nagda, Jihyeon Lee, Matthew Thompson, Chunjong Park, Tim Strother, Valentin Liévin, Roma Ruparel, Akshay Goel, Teya Bergamaschi, Suhana Bedi, Meet Shah, Pavel Dubov, Liviu Panait, Toshiyuki Fukuzawa, Sam Schmidgall, Craig Schiff, Joseph Xu, Aliya Rysbek, Yana Lunts, Jan Freyberg, Rebecca Hemengway, Sunny Virmani, David Racz, Carey Radebaugh, Joëlle Barral, Kavi Goel, Dale R. Webster, Katherine Chou, Avinatan Hassidim, Yossi Matias, James Manyika, Gregory Wayne, Tao Tu, Yun Liu, Ethan Goh, Christina Chen, Ryutaro Tanno, Po-Hsuan Cameron Chen, Mike Schaekermann, Anil Palepu 2026-08-10

TL;DR - A Gemini-based multi-agent system, AMIE (Video), conducts real-time audio-visual clinical consultations and matched or beat primary care physicians in a randomized OSCE study — the first demonstration of expert-level medical AI in live video consultations.

AMIE (Video) combines low-latency dialogue, clinical reasoning, and real-time audio-visual perception; the authors also built a taxonomy and automated evaluations for clinical audio-visual cues in telehealth.
Randomized OSCE with 30 PCPs, 15 patient actors, and 100 scenarios compared AMIE (Video), AMIE (Text), and PCPs over video; clinical evaluators rated AMIE (Video) on par or better for history-taking, diagnosis, management, and physical observation/examination.
Patient actors preferred AMIE's assessment and explanation of conditions and preferred the video interface over text chat for communicative effectiveness, convenience, and feeling understood; PCPs were still preferred for rapport and partnership building.
Reported limitations: fine anatomical precision, subtle affective nuances, and high-frequency movements — authors state further research is needed before real-world deployment.

view merged work →