Synaphi: Accessible AI Medical Communication

DEV Community

Beyond Words: Building Synaphi, an AI Medical Assistant for Everyone

This is a submission for the MLH × DEV Writing Challenge, featuring Synaphi, built at EmoryHacks 2026. What I Built Imagine understanding every word your doctor says, but being unable to answer a single question. "Where does it hurt?" For someone recovering from a stroke, living with severe paralysis, or temporarily unable to speak, that simple question can become an enormous communication barrier. And the problem isn't necessarily understanding. It's having a reliable way to respond. At EmoryHacks 2026, we built Synaphi — an AI-assisted patient–clinician communication platform designed around a simple principle: everyone deserves to be heard. Synaphi brings clinicians and patients into a synchronized communication experience. The clinician asks questions, and the patient responds through an accessible interface designed to support different physical and communication abilities. The patient can use: Large visual and pictogram-based choices Keyboard and single-switch scanning Dwell-based selection Touch controls Browser-supported speech input Experimental webcam-based blink and head gestures The platform also supports five languages and can help patients communicate pain location, intensity, and quality. But we made one architectural decision that defines the entire project: AI can help decide which choices to offer. It cannot decide what the patient means. The patient selects the information. The patient confirms the final message. The clinician remains responsible for clinical decisions. Why not just build another medical chatbot? Because a chatbot that guesses what a person wants to say can introduce a serious problem: it might express something the person never intended. For example, if someone indicates pain, AI should not invent additional symptoms or make statements about consent. Our solution separates AI-powered interpretation from deterministic message construction. The intelligence assists communication. It does not replace the communicator. How Synaphi works The system follows a six-stage interaction: A clinician asks a question, such as "Where does it hurt?" Google Gemini interprets the question into structured intent, response type, and search terms. Typesense retrieves relevant choices from a curated communication library. Synaphi displays large, accessible response cards. The patient selects and reviews a response using their preferred controls. Only after confirmation is the message spoken aloud and sent to the clinician. Our frontend uses React and TypeScript. A Python FastAPI backend handles communication and AI orchestration, with WebSockets synchronizing clinician and patient views. We designed the experience to support two devices or a split-screen demonstration. The hardest accessibility problem: one button A patient with very limited mobility may be able to activate only one switch. That means ordinary buttons, dropdowns, and text entry become difficult to use. We implemented two-level switch scanning. Instead of cycling through every possible button individually, the interface highlights groups of actions first, then individual choices within the selected group. The patient can navigate and respond using the Space key as a switch. We tested a complete question-answering flow using only Space-key input. Our emergency Help action remains available independently of Gemini and Typesense. Even when the AI services fail, the patient can still trigger the prototype's local help notification. Experimental facial controls Synaphi also implements facial-landmark detection using MediaPipe inside the browser. For intentional-blink selection, the detection logic includes a calibrated threshold, a deliberate hold period, hysteresis, cooldown, and a requirement to reopen the eyes before another selection. These safeguards are intended to reduce accidental activations. The detection logic has unit tests, and the MediaPipe model has been loaded in automated browser testing. However, reliable blink/head operation on ...

Powered by Readybot.io