Built a stroke detector and contactless heart rate monitor for Meta Ray-Ban glasses. Used Claude for the GraphRAG schema and signal processing logic. Here’s what each AI was actually good at.
By Holidays in Europe / April 27, 2026 / No Comments / Uncategorized
Innovative Development of a Contactless Stroke and Heart Rate Monitoring System for Meta Ray-Ban Glasses
In an exciting hackathon project awarded third place and now licensed under MIT, a team successfully developed a novel medical monitoring system integrated seamlessly into Meta Ray-Ban glasses. This system not only detects facial drooping—a key indicator of stroke—with high accuracy but also monitors heart rate contactlessly, providing real-time guidance via an AI-powered voice agent. Here’s a comprehensive look at the project, the AI tools utilized, and the insights gained from their respective strengths.
Project Overview
The custom system operates directly on Meta Ray-Ban smart glasses, leveraging their capabilities to perform critical health assessments in a contactless manner. The core features include:
- Facial Droop Detection: Achieving an Area Under the Receiver Operating Characteristic curve (AUROC) of 0.985, allowing for highly reliable identification of stroke symptoms.
- Heart Rate Monitoring: Using remote photoplethysmography (rPPG) to estimate heart rate without direct contact.
- Voice Guidance: An AI-driven conversational agent, powered by Gemini, delivers real-time clinical guidance aligned with JRCALC 2022 protocols, enabling prompt action in emergency situations.
Developing with Multiple AI Tools
Throughout the development process, the team leveraged a variety of AI assistants, each contributing uniquely at different stages of the project.
ChatGPT: Rapid Development and Boilerplate Tasks
ChatGPT proved invaluable for building the foundational infrastructure. Its strengths include:
- Rapid iteration of the FastAPI WebSocket backend and the React-based dashboard interface.
- Generating complete, functional code snippets when project specifications were clear, accelerating the development timeline.
- Handling boilerplate-heavy components such as Meta Data Access Toolkit (DAT SDK) integration, providing working templates that sped up initial setup.
Claude: Schema Design and Reasoning
Claude excelled in the more complex aspects of designing the system’s core logic:
- Schema Design for GraphRAG: Collaboratively crafted the graph schema representing hierarchical clinical documents, focusing on node types and edge semantics. Claude’s ability to reason about node relationships and tradeoffs resulted in a robust, flexible data structure.
- Debugging Signal Processing Algorithms: Assisted in troubleshooting the harmonic-weighted FFT peak selection method, which required in-depth understanding of frequency domain behavior—a task where Claude’s reasoning capabilities shone.
Gemini Live: Real-Time Voice Interaction
The Gemini platform was employed to manage the real-time voice interface, taking advantage of its native audio streaming support:
- Delivered smooth, responsive voice guidance to users in emergency scenarios.
- Simplified the integration of live audio capabilities, ensuring immediate feedback and interaction.
Key Insights and Workflow Recommendations
The team’s experience underscores a crucial insight: utilizing a combination of AI tools can lead to more efficient and effective development workflows. Each AI had distinct strengths:
- ChatGPT: Best suited for rapid code iteration and boilerplate generation.
- Claude: Excelling in complex reasoning, schema design, and troubleshooting.
- Gemini Live: Optimized for real-time audio and voice interaction.
By employing all three strategically throughout the development lifecycle, the team achieved a sophisticated, reliable system that leverages the unique strengths of each AI.
Conclusion
This project exemplifies the power of integrating multiple AI tools to create innovative health monitoring solutions. The system’s deployment on Meta Ray-Ban glasses demonstrates how wearable technology can be harnessed for critical medical diagnostics and emergency response. Moving forward, adopting a multi-AI workflow can open new frontiers in rapid, intelligent, and contactless health tech development.
This work is now publicly available under an MIT license, reflecting a commitment to open innovation and collaborative progress in healthcare technology.