Project Title: AI Documentation for Clinicians with Smart Glasses

Problem:
Doctors have to document their patient meetings, which takes up a lot of time. There is a need to make the process more efficient for them so that they are able to prioritize their patients instead of doing administration work.

Solution:
The proposed solution is to use smart glasses to capture audio during doctor-patient conversation which sends over an audio file to be processed by AI models. Once the audio file is sent via Bluetooth, it gets turned into a transcript with speaker tags accessible on the user-interface.

The physician can then generate a report from this transcript on the user-interface and use the fact-checker if they wish to compare accuracy between transcript and report.

8 min Video Demo:

By using wearable smart glasses, the doctor is not required to be facing the computer directly when talking with patients, and doesn’t have to rely on a computer microphone especially if a computer doesn’t have one.

By offering direct processing of audio to text, the doctor doesn’t need to recall conversations or take notes while talking to the patient.

With the use of an LLM, the doctor can summarise their conversation with the patient in a structured format.

Our project doesn’t aim to replace the human element in clinical documentation but rather help enhance speed and efficiency. Our design centers on human validation — the doctor always reviews the final product and can look back at the transcript and report to edit as needed.

Since data privacy is a concern, the entire application runs offline. All data is stored locally — within the doctor’s personal laptop using local files or local databases — and is never uploaded externally.

Development Team

Annaelle Mansuy:
Role: LLM summarisation, RAG Pipeline, AudioProcessing Pipeline (Diarisation and Audio to Text), Backend and Frontend communication

Deependra Datta:
Role: Database and Frontend Developer

Cici Liu:
Role:NPU, Frame Glasses Specialist, "Hardware Integration"

Ming Yao:
Role: AudioProcessing (Diarisation and Audio to Text) OpenVino Specialist and Deployment "Packaging"

Project Management (Gantt Chart)

Monthly Update Video Links:

November: November Update

January: January Update

February: February Update

March: March Update