Appendicies
GDPR
1. Introduction and Scope
This document sets out the data protection and privacy guidelines for our AI clinical documentation project. Our goal is to support doctors by reducing the administrative burden through the use of audio-capture glasses, real-time processing, transcription, and report generation. While our solution aids clinical documentation, it does not replace the clinical judgment or the traditional methods of record keeping. This policy applies to all data collected, processed, and stored through the project’s various modules, including audio recording, transcription, summarization, and user interactions.
2. Data Controller and Data Processor Responsibilities
Data Controller: The project management team is responsible for ensuring that all data processing activities comply with GDPR. They determine the purposes and means of processing personal data.
Data Processors: All developers and third-party service providers involved in processing data (e.g., those handling transcription or summarization) must adhere to this GDPR policy and any relevant data processing agreements.
3. Data Collection and Processing Activities
3.1 Types of Data Collected
- Audio Recordings: Captured using glasses during clinical interactions.
- Transcripts and Speaker Tags: Generated from audio to provide an accurate record of the consultation.
- User Data: Information related to doctor and patient profiles (e.g., account details, patient identifiers).
3.2 Purpose and Legal Basis
Purpose: To reduce documentation time by converting clinical conversations into structured reports and transcripts, thereby improving accuracy and efficiency.
Legal Basis: Data processing is conducted on the basis of explicit consent from the doctor (and patients, where applicable) and as necessary for the performance of a service in the area of public health. In clinical contexts, additional legal safeguards will be observed for processing health-related data.
3.3 Data Minimisation and Purpose Limitation
Minimisation: Only data strictly necessary for generating the transcript and report is collected.
Purpose Limitation: Audio and transcript data will only be used for the purposes of:
- Generating and verifying clinical documentation.
- Enabling user interactions such as editing reports and managing accounts.
4. Data Storage and Security
4.1 Local Processing and Storage
Data (audio recordings, transcripts, and reports) will be processed and stored locally on the user’s device to minimise exposure and potential data breaches.
4.2 Security Measures
Access Control: Authentication mechanisms (login/logout and password management) are in place to ensure only authorized users can access the system.
5. Data Subject Rights
All data subjects (e.g., doctors, patients) have the following rights under GDPR:
- Right of Access: They can request details of all personal data processed.
- Right to Rectification: Inaccurate or incomplete data will be corrected upon request.
- Right to Erasure: Under appropriate circumstances, individuals can request that their data be deleted.
- Right to Restrict Processing: Data subjects may request limitations on how their data is processed.
- Right to Data Portability: They can request to receive their data in a structured, commonly used format.
- Right to Object: Data subjects may object to data processing based on their specific circumstances.
- Automated Decision-Making: If decisions are made solely on automated processing, individuals will be informed and given the opportunity to seek human intervention.
6. Data Retention and Deletion
Retention Period: Personal and sensitive data will be retained only as long as necessary for the purposes outlined in this document.
Deletion Procedures: Once the data is no longer needed, secure deletion protocols will be implemented. Users can also request data deletion in accordance with their rights under GDPR.
7. Data Breach Procedures
- Notification: In the event of a data breach, the data controller will notify the relevant supervisory authority within 72 hours, unless the breach is unlikely to result in a risk to the rights and freedoms of data subjects.
- Mitigation: Steps will be taken immediately to contain and mitigate any data breaches.
- Communication: Affected users will be informed if the breach is likely to result in a high risk to their rights and freedoms.
8. Data Transfers and Third-Party Processors
Data Transfers: Data will not be transferred outside the secure environment (local processing) unless explicitly required and only after ensuring that adequate safeguards are in place.
Third-Party Processors: All external service providers (e.g., cloud services, transcription services) must enter into a data processing agreement ensuring compliance with GDPR.
9. Data Protection Impact Assessment (DPIA)
A DPIA will be conducted periodically to assess and mitigate any risks associated with data processing activities, particularly in handling sensitive health data. The DPIA will evaluate:
- Potential impacts on data subjects’ privacy.
- Adequacy of security measures.
- Compliance with GDPR obligations.
User Manual for Healthcare Management Application
This manual provides step-by-step instructions on how to use the application—from creating an account to managing patient meetings. Follow the sections below to navigate through the key functionalities.
1. Creating an Account
- Enter Your Personal Details: Input your First Name and Last Name.
- Set Up Your Credentials: Create a unique username and choose a secure password.
- Complete Registration: After entering your information, click the Create Account button. Your account will be set up, allowing you to access the application.
2. Logging In
- Access the Login Screen: Enter the username and password you created during registration.
- Successful Login: Click the Login button. Once authenticated, you will be redirected to the Main Page.
3. Main Page Overview
- Patient List: A complete list of your patients is displayed.
- Search Functionality: Use the Search Bar to quickly find patients by name.
- Sorting Options: Click the Sort Button to arrange the patient list either alphabetically by last name or by MRN number in ascending order.
- Adding a Patient: Use the Add Patient button to input new patient details.
4. Adding a Patient
- Initiate the Process: Click the Add Patient button on the main page.
- Enter Patient Details: Provide the MRN Number (unique identifier), First Name, Last Name, and Date of Birth.
- Confirm Addition: Click the Add Patient button to save the new patient in your system.
5. Viewing Patient Details
- Access Patient Page: Click on a patient from the list on the main page.
- Patient Page Features: View a list of all previous meetings held with the patient.
6. Managing Meetings
Adding a Meeting
- Navigate to the Meeting Page: On the patient’s page, click the Add Meeting button.
- Input Meeting Information: Specify the Date and Time and provide a Summary of the meeting’s content.
- Recording the Meeting:
- Start Recording: Tap the glasses icon or click the Start Recording button located at the top right.
- Stop Recording: Tap the glasses icon again or click the Stop Recording button. A transcript will be generated a few minutes after stopping the recording.
- Generate and Review Report: Click the Generate Report button to create a report from the transcript. To verify the report’s accuracy, use the Compare button in the Fact Check section.
- Save the Meeting: Click the Save Meeting button to store all meeting details.
Viewing and Editing a Meeting
- Accessing Meeting Details: From the patient page, click on a meeting from the list.
- Displayed Information: The meeting page will show the Date and Time, Summary, Transcript, and Report.
- Editing a Meeting: Click the Edit Meeting button, make necessary changes, and click the Save button to update the details.
7. Logging Out
- End Your Session: Click the Logout button on the main home page to securely log out of the application.
This manual provides a complete guide to managing your account, patients, and meetings. Follow these steps to efficiently use the application and maintain organized records of your patient interactions.
Deployment Manual for Team18 Smart Glasses Project
This manual outlines the steps to deploy the Team18 Smart Glasses Project, a Smart Glasses AI system designed for doctors to record consultations, transcribe audio, and generate summary reports — all processed offline using local LLMs. Follow the sections below for a smooth deployment process.
1. Project Structure
The project is organized as follows:
- frontend/ – Electron-based frontend interface
- backend/ – Python backend powered by FastAPI
- models/ – Contains the required LLM and embedding models
2. Prerequisites
Before deployment, ensure you have the following installed:
- Python 3.7+ – Required for the backend.
- Node.js and npm – Required for the frontend.
- Poetry – For managing Python dependencies.
- Electron – For running the frontend.
3. Installing Dependencies with Poetry
3.1 Install Poetry
If Poetry is not installed, run the following command in your terminal:
curl -sSL https://install.python-poetry.org | python3 -
For additional details, refer to the Poetry installation guide.
3.2 Install Backend Dependencies
Navigate to the backend/ folder:
cd backend
Install the dependencies specified in pyproject.toml:
poetry install
This command will create a virtual environment (if one doesn't already exist) and install all required packages. To run commands within the virtual environment, use:
poetry run python main.py
4. LLM & Embedding Models Setup
Download and place the following models in their respective directories within the models/ folder:
-
granite3.1-dense:8b
You can replace the file path with any llm gguff file you download but just make sure there is right file path and change the name or the path to match
Purpose: Generating text embeddings
Purpose: Summarization and generation
File Path:
project_root/models/library-granite3.1-dense-8b/model-44d19d212d76.gguf
-
granite-embedding-125m-english-Q4_K_M
You can replace the file path with any embedding gguff file you download but just make sure there is right file path and change the name or the path to match
Purpose: Generating text embeddings
File Path:
project_root/models/embedding/granite-embedding-125m-english-Q4_K_M.gguf
-
deepseek-r1:7b
You can replace the file path with any gguff deepseek file you download but just make sure there is right file path and change the name or the path to match
Purpose: Generating text embeddings
Purpose: Fact checking
File Path:
project_root/models/library-deepseek-llm/model-60cfdbde0472.gguf
Ensure that each model is correctly downloaded and placed in its designated folder.
5. Running the Project
5.1 Frontend Setup
Navigate to the frontend folder:
cd frontend
Install Electron:
npm install electron --save-dev
Start the Frontend:
npm start
This command launches the Electron-based interface of the project.
5.2 Backend Setup
Navigate to the backend folder:
cd backend
Run the Backend Server (using Poetry):
poetry run python main.py
The backend server, powered by FastAPI, will start and be ready to handle requests from the frontend.
6. Test Login Credentials
For testing purposes, you can use the following credentials to log into the system:
- Username:
doctor
- Password:
password
7. Additional Considerations
Environment Variables: Ensure any required environment variables for your deployment (such as API keys or configuration settings) are properly set in your environment or via a configuration file.
Deployment Environment: For production deployments, consider using process managers (like PM2 for Node.js or systemd for Python) and setting up proper logging and monitoring.
By following this manual, you should be able to successfully deploy the Team18 Smart Glasses Project. For any issues or further customization, refer to the documentation provided with each component or contact your development team.
Blog Posts
Blog Post #1: Project Update
We’re excited to share our first progress update on this innovative project involving smart glasses for medical applications. Our team began by completing an HCI report, which included submitting a detailed presentation. This helped us establish a strong foundation for understanding how users might interact with the technology.
Following that, we held a productive discussion with Dr. Andrew Tan, where we explored the different medical fields that could benefit from using these smart glasses. We also examined how doctors typically store and manage their notes, which gave us valuable insights into the data-handling requirements of the device.
Next, we had a meeting with Dr. Kavin Raju, who provided an overarching view of the project’s potential and guided us on setting realistic goals for our development timeline. These conversations have been instrumental in shaping our objectives and ensuring we’re focusing on features that truly matter in a healthcare setting.
On the technical side, our team conducted an initial test of the glasses’ microphone and camera capabilities using the NOA app. We delved into the open-source code to better understand how we can integrate and leverage the hardware. This hands-on experience has set the stage for more advanced experimentation and refinement.
Looking ahead, we plan to continue working closely with our collaborators to finalize our approach. Our next steps include scheduling another meeting with Dr. Raju for further discussions and receiving our own pair of Frame Smart Glasses to expand testing. We’re looking forward to sharing more updates soon as we move deeper into the development phase.
Stay tuned for our next post, where we’ll delve into the technical challenges we’re tackling and highlight any new milestones we’ve reached. We appreciate your support and can’t wait to see where this journey takes us!
Blog Post #2: Requirements, Research, and More
We’re back with another update on our ongoing project! Our team recently put together a MoSCoW requirements document—breaking down the features into must-have, should-have, could-have, and won’t-have categories. This has helped us prioritize development tasks and ensure we’re focusing on the most crucial elements first.
We also had a productive discussion with Dr. Kavin Raju to review these requirements. During the conversation, we examined some of the challenges posed by the smart glasses hardware and brainstormed potential solutions. This exchange highlighted a few key areas we need to refine before we can move forward with confidence.
On the technical side, we set up a GitHub repository to host our code and collaborate more effectively. This platform will serve as our central hub for version control, documentation, and continuous integration. In addition, our team dived deeper into researching the Frame Smart Glasses—specifically, how to integrate the open-source components and explore text-to-speech mechanisms. We’re also investigating ways to transmit recordings from the glasses to phones or nearby devices.
Looking ahead, we’ll continue refining our requirements based on feedback and ongoing research. Our next steps involve testing the hardware in different environments and working on advanced features like real-time data processing and speech integration. Stay tuned for our next update, where we’ll share more about the progress we’re making in these areas!
Blog Post #3: Exploring AI Transcription, Refining Requirements, and Moving Forward
We’ve made significant progress on our project since the last update. Our team recently explored the potential of using OpenAI’s Whisper technology to transcribe doctor-patient conversations. This sparked discussions about how we can ensure accurate, efficient transcription while maintaining patient privacy. Alongside this, we’ve been looking into various large language models (LLMs) to enhance our data processing and summarization capabilities.
On the hardware side, we tested the smart glasses’ microphone under different conditions. The results highlighted some challenges with picking up quiet or distant voices, prompting us to investigate ways to improve audio capture and clarity. We also produced a brief progress video to demonstrate the glasses’ current capabilities and outline our vision for upcoming features.
We then held a meeting with our teacher assistant and project supervisor to review our updated requirements. During these discussions, we were encouraged to refine our approach to transcribing and summarizing conversations, ensuring we’re focusing on the most critical features first. We also revisited our project timeline, updating our Gantt chart to reflect new deadlines and milestones.
Looking ahead, we’ll continue iterating on our requirements based on the feedback we received, with an emphasis on achieving reliable transcription and exploring advanced AI solutions. Stay tuned for more updates as we dive deeper into development and move closer to delivering a solution that transforms how doctors interact with patient information!
Blog Post #4: SRS, Experimentation, and the Elevator Pitch
We’re excited to share the latest progress on our project. First, our team finalized a Software Requirements Specification (SRS) document. This outlines our core vision for the solution, including detailed features and functionalities. Having a clear roadmap for development is helping us stay organized and focused on the most critical tasks.
Meanwhile, we dove deeper into experimentation with various large language models for report generation. This involved testing prompt engineering techniques and exploring diarization methods—an essential step to accurately separate and identify different speakers in recorded conversations. These experiments are giving us valuable insights into how we can optimize our approach to transcription and summarization.
On the presentation front, we’ve been hard at work crafting an elevator pitch. We drafted the speech and created a set of slides to succinctly explain our project’s purpose and potential impact. This process helped us clarify our value proposition and sharpen our messaging.
Finally, we met with our client to discuss our progress and refine our pitch strategy. Their feedback has been invaluable in guiding our next steps, and we’re looking forward to integrating their suggestions into the final presentation.
Stay tuned for more updates as we move closer to delivering a polished, user-friendly solution!
Blog Post #5: Presenting to IBM and Making Prototype Progress
Our project continues to gain momentum! Recently, we had the opportunity to present our elevator pitch to IBM, showcasing the core concept and potential impact of our smart glasses solution. This was an exciting chance to receive feedback from industry experts, and it helped us refine our messaging and confirm the feasibility of our approach.
On the development front, our team began working on the prototype using llama.cpp. We integrated a “granite” model into the project, though we encountered challenges when trying to upload large files to our GitHub repository. This experience is guiding us to explore more efficient ways of managing and sharing large model files—an essential consideration for scaling the solution in the future.
In addition, we started developing the frontend portion of the project, laying the groundwork for a user-friendly interface. Our goal is to create an intuitive experience that seamlessly integrates transcription, summarization, and other core functionalities.
Stay tuned for more updates as we continue to refine our approach, overcome technical hurdles, and bring this project closer to a polished, real-world solution!
Blog Post #6: Design Sketches, Vector Search, and Whisper Testing
We’re excited to share our latest developments on the project! Our design team has been hard at work creating hand-drawn sketches to shape the final look and feel of our frontend. From menu layouts to user interaction flows, these sketches are helping us visualize the user experience before we move into more detailed design work.
On the technical side, we successfully set up vector search for report generation, integrating FAISS along with a granite embedding model and LLM. This step allows us to efficiently store and retrieve information, which is crucial for features like quick lookups and summarization. It’s a big leap toward ensuring our system can handle large volumes of data in a streamlined way.
Meanwhile, we used Figma to build out a wireframe for the app’s UI, giving us a clearer picture of how users will navigate between different pages and features. By focusing on usability early on, we can refine our interface to ensure it’s both intuitive and visually appealing.
Lastly, we conducted tests on different versions of Whisper—including small, tiny, and various diarization tools like pyannote and tinydiarize. The goal was to compare efficiency and accuracy across these setups, helping us pinpoint the optimal balance for real-time transcription. These insights will guide our next steps in refining the transcription engine.
That’s it for this update! We’re looking forward to sharing more progress soon as we continue refining the design, enhancing our data processing capabilities, and working toward a polished user experience. Stay tuned!
Blog Post #7: Frontend Development, Audio Recording, and Database Integration
Welcome back for another update on our project! Here’s a quick rundown of what we’ve been working on:
Frontend Progress
Our team created a basic login page along with a dedicated view that displays a list of all patients. These pages form the core structure of our user interface and set the stage for more advanced features to come.
Audio Recording with Smart Glasses
We successfully connected the glasses to an audio-monitoring tool, enabling us to start and stop recordings by simply tapping the glasses. This setup automatically stores audio files on a laptop, making it easier to manage and organize recordings for further processing.
Database Creation
Using SQLAlchemy and FastAPI, we designed and implemented a new database. We also created an ER (Entity-Relationship) diagram to guide our schema design. This step ensures our data is structured efficiently, paving the way for scalable and reliable performance.
Exploring Intel OpenVINO
To improve the efficiency of our transcription process, we began experimenting with Intel OpenVINO. By optimizing models like Whisper, we hope to achieve faster inference times and smoother real-time performance—key factors in delivering a seamless user experience.
We’re excited to keep building on these foundations, and we look forward to sharing more progress soon. As always, thank you for following along on this journey!
Blog Post #8: New Pages, Database Integrations, and Fact Checking
We’re excited to share our latest updates! Our team has been hard at work creating and refining various features to enhance both the functionality and user experience of our project:
Add Patients & View Meetings Pages
We introduced an “Add Patients” page and a “View Meetings” page, both of which are now fully integrated with our database. This allows for streamlined data entry and retrieval, making it easier for users to manage and review patient information as well as scheduled or past meetings.
Fact Checker Implementation
One of our most anticipated features—a built-in fact checker—is now up and running. This tool helps ensure the accuracy of transcribed content and generated reports by verifying key pieces of information. It’s an important step toward maintaining reliable documentation and reducing potential errors.
Add Meeting Page & Report Generation
We created a new “Add Meeting” page that not only captures meeting details but also links seamlessly to our transcript and report generation modules. By connecting the backend logic to the frontend interface, users can quickly add a new meeting, record its audio, and generate a comprehensive summary or report—all from one centralized location.
With these enhancements in place, we’re getting closer to delivering a cohesive, user-friendly solution that addresses the challenges of efficient data capture and management. Stay tuned for more progress updates as we continue refining our features and expanding the project’s capabilities!
Blog Post #9: Database Integration, Client Feedback, and UI Enhancements
We’re excited to share our latest round of progress updates on the project! Here’s what our team has been working on recently:
Database Integration
We successfully connected the transcript and report generation features to our database, as well as integrated them into the “View Meetings” page. This update allows users to seamlessly generate, store, and retrieve both transcripts and reports—streamlining the entire workflow from start to finish.
Client Demo
We had the opportunity to present our current progress to our client. They were pleased with how the transcript and report generation functions are coming together, and their feedback validated many of our design decisions. This positive response motivates us to keep refining and enhancing these core features.
User Interface Improvements
Our design team focused on polishing the color scheme and overall layout of the application. By moving certain buttons to more accessible locations, we’ve made it easier for users to navigate and interact with the system. These UI tweaks are all part of our commitment to ensuring a smooth, user-friendly experience.
With these updates in place, we’re moving closer to a fully integrated solution that combines efficient data management, accurate transcription, and user-friendly design. Stay tuned for more progress as we continue refining the app and incorporating new features!
Blog Post #10: Finalizing the Project
As we wrap up this journey, we’re proud to announce that our project has reached its final stage! Here’s a brief look at what we’ve accomplished and where we stand:
Fully Integrated Solution
We’ve successfully brought together all the major components—smart glasses integration, transcription and report generation, database management, and a user-friendly interface. Everything now works in harmony to streamline meeting documentation and make it easier to review and manage important information.
Refined User Experience
Our design and development teams focused on delivering a smooth, intuitive interface. From color schemes to button placement, every detail has been carefully considered to ensure ease of use. This means that healthcare professionals (and others) can quickly capture and review critical data without feeling overwhelmed by unnecessary complexity.
Client Satisfaction
Throughout the project, we held regular demos and feedback sessions with our client. Their insights helped us fine-tune each feature, ensuring the final product meets real-world needs. It’s been incredibly rewarding to see their positive reactions as we’ve implemented each improvement.
Future Possibilities
Although this marks the end of our current development cycle, we’re leaving the door open for potential future enhancements—like expanding our AI capabilities, refining the hardware integration, and exploring additional use cases in different fields. There’s plenty of room for growth should the project continue to evolve.
Thank you for following along on this exciting journey! We hope our updates have given you insight into the hard work and collaboration that brought this vision to life. We appreciate all the support and enthusiasm from everyone who helped us reach this milestone.
Here’s to wrapping up this chapter—and looking ahead to the possibilities on the horizon!