Design Changes

Main.py: Post “/start-audio” and “/stop-audio”

Initially, for the POST endpoints in main.py, the design was to start and stop observers when they start recording. However, this led to many bugs in timing and unexpected issues. The observers take some time to initialize and stop, which leads to delays, and the observer shuts down before processing the files. Hence, the design was changed to a permanent observer pattern.

• Permanent Observer with Async Context Manager:
  • Simpler to implement.
  • Observer runs as a background task.
  • Doesn’t interrupt other operations.
  • Automatically cleans up once the server shuts down.
• Start and Stop Observer Dynamically:
  • Timing issues.
  • Misses files between stopping and starting.
  • Risks clean-up problems.

Also, initially, the program was designed with global variables, which led to many issues as it wasn’t thread-safe. Later, it was made thread-safe by creating AppState.

AudioHandler Class and TranscriptWatcher

To detect new files that go into the audio folder or new transcript files, the program uses the watchdog library to monitor specific directories.

TranscriptWatcher Class in monitorAudio.py

TranscriptWatcher watches over the processed_text directory and triggers when a TXT file is created or modified.

on_created and on_modified

These are functions that are file system events (from the watchdog library) triggered when it detects changes in the directories, such as when a file is modified or created.

• last_modified_times: A dictionary with the last modified timestamps.
• debounce_interval: An arbitrary value that acts as a cooldown timer to prevent events from occurring too frequently, which could result in errors.

When the function is triggered, it sets current_time as the time and retrieves the last_modified_time for the file path. If it exists, it gets the time; if not, it sets the last_modified_time for the file path as 0. If the difference between current_time and last_modified_time is larger than the debounce_interval, it allows processing to continue and calls process_event.

process_event()

This function creates a unique key path_key for the file path with the file size.

• processed_paths: A set of all the paths already processed (no duplicates allowed).
• If the path_key already exists in this set, it will not be reprocessed.
• If not, it adds the new path_key to the set and calls update_processing_state.
• It keeps a cached set of files, but once it exceeds a certain limit, it cleans up after itself.

update_processing_state

All transcripts must take a queue and wait their turn to be processed.

• processing_state: A class in main.py with attributes such as:
  • current_path: A string representing the current file being processed.
  • is_processed: A boolean indicating whether the file has already been processed.
• processing_lock: A threading.Lock ensuring that processing is done one at a time, maintaining thread safety.

The function checks if the current_path from processing_state matches the file currently being processed and whether is_processed is true. If so, it avoids duplicate processing. If current_path is None or is_processed is false, it starts a new path, sets the file path as current_path, and sets is_processed to false. Otherwise, it adds the file path to the queue if it’s not already there.

start_transcript_watcher

This function is called in main.py. It requires the processing_state class, the lock, and the queue as inputs.

• It ensures the correct transcripts/processed_text directory exists.
• It sets up an instance of the observer watching this directory.
• It starts the TranscriptWatcher handler and begins monitoring the folder.
• Returns the observer instance so it can be controlled in main.py.

AudioHandler Class in monitorAudio.py

Initialization Phase

Creates an instance of the AudioProcessor class with the Pyannote authentication model key (note that the Pyannote model is open source and free to use, but you need to provide your information). Audio files are processed one at a time, so they need to wait in a queue patiently for their turn.

• Sets up self.processed_files and self.pending_files, which is a list used as a queue (FIFO) to track processing and pending files.
• self.handler_lock: A lock (mutex) for thread safety for operations on pending_files.
• Stores watch_folder: The directory to observe where the audio files are coming from.
• Upon initialization, it calls start_processing_thread.

start_processing_thread

The process_files (main program) is contained within start_processing_thread and runs in an infinite loop. It checks for new files to process and independently handles each of them.

• The lock handler_lock ensures only one thread can access pending_files at a time.
• Error handling ensures that if there is a problem with processing an audio file, an error message is logged.
• There is a short delay at the end to avoid excessive CPU usage.
• start_processing_thread creates a daemon thread that runs process_files in the background, so the audio processing process doesn’t block the main program and ensures proper program closure.

add_file_to_pending

Uses a thread lock when accessing the list of pending files and adds a new file to the pending queue.

on_created

Monitors when a new file is created, checks if the file has been processed, and adds it to the processed_files set and queues it for processing.

on_moved

Handles when files are moved into the watch_folder and updates them for processing.

Fact_Check.py

The purpose of Fact_Check is to compare the transcript with the generated report. It uses a RAG (Retrieval-Augmented Generation) approach by performing a similarity search and then using an LLM to compare the claim with the searched evidence and reason a conclusion about whether the claim in the report is accurate to the transcript.

SessionManager Class

Manages the session state for the fact-checking process, which main.py (server) requires. It includes:

• A function to create a session UUID and store various information inside the sessions.
• A function to clean up the session.

generateEmbeddingsForTranscript

This function takes the transcript as a string, calls a function to split sentences with speaker tags (to split it into chunks), and then calls a function to generate embeddings. The result is a np.ndarray, which is appended to a list. It returns a chunk containing a list of chunked text, their embeddings, and their chunk indexes.

generateEmbeddingsForReport

Similar to generateEmbeddingsForTranscript, but it chunks differently without speaker tags and processes sentence by sentence.

process_chunks

Takes the chunks and returns an np.ndarray of embeddings and text.

index_embeddings

Takes the numpy array embeddings and uses FAISS (see more details about the library in the technology review section) to index them, returning the texts and index.

query_faiss_index

Takes the claim, the FAISS index, the texts accompanying the index, the query vector (embedding), and the number of closest results. It performs a search to match the embeddings and returns results, including the claim and the search results.

similarity_search

Takes the transcript string and report string. It:

• Generates embeddings for the transcript and report.
• Processes the chunks (process_chunks).
• Creates a FAISS index from the transcript embeddings and texts.
• Matches each claim in the report to the most relevant evidence in the transcript using query_faiss_index.

verification_prompt

Contains the prompt used by the LLM to generate a response. It takes the claim and the results of the evidence from the similarity search as parameters.

Fact-Check Function in main.py

• @app.post("/create-session"): Creates a session ID using an instance of the SessionManager class.
• @app.post("/process-documents"): Gets the session ID, retrieves the transcript and report from the frontend, and uses the similarity_search function. Transmits the session ID, claims, and evidences.
• @app.get("/stream-fact-check"): Gets the session ID from the event source in the frontend (passed as a query parameter in the URL). Compares it to the session ID in the SessionManager instance, processes each claim, and streams responses using the DeepSeek model and the prompt.

Splitting_chunks.py

chunk_report_claims

This function chunks reports by:

• Removing markers and splitting into sections, removing the headers of the report.
• Splitting into sentences and adding them to the list of claims for the fact-checking pipeline.

sentence_split_with_speaker_tag

Processes a transcript of dialogue between speakers and splits it into smaller chunks while preserving the speaker tag.

Report_Generation.py

Contains generate_medical, which includes the prompt the program uses to generate a report.

Run_granite.py

Classes

Includes classes such as GraniteModel and DeepSeekModel. Each class represents the loading of one LLM model using a singleton pattern to avoid loading large LLM models multiple times.

• A lock ensures thread safety, preventing multiple threads from creating an instance simultaneously.
• There is an option to use optimum.intel or llama.cpp libraries to load models.

Design Changes

Initially, the program was designed to reload the AI model each time it was used, which made things slow. This was changed to load only a single instance, allowing the AI model to be loaded once and reused.

run_granite and run_deepseek

These functions run the LLM model by getting an instance of the model and producing a streaming response.

Recorder.py

This code is designed to simulate the frame glasses, in case the frame is not available. PyAudio is used as it provides configurable parameters such as 16kHz sample rate, 1024-byte chunk size, and mono-channel PCM-16 format. This configuration balances audio quality with system responsiveness, particularly important for wearable device simulations.

The recording lifecycle follows a sequence beginning with initialization through a simulated tap gesture (triggered by keyboard input). Once activated, the system enters a continuous recording state where audio data gets buffered in memory. A dual-locking mechanism ensures thread safety during this process - using threading.Lock for synchronous operations and asyncio.Lock for asynchronous coordination. This prevents race conditions while maintaining system responsiveness.

Audio processing occurs through a dedicated pipeline that handles the conversion of raw audio data into WAV format files. The implementation includes atomic file operations to prevent corruption, first writing to temporary files before finalizing the save operation. For backend integration, the system communicates with a FastAPI service using asynchronous HTTP requests through the httpx library, allowing for non-blocking API calls to start and stop recording sessions.

The tap gesture simulation provides an intuitive user interface, mapping keyboard input to device interaction patterns. This abstraction allows for realistic testing of the recording functionality without requiring physical hardware. The system maintains state consistency throughout the recording lifecycle, properly handling concurrent start/stop requests and ensuring all audio data gets preserved.

Performance considerations guided several design choices, including the use of ThreadPoolExecutor for blocking I/O operations and memory-efficient bytearray buffers for audio storage. The architecture supports future enhancements such as audio streaming, real-time processing, and additional gesture controls while maintaining the current system's reliability and responsiveness.

The recorder.py helped us diagnose an issue of why sometimes events couldn’t/weren’t triggered or detected as sometimes the watchdog is not sensitive, and you might need to explicitly trigger an event.

Frame.py

This code implements an audio recording application for Brilliant Labs Frame glasses. The Audio Recording and Saving feature is a core functionality of the Frame smart glasses application, enabling users to record audio and save it as .wav files. This feature is implemented using asynchronous programming with Python's asyncio library. The frame_sdk library is utilized to interact with the glasses' hardware, specifically the microphone, motion, and display. The microphone captures audio data in chunks, which are stored in a byte array. When recording stops, the wave module is used to save the audio data as a .wav file, ensuring the correct format (mono, 16-bit, 16 kHz sample rate).

User interaction is handled through tap gestures on the glasses. When tapped, the application either starts or stops recording by calling the start_recording() or stop_recording() methods, respectively. These methods manage the recording state and initiate or terminate the record_continuously() task, which captures audio data in the background. Additionally, the application communicates with a FastAPI backend using the httpx library to control recording via API endpoints. If the API call fails, the application falls back to direct control, ensuring robustness.

Visual feedback is provided on the glasses' display, showing "Recording" or "Stopped" messages to inform the user of the current state. The recorded files are saved in an audio directory within the project root, with filenames incremented for each new recording. This feature combines hardware interaction, asynchronous programming, and user input handling to deliver a seamless audio recording experience on Frame smart glasses.