ReadingStar

There are two main backend requirements: music player and lyrics matching.

Looking at how to add the music into the game, we had to find a method that did not infringe intellectual property laws and integrated seamlessly. The choice was made for us by the restrictions of law, we could not license songs individually, and we could not attain them otherwise. Therefore we had to integrate an existing provider into our application, where the only possible choice was the Youtube API. Hence, we decided to embed the Youtube player into the application and stream the songs.

For matching lyrics, we found many different models and methods. We looked at open source models and tested their accuracy on prototypes. Since we needed Intel optimisation, we used OpenVINO to increase efficiency on Intel hardware by converting models into an Intermediate Representation (IR).

We compared various speech recognition models during our research using various metrics such as the "librispeech performance benchmark", and testing the models locally. Nvidia's canary-1b model outperformed other ASR models in most metrics including Word Error Rate, however when testing with audio from neurodiverse children provided by the National Autistic Society, it failed to accurately transcribe most of the audio. Another model we tested was Vosk, which excels in real-time transcription, essential for our application, however the models are stored in Kaldi nnet3 format which proved difficult to integrate into an openvino pipeline for inference. We finally decided on OpenAI's tiny.en whisper model. Whisper is trained on a much more diverse dataset than Nvidia's Canary, and therefore performed better with neurodiverse speech and different accents, and the small size of the tiny model allowed us to engineer an effective and efficient algorithm for live transcription. Additionally, OpenVino had already created a pipeline for utilizing the whisper model which allowed us to optimize inference on Intel CPUs and NPUs and create an accurate and efficient pipeline for live speech recognition.

The final component is the python framework to implement the backend technologies. We opted for FastAPI due to its support for Asynchronous Server Gateway Interface (ASGI) ensuring low-latency handling, which is crucial to our application. FastAPI’s lightweight design and seamless integration with our AI pipeline, allowed us to build a responsive and scalable backend.