ReadingStar

Our aim was to provide an application which allowed users of any demographic to easily improve their speech by performing karaoke with their desired song, providing real-time feedback and rewards based on similarity. We achieved this, and managed to build on this with further analysis on the recording of their voice during the song to finally score their overall performance.

The Whisper model we utilised for transcription and analysis in this project is world-leading in terms of performance and WER, when standardized for all demographics, but unfortunately its limitations for live ASR emerged in this project.

One point of note was its occasional inaccuracy for medium-length or incomplete phrases. Due to this, perfectly valid speech to the human ear would not be deemed similar enough to the lyric in real-time, and subsequently not be awarded in-game points. This is because the model adapts transcriptions based on the surrounding context, which it is starved of in the short window for the lyric matching (no more than a few seconds). Therefore, the final accuracy score presented at the end of a song is always the better (and correct) performance assessment, compared to the points score.

The project as a whole is now very stable, though at times during development it demonstrated quite volatile behaviour. For example, developing a React Native front-end for Windows suffers from a limited range of libraries which required us to find tricky workarounds for seemingly simple tasks, such as microphone access. Furthermore, the app was prone to crashes in the early stages of UI development, usually due to errors such as failed library imports or buttons disappearing at runtime, despite being present in the code. However, as we became familiar with the platform, these spontaneous errors decreased in frequency and we are very confident that the current application is robust enough to handle a vast range of events.

For our project, additional efficiency was provided by Intel’s OpenVINO inference pipeline. As demonstrated in the performance testing section, OpenVINO provides optimised performance at no cost to speed, and thus uses less power as a result. This is critical for users, especially on laptops, who can still run the model at great speed without having to worry about excessive power usage.

ReadingStar aims for low latency inference for real time ASR, making the NPU the most suitable inference device. This is because the NPU excels at low-latency and energy-efficient inference whereas the GPU is optimised for parallel, high-throughput tasks, which is more suitable for training and less ideal for real time inference tasks.

The very choice of React Native for our UI was made with the foremost thought of compatibility in mind. In a world where Windows PCs are ubiquitous in households and educational institutions alike, with the long-term aim of distribution as a Microsoft Store installable, it is imperative that the app is supported by as many of them as possible.

Most Windows PCs run on a form of Intel hardware, meaning that the OpenVINO pipeline included will improve efficiency on almost every device that uses ReadingStar. Furthermore, this app can also run at its peak performance due to OpenVINO’s NPU compatibility on devices that possess one.

In the future, a mobile version could be considered, and as we already have a React Native framework it should not be difficult to port it to iOS- or Android-specific usage. This would allow for a wider range of users to access the app, such as parents of neurodivergent children or even the children themselves.

Throughout our project, our team strived to write quality and self-documenting code, ensuring an intuitive “pick-up-and-go” development process. Because of this effort on our part, we are very confident that any future engineers can continue effectively without difficulty understanding the algorithms.

We also worked hard to devise an optimal workflow which was reflected in the code. This means that there are no convoluted algorithms or redundant functions to trawl through before a change is made.

Throughout the course of our project, our team was very interactive over every channel. Outside of the lab sessions which we all maintained consistent attendance to, we would plan and discuss the execution of solutions over our WhatsApp group chat, and by logging our progress on GitHub with issues, descriptive commits, and thorough code reviews.

Through all these media, we would delegate or share responsibilities for upcoming tasks, holding each other accountable and working quickly to maintain steady progress. In addition, our team was very flexible in terms of roles, with each member contributing to all aspects of the project. This was particularly useful when one member was struggling with a task, as another could step in and provide assistance.

Our agile approach as the requirements evolved, from feedback at the BETT Conference to the comments of pupils at the Helen Allison School, was kept streamlined through receptive communication on the WhatsApp group chat again. We also made sure to check off objectives at regular points in development, and keep the Gantt chart updated.