ReadingStar

To test performance of the application, we looked at whether individual units were executing as expected, and analysed the real metrics of the application functions, through speed and resource use.

Integration testing was included with these API calls, such as playlist management functions, as React Native Windows does not inherently possess capabilities for UI tests.

We began unit testing with PyTest for all endpoints of the backend API in the file api_test.py. They were performed with the instruction python -m unittest api_test.py.

API unit testing

We analysed the inference time of the Whisper model on 500 .WAV tracks, with length between 30 seconds and 5 minutes, with and without Intel’s OpenVINO inference pipeline, on an average Windows laptop CPU.

The CPU specs:

• Intel Core i5-12450H
• 16GB RAM
• 8 cores, 2.00 GHz clock speed

Note: the CPU usage before the model was run in either case sat at overheads of ~28%.

As shown by the first figure, without the OpenVINO pipeline, performance is deprecated, because there is nothing to optimise the Whisper model when transcription. This would make its use without the pipeline intractable for the speeds required for near-instant feedback.

Furthermore, analysis of power usage in complete inference, from the second figure, further asserts how crucial OpenVINO is for exceptional performance in ReadingStar. On average, it uses 5.5x less energy while producing output in the exact same time as a non-OpenVINO model.

For hardware testing, we compared the performance of ReadingStar on a range of devices to understand how the application behaves on different hardware configurations.

The OpenVINO pipeline can optimise AI processes on either the native CPU, GPU, or, if available, the NPU (neural processing unit). We tested OpenVINO on the ​​Intel® Core™ Ultra 5 Processor 125H, which contains three devices:

• Intel Core Ultra 5 Processor 125H CPU (14-core, 4.50 GHz)
• Intel Arc Graphics GPU
• Intel AI Boost NPU

The following graph illustrates the total package power consumption when using different inference devices over a complete session in the ReadingStar application. In all cases, the majority of the power consumption is done by the CPU, as it is responsible for running the application. The key difference lies in the additional power consumed during inference, depending on the device used.

Comparing the CPU and NPU utilised graphs, the peaks of package power are considerably higher when not utilising the NPU; the maximum power consumption when utilising CPU is 32.28 W in comparison to 21.46 W when using the NPU for inference. Additionally, the power consumption when utilising CPU for inference is much more erratic, this is because during batches of live transcription, the computer is using CPU for automatic speech recognition inference, which leads to frequent fluctuations in power usage. These fluctuations are still visible when using the NPU for the batch inference, but as the NPU is much more efficient for AI inference, the variations are much smaller and more stable.

As for the GPU graph, its power consumption shows even greater fluctuations than the CPU. This is expected, as GPUs are designed for training rather than inference, making their power consumption highly variable when carrying out the batch inference in the application session. The GPU has a max of 32.07W, similar to CPU, but the batch inference fluctuations are unstable and less consistent as the GPU attempts to optimise through parallel processing and high-throughput workloads.