HCI

Sketches

As part of the HCI requirement we created sketches to visualise how our solution would work. However, as the app does provide visual interaction for the user, we have made an overview of the components that it may consist of. The sketches represent how (before realising that we would require a different approach) the sensors will interact with movement. The initial high level sketch shows the position of all the sensors that will be used to track the position of the body in 3D space, thus allowing us to, in terms of those positions, know whether or not the user is standing using a correct pose or not. The Sensor representation shows how movement will be processed by the sensor, then outputted as useful data that we can then manipulate to make the user stand correctly . Below are the 2 main sketches.

Component Sketch

The sketches then allowed us create addtional architectural designs. It is important to note that this is not the way the solution is implemented, but it is fact the visualisation of two main design iterations. The final desgin was crated after conversing with the client and project support. We have placed further information regarding the final iteration in our Design Section. In the designs shown, we created an initial sketch that had the sensors outputting their data to an Arduino, which is an open software based on simple hardware and software, ideal for a project such as this, where the amount of data collected isn't too large. This Arduino would then process the data using algorithms that our team has created, to produce 3D sound and guide the user into a standing position. After some communication with the client, we decided to make use of the Hololens as a 3D sound device to output sounds to the user.

Wireframes

The wireframes below show the last form of iteration we took. The sketches were a preliminary design, displaying our initial idea of approaching the project. We must stress the fact, that the application is not based on visual input as it is a 3D sound project. Hence the lack of suitable wireframes.

Prototype Screenshots

Below are the prototype images of our application.

Design

System Architecture Diagram

Component A - User

This part of the system architecture diagram shows how the user interacts with the overall system. The headphones indicate the usage of 3D as well as granting the immersive expierence. The user here is intended to be Mark Pollock.

Component B - Kinect Sensor

This represents the Kinect 2 Sensor which retrieves data from component A (the user), using its depth sensor. It captures continuous frames of movement. It must be noted that this piece of hardware must be placed at a suitable height and a distance from the user to fully capture the skeleton image.

Component C - Database Module

It has the role of organising the bones of the avatar in serializable data structures that can be transformed into JSON format. In addition, it has classes that handle the transmission of the JSON to the server component in an asynchronous way.

Component D - 3D Sound

This component represents the output of the previous module and is what the patient hears. It can be modelled separately from the other parts, by choosing different sound scripts and by modifying their settings in the Unity Editor.

Component E - Avatar and Therapy Module

This is the core module of our application. It has the role of analysing the posture of the avatar and give feedback based on that. Furthermore, it coordinated which sound scripts to play and when allowing the correction of one joint at a time. In this way, we avoid a mixture of different noises that would not tell anything useful for the user.

Component F

This part receives the request from the main application and saves the JSON documents in the database. It acts as a secure bridge between the app and database. By running on the server it makes difficult for third-party to access the security information about the database.

Component G - Database

We used the database to store the poses a user might want to achieve during the exercises. It contains just the positions and rotations of the user in relation to the avatars bones

Component H - Kinect Module

This section gets the data from the sensor and transforms it to the Unity format. Also, we map it or our avatar so that it can mimic the patient’s movements.

Class Diagram

Sequence Diagram

Design Patterns used

Observer Pattern

This pattern is used to notify the observers when an event has occurred. We create loosely coupled code by just notifying the listeners when a changed has occurred, without calling specific methods. In our implementation, we used an Observer pattern to update the avatar when a changed is seen in the patient’s position. Also, we needed it for classes that wait for user’s command in order to execute some code. Once a word is recognised, the state of the subject changes and all its observers are alerted.

Command Pattern

This pattern was used to separate the invoker of an action from the receiver. The code is separated such that the object that calls the command has no idea about the specific implementation of it. A concrete example is the voice command menu. When we require to save something to the database, the code that recognises the user’s voice will invoke the “save pose” function call on the Button Manager class. This will call the execute method of “DataSender” class that contains the code for sending information to the server.

Singleton

A singleton was used when it was unnecessary to have more objects of the same type. For example, having multiple speech managers on runtime would mean having more calls on code that executes commands, while the user wanted to perform the action only once. Despite this, the design pattern has a clear scope, it is dangerous and not beneficial to overuse it - For example it can break thee SOLID principles of object-oriented design. Also, if not used adequately it can cause numerous bugs, especially in multithreaded applications.

Strategy Pattern

We had used this as a stronger alternative to the Template method pattern because it allows the code to be more flexible as it respects the open-closed principle. This means that future developers should just add new implementations that obey the current interfaces, instead of changing the current work. This considerably reduces the maintenance costs. The pattern was applied to the algorithm that recognises different poses and to the part that chooses which sound is played, making it possible to easily change the approach, if advanced solutions are found in the future.

Adapter Pattern

The adapter was required for connecting the Sensors raw data format into Unity spatial coordinates. We used some existing wrapper classes, which are the Adapters to call a specific function on the Kinect interface and change the returned result in a way that fulfilled our needs.

Decorator Pattern

This pattern allowed to regain more control on sounds that are played during runtime. The simple Unity behaviours “Play ()” and “Stop ()” were not enough to coordinate multiple sound sources on the same Game Object. We had to create a decorator class called “Sound Settings” that would add extra functions to the original methods of Unity’s AudioSource component. Doing this we avoided direct modifications of existing work, which would have been impossible.

Dependency Injection Pattern

TDD (test driven development) made use of loosely coupled code. We had to use the dependency injection to avoid creation of an object in concrete classes. Instead, we passed references in the constructors, making the code more flexible, reusable and testable.

Data Storage

Although this was not one of the main requirements, we needed a someway to store different postures that user might want to achieve during the exercise. We used the Mongo database which was deployed on mLab (a cloud service for mongoDB) because of its fast and easy configuration. To connect with it we created a server file in node.js that was deployed on Azure. For deploying we created a GitHub webhook that would put the new code on the cloud each time a pushed was made on the production branch.

Key functionalities

Gather 3D joints from the environment and map them to an Avatar

In order to track the movement of the user’s body in real time we used the Kinect device and the Kinect SDK created by Microsoft. It must be mentioned that the tracking algorithm implemented in the sensor works with face detection, which means that the patient must face the Kinect to have his limbs position monitored. Once data is received from the device, we must process it and transform from the raw format into Unity spatial coordinates. This transformation is done using wrapper classes from the MS-SDK library we imported.
Furthermore, we filtered the converted data using the double exponential filter method, described in the research part. The process of filtering is necessary because occlusion (misalignment of joints) can occur, meaning that Kinect gives inaccurate results when compared with real coordinates. Also, an approximation might be needed when some joint changes its position between different frames. If no approximation is done, a sudden difference in avatar’s position will be noticeable, making it desynchronized from the user’s movements.
After correctly configuring the above steps, we must map the retrieved information to game objects which are represented as bones of the avatar. All virtual bones are arranged into a hierarchy which denotes the human skeleton.

Develop an algorithm that detects incorrect pose of the user

This requirement had been quite challenging due to the numerous ways of tackling this problem. We required a unique which applied to a simplified version of our task. There are many machine learning algorithms that can indicate and perceive the pose of the user, but are computationally expensive. This factor is an obstacle for giving real time feedback. To avoid this problem, we designed a rule-based system, that gives feedback on the pose by considering the relation between multiple joints. For example, to determine how bent the left knee is, we calculate the angle between 3 points: left hip, ankle and knee. Those joints form 2 vectors in 3d space, which makes it easy to calculate the angle using this formula:

$𝜃={\cos }^{-1}\frac{\mathrm{|P|\; x\; |Q|}}{\mathrm{P\; x\; Q}}$ where P, Q are the vectors defined by the joint points
We know that the knee has a correct position when the value of the angle is inside an interval of error.

As the definition of correct joint alignment depends on the user’s preferences, we define the conditions as variables which are calculated from the predefined choice of the avatar model. In our case the clients clearly defined what a correct pose means. Below is an example of what a imbalanced and balanced skeleton looks like. Different body parts can affect human position independently of the other limbs, which is the case of spine base. The solution is to simply take their rotation in space. Despite there are distinct conditions, our system can handle all different approaches, because there are just a limited number of possibilities to consider.

Transform feedback in 3D sound modulation

As our end user was visual impaired, he required audio guidance to know how to correctly align his joints. Simple verbal instructions were not enough, because they could not reflect how the user was progressing in terms of his current state. In other words, the verbal instructions did not suggest when knees are bent at an angle of 5 degrees and when at 60 degrees. There was not differentiation between the two. Instead, we took the approach of trying to localise the joints with 3D sound. In real life, we can distinguish sources of noise by how far they are and from which direction. To avoid an unnecessary large learning curve, we can allocate keywords to each joint. This means we can combine the 3D sound affect principal along with this to create a tailored experience for the user. To accomplish an immersive experience, we used the Microsoft’s HRTF spatializer (Discussed in the research part) and attached sound to each bone of the avatar. When he moved the sound source moved as well.

As the role of avatar has been clearly defined, we need to create the 3D sound effects that represent the source of noise to be present in the scene. Taking the head of the humanoid (The avatar) as listener, position of those objects can be easily deduced from sound modulation. The patient will have the feeling of being substituted in place of the avatar in the virtual world hence being be able to realise where his joints are localised in space.

Despite the solution being good, we faced numerous challenges. Firstly, the spatializer was built to imitate reality as much as possible. This meant that large distances have a noticeable effect on the sound volume. In our case, even 5 degrees of motion were extremely important and needed to be signalled to be changed. To solve this, we created mock joints. In the Unity scene, these move exponentially in relation to the user’s joints on a specific axis. For example, if the left hip is not properly aligned with the rest of the body (pushed too far on the left) than there is a difference Dx between the correct location and the current location on the x axis. Considering that value of Dx is constrained by the human body shape and is rigid (i.e it belongs to [-1, 1]) we can compute a new position for a mock hip by applying the function . The newly created point will have the same values on y and z axis as the human joint while the x value will be given the previously mentioned function. The new object will then have a sound source attached to it. The schema below explains the concept:

Secondly, a wrong pose can be the result of multiple joints that need correction. In this case, we cannot play at once all the various noises mapped to those joints. This will cause large amount of confusion for the user. Instead we try to choose the correct joint that has the maximal error. However, different limbs have different acceptance intervals of distinct ranges. We solved the issue by considering the proportions of joints and limbs and calculating the percentage error obtained from each proportion.

Percentage Error
Main Menu with Voice activated Commands

All users should be able to interact with the application in a friendly and easy-to-use manner. Since our target audience are visually impaired people we had agreed with the client to construct a menu based on speech recognition. Our solution utilises a small grammar of several instructions combined with the speech recognition features of Windows 10. Furthermore, Kinect has a very performant microphone that allowed us to create a reliable user interface. The only problem with the speech recognition system is its vulnerability to external sounds. This is the case for all speech engines. Below is the getVoice command.

Database and Server

We needed a database to store possible posture that user might want to achieve. To connect that with our application we needed a server. On the client side we send post request to the server using C# WWW forms: The nodeJs part had the role to parse the JSON received, and save it to the database if the correct form was recognized. This consisted of the joints having 4 features: rotation, local-rotation, position and local-position. Local-position and local-rotation are just rotations and positions with respect to the parent bone.

Testing

Various testing stratigies were used to ensure that functionalities operated smoothly. We will discuss the various testing techniques below and the situation in which they were performed.

Functional Testing

This testing strategy is used to ensure that every function of the system works in relation to the initial requirements. The main technique used is called black-box [16] testing which does not involve any source code. The test is run on the input requirement which is then checked against a test with a defined expected output.
We performed functional testing using a step by step procedure [16]:

1. Identify the requirements. We had done this at a previous phase of our project iteration
2. Create appropriate input data for the test cases. Because the combination of joint alignments is a large number, we had to identify which joints are the most common. We then stored their positions and provided an input for our test
3. Determined the output depending on how functionality was defined: We had to consult with Dr Sayenko to find out what are the possible ways a patient tries to correct the pose. This is because the order of joints that are corrected affects the output of our application. In addition, we had a clear vision of how a corrected posture would like due to numerous previous meetings with our supervisors.
4. Run the test: Depending if our test was automatic or manual, we executed the test cases and obtained an output
5. Compare the output against the expected values: In this section we could define if the test was failing or not, giving us an idea of what was wrong with our software.

How we approached Testing

By now we had tested the functional requirements (described in the Requirement section) and the UI. We mainly used the black box technique as we were interested in how certain interfaces behave and not how they are implemented. When we discovered a test that failed and was hard to debug we had to refer to the Transparent technique (White box testing) and investigate the internal behaviour of the code. The main argument for not using it as much - is because the first method tends to focus on the implementation details, rather than of the main functionality. In the near future, we plan to test the application with our end user. His location in Ireland prevented us to do so. However, we discussed it with our clients and hope to ship the software to him or find a way to arrange a meeting.
Different Companies have their own way of defining the next test types, so we follow the ISTQB (International Software Testing Qualifications Board) definitions.

System Testing

We had to test our application from one end to another, so after we finished the requirements, Unit and Integration tests we performed this type of testing. In Unity and other Game Engines, it is very hard to automate testing, because you need to check interactions between objects. Unity provides a Test Runner tool, but it is appropriate for only Unit and Integration Tests. To automate the verification of the entire app we had to recreate the scenes from runtime, which would have been time consuming and inefficient. Instead, we manually checked the behaviour running through all features once. When the found faults were fixed we repeated the cycle, gradually improving the software.

Stress Testing

Stress testing was used to observe specific results for parameters near the breaking point. Regarding the project, it used to verify if the system handles the post requests from the unity application and successfully stores the data into a database under thousands of requests per second. On the Unity part of the app, we tested how well the algorithm gives feedback on small and fast changes in pose. Each test case has 4 different graphs named as Performance, Throughput, Errors and Tests. The user load is static throughout with the test duration lasting 1 minute. The performance graph indicates the average response time along with the throughput which shows how many requests it is receiving within a second. The error and test graph are self-explanatory.

On the Unity side, we test how the program detects normal movements of the joints at a constant rate. Furthermore, we examined how the speech recognition API behaves with different voices or accents

Performance Testing

We carried performance tests to see how the system behaves under conditions we expect, which are in the specification of the application. It was used to determine the QOS (quality of service) based on a large number of user requests. We created virtual users which simultaneously accessed the URL. It granted us critical information such the average response time and the number of errors generated. It increased our confidence in the scalability and reliability of the system as it could handle a large number of requests. The results below show a total of 2923 requests being received over a span of two minutes.

Evaluation

Management

Legal Implications

Click here if the PDF does not display

References

• [1] Polhemus Fast-Track: Available at: https://polhemus.com/
• [2] XSENS: Available at: https://www.xsens.com/tags/sports-biomechanics/
• [3] NOTCH: Available at: https://wearnotch.com/
• [4] Kinect: Available at: https://msdn.microsoft.com/en-us/library/dn758675.aspx
• [5] Paper Kinect: Available at: https://www.cla.purdue.edu/academic/vpa/ad/act/resources/ad41700_barrett_kinect_unity.pdf
• [6] OpenNi: Available at: https://s3.amazonaws.com/com.occipital.openni/OpenNI_Programmers_Guide.pdf
• [7] MySQL: Available at: https://dev.mysql.com/doc/refman/8.0/en/
• [8] Mongo Db: Available at: https://docs.mongodb.com/manual/
• [9] OpenPose: Available at: https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/README.md
• [10] SMPLify: Available at: http://files.is.tue.mpg.de/black/papers/BogoECCV2016.pdf
• [11] Speech Microsoft: Available at: https://msdn.microsoft.com/en-us/library/jj127860.aspx
• [12] Joint Filters: Available at: https://msdn.microsoft.com/en-us/library/jj127860.aspx
• [13] Double Exponential Microsoft: Available at: https://msdn.microsoft.com/en-us/library/jj127860.aspx
• [14] Command Pattern: Available at: https://sourcemaking.com/design_patterns/command
• [15] Gamma, Erich, et al. Design Patterns: Elements of Reusable Object-Oriented Software. Pearson Education, 2015.
• [16] Functional Testing: Available at: https://www.guru99.com/functional-testing.html
• [17] NSubstitute: Available at: http://nsubstitute.github.io/help.html
• [18] Unity Tests Documentation: Available at: https://docs.unity3d.com/Manual/testing-editortestsrunner.html

© All rights reserved