This section contains the evaluation of our project, including future improvements and achievements.
Summary of Achievements
Requirements Status
| Requirement | Type | Successfully implemented (Y/N) |
|---|---|---|
| The robot should incorporate voice UI by allowing the user to create and update a grocery list by speaking to it. | Functional | YES |
| The robot should verbally respond to user input. | Functional | YES |
| The robot should update grocery lists which should be accessible to the user. | Functional | YES |
| SOTA should also incorporate Image recognition by being able to identify ingredients that the user might not know. | Functional | Partial |
| SOTA should be intelligent enough to recommend a recipe to the user depending on the ingredients the user has suggested to SOTA. | Functional | YES |
| SOTA should also be able to walk the user through recipes using voice UI. | Functional | YES |
| SOTA should be able to carry out basic conversions e.g ounces to grams. | Functional | YES |
| SOTA should be able to set alarms and timers. | Functional | YES |
| SOTA should be able to recognize ingredients under various light conditions. | Non-Functional | YES |
| The instructions provided by the robot should be vivid and easy to understand for the user. | Non-Functional | YES |
| SOTA should have a quick response time. | Non-Functional | YES |
| SOTA should be dependable i.e deliver services when requested. | Non-Functional | YES |
| SOTA should be able to work normally without threat to life or environment. | Non-Functional | YES |
Challenges
Devising the initial design was quite challenging, as we were not familiar with the nature of Voice User Interface. To familiarise ourselves with Voice User Interface, we referred to web articles and academic papers online that explained the characteristics of Voice User Interface and how it has evolved through the years. We also described our interface to potential users in the early phases of development to identify key areas that we should be focussing on, which are those areas that users are likely to find more significant to their overall user experience.
Furthermore, working with hardware components was, as usual, quite unpredictable. Connecting the robot to a WiFi network was frequently a difficult task and losing connection between the robot and our device was common.
Switching the Speech API from Bing Speech to Google Cloud Speech was quite challenging, because the Google Cloud Speech API required us to set up the environment to allow API credentials authentication and SSL communication on the robot. This involved a lot of debugging and looking at what others around the Internet had done when they encountered this problem.
Known Incomplete Features
One of our most significant incomplete features is the absence of an independent voice user interface. Our system currently relies on clicking the button on Sota to give it a request. Ideally, we would have Sota listen for a command phrase, like “Hey Sota”, and start processing requests from there. Although our plan from the beginning was to implement the system that way, we were ultimately not able to due to multiple reasons. We considered making Sota listen continually and make multiple requests to the Speech API to search for the command phrase. However, since the Sota library functions only allow developers to record a .wav file of fixed length and since that many calls to the speech API would not be economically feasible, we were not able to continue with that approach. Through our client, we were able to ask the NTT Data Italy team that is currently working on a Sota project for advice on this issue. We discovered that they had used external hardware components to allow the robot to continually listen and process recordings. As this was not something we could carry out or that we had access to hardware to implement, we decided to substitute it with using the button as an interface.
Another incomplete feature is the image identification feature. After a firmware upgrade, the camera driver sometimes does not work, so the image recognition does not fully work.
Our grocery list feature is also not completely what we envisioned it to be. Although we did manage to link the system with Dropbox such that there was a link that users could go to for our application, sign into their Dropbox and obtain a key that they can give our system in order to set up the connection, connecting it to specific users’ Dropbox accounts would have needed a dashboard of some kind to allow the user to input their key. We considered using Voice UI, but that would have been tedious for users and potentially have a detrimental effect on User Experience. We also considered building a dashboard for the robot, but realised that there was no existing structure for that on which we could build on and it would be beyond the scope of our project.
Vision
Our vision for the project was to have a completely hands-free cooking assistant. We ideally wanted our users to walk into the kitchen and be able to go “Hey Sota, what’s on my groceries list?” or ask for a recipe using their remaining grocery items then have both hands busy on preparing food while being able to ask “What’s the next step?”. The following is a story that would demonstrate our vision for this product.
It’s late at night, Adam has just gotten home from class. Since all the stores were closed, he was not able to buy dinner before returning home. Although he has only cooked a handful of times before, he would have to settle with preparing his own food tonight. He opens the fridge and realises that he had forgotten to buy eggs yet again. Determined to buy it the next time he goes grocery shopping, he says “Hey Sota, open my groceries list”. “Your grocery list has been selected”, he hears. “Update: add eggs,” he replies. “Your list has been updated”, Sota says. Adam checks the Dropbox application on his phone and the item has been added to his grocery list, ready for his next round of grocery shopping!
Now, what is Adam going to cook without eggs? He has pasta but is clueless on what else to add. On discovering an ingredient that his roommate has in the fridge, he is curious and wants to find out what it is. “Hey Sota, identify this,” Adam queries. “This is an avocado,” Sota replies confidently. Adam is sure his roommate wouldn’t mind, so he decides to take an avocado and make a dish with it. He begins by boiling the pasta. Since he is not very used to multitasking, he asks Sota for help to keep time. “Hey Sota, timer for five minutes”. “Your timer has been set”.
Without having seen this before, he decides to ask Sota for a recipe. “Hey Sota, recipe for pasta and avodacos”. Sota begins reading out the recipe and Adam follows it step by step, at his own pace, saying “Hey Sota, next step” after he has completed the previous step. He is not very familiar with measurement unit conversion, so he relies on Sota whenever he encounters an unknown measurement by querying “Hey Sota, convert…” followed by the units and quantity.
With Sota, Adam was able to prepare a scrumptious meal quickly and efficiently, without getting food all over his tablet.
Critical Evaluation
Architecture Design
We were forced to change the architecture of the project during the implementation of the Proof of Concept. The reason was that the original proposed intergration of the robot with cloud-based service such as Google Drive or Dropbox to save the results was unsuccessful, as the user would have to include his own personal details (API token for the chosen cloud-based service) in order to be able to sync the results there.
After the discussion with the client we decided to drop this feature, however it still remains open to implementation as a future improvement.
User Interface Design and User Experience
Some of the most important heuristics that we followed are found in our HCI section, along with research into them. Since Voice User Interface is significantly different from the usual screen interface, we decided to look at what others have done in devices like Google Home and Amazon Alexa.
Our central focus for User Experience is convenience and naturalness of interaction. To achieve these, we have looked into current problems in these areas that exist in voice interface and we designed our tests to evaluate these aspects of the system.
As mentioned before, we were not able to achieve the User Experience that we wanted to achieve, and had to incorporate an extra layer in the interface, the button. Furthermore, we would have liked to allow the user to access their grocery lists through a platform like Dropbox, to compensate for the absence of a screen interface. However, after linking the system to Dropbox, we found that the process of linking the system to the user’s Dropbox account solely through voice interface would be difficult. While looking for another alternative, we considered building a dashboard for the user to connect to the robot, but this idea was later written off because it was beyond the scope of our project.
Functionality
We have implemented all our functional and non-functional requirements fully, with the exception of the image recognition.
Stability
Sota operates under various environments without crashing in various environment.
Compatibility
As our system was developed for the Sota robot alone, it makes use of the Sota library and is not compatible with other hardware platforms. Sota is built on an Intel Edison module, which uses Java code and can be reached through Eclipse’s Ant Build mode, both of which we have made use of.
User Acceptance Testing Evaluation
The full report for User Acceptance Testing can be found here.
To summarise our User Acceptance Testing, we asked several users to read the system manual then use each of the features of the system and asked them several questions after that regarding both the system manual format and the system performance. Our decision to also focus on the system manual stems from our belief that a good system manual is required to be able to guide the user well, and similarly a bad user manual would be off-putting to potential users. We gathered some interesting comments, which are listed in the report. Some of these comments include being confused about the grocery list setting feature, since it involved many commands, and were not comfortable with several command words, such as “timer for” or “select”.
After the first round of User Acceptance Testing, we made the appropriate changes to the system and the system manual then conducted a few more user tests. The feedback was more positive this time, as users generally felt they were given more flexibility and understood the system better.
Integration Testing Evaluation
The full report for Integration Testing Evaluation can be found here.
Functional Testing Evaluation
We used 2 methods to perform functional testing on Sota: + Boundary Value analysis + Exploratory testing
Boundary value analysis helped us test how Sota responds to extreme user inputs e.g. creation of grocery lists of various sizes. We performed boundary value analysis to ensure that the system does not break under such user input. Exploratory testing allowed us to test various features of Sota which were stated in the functional requirements documentation. This testing method was used to ensure that all features worked properly and that they conformed to the stated system goals in the initial design phase.
We performed 6 tests on Sota which are explained in detail below:
1) Grocery List creation: We performed a test to ensure that the user was able to create simple grocery lists through voice UI. This test was important as it was one of the basic functional requirements of Sota. Grocery lists were created in various environments with differing noise levels. Sota was able to comprehend what the user meant most of the time and hence this test was a success.
2) Update Grocery List: We also performed a test to ensure that the user can easily update their grocery list. The initial test involved updating the grocery list with a single item. To test the flexibility of the system we updated the grocery list with 50 items and Sota did just fine. This test was also a success as Sota had no problems dealing with a huge list of items. An important aspect of grocery update was to ensure that Sota understood the ingredients spoken by the user - we can comfortably say that Sota successfully comprehends these ingredients most of the time.
3) Recipe suggestions: Another functional feature we tested was whether Sota can suggest valid recipes depending on the ingredients that the user has. We performed a simple initial test with 3 ingredients to test whether Sota returns valid recipes on simple user input. To test boundary values on this feature we provided Sota with 15 ingredients and Sota was easily able to provide the user with valid feedback on potential recipes. This test was a success too.
4) Image recognition: Sota should be able to recognise ingredients for the user. This can be helpful when the user does not know the ingredient. We tested this feature by varying the light levels under which the ingredient was shown to Sota. Sota performed reasonably well and detected the ingredient correctly in most cases. After performing various tests on Sota’s image recognition feature we decided that it correctly works.
5) Unit conversion: Sota should be able to convert between various units e.g teaspoons to milliliters. We performed various tests on Sota to ensure that the unit conversion was working correctly. We tested all the unit conversions with extreme values and Sota returned accurate results. We deemed this test to be successful as it worked correctly under all test cases possible.
6) Timers/Alarms: The system should be able to keep track of timers set by the user. To check this feature worked correctly, we started our test by setting various timers. Sota kept track of timers correctly and alerted the user once the timer was over. Timers were created of various lengths and at the same time. This test was successful.
After performing a variety of tests on Sota, we believe that Sota is functioning correctly as required. It conforms to the goals set out with the client at the beginning of the project. We have tested various extremes and seen how Sota responds to such user input. When testing Sota’s feature for recipe suggestion, we provided Sota with 15 ingredients - the system took a while to provide feedback to the user but we believe that even under extreme cases Sota was considerably efficient. Sota passed all test cases and hence functional testing has been a success.
Project Management
Our team has been able to manage the project well through consistent communication, the Agile development framework and constantly updating the Trello development board with tasks for each week. Although we have had several unexpected delays throughout the development, we were able to plan our time well and get ourselves back on track. Our main method with which we kept track of progress was the weekly meetings during schedules labs.
Achievements
We are glad to have been able to work with NTT Data, and our client, Gaye Soykok, for these two terms. We enjoyed the scope of our project, allowing us to learn more about Sota’s technologies and hardware characteristics, and giving us the flexibility put our ideas into the project. The tremendous amount of support and help that both our client and our TA have provided during this project has allowed us to perform better and achieve the targets that we set at the beginning of the project.
During the course of this project we have learnt a lot from using the agile software development methodology to experimenting and implementing various APIs in order to satisfy the project requirements in the best way possible. We embarked on the project by designing several use cases. We decided that each of the team members would come up with at least three use cases. This allowed us to think about the project in a broader perspective and it allowed us to gather a variety of potential ideas. We believe this was an important decision that ensured that each teammate understood what the project was about and what was needed by the client.
On the other hand, reflecting on the technical aspect of the project we have acquired knowledge on implementation of Voice UI, Agent AI and the Internet of Things. We performed extensive research into these topics and structured our project on how similar products are performing in the current market. We carried out product analyses on Amazon Echo and Google Home to see how we could best implement our project.
Working with a device, Sota, rather than working solely with software, was a unique experience that taught us a lot as programmers. To develop a system with a voice user interface instead of an application or website, we had to approach the problem with a different mindset and learn from existing products. Having the opportunity to develop on a device that was new to us was challenging but improved our ability to quickly adapt to new technologies.
Our project involved the use of various APIs. Research was also carried out to ensure that our system would use the most efficient APIs. We made use of Java in line with Clarifai, Spoonacular API and Google Speech API. This project helped us improve on writing Java code for large projects and most importantly taught us the importance of testing at each stage. Each sub-implementation of the project was tested before being passed on for system integration. Each of us performed unit tests to ensure that our implementations were working correctly under various circumstances. This made code handling easier as the project progressed since testing at each stage meant less bugs later on.
We also performed unit and functional tests to assure ourselves and the client that the system conforms to the initially stated requirements. Moreover, the last stage involved carrying out Integration tests and User acceptance testing. Integration testing was important in order to establish the robot as a fully working system with properly functioning subsystems. We performed various user acceptance tests to gain insight on our project from different perspectives. This project has trained us well on software engineering practices and equipped us with the necessary skills to carry out projects in the industry.
The project allowed us to learn from each other and we believe it has strengthened our skills to work as a team. It has provided us with the tools and knowledge which have enabled us to work as effective members of the team and move ahead as a team. Moreover, this project has helped us develop life-long learning through reflection since it involved continuous reflection on the software development life cycle.
Throughout the project, all decisions were made as a team with our client as long as the reasoning behind the suggestions was strongly supported. In addition to effective communication, we have gained the crucial skill of relaying information efficiently and precisely to the supervisors and commercial stakeholders.