In this section, outline the tests we carried out and the research we carried out on them beforehand. Our findings and evaluation of these tests are shown in the Evaluation section.

Testing Strategies

During and after the development of PoC finishes, we will conduct testing to ensure that the product works as intended. After researching on various approaches to and tools for testing, we decided to use the following testing methods:

Unit Testing

Research

We will use Unit Testing to test the functionality of the separate components, including units conversion, grocery list and ingredient recognition. We have compared multiple testing frameworks, such as Arquillian (highly innovative and extensible testing platform for JVM allowing developers to create automated integration, functional and acceptance tests), JTest (automated software testing and static analysis toolbox) and JUnit (unit-testing framework).

In the end we decided to use JUnit; writing JUnit tests is simple and elegant, JUnit tests can be bundled into multiple test-suites testing different parts of the system and they are written itself in Java, bringing the learning curve of a new framework to minimum.

Unit Testing has several main advantages that relate to our system1

  • They increase confidence in changing and maintaining code. This is especially useful when developing code concurrently and in iterations, where we have to go back to previously written code to improve it. After every change, we are able to quickly verify that the units still work as intended by running the tests. Furthermore, unit tests can be useful for future developers of the project when they are maintaining the code.
  • Making unit tests encourages us to develop more reusable code because the functions used in unit tests should be modular to enable its unit test to be written.
  • Costs of fixing problems found at the unit testing level is less compared to if they were discovered at higher levels, since it usually take a shorter time and less effort.
  • Once unit tests are written, they can be run after every small change, meaning that developers only have to debug the latest changes. Narrowing the scope in which developers have to look saves a considerable amount of time.
  • Unit testing enables us to write more reliable code, as the code works according to our end goal for the program.

We plan to write test suites for the main algorithm, including, but not limited to, these features: + The conversion function for all measurement units + Updating to and reading from grocery lists + Recipe finding functions

Implementation

With Java’s JUnit, we used unit testing to test individual components of our system before integration and functional tests. The purpose of unit testing is to “validate that each unit of the software performs as designed”.1 Units are the smallest parts in the system that we can test and usually have a single output. We have chosen individual methods as our units and have written unit tests for them with ordinary and edge cases.

User Acceptance Testing

Research

We will employ User Acceptance Testing to test the product on our end users. User Acceptance Testing is a method of checking that previously defined requirements are still on track. They portray user requirements in a directly verifiable way and provide an overview of how complete our system is. Carrrying out User Acceptance Testing in addition to unit and integration testing will ensure that we identify problems that those have missed, since it defines behavior instead of functionality. We would like to test LeChefu’s features in its environment and judge how easy it is to use the Sota for our use case.

The specific approach that we are taking for this is Alpha & Beta testing. This involves two User Acceptance tests, once during the development of the system and another at the end of development. The initial Alpha test enables us to catch and fix problems before testing our final product. The Beta test allows us to test our final product with our client and potential users.

To create effective User Acceptance Tests, our tests will be based on our User Stories and we will define the acceptance criteria according to the system’s requirements.

Implementation

We will carry out Alpha User tests then improve our system based on findings. After that, we will conduct Beta User tests on our final product and improve the system after that if needed.

Integration Testing

Research

The phase in software testing in which individual software modules are combined and tested as a group. It occurs after unit testing and before validation testing. It takes, as its input, modules that have been unit tested, groups them in larger aggregates, applies tests defined in an integration test plan to those aggregates, and delivers as its output the integrated system ready for system testing.3

Integration tests are used to show that the major parts of a system work well together.4

There are two methods of Integration Testing:

Bottom-up

  • requires the lowest-level units be tested and integrated first (referred to as utility modules)
  • utility modules are tested early in the development process
  • need for stubs is minimized
  • the need for drivers complicates test management and high-level logic and data flow are tested late. Like the top-down approach, the bottom-up approach also provides poor support for early release of limited functionality.

Top-down5

  • requires the highest-level modules be test and integrated first
  • allows high-level logic and data flow to be tested early
  • minimizes the need for drivers
  • the need for stubs complicates test management and low-level utilities are tested relatively late in the development cycle.
  • poor support for early release of limited functionality.

Implementation

After all features have been unit tested, we performed integration testing to discover that all major parts of the system work together well.

We employed bottom-up integration testing, which is integrating and testing the lowest-level units first, followed by higher-level components.

Functional Testing

Research

What are functional tests? Functional testing involves testing the system against the preset functional requirements and specifications. In functional testing, the structure of the program is not considered. “Functional testing is normally called Black Box testing. The test cases for the entire system are designed from the requirement specification document for the system.”6

Typically functional testing involves the following steps:

  • Identify functions that the software is expected to perform.
  • Create input data based on the function’s specifications.
  • Determine output based on function’s specifications.
  • Execute the test case.
  • Compare actual and expected outputs.

These are some of the techniques for Black Box testing:

  • Equivalence class partitioning: This method involves dividing the domain of all inputs into a set of equivalence classes, so that if any test in an equivalent class succeeds then every test in that class succeeds. This technique identifies classes of test cases such that if one test case passes then so do the rest. Without looking at the internal representation of the program it is difficult to pick out these equivalent classes. This method tries to approximate this ideal. This approximation is an assumption that if the specifications require exactly the same behavior for each element in a class of values, then the program is likely to be constructed so that it either passes or fails for each of the values in the class.6

  • Boundary Value Analysis: Many systems have the tendency to fail on boundary values, hence making it important to test these boundary values.This testing technique is where the extreme boundary values are chosen Boundary values include maximum, minimum, typical values, and error values. This testing technique forces attention to exception handling.7

ADVANTAGES OF BLACK BOX TESTING:

  • Tester is free from any pressure of knowledge of specific programming languages to test the reliability and functionality of an application / software
  • Facilitates identification of contradictions and vagueness in functional specifications
  • Test is performed from a user’s point-of-view and not of the designer’s. It simulates actual system usage.
  • Test cases can be designed immediately after the completion of specifications. 7

DISADVANTAGES OF BLACK BOX TESTING:

  • Some tests can be redundant if they have already been executed by the designer.
  • Test cases can be very difficult to design if the specification is unclear.
  • This method can not be used to test complex code segments. [3]

Black Box testing does not test a system thoroughly but it is reliable when it comes to functional verification and ensuring a bug free system.

How are we planning to carry out our functional tests?

We are planning to use the Boundary Value Analysis technique to test extreme boundary values in the system. We plan on using this method to ensure the system does not break when the user enters too many ingredients in their grocery list or when searching for a recipe. We will also carry out some exploratory testing to ensure that the system fully conforms to the stated functional requirements. This will involve designing use cases from the functional requirements specifications.

FUNCTIONAL REQUIREMENTS:

  • The robot should incorporate voice UI by allowing the user to create and update a grocery list by speaking to it: This feature will be tested by creating various grocery lists of different sizes. We will also try recording in noisy environments to check how the system performs under such conditions.

  • The robot should verbally respond to user input: This can be tested by asking Sota simple conversion questions such as “Convert 5 cups to ml”.

  • Sota should also incorporate Image recognition by being able to identify ingredients that the user might not know: We plan to test this feature by showing Sota a variety of ingredients. We will display these ingredients to Sota in various light intensities to check how it responds under such conditions.

  • Sota should be intelligent enough to recommend a recipe to the user depending on the ingredients the user has suggested to Sota: This can be simply tested by suggesting certain groups of ingredients and ensuring the response from Sota is actually correct.

  • Sota should also be able to walk the user through recipes using voice UI.
  • Sota should be able to carry out basic conversions e.g ounces to grams.
  • Sota should be able to set alarms and timers.

Implementation

We used 2 methods to perform functional testing on Sota: + Boundary Value analysis + Exploratory testing

References

[1] Theme and Powered, C. (2016) Unit testing. Available at: http://softwaretestingfundamentals.com/unit-testing/ (Accessed: 26 February 2017).

[3] Martyn A Ould & Charles Unwin (ed), Testing in Software Development, BCS (1986), p71. Accessed 31 Oct 2014

[4] Hunt, A. and Thomas, D. (1999) The pragmatic programmer. Available at: https://www.amazon.co.uk/Pragmatic-Programmer-Andrew-Hunt/dp/020161622X (Accessed: 28 February 2017).

[5] Microsoft (2017) Integration testing. Available at: https://msdn.microsoft.com/en-us/library/aa292128(v=vs.71).aspx (Accessed: 28 February 2017).

[6] Tutorials Point (ed), Functional Testing. Available at: https://www.tutorialspoint.com/software_testing_dictionary/functional_testing.htm (Accessed: 9 March 2017)

[7] TechTarget (2017), Six functional tests to ensure software quality. Available at: http://searchsoftwarequality.techtarget.com/report/Six-functional-tests-to-ensure-software-quality