Testing method

Due to the nature of our system, we have decided to conduct simple alpha testing. Alpha testing is a type of User Acceptance Testing which will be very useful to try the system in a semi-controlled environment [10]. We will be able to directly pick out specific functions we want to test whilst also being able to analyse the code in conjunction. While other forms of testing such as unit testing is incredibly useful, unit testing may not be appropriate here. We are unable to test specific and individual modules. Modules within a video game will always run in conjunction together and therefore it would be very difficult to isolate one single function to conduct unit testing on. Therefore, we have decided to continue with alpha testing. This involved writing out a testing criteria based on the core mechanics of MineACT. The tests were done by a member of our team. They had set up a Spigot server and tested the plugin within the environment, going along the different cases. This also involved one member creating sample conversations in Voiceflow, ranging from varying degrees of complexity (whether the test was a simple "Hello" from the bot to allowing complex levels of branching). We were also given Voiceflow conversations from our client which included a high degree of branching.

Test	Result
Parser detects OS and correct python command is called when running the intents process.	Works correctly, tested on Windows and Linux servers.
Python script runs and fetches all intents from all sheets using NoCodeAPI and returns it in a String format.	Works correctly, message returned as a String with “, “ as a delimiter.
MineACT parses all of the intents and converts it into an array for fast and easy access.	Works correctly, the array has all of the correct intents, including ones with spaces.
Parser converts any ‘.vf’ file correctly with the variable map, intent map, and node map.	Works correctly, the maps have the correct key and values put in.
Parser successfully finds the ‘.vf’ file from the vf_directory folder, else returns the correct error.	Works correctly, searches for extension and parses. If no file, returns an error and exits program.
MineACT bot is able to iterate through the nodes and carry out a conversation until the end.	Works correctly, all possible nodes can be traversed with the correct blocks that the parser allows.
MineACT bot is able to parse “Speak” blocks (i.e.: bot is able to speak to the user and replace variables with respective value).	Works correctly, tested with 1 and many variables, all replaced properly.
MineACT bot is able to parse “Choice” blocks (i.e.: allow for simple branching to take place with two choices).       Check to see if conversation goes to first choice.       Then check to see if conversation goes to second choice.	Works correctly on tested conversations. Choice allows the user to branch out to any of the blocks. If invalid output, it will return 0 and go to null which will end the conversation. It will return the correct node ID and branch.
MineACT bot is able to parse “Capture” blocks (i.e.: allow the user to input a string which will be stored in a variable to be used for nodes)	Works correctly, will add the variables into the variable map. It waits until the user inputs a word.
MineACT bot is able to parse “Condition” blocks (i.e.: compares user input to variables/values ).       Check to see if first condition can be met.       Check to see if other conditions (including else statements) can be met.	Works correctly, will add the condition properly and parse it. Currently, the ‘ equals’ condition works. It will check for both variables AND values and return the correct node ID and branch.
MineACT bot is assigned to a user and added to a map so that two bots may not be assigned to one user. (Also allows for multiple users to have their own unique bot)	Works correctly, the MessageListener class adds a bot to the player. If already assigned, it will update userInput instead of adding again.
MineACT bot listens to messages sent by user and not respond until triggered.	Works correctly, tested with the intent list from the Google Sheets.
MineACT bot completely parses file and has bot ready upon start-up of the server.	Works correctly, system prints line onto console when plugin loaded.
MineACT bot is able to successfully retrieve all intents from linked Google Sheet.	Works correctly, system prints line onto console fetching and returning the list.
User is able to start a conversation with the bot if an intent has been detected.	Works correctly, when triggered it checks if assigned already, if not, it will assign a bot to the player.
User is able to input varieties of inputs (i.e.: if an intent within the Google Sheet is “will be impossible”, the user is therefore able to type in “ everything will be impossible”.	Works correctly, lowercase and uppercase are also checked. Uses a contain() function to check if it is a substring.
User is able to start a “treasure hunt” and complete it once the conversation with the bot has been completed.	Works correctly , treasure hunt begins upon end of conversation and will continue until solved.
User may still continue to play the game when the bot is active, and conversation triggered.	Works correctly, game still runs whilst the bot is waiting for user input.
Treasure hunt spawns in a random x, y, z coordinate near the user.	Works correctly, treasure generated randomly within a reasonable range of the player.
Treasure does not spawn in the air and will always be below the player.	Works correctly, treasure is always on the ground or below.
Treasure does not spawn below 20 in the y-axis.	Works correctly, treasure is never below 20 on the Y axis.
Treasure amount is never below minimum nor above maximum the possible 3-D range.	Works correctly, treasure is within range for the x, y, and z axis.
Treasure type is close to the probability distribution of the rewards.	Works correctly, probability is calculated, and material type is generated.

Results and conclusions

As shown by our results above, we had faced no bugs or errors along the way. Each of our tests had worked perfectly according to the criteria we had set. Perhaps if we were able to conduct wider and more extensive user-acceptance testing with a larger pool of testers, we could have uncovered more bugs. However, from our own in-house testing, we had concluded that no bugs are currently present within the MineACT plugin.

References

[10]T. Peham, "5 Types Of User Acceptance Testing", Usersnap, 2021. [Online]. Available: https://usersnap.com/blog/types-user-acceptance-tests-frameworks/ [Accessed: 03-Mar-2021]