We have used various technologies and dependencies to work on the system, including languages, libraries, and APIs. They each work together in conjunction and will be important for future developers for this project.
Java and Python are the two primary languages we have decided to use. Java is a necessity since Minecraft is a Java-based application. Python has been used to communicate with the Google Sheets API which will then send back a list of intents back to the plugin. The two are able to communicate through the ProcessBuilder library in Java. Java will call an instance of the Python script. During development, we had used between Python 3.7.0 and 3.9.0; they both have worked successfully. Though, naturally, we will recommend using the latest stable build that supports the other library dependencies.
The use of Google Sheets and NoCodeAPI are key components of our project. NHS clinicians will be storing intents on a Google Sheet. To connect the Python script to the Google Sheet, a JSON API will have to be implemented through NoCodeAPI. Once the API has been set up for use, an API endpoint URL will be used to make GET requests to the Google Sheet in order to retrieve all intents.
Jackson is a Java library that allows the parsing of JSON. By using this library, we can turn JSON into JsonNode objects. This will allow us to use methods such as .get() to
retrieve specific information from a JSON node, alongside various other important methods. This will be useful to retrieve node types from the Voiceflow JSON file and also the information stored
within it such as the bot's lines in the conversation.
Minecraft is a game where multiple processes may take place at once. The player may be able to use the in-game mechanics whilst also chatting with their friends through the chat function.
However, with the inclusion of our plugin, this had brought the problem of how a player can continue interacting with the game whilst also interacting with the bot. We did not want to stop
the player from being able to play the game so that the bot can parse the conversation into Minecraft. Therefore, we had implemented multithreading into the plugin. The Spigot library has allowed us to
use new Java methods including AsyncPlayerChatEvent and BukkitScheduler. We had decided the best course of action would
be to ensure all interactions with the bot, including parsing, are completed asynchronously on another thread. This way, the game will not have to stop to wait for the interaction with the bot.
Along with this, each player will be assigned a bot to them if there is the need for multiple players on one single server. The player and the assigned bot will be put into a HashMap in which they
can have a conversation with their specific bot. The code below will run if the player does not have a bot assigned to them.
In order to put the Voiceflow conversation into Minecraft, we had to create a parser. Currently, the parser is able to parse a limited number of blocks from Voiceflow, but includes any of the main blocks a clinician may use when creating a conversation. To build the conversation into a readable format, we have to first retrieve the root node of the conversation (where it will begin). For each node, we have to also retrieve the block type which will be given a corresponding function within Minecraft. The parser will be reading the JSON file that is plugged into the server files. The nodes will be added to a HashMap called nodeMap so that it can be easily parsed. The code below will retrieve information including the root conversation node, a node containing all hardcoded Voiceflow intents, and a node containing all hardcoded Voiceflow variables.
As stated before, we have used the Jackson library to parse the JSON data. This will allow us to extract specific information from the JSON at hand. The specific information that we need include the node type (which determines what type of interaction the bot will have with the user and vice versa) and the node text (which is the text that the bot may say to the user). From a wider perspective, Jackson is essentially the library which allows us to move from one conversation node to another. This is because the Voiceflow JSON is formatted in such a way that each node has a unique hash ID as well as a "target" ID which determines which node it will go to depending on the interaction that is meant to take place. Below, you will see the JSON of a "speak" block. This block allows the bot to speak to the user. The parts that have been underlined in red show the ID of the current node and the ID of the target node. Since this is a speak node, it will move to the target node without waiting since the user is not inputting any strings just yet.
The speak block allows the bot to speak to the user. It only has one target node. This bot contains strings which will be outputted into Minecraft. The user cannot interact with this; the bot does not wait for an input from the user to output the contents of a speak block. The code below shows how the speak block is parsed.
The choice block has multiple target node IDs since this is a block which allows branching. Branching will take place if a string hardcoded in Voiceflow has been detected. The Voiceflow interface allows for the storage of intents, but it is best that these remain limited to simple "Yes" and "No" statements; Voiceflow does not have NLP and therefore expects an exact match of the string. These intents are separate from the intents stored in the Google Sheet. These will only be used for the choice block. These hardcoded intents will be stored in a HashMap called intentsMap. This HashMap will be referenced to when the user has landed on a choice block. The parsing of the choice block is shown below.
In the code snippet given below, a JsonNode called "expression" is being created. This part of the code works to parse the "Condition" block in Voiceflow which allows for the use of IF and ELSE statements, another form of branching. As shown by the code below, the code first checks to see the data type of what is being compared. It will do the same with what this is being compared to (for example, determining if the first item is a user's captured input and the second item is a pre-set intent hardcoded in Voiceflow). The code will then check to see if these two variables are equal to each other, which will lead to branching.
Another block that we had implemented is the "capture" block. The capture block allows the user to enter an input regardless of conditions. This input will be stored in a variable which can then be used later in the conversation. When the conversation has moved to a capture block, the bot will wait for the user's input. Once the user has entered a string, it will assign the variable with the value of the user's input. This will then be stored in a HashMap called variableMap which will be referenced to when called later in the conversation. The parsing of a capture block can be seen with the code snippet below.
All intents will be stored and updated within Google Sheets. To retrieve all the intents, we had to connect a Python script to NoCodeAPI by using an API endpoint URL which can be retrieved from the service.
All intents will be retrieved from the Sheet as JSON by using the requests library. The JSON contains a list of dictionaries. The script will iterate through this list and retrieve the values from
each dictionary. This is because the intents are stored as values while the headings are the keys. Java is able to call the Python script by using ProcessBuilder, a library which allows Java to call
instances of external programs. The code below shows how the GET request is made to the NoCodeAPI endpoint. An API endpoint URL must be placed here. Once this request has been sent, a response containing all the
data as JSON will be returned, encoded, and then placed in a readable format (i.e: a list of all the values).
The array will be stored for later use. When a player enters an input, the plugin will check to see if an intent exists as a substring within the user's input. If found True, then the conversation will begin. Else, the bot will continue waiting. Since we are checking to see if the intent exists as a substring within the user's input, the user is able to type in more diverse range of strings. Rather than having to type exactly what is within the spreadsheet of intents, such as "will go wrong", users will be able to type in other inputs like "everything will go wrong :(". This is shown here in the code below. This function is case insensitive and therefore does not consider case.
One final function that we had implemented is the treasure hunt. This adds a level of fun to our plugin. Once the player has completed a conversation, they are able to begin a treasure hunt. It is a reward for completing the therapy session. The treasure hunts will give the players randomly generated clusters of in-game ores such as diamond, iron, or gold with varying degrees of probability. The cluster is generated near the player and will go down several blocks. If the root of the cluster touches an empty space, then it will continue to go downwards before setting itself. The player will be given a message of "warmer" or "colder" based on how much closer or further away they move from the cluster. The code below shows how the spawn location of the treasure will be determined. This is relative to the player's current location and will take random X and Z coordinates which are 30 blocks away at most.
The image below shows the point-of-view of the recipient. We are able to successfully enter strings through the in-game chat to start a conversation with the bot and continue it until the end.
The image below shows the beginning of the treasure hunt game after the therapy session has been completed. This particular part of the code is setting the spawn location of the treasure. The player will be prompted with "Warmer" or "Colder" depending on their position relative to the treasure. As they get closer, they will be prompted with "Warmer", and as they get further away, they will be prompted with "Colder".
System developed by Arifuzzaman Choudhury, Nishan Sen, and Ruochen Sun
With the support of our industrial partners Dr Joseph Connor (NHS) and Professor John McNamara (IBM)
Supervised by Dr Dean Mohamedally and Hongxiang Chen
University College London
