Testing
Overview
We used a variety of testing methods to evaluate our tool, with a strong focus on user testing because of the interactive nature of our project. As the primary aim of our project is to deliver a reliable high-performance, and user-friendly 3D environment we have conducted comprehensive testing, including user acceptance testing, unit testing, compatibility checks, and responsive design evaluations. Given that our project focuses on visual 3D scenes, it was essential to design a user interface that aligns with user needs to create a smooth and engaging experience. Since the user remained central to our design process, we gathered feedback continuously from early stages to the final implementation.
Section 1 - Unit Testing
We used the popular vitest library to unit test the core functionality of our app. Each unit test was designed to test the functionality of a single block. This allowed us to rigorously test all the side-effects of each block, ensuring that upon composition, each block would function exactly as intended, without interfering with any other block. The unit tests function as follows:
- A headless block workspace (not rendered to the screen, as testing is run in a Node.JS console) is created
- The block to be tested and any dependencies are injected.
- For example, the material color change block would also have a parent object block and a colour picker block so it would function properly.
- The workspace is used to generate code, using the same function that the webapp uses.
- Then, a headless scene context is created.
- The generated code is run on the scene context, using the same function that the webapp uses.
- Then, the scene is verified to have changed state according to the block
- For example, for the material color change block, the
albedoColorproperty is verified to be set to the appropriate color.
- For example, for the material color change block, the
Section 2 - End-to-end Testing
After unit testing each individual block, we used the application to create multiple complex scenes, using all the features in each of them.
For example, one of our test scenes consisted of a simplified model of the solar system, with the Sun, Earth, and Moon. Each object was textured appropriately. Both the Earth and Moon were animated to orbit around the Sun and Earth, respectively.
This type of end-to-end testing showed that our system worked flawlessly in a real-world project similar to the scope of our most advanced users, and not just synthetic unit tests. While building the test scenes, there was also experimentation and exploration of what we (as a user of the software) wanted to achieve. This tested the resilience of our software to many workspace edits, block deletions, and page refreshes, which would not have been possible in our other user tests, due to the limited amount of time we could test with each individual user.
Section 3 - User Acceptance Testing
Although users and clients were actively involved in the UI design process and provided ongoing feedback, it was still important to conduct comprehensive testing of the final product with users.
During client presentation of product sessions, clients were shown a nearly complete version of the platform. They responded positively, especially appreciating the simple and user friendly interface. Additionally, they proposed the idea of including a JavaScript panel, suggesting it could help children who wish to go beyond visual programming and start learning 3D coding with Babylon.js.
To evaluate Babylon Studio with real users,we conducted structured user testing with 10 students aged between 11 and 14. Participants were introduced to Babylon Studio through an initial demonstration, followed by a hands-on exploration period of approximately 30 minutes, during which they experimented with creating, editing, and animating 3D objects, utilizing the Blockly visual programming interface and interacting with the AI assistant.The students were instructed to explore the platform thoroughly, making use of all core features in order to provide meaningful responses. After the session, we distributed a link with a Microsoft Form designed for gathering feedback. In the first part, users answered a combination of open-ended questions and multiple-choice questions to gather their opinions about the implemented features. This included assessing their overall engagement, clarity of feature explanations, and their subjective preferences regarding usability and intuitiveness. We collected 10 responses, and below we’ve shared and discussed some key insights:
Section 3.1 - User Testing Questions
| Question | Response Type |
|---|---|
| 1. Did you enjoy using Babylon Studio? | Yes / No |
| 2. Do you think Babylon Studio would help beginners with no prior experience in 3D design learn effectively? | Yes / No |
| 3. Did you find Babylon Studio engaging? | Yes / No |
| 4. Was the AI assistant useful to you when using Babylon Studio? | Yes / No |
| 5. Do you have any suggestions on how we could improve Babylon Studio? | Open-ended |
Section 3.2 - Responses
| Student | Q1 | Q2 | Q3 | Q4 | Q5 |
|---|---|---|---|---|---|
| 1 | Yes | Yes | Yes | Yes | Add some example scenes to start from. |
| 2 | Yes | Yes | Yes | No | It would be nice to have an undo button. |
| 3 | Yes | Yes | Yes | Yes | Include a library to share projects with friends. |
| 4 | Yes | No | Yes | Yes | More detailed AI explanations needed. |
| 5 | No | Yes | No | Yes | Make the interface more engaging for younger users. |
| 6 | Yes | Yes | Yes | Yes | Allow customization of UI colors. |
| 7 | Yes | Yes | Yes | Yes | Create tutorials accessible from within the app. |
| 8 | Yes | Yes | Yes | No | AI suggestions sometimes irrelevant; improve accuracy. |
| 9 | Yes | Yes | Yes | Yes | Add a gallery feature to show user creations. |
| 10 | Yes | Yes | Yes | Yes | Make the Blockly blocks more visually appealing. |
Section 3.3 - Summary of Responses
| Question | Yes (%) | No (%) |
|---|---|---|
| Enjoyed using Babylon Studio? | 90% | 10% |
| Helpful for beginners to learn 3D design? | 90% | 10% |
| Found Babylon Studio engaging? | 90% | 10% |
| Found AI assistant useful? | 80% | 20% |
Section 3.4 - Custom UI/UX Evaluation Framework – User Centered
Evaluation Questions
| Principle | Evaluation Question |
|---|---|
| Clarity & Feedback | Did Babylon Studio clearly indicate actions such as loading, saving, and displaying errors? |
| Language & Labels | Were the labels and language used on buttons and menus easy to understand? |
| Freedom & Reversibility | Could you easily undo your actions if you made a mistake? |
| Consistency & Flow | Did the interface design remain consistent throughout your usage? |
| Ease of Use | Was Babylon Studio easy to use without extensive guidance? |
| Memory Load | Did you find it easy to remember how to use Babylon Studio after your initial experience? |
| Accessibility & Efficiency | Would additional accessibility features (eg. shortcuts ) enhance your experience? |
| Visual Appeal | Did you find the visual design appealing and well-organized? |
| Error Handling | Were error messages clear and helpful for resolving issues? |
| Support & Helpfulness | Were the AI instructions and step by step guidance useful to you? |
Section 3.5 - Responses (out of 10):
| Principle | Positive Responses (out of 10) |
|---|---|
| Clarity & Feedback | 9 |
| Language & Labels | 10 |
| Freedom & Reversibility | 7 |
| Consistency & Flow | 9 |
| Ease of Use | 8 |
| Memory Load | 8 |
| Accessibility & Efficiency | 6 |
| Visual Appeal | 9 |
| Error Handling | 7 |
| Support & Helpfulness | 8 |
Section 3.6 - Summary
| Principle | Positive Response (%) |
|---|---|
| Clarity & Feedback | 90% |
| Language & Labels | 100% |
| Freedom & Reversibility | 70% |
| Consistency & Flow | 90% |
| Ease of Use | 80% |
| Memory Load | 80% |
| Accessibility & Efficiency | 60% |
| Visual Appeal | 90% |
| Error Handling | 70% |
| Support & Helpfulness | 80% |
Section 3.7 - Babylon Studio in Use
| Question | Positive Responses |
|---|---|
| Would you recommend Babylon Studio to a friend or classmate interested in 3D programming? | Yes (10/10 – 100%) |
All students enthusiastically recommended Babylon Studio, highlighting its engaging, intuitive, and educational value.These insights indicate that there is significant room for improvement, and the identified issues will be addressed in subsequent iterations following the release of the initial version.
Section 4 - Responsive Design Testing
Responsive Design Testing Results (Simulated via Matt Kersley Tool)
To assess Babylon Studio's responsiveness across various screen resolutions, we used the Matt Kersley Responsive Design Testing Tool, which simulates different device widths commonly used across desktops and mobile devices. The simulated viewports tested included:
240px (older smartphones)
320px (standard mobile)
480px (small tablets or landscape phones)
768px (tablet portrait)
1024px (tablet landscape and small desktops)
1280px+ (standard desktop monitors)
Key Findings:
| Viewport Size | Observations |
|---|---|
| 240px & 320px | The interface was not usable at these resolutions. Elements overlapped, Blockly blocks were cut off, and buttons became too small to interact with. Navigation menus either disappeared or required excessive horizontal scrolling. This confirmed that Babylon Studio is not suitable for smartphones. |
| 480px | Slight improvement in layout, but critical UI components such as the Blockly editor and 3D canvas remained unusable. Text readability was compromised, and block dragging caused glitches due to limited viewport height. |
| 768px | Layout began to stabilize. The interface was functional, though cramped. Usability was possible but not ideal. Users could navigate and interact with features with some difficulty, but the learning experience was hindered by constrained space. |
| 1024px & 1280px+ | Babylon Studio performed best in these environments. The interface scaled smoothly, all components were visible and functional, and interaction with Blockly and the 3D scene was fluid. Visual clarity and usability were maintained without compromise. |
Section 5 - Browser Compatibility Testing
To ensure broad accessibility of our web application, we conducted compatibility testing across several browsers, like Safari, Google Chrome, Firefox, Microsoft Edge. In all cases, the application displayed and functioned as expected. Additionally, we assessed performance across various device types. The platform was not optimally displayed on smartphones or tablets and we decided not to support smartphone or smaller laptops and tablets devices to maintain the quality of user experience intended.