UCL, GOSH and Avanade are working together on this second iteration of this open source project. This work will be led with the simulation team at GOSH. The project will involve building an editor with layers and interactive timeline hotspots that apply situated cognition 360 videos to a handful of human factors training sessions during a series of emergency scenarios. Speech transcription with Deepspeech or other local speech recognition should be included to generate searchable summary notes that allow you to jump through the timeline and position in 360, as well as generate multilingual text for all users. The open source project will reside in the NHS index at Apperta Foundation.
Requirement gathering
We collect the requirements in several meetings with our clients in first few weeks
Persona
User Case List
MosCow List
Functional
must have
Adding hotspots by click
Hotspots at static positions
Adding name and text to hotspot
Photos attached to hotspots
Hotspots details store to json files
Hotspots details read from json files
Basic player functionalities for editor- progress bar, play button, sound button
should have
Hotspots only appears at certain time period
Hotspots can be deleted
Switch to other videos and switch back
Outcome of editor delivered as a pack
Rename and organise the files attacked to hotspots
could have
When jumping back from the branched video, the video player is at the previous time spot and facing the previous hotspot
Timeline showing the hotspots attached to videos
Timeline showing the hotspots of all branches
Separate player that can unpack the package produced by the editor
Separate player that can view everything produced by the editor
Basic player functionalities for separate player- progress bar, play button, sound button
Choose to include player app when editor publish package
Auto human hand touch order recognition
Hotspots and its photos and branches can be renamed
Timeline can navigate the users to the timespots
Timeline can be updated after adding hotspots
Player would delete the pack generated from the package
Own Logo for the project
Auto explanation of functionality when hover on buttons
Colored timeline to represent state of branches
Auto speech transcription for video imported
Branches can be renamed
Non-Functional
must have
The software needs to be precise and accurate about the information it helds.
Safe and Secure: Files and its relative information would be held secure
The whole system should be held offline.
User-Friendly: Files should be held in a organized and understandable way.
should have
User-Friendly: Reduce the probability for the users to lost their work by accident
There should be help pages/ guidance on the page to help the user.
The software should be designed to deliver the files it produces easily.
could have
Clear and understandable interface
Buttons with clearer and more understandable icons
would like to have
All the icons should of the same style.
The panels display the information clear.
Research
Related Project
We received a web-base 360 video player from our module last year. The similarity is we both play 360 video and have an editable layer. But our project is completely offline and have much more feature in editing the video.
Possible solution A: Just like the project done last year, we build the web app on electron and attach a unity player/editor to it.
Pros: The web-based website can be deployed on different machines.
Cons: There might be bugs involved when merging the 2 systems, and web-base application have more security risk
Possible solution B: All parts of the project are done in Unity.
Pros: Unity is open and flexible to changes, completely offline app is convinient
Cons: Unity itself is very complex and has different versions, so project built in old version of unity will have bugs when it is deployed into a newer version of unity.
Unity only support C# as script, so we didn't consider other languages as we work in Unity.
We decided to implement the whole project inside unity, so the project will be safer compared to web-based. Also, many NHS videos has its own GDPR regulation with it so it’s better to hold everything locally.
UI design
Since the users of our app will be clinicians and students, we designed the UI to be simple and clean. The app only have one main page and is divided into three sections: menu, player, and timeline.
We use icons instead of text on the buttons of the menu, which make the app simpler but harder to understand the meaning. So we build an extra help page to guide the users, since this editor have multiple functionalities.
We come up with many sketches and build a prototype with invisionApp Which we still make a few changes to the design during the development process, but the main theme and idea remain unchanged.
The doctors(user) provide 360-videos to the editor and the editor will generate a project file in file system, which Json file record the modification of the user. The editor then publish the Learning Pack(Packed project file) which the user can send to students(audience). The audience need the player to enjoy the package
Implementation
The app we build have architecture similar to web pages, that UI elemtns and scripts are seperately implemented. All elements you see in the app is build with Unity scene, and the scripts we wrote in Assets folder provide the reaction of each elements.
In Unity scene, the elements are implemented in modules: Menu/Player/TimeLine/Panels are all in main canvas, but are independent from each other. Similar to html web page, each element can have parent/child elements which can inherit properties; but unlikely to html, we can modify the properties like layout, image, script and size directly in Unity instead of coding.
We have written 28 scripts in this project, and each of them are responsible for a subsection of functionalities.
We only use one library in the entire project, which is file selection function. It is located in the bottom of VideoManager.cs script(line 76).
Timeline is one of the key features in our project, its mainly implemented through the script TimelineController.cs . The script will read the Json file in the project and generate an image to fit in the timeline element. Our project file is designed to have same structure in every video branches, so the script can recursively walk through every branches and the hotspots in them.
We execute the script each time we create a hotspot or save the project so the image can be regenerated and updated. The "click to locate" function of the timeline hotspot is implemented by reading the x,y location and start time location of hotspots in Json files, then replace the current view location and video progress time to locate the target hotspot.
Testing
We do the testing in the last two weeks. Although we didn't do unit test or branch coverage test with our code, we do white-box integration test our app on multiple machines by development team member.
We also demostrate our app to our clients and our teaching assistance for some advice and feedback.
Our clients has pointed out few possible improvement and we found number of bugs during the test. From user perspective, clients had suggested wording of the buttons and instructions are confusing, and our timeline could present more information about the hotspots and branches.
We changed the wording to simpler one and build a help information panel to guide the users. But we give up the improvement of timeline since we need to rebuild the timeline logic and we are out of time.
