First Bi-weekly report

Aleksandar Rusinov , Jetnipat Sarawongsuth, Dingzhong Weng

We have been assigned a task to create an occupancy predictive engine. Our client is Qualcomm. We will be closely collaborating with Team 21 which will be working on the hardware side of the project (Project name: Qualcomm Presence with Unifying Signals) while our team, Team 20, will be working on the software side of the project . On 06/10/2014, all of the team members of both teams had a joint client meeting to gather more information about the project.

First Client Meeting (Date: 06/10/14)

The followings are the items discussed during the client meeting:

Stretched Goals:

• Using the data we have to control external devices such as light, heater, etc.

Additional Ideas:

• The possibility of using machine learning.
• To start with, using the information obtained from the sensors as well as our own visual observations, train the system to detect the population of the room with the sensors and double check the accuracy of the system from the sensor using our observation.
• Experiment with different sensors and detectors and analyse the accuracy and effectiveness of the data obtained from these sensors before narrowing down the list of sensors to be used
• The effectiveness of using Hadoop.

Group Meeting (Date: 07/10/2014)

We looked through the advantages and disadvantages of using different types of sensors in our project. We decided to work jointly with the other team on the hardware solutions at first. The following are some of what was discussed:

1. IR distance sensors – most suitable and cheap.
2. Microwave sensors – relatively cheap but it is still unclear as to whether it will be a convenient way of doing so.
3. Laser sensor – very good and precise but expensive, each costing around £300.
4. Beacon – enables to track a specific person that is upto 50 metres from a receiver, and even up to 100m in open spaces..

We also brainstormed ideas about the data received and how it will be obtained:

We were discussing about the possibility of mounting an array of IR sensors on top of the two sides of each door pointing vertically downwards so we will receive a signal which notifies the system when an object passes through the sensors. The data will be captured from the IRs with the frequency of 10 per second (860000 per day) and decide if there is a person present.

If the person is present we set:
outer IR as OIR                 (this IR sensor is located on the outside of the door)
and inner IR as IIR           (this IR sensor is located on the inside of the door)
If a person walks (abnormal value on the sensors) through OIR and then to IIR  we sent Timestamp for person entering. 
If a person walks (abnormal value on the sensors) through IIR and then to OIR  we sent timestamp for person exiting.

Example of how the data may be stored:

SensorValues[]{
OIR[]
OIR_TIMESTAMP
IIR[]
IIR_TIMESTAMP
MICROWAVE
VIDEO
TIMESTAMP
}

Example of simple processing of the data:

IF OIR_TIMESTAMP<IIR_TIMESTAMP – ENTERING
IF OIR_TIMESTAMP > IIR_TIMESTAMP  - EXITING

We will set a delay limit to a specified figure, for example, 1 minute, so unusual behaviours, such as OIR being triggered but IIR not being triggered within a set timeframe, will be discarded; If one sensor is triggered but not the other within the period of 1 minute, then this will not be recorded onto the system.

Looked at Hadoop – Two main parts – MapReduce and HDFS (hadoop data file System) with a lot of tools providing different functionalities :
Hive – allows one to write queries in HiveQL – language similar to SQL
Pig - Pig Latin – language based on Hadoop can adept at very long data pipelines.
HBase – non-relational database that allows low-latency lookups in Hadoop
Flume – enables populating Hadoop with web servers, mobile devices to collect data to Hadoop
Oozie – workflow processing system allowing one to define schedule of jobs written in multiple languages.
Ambari – set of tools that allow the managing of Apache Hadoop clusters.
Avro - serialization system allowing encoding the schema of Hadoop files. Good for executing removed procedure calls.
Sqoop - moving data from other kind of data stores into Hadoop (i.e SQL database)
HCatalog – metadata management
Mahout - Data mining library. Performs regression testing , clustering and statistical modelling implementing map reduce. (may be important for the predicting engine)
Tutorials from You Tube - https://www.youtube.com/watch?v=xWgdny19yQ4
Meanwhile, we are learning from previous works. The relating articles are listed as the following:
http://www.eng.yale.edu/enalab/publications/human_sensing_enalabWIP.pdf

Second Bi-weekly report

Much progress has been made for our team this week. Whilst the other team (Team 21) is still deciding on which sensors they will be using in this project, our team has started writing a basic java program that simulates the input and compute the number of people in rooms. We have taken into account the possibility of each room having more than one door as well as how each room are connected to each other. We have come up with two different approaches that might do the job for us. They will be explained later in this report. We also had another short joint meeting with the client. The minutes of the meeting are shown below.

Client Meeting (Date: 20/10/2014)

This week we clarified to the client about the timeline of the project. Our aim by the end of the first term is to do as much research about the topic as possible as well as creating the first working prototype of the product. We then talked to the client about our progress and kept them up to date about what we are currently working on as well as what we will be working on in the next weeks or so. We raised the idea to our client about the possibility of using buffers to aggregate the data from the sensors then sending them all together at a regular interval. However, the client are reluctant about us using this approach as the data obtained will not be in real time. Later on in the meeting, we went on to discuss the problems that could arise should the current system (having sensors mounted at the doors) fail to detect a person enter or leaving. They raised the question about the effectiveness of this idea and how difficult it could be to error correct and suggested that we use more than one type of sensors. For example, as well as using the doors sensors, we may also use thermal camera to count how many people are in a room or look at how many devices are connected to Wi-Fi.

Considered different approaches and design:

First Approach

Here, the building will be represented as a graph (fig 1) and all the doors are represented by the edges connecting the nodes (room) together. The weight function is on the vertices and it shows the number of people in that room. All the Rooms have their individual IDs that are in strict weak ordering form. There is a buffer that is send every 0.1 second* and it indicates the direction and the number of people passing through each door. If the we receive a signal indicating that someone is passing through a door and it is from the room with higher ID to the room with lower ID, then we assign -1 to it and vice versa. This buffer then changes the graph. Tuples with fields containing the time, number of people, etc. will then be sent to relational database every half an hour* for each room.

Figure 3 Graph Model of a Building

Figure 4 Handling the Message Received from the Sensors

Second Approach

We assign doors as objects with names indicating the rooms they are adjacent to. For example, a door might have a doorID of “F01R12F01R13”. This means that this door connects the room with ID “F01R12” and “F01R13” together. When a person passes through a door, a string is sent in JSON format with the two RoomIDs – “F01R12F01R13”**. This gives us the information on the direction of travel. Now we know that the person is going from the first room “F01R12” to the second “F01R13” and if the person is travelling in the opposite direction the RoomID would be “F01R13F01R12”.

This week each of the members of the team has done the following tasks:

Aleksandar:
Done sketches of the variants considered.
Organised meeting between the teams and Steve Hailes.
Java application:
Looked at and used JSON processing – JSON simple (https://code.google.com/p/json-simple/) and JSON processing API (https://jsonp.java.net/)
Looked at and used Guava: Google Code Library - https://code.google.com/p/guava-libraries/
Considering Java2D and Swing for the simulation - http://zetcode.com/gfx/java2d/ , http://zetcode.com/tutorials/javaswingtutorial/ .
Started developing Java app for the engine.

Jetnipat Sarawongsuth:
Done minutes of meeting with client.
Looked at Java MySQL API and how to implement it. - http://www.vogella.com/tutorials/MySQLJava/article.html
Wrote a java class file that communicates with MySQL.

Dingzhong Weng:
Acquainted the team with Github.
Done repository for the project.
Done UML diagrams.
Done UML Use case and class diagram.

*The time period might be different.
** This may not be the optimal way of representing the RoomId.

Third Bi-weekly report

Over the past week, both teams, team 8 and team 9 have been cooperating and discussing over the data format that our team will receive from the other team. Jetnipat Sarawongsuth was tasked to look at and decide on the different hardware device that can be used to connect the sensors that the other team had been experimenting with. A decision was then made that the sensors are to be connected to an Arduino board version YUN. An Arduino is very easily programmable and is very versatile in nature. Unlike other version of Arduino, Arduino YUN’s board has a built-in Ethernet and WiFi support. This will turn out to be very handy when we are looking at sending the data wirelessly from the Arduino’s sensors to a database when the sensors are interrupted. As an experiment, two IR sensors are to be connected to an Arduino YUN which will then be mounted on top of a door frame. A simple program has been written by the other team that can detect movement when something passes through the IR sensors. Over the next weeks or so, I will be looking into creating a simulator of the floor plan in to test our system and the error handling module.

Client Meeting (Date: 10/11/2014)

This week we had another joint meeting with the other team with the client from Qualcomm. We mostly explained to the client the progresses of both teams. The following are what was discussed.

The other team has been making progresses on Arduino and the IR sensors. They have written a simple piece of code and detects movement when someone walks pass the sensors. However, the data here are not being stored and is sent over the USB cable. In the near future, we will be looking at using Arduino YUN which will make it possible to send the data from the sensors directly and wirelessly to a computer, database, or a tablet. At the moment, there is a slight issue with the batter power. This means that the IR sensors do not have enough power to cover the whole length of the door. They are still looking into the possibilities of using alternate power source that will provide enough powers to the sensors. The usage of Kinect was also briefly gone over. Kinect could be used as the secondary sensors to the IR sensors. After having discussed that, our group explained in a fair amount of details in what the database structures may look like. We also explained to the client on how the basic java application that we wrote works as well as explaining to them about the usage of MySQL table as a mean to store data. During the explanation, the clients once again brought up the idea of hardware errors and how our current software might not be able to error correct. They suggested that we have an error correcting module or inconsistency spotters that gives a flag when the data seems incorrect. They introduced the idea of using confidence level when we calculate how many people are in a room by looking at the reliability of the sensors. This could be done by putting weightings on the different sensors. For example, sensors that are not 100% accurate might get a smaller weighting in the result so the result will not be entirely based on the inaccurate sensors. This is something that our group might be looking into implementing at later stage of the development of the application.

Client Meeting (Date: 13/11/2014)

After a brief meeting on Monday, the client agreed to come to UCL to have a face-to-face meeting with both Qualcomm teams. The meeting started off with the Qualcomm representatives explaining what they had done earlier this year with a Kinect device. They talked about how they were using different software to track hands movement. They used skeleton tracker which track the hands as well as the elbow. This eliminates the problem where the motion tracker would lose the hands when the two hands cross over each other. This is very relevant to our project since the hardware team is also currently working and experimenting on the usage of Kinect as one of the sensors. Instead of tracking the hands, our team is looking at researching and developing a program for Kinect that can be used to track people’s head as they walk through a doorway. The Kinect device will be fitted on top of the doorway. However, since it is a complicated device, the cost is significantly higher than the cost of our other sensors. This was discussed in the meeting and the client suggested that the benefits of using such sensor outweigh the cost of using and operating it so we will continue working with it for the time being. The hardware team also went on to explain their progress on the current Arduino work they have been working on. They showcased the IR sensor working alongside an Arduino and displaying the result on a laptop. As an object cuts through the IR beam, the counter increments and display the total on the laptop. However, there are a few different problems that were presented such as the rate of detection can be slightly slow. Alongside IR sensors, we also talked about using ultrasound as another sensor. This will also be mounted at the entrances to a room. Our group then started explaining our progress on the database as well as the user interface that we have decided to go with. We decided to go with Aleks’ user interface design as it is very practical and informative since it provides a graphical representation of the building plan, the current number of people in each room as well as some statistics and graphs from past data. We showed the client our basic user interface design and they seemed to be pleased with it. The client brought up the idea of the different sensors sending different values for what was supposed to be the same. For example, an ultrasound mounted on door A might not suggest there’s a person passing through but the IR sensors might suggest otherwise. The client asked a question on the reliability of the sensors and which sensors to disregard in such event. To answer this problem, we will be looking into the details in the coming weeks but we briefly explained our solution to this problem. Our solution to this issue is to come up with an algorithm that takes into account the reliability and the preset confidence levels of the each of the sensors and come up with an overall confidence level. Each sensors will carry and confidence level weightings which will be set according to their real life accuracy and reliability. Lastly, we explored over the different ways in which the data from the sensors can be delivered or received. We talked about the two ways in which the data can be sent. The first being event based data. This is when the data is sent automatically every time an event is occurring. For example, a data is sent when the IR sensors detect movement. The second being active data sending. This comes in the two forms of data pushing and pulling whereby a system pushes the data into the database every specified period of time for data pushing and the data is sent only in the event where the database requests it for data pulling.

During the last lab classes our team met with the HCI supervisor for the module. We discussed the user interfaces for our system we made for course work 3. We agreed to step on the basis of the UI presented by Aleksandar Rusinov. We were given few advices how to enhance and make it better. Our supervisor recommended checking Google nest Thermostat applications and looking at the interface there.

Nest learning thermostat UI research report

Nest learning thermostat is a wi-fi enabled thermostat that has a lot of different functionalities such as personal scheduling, program choosing and history viewer. You can also set and view the temperature remotely from your mobile phone. Also the thermostat allows you to view the recordings of the temperature for each room for a certain period of time. This functionalities make Google Nest a valuable product to look at since one of the main goals of our UI will be presenting similarly structured data.

Here is how the temperature in the moment for a specific location is presented. Similarly for our project one may have the number of people instead of the degrees. Depending on how many people are there in a location the color of the circle will indicate how crowded is the space there. There also may be sensors in a queuing place - café, theatre, cinema, zoo and and a time estimation that says how much time will pass until all the current people have passed.

This is how the recorded data is presented in the Nest app. The energy consumption history is displayed with different variants. We may include the occupancy level when we present the data in one of the future UI mocks. That is how you can figure out in which hours a place gets crowded usually and not wait a really long queue for hours.

Intial PoC Web Design

Dingzhong Weng will work with Lazlo de Brissac from the hardware team to create initial PoC report on a website. They agree to implement Ruby on Rails as their framwork for entire website development. As for the style of presentation, they have decided to design a simplistic single page website and to section the important aspects of initial PoC, so that the client can have a straight-forward understanding of all the information presented. Also, both team have agreed to share the layout of the website. It will have two navigation button clearly indicating the name of the team, so that the client can directly click and view the respective PoC for the team selected.

Jobs done:

Aleksandar:
Looked at Google Nest
Done accepted UI design
Construct Tables for the new concept
Done Proof of Concept draft skeleton

Jetnipat Sarawongsuth:
Done minutes of Skype meeting
Done minutes of general in campus meeting
Done research for a connection device

Dingzhong Weng:
Done draft for the project site, link: https://qualcomm.herokuapp.com (temporary)

Fourth Bi-weekly report

Progress

During the last two weeks, we have made much progress and we approached the issues of setting up an online system. During the first week, after migrating from a common Java application to a dynamic web application, we have done research on servlet and we conducted some tests and examples. Unfortunately, the deploying of the Java application on a server took longer than we previously expected so we settled down with a PHP solution that sends the data to the database and displays it. We discussed that issue with our client-Nadine Macrae on Monday and we were able to develop a PHP prototype that satisfies some of our objectives and points that the client mentioned by Friday 28th November. We plan to show it to the client on the next client meeting on Monday 1st December and ask for their feedback and improvement suggestions.

Client Meeting (Date: 17/11/14)

We had another joint weekly meeting with Qualcomm and we had a discussion on the following subjects. The hardware team had been experimenting with the ultrasound sensors which they had been planning to use and they raised an issue with them but this will be looked at in the coming weeks. Alex’s team uploaded code on github. Also, some fixes on UI design are on schedule. Qualcomm and both team decided that the meetings will hold in two periods in the future. First one at 11 am will be meeting with Alex’s team, and second one at 11:30am will be meeting with Lazlo’s team. Speaking of the UI design, Alex brought up that the team has talked to its supervisor for some advice over the design. Before next meeting, Alex will set up a database for both team to communicate data with. Qualcomm suggested to take a look into REST design pattern. Both team decided to continue researching in ways to communicate the data with.

Client Meeting (Date: 24/11/14)

Today the hardware and the software teams of Qualcomm project had a Skype meeting with the client, but this time we had separate sessions with each team explaining their progresses on the project. The following is our part of the meeting. We started the meeting off by explaining to the client what the team members have been working on over the past weeks. Dingzhong has been working on creating a template for a website for the Proof of Concept part of the project as well as working on the user interface. We briefly talked to the client about the user interface for the project website to monitor the number of people in rooms as well as our approach to colour coding the rooms to give the user the idea of how crowded each room is. Jetnipat has been researching and coding PHP scripts that will allow the data received to be sent to MySQL database as well as the scripts that handle receiving data over a port. Meanwhile, Aleks has been working with Dingzhong on the user interface as well as working more on the simulator. After that, we informed them that due to the time constraint this term, we will not be able to complete the process of having a Java app, which deals with receiving and storing data, running on a server over a virtual machine that we have recently requested to UCL. Thus for now, we will be working with PHP scripts that will work as a replacement instead. We explained to the client that the MySQL will be stored in MS Azure and that the data that we will be receiving from the other team will be sent through SSH socket. The client then suggested that we look at another, possibly, option of sending the data through HTTP instead as the client suggested that it will be easier to work with. The last thing we discussed was how our simulator works. Currently, our simulator simulates only one transition but it is done several times. We then discussed over the fact that our simulator currently is not using SSH socket so the client advised us to use as much code that’s to be used in the real code as possible for the simulator.

Lab Session (Date: 25/11/2014)

Today in the two-hour lab session, we made some physical progresses in the project. We first set up a database of event snapshot on Microsoft Azure. The event snapshot will contain information about people entering or leaving the room. The table that stores the information contains these fields: doorID, event_time, transition, confidence level. DoorID field stores the string that represent the door that someone has walked through. Event Time stores the date time format of when someone walked through the door. Transition tells us which room the person is walking to. For example, either room 1 to room 2 or room 2 to room 1. Confidence level shows how confident we are that someone has walked through the door. We also set up PHP scripts that deals with data insertion into MySQL events_snapshot table as well as a php script that listens to a port (socket) for data.

Jobs done:

Aleksandar:
Researched on ways to implement the simulator on Java (Multi-threading and socket listening).
Made PHP scripts for the Azure server (http connection that adds a row into the MySQL table and display the table).
Set up the server on Azure.

Jetnipat Sarawongsuth:
Researched on PHP script that deals with sending data to MySQL table and script that deals with data retrieval from a port.
Conducted minutes of meetings.
Conducted biweekly report with Aleks.
Worked on the project during the lab session.

Dingzhong Weng:
Working on the Proof of concept website.
Conducted minute of the meeting (17/11/14).

Fifth Bi-weekly report

Progress

During the last two weeks, we made a progress on creating and finalising the group site and worked on the video as a Proof of Concept work for the project. During the first week, we worked on further developing the PHP simulator and successfully deployed it on azure. After migrating from a common Java application to a dynamic web application, we did some researches on servlet and we conducted some tests and examples. We successfully presented the PHP prototype, the simulator and the mockup user interface to the client during the previous two meetings. The mockup models for our system were further developed. We then looked into the issues about the UI and, through heuristic evaluation, were able to identify multiple problems with the UI including inconsistencies in the words used. Rounding up the first term, we should focus on the PoC for term two and separated issues as outline at the end of the report.

Client Meeting (Date: 01/12/14)

This Monday, our group had another weekly meeting with the client. We had a representative from the hardware group with us as well in during the meeting. The meeting started off with us explaining our progress that has been made in the last weeks. We explained how our PHP connection to the MySQL database hosted on Microsoft Azure and what it does to our client. The database contains the event snapshot table that stores the information on the events triggered by the sensors. We then sent the client the link to the HTML page that contains the PHP script that store data into the database as well as displaying it in a table on the HTML webpage and asked for their feedback on it. They were pleased to see the data being sent to the event snapshot database through HTTP and the results of the database displayed neatly in a table. We explained the current difficulties with the DateTime format for the date field in the database. After that, we talked about the process of inputting the data into the database. The representative from the other team briefly talked about how they will generate a HTTP link that has all the data regarding an event and that will be stored into the database with the written PHP script. However, as of now, we clarified with the client that we are not able to use Arduino to send the data into our existing event snapshot database since there are certain difficulties with the different fields in the database and the current Arduino program itself. Finally, we sent the client the general layout of the website for the Proof of Concept part of the project. It’s not yet completed. They will get back to us with an email about the feedback for the website at some point during the week.

Client Meeting (Date: 08/12/14)

Today we had a final meeting with Qualcomm for this term. We told them about the progress made since last week. We showed them the webpages that has got PHP scripts running in the background sending and requesting data. We also showed them the ‘read_simulator’ PHP page. This sends some data acquired from the simulator that was created to the database. The simulator has been improved in that the date and time now will now come from the device. Though right now, it is not currently connected to the Arduino. We are waiting for the hardware team to work on transferring the data from Arduino through HTTP protocols. We later explained to them about a couple of new tables that we decided to create in our database, the first one being the OccupancyLevels table. This table stores the information on how crowded a room is. For example, we might have a label Empty when there is nobody in the room or we can have a label Crowded when there are more than 30 people in a room. We can also have associate colours for each level of crowdedness. Another table is ConfigRoomLimits. This store the information on what action should be done depending on how many people are in the room. For example, we might set a condition in this table for a room to turn the volume of the speakers at a club up if there are more than 20 people in the room between 2AM and 4AM. Finally, we had a discussion over the Proof of Concept website and they were quite pleased with the layout of the page. We will be putting in actual content on the page in the coming days.

Main tasks completed

Task	Contributors
Created main project report	Aleksandar Rusinov, Jetnipat Sarawongsuth
Created Simulators	Aleksandar Rusinov
Recorded minutes of meetings	Dingzhong Weng, Jetnipat Sarawongsuth
Recorded Video	Aleksandar Rusinov, Jetnipat Sarawongsuth, Dingzhong Weng
UI Heuristic Evaluation	Aleksandar Rusinov
Set grounds for prediction engine	Jetnipat Sarawongsuth, Aleksandar Rusinov
Modules Desciption Elaboration	Aleksandar Rusinov, Dingzhong Weng