Project Management
This page details all aspects regarding the management of this project, including the tools we used and the list of issues in sprints. Click the buttons below to jump to the corresponding section.
1. Software and Tools 2. Work Packages 3. Cumulative Flow 4. Issues Archive
Software and Tools
JIRA
We were following the Agile development methodology for this project and as part of this approach, we did weekly sprints. The team and the client will meet every week to discuss the progress on the tasks/issues in the previous sprint and plan issues to do for the coming sprint. To make these issues easier to manage, we used JIRA, a software development tool. JIRA allows us to plan and track issues by putting them into sprints and the team can mark issues as being in progress or done. JIRA also creates reports with diagrams such as burndown chart and cumulative flow diagram to assist the team to access the overall progress of the project.
The issues archive and cumulative flow diagram sections below are generated from our JIRA board.
Slack
To enable the team to communicate with the client effectively, we used Slack as our main platform for sharing quick messages. The Slack channel for this project is connected to the other services we were using, including JIRA and Google Drive, so we would be notified when a task status is updated or a file is uploaded to the shared folder.
Work Packages Completed
Shivam Dhall - Group Manager & Chief Researcher
As the Group Manager, Shivam was responsible for overlooking the progress of the project, distribute work among team members, and assist team members when necessary. Shivam also led the research aspect of the project, this included researching on existing systems and libraries that can be used during development. Additionally, Shivam was involved with the development of some features of the system. Shivam was also the testing lead and was responsible for performing unit tests and integration tests during the development phase of the project.
- Background research
- Implement file uploading in JSON, XLS, XSLX format with partial loading functionality
- Implement file downloading/saving functionalities
- Implement text data analysis functionality
- Implement text data visualization (word frequencies) functionality
- Unit tests for backend module
- Manual functional tests
Bandi Enkh-Amgalan - Technical Lead & Client Liaison
Bandi was the technical lead of the team and was responsible for overlooking the technical aspect of the system in development. Bandi researched and set up the architecture for the system, and implemented many of the features on both the frontend and the backend. Bandi was also the Client Liaison of the team and was responsible for communicating with the client on the progress of the project as well as setting up regular meetings with our client.
- Research and design architecture
- Set-up infrastructure
- Implement data viewing features (Search, Filter, Sort)
- Implement data analysis functionality
- Implement data visualization functionality
- Set-up deployment
- Edit technical documentation
Gordon Cheng - UI Lead & Chief Editor
Gordon was the User Interface Lead of the team and was responsible for designing and implementing the web app’s UI, making sure that it is both functional and aesthetically pleasing. Gordon was also the Chief Editor and overlooked the various documentation generated by the project. This included designing and developing the project website as well as editing documentation such as the user guide and the project videos. Gordon was also responsible for developing some features of the system, including many of the data cleaning functionalities.
- Design and develop project website
- Implement file uploading in CSV format with support for partial uploading
- Implement data cleaning features (Missing data, Find & Replace, Feature scaling etc.)
- Design and implement UI for data analysis feature
- Design and implement UI for data visualization feature
- Produce project videos
- Edit user guide
Cumulative Flow Diagram
This diagram shows the statuses of issues over the first and second term.
This diagram shows the statuses of issues over the first term.
Issues Archive
| Issue Type | Key | Summary | Status | Resolution | Created | Updated |
| Story | UCL-80 | Test all features, uncover & fix bugs | In Progress | Unresolved | 20/3/2016 12:49 | 20/3/2016 12:50 |
| Story | UCL-79 | Fix missing analysis bug (zero valid values) | To Do | Unresolved | 4/3/2016 15:23 | 20/3/2016 12:49 |
| Story | UCL-78 | Fix rename & change data type of column bug (name becomes blank) | Done | Done | 4/3/2016 15:23 | 20/3/2016 12:50 |
| Story | UCL-77 | Undo | Done | Done | 3/3/2016 17:06 | 20/3/2016 12:48 |
| Story | UCL-76 | Improve delete rows | In Progress | Unresolved | 3/3/2016 17:03 | 20/3/2016 12:49 |
| Story | UCL-75 | Duplicate column | Done | Done | 3/3/2016 17:01 | 7/3/2016 12:10 |
| Story | UCL-74 | Import JSON | Done | Done | 3/3/2016 17:01 | 8/3/2016 10:00 |
| Story | UCL-73 | Fix deployment issues with Windows | Done | Done | 1/3/2016 12:09 | 8/3/2016 13:34 |
| Story | UCL-72 | Save generated charts as image | Done | Done | 1/3/2016 12:08 | 8/3/2016 10:00 |
| Story | UCL-71 | Search (+ with regex) | Done | Done | 1/3/2016 12:07 | 20/3/2016 12:48 |
| Story | UCL-70 | Edit individual cells | Done | Done | 1/3/2016 12:06 | 12/3/2016 12:36 |
| Story | UCL-69 | Export dataframe as JSON, CSV | Done | Done | 1/3/2016 12:04 | 1/3/2016 15:09 |
| Story | UCL-68 | Find & show outliers for numeric columns | Done | Done | 2/2/2016 16:45 | 6/3/2016 18:22 |
| Story | UCL-67 | Find & view duplicated rows | Done | Done | 2/2/2016 16:43 | 6/3/2016 18:22 |
| Story | UCL-66 | Find & replace feature with regular expressions | Done | Done | 28/1/2016 20:41 | 6/2/2016 22:04 |
| Story | UCL-65 | sorting of rows by column | Done | Done | 28/1/2016 20:41 | 20/3/2016 12:48 |
| Story | UCL-64 | Date variable type: implement conversion to/fro & integration with existing clean, visualize and analyze features | Done | Done | 28/1/2016 20:40 | 5/2/2016 23:58 |
| Story | UCL-63 | refactor CleanController: move cards in toolbar to individual Angular directives, use pubsub to communicate changes in state | Done | Done | 22/1/2016 23:32 | 28/1/2016 20:42 |
| Epic | UCL-61 | Data Analysis | To Do | Unresolved | 8/12/2015 14:49 | 8/12/2015 14:49 |
| Story | UCL-60 | show text analysis such as most frequent word for string type data | Done | Done | 8/12/2015 14:47 | 30/1/2016 22:49 |
| Story | UCL-59 | data analysis must haves (mean, median, range, stddev etc) | Done | Done | 8/12/2015 14:45 | 30/1/2016 21:28 |
| Story | UCL-58 | show the unique values and their count of every column of the dataset | Done | Done | 8/12/2015 14:44 | 27/2/2016 18:20 |
| Story | UCL-57 | UI for graph screen | Done | Done | 8/12/2015 14:39 | 20/1/2016 14:09 |
| Epic | UCL-56 | Data Vlsualisation - Must | To Do | Unresolved | 8/12/2015 14:25 | 8/12/2015 14:43 |
| Story | UCL-54 | visualise data using graphs (should have): scatter plots, time-series plots, pie charts | Done | Done | 8/12/2015 14:23 | 1/3/2016 1:42 |
| Story | UCL-51 | visualise data using graphs | Done | Done | 8/12/2015 14:23 | 28/1/2016 20:38 |
| Epic | UCL-50 | Clean the data | To Do | Unresolved | 8/12/2015 14:22 | 8/12/2015 14:25 |
| Story | UCL-49 | support cleaning of missing values by interpolation | Done | Done | 8/12/2015 14:22 | 28/1/2016 20:43 |
| Story | UCL-48 | support cleaning of missing values by filling with the most recent value | Done | Done | 8/12/2015 14:21 | 22/1/2016 14:56 |
| Story | UCL-47 | support cleaning of missing values by inserting an average | Done | Done | 8/12/2015 14:21 | 24/1/2016 20:36 |
| Story | UCL-46 | show rows with missing values in specified column | Done | Done | 8/12/2015 14:21 | 26/1/2016 17:48 |
| Story | UCL-45 | show rows with invalid numbers in specified column | Done | Done | 8/12/2015 14:21 | 26/1/2016 17:48 |
| Story | UCL-44 | inserting user-specified values as a universal cleaning operation | Done | Done | 8/12/2015 14:21 | 24/1/2016 20:04 |
| Story | UCL-43 | removing rows as a universal cleaning operation | Done | Done | 8/12/2015 14:21 | 22/1/2016 1:36 |
| Story | UCL-42 | User can specify encoding and other options on upload screen | Done | Done | 7/12/2015 22:53 | 11/1/2016 17:16 |
| Story | UCL-41 | Project Video | Done | Done | 1/12/2015 14:43 | 11/12/2015 20:47 |
| Story | UCL-40 | Export data in JSON or CSV | Done | Done | 1/12/2015 14:39 | 11/1/2016 17:16 |
| Story | UCL-39 | First end-to-end data cleaning use case | Done | Done | 1/12/2015 14:36 | 28/1/2016 20:38 |
| Story | UCL-38 | Explore and select UI framework | Done | Done | 1/12/2015 14:33 | 5/12/2015 19:07 |
| Story | UCL-37 | Create initial UI for testing | Done | Done | 1/12/2015 14:33 | 7/12/2015 10:54 |
| Story | UCL-36 | The DCS shall allow users to specify variable names and types. | Done | Done | 1/12/2015 14:30 | 11/1/2016 17:16 |
| Story | UCL-35 | First Github commit | Done | Done | 24/11/2015 14:50 | 29/11/2015 21:39 |
| Story | UCL-34 | Deploy skeleton into a VM | Done | Done | 24/11/2015 14:48 | 29/11/2015 21:39 |
| Story | UCL-33 | Build skeleton infrastructure | Done | Done | 24/11/2015 14:47 | 26/11/2015 12:52 |
| Story | UCL-32 | Investigate Flask | Done | Done | 24/11/2015 14:45 | 26/11/2015 12:52 |
| Story | UCL-31 | Review Seldon 0.99 Python Pipelines | Done | Done | 17/11/2015 15:17 | 20/1/2016 14:25 |
| Story | UCL-30 | Map out the Python module for backend features | Done | Done | 17/11/2015 15:13 | 4/12/2015 17:46 |
| Story | UCL-29 | Investigate graph libraries to minimise front-end UI build | Done | Done | 17/11/2015 15:06 | 30/11/2015 12:31 |
| Story | UCL-28 | Outline the architecture | Done | Done | 17/11/2015 14:51 | 24/11/2015 14:35 |
| Story | UCL-27 | Investigate Pandas | Done | Done | 17/11/2015 14:51 | 24/11/2015 14:35 |
| Sub-task | UCL-26 | UCL-15 Ask TA/Dean about possibility of UCL hosted dev server | Done | Done | 30/10/2015 14:58 | 17/11/2015 14:47 |
| Sub-task | UCL-25 | UCL-19 Prepare Interview Questions | Done | Done | 30/10/2015 14:55 | 17/11/2015 14:47 |
| Story | UCL-24 | Remove references to other products in features document | Done | Done | 30/10/2015 14:49 | 30/10/2015 14:55 |
| Story | UCL-23 | Bi-weekly Report | Done | Done | 29/10/2015 10:46 | 1/11/2015 15:51 |
| Story | UCL-22 | Trying some of the techniques tools on the sample dataset [AZURE] | Done | Done | 27/10/2015 15:03 | 17/11/2015 14:48 |
| Story | UCL-21 | Trying some of the techniques tools on the sample dataset [ZEPPLIN] | Done | Done | 27/10/2015 15:03 | 17/11/2015 14:48 |
| Story | UCL-20 | Trying some of the techniques tools on the sample dataset [JASP] | Done | Done | 27/10/2015 15:00 | 12/11/2015 20:03 |
| Story | UCL-19 | Interview the company about potential features and data cleaning process | Done | Done | 27/10/2015 14:58 | 17/11/2015 14:47 |
| Story | UCL-18 | Send the questionnaire and collate results | Done | Done | 27/10/2015 14:57 | 28/1/2016 20:38 |
| Story | UCL-17 | Github setup and sandbox | Done | Done | 20/10/2015 16:49 | 27/10/2015 7:46 |
| Story | UCL-16 | Create preliminary content and load onto project website for university | Done | Done | 20/10/2015 16:38 | 26/10/2015 17:40 |
| Story | UCL-15 | Setting up a dev server for collaboration and demos | Done | Done | 20/10/2015 16:34 | 17/11/2015 14:47 |
| Story | UCL-14 | Wireframes of potential UI/UX | Done | Done | 20/10/2015 16:32 | 18/11/2015 15:33 |
| Story | UCL-13 | Look at outcomes of the Springleaf competition | Done | Done | 20/10/2015 16:30 | 27/10/2015 14:54 |
| Story | UCL-12 | Find another cleaner dataset for testing purposes | Done | Done | 20/10/2015 16:29 | 26/10/2015 22:12 |
| Story | UCL-11 | Document the options / choices to be made for the final solution | Done | Done | 20/10/2015 16:28 | 22/10/2015 11:33 |
| Story | UCL-10 | Bi-Weekly Report | Done | Done | 13/10/2015 21:01 | 20/10/2015 15:58 |
| Story | UCL-9 | Requirements questionnaire for Seldon users | Done | Done | 13/10/2015 16:48 | 23/10/2015 17:13 |
| Story | UCL-8 | Initial data analysis tests in Zeppelin (or another tool) | Done | Done | 13/10/2015 16:47 | 20/10/2015 16:26 |
| Story | UCL-7 | Initial data analysis tests in JASP | Done | Done | 13/10/2015 16:43 | 20/10/2015 15:57 |
| Story | UCL-6 | Research data cleaning methods | Done | Done | 13/10/2015 16:36 | 20/10/2015 16:26 |
| Story | UCL-5 | Research data analysis methods | Done | Done | 13/10/2015 16:34 | 19/10/2015 13:50 |
| Story | UCL-4 | Research Kaggle knowledge sources | Done | Done | 13/10/2015 16:29 | 17/11/2015 14:48 |
| Story | UCL-3 | Investigate Amazon ML ETL features | Done | Done | 13/10/2015 16:28 | 15/10/2015 11:03 |
| Story | UCL-2 | Investigate Azure ML ETL features | Done | Done | 13/10/2015 16:27 | 15/10/2015 11:03 |
| Story | UCL-1 | Download kaggle data | Done | Done | 13/10/2015 8:40 | 13/10/2015 17:35 |
Term 1 Archive
| Key | Summary | Created | Resolved | Story Points | Sprint |
| UCL-38 | Explore and select UI framework | 1/12/2015 14:33 | 1 | Sprint 7 | |
| UCL-39 | First end-to-end data cleaning use case | 1/12/2015 14:36 | 2 | ||
| UCL-41 | Project Video | 1/12/2015 14:43 | 3 | Sprint 7 | |
| UCL-40 | Export data in JSON or CSV | 1/12/2015 14:39 | 2 | ||
| UCL-31 | Review Seldon 0.99 Python Pipelines | 17/11/2015 15:17 | 2 | Sprint 5, Sprint 6, Sprint 7 | |
| UCL-37 | Create initial UI for testing | 1/12/2015 14:33 | 3 | Sprint 7 | |
| UCL-36 | The DCS shall allow users to specify variable names and types. | 1/12/2015 14:30 | 3 | Sprint 7 | |
| UCL-30 | Map out the Python module for backend features | 17/11/2015 15:13 | 3 | Sprint 5, Sprint 6, Sprint 7 | |
| UCL-18 | Send the questionnaire and collate results | 27/10/2015 14:57 | 1 | Sprint 3, Sprint 4, Sprint 5, Sprint 6, Sprint 7 | |
| UCL-29 | Investigate graph libraries to minimise front-end UI build | 17/11/2015 15:06 | 30/11/2015 12:31 | 2 | Sprint 5, Sprint 6 |
| UCL-34 | Deploy skeleton into a VM | 24/11/2015 14:48 | 29/11/2015 21:39 | 3 | Sprint 6 |
| UCL-35 | First Github commit | 24/11/2015 14:50 | 29/11/2015 21:39 | 1 | Sprint 6 |
| UCL-33 | Build skeleton infrastructure | 24/11/2015 14:47 | 26/11/2015 12:52 | 2 | Sprint 6 |
| UCL-32 | Investigate Flask | 24/11/2015 14:45 | 26/11/2015 12:52 | 1 | Sprint 6 |
| UCL-28 | Outline the architecture | 17/11/2015 14:51 | 24/11/2015 14:35 | 3 | Sprint 5 |
| UCL-27 | Investigate Pandas | 17/11/2015 14:51 | 24/11/2015 14:35 | 2 | Sprint 5 |
| UCL-14 | Wireframes of potential UI/UX | 20/10/2015 16:32 | 18/11/2015 15:33 | 3 | Sprint 5 |
| UCL-21 | Trying some of the techniques tools on the sample dataset | 27/10/2015 15:03 | 17/11/2015 14:48 | 2 | Sprint 3, Sprint 4 |
| UCL-22 | Trying some of the techniques tools on the sample dataset [AZURE] | 27/10/2015 15:03 | 17/11/2015 14:48 | 2 | Sprint 3, Sprint 4 |
| UCL-4 | Research Kaggle knowledge sources | 13/10/2015 16:29 | 17/11/2015 14:48 | 2 | Sprint 1, Sprint 2, Sprint 3, Sprint 4 |
| UCL-15 | Setting up a dev server for collaboration and demos | 20/10/2015 16:34 | 17/11/2015 14:47 | 3 | Sprint 3, Sprint 4 |
| UCL-26 | UCL-15 Ask TA/Dean about possibility of UCL hosted dev server | 30/10/2015 14:58 | 17/11/2015 14:47 | Sprint 3, Sprint 4 | |
| UCL-19 | Interview the company about potential features and data cleaning process | 27/10/2015 14:58 | 17/11/2015 14:47 | 2 | Sprint 3, Sprint 4 |
| UCL-25 | UCL-19 Prepare Interview Questions | 30/10/2015 14:55 | 17/11/2015 14:47 | Sprint 3, Sprint 4 | |
| UCL-20 | Trying some of the techniques tools on the sample dataset | 27/10/2015 15:00 | 12/11/2015 20:03 | 2 | Sprint 3, Sprint 4 |
| UCL-23 | Bi-weekly Report | 29/10/2015 10:46 | 1/11/2015 15:51 | 2 | Sprint 3 |
| UCL-24 | Remove references to other products in features document | 30/10/2015 14:49 | 30/10/2015 14:55 | 1 | Sprint 3 |
| UCL-13 | Look at outcomes of the Springleaf competition | 20/10/2015 16:30 | 27/10/2015 14:54 | 2 | Sprint 2 |
| UCL-17 | Github setup and sandbox | 20/10/2015 16:49 | 22/10/2015 11:50 | 2 | Sprint 2 |
| UCL-12 | Find another cleaner dataset for testing purposes | 20/10/2015 16:29 | 26/10/2015 22:12 | 2 | Sprint 2 |
| UCL-16 | Create preliminary content and load onto project website for university | 20/10/2015 16:38 | 26/10/2015 17:40 | 3 | Sprint 2 |
| UCL-9 | Requirements questionnaire for Seldon users | 13/10/2015 16:48 | 22/10/2015 18:04 | 2 | Sprint 1, Sprint 2 |
| UCL-11 | Document the options / choices to be made for the final solution | 20/10/2015 16:28 | 22/10/2015 11:33 | 2 | Sprint 2 |
| UCL-6 | Research data cleaning methods | 13/10/2015 16:36 | 20/10/2015 16:26 | 2 | Sprint 1 |
| UCL-8 | Initial data analysis tests in Zeppelin (or another tool) | 13/10/2015 16:47 | 20/10/2015 16:26 | 2 | Sprint 1 |
| UCL-10 | Bi-Weekly Report | 13/10/2015 21:01 | 14/10/2015 16:41 | 2 | Sprint 1 |
| UCL-7 | Initial data analysis tests in JASP | 13/10/2015 16:43 | 20/10/2015 15:56 | 2 | Sprint 1 |
| UCL-5 | Research data analysis methods | 13/10/2015 16:34 | 19/10/2015 13:50 | 2 | Sprint 1 |
| UCL-2 | Investigate Azure ML ETL features | 13/10/2015 16:27 | 15/10/2015 11:03 | 1 | Sprint 1 |
| UCL-3 | Investigate Amazon ML ETL features | 13/10/2015 16:28 | 15/10/2015 11:03 | 1 | Sprint 1 |
| UCL-1 | Download kaggle data | 13/10/2015 8:40 | 13/10/2015 17:21 | 1 | Sprint 1 |