Architecture Research and Design

Performing the data processing locally will make for a simpler architecture, but it is a worse option in every other regard. Choosing this approach will make the performance of the end product be bottlenecked by the user's computer and browser capabilities. It also limits development options, as we would most likely only be able to code in JavaScript. Lastly, we would end up with an architecture whose computational capability cannot scale beyond the user's browser.

Although choosing to introduce a back-end component to the project on which data processing takes place will make the architecture of our product more complex, the benefits outweigh this minor disadvantage.

Moving data processing to a back-end instead of performing it will overcome all the shortcomings of performing it locally. The end product will have higher performance and an architecture that can scale much more easily. It gives us greater freedom in terms of development, as there are many more choices of platforms for a backend.

There is no use avoiding a backend as one of our "could have" requirements is the option for a notebook style interface which would most likely necessitate some form of backend anyway.

It would be a waste of time re-inventing the wheel when there are readily available open-source libraries that we can integrate into our project.

Apache Spark is the option that is the most scalable and thus would afford the best performance, as Spark being a cluster computing framework is inherently scalable and designed for performance. However, we were advised by our client that it would be overkill for our project and that such a level of performance and scalability is beyond the scope of this project.

We are also quite aware, through our investigation on Zeppelin, that Apache Spark does not have much built-in functionality and is actually just a bare-bones distributed computing framework. In fact, choosing to use Spark is no different from choosing to implement everything ourselves, except an added bonus of running on top of a scalable platform.

Our research revealed that there is actually not much difference in the functionality provided by Pandas and R + its built-in libraries. In fact, R actually has an upper hand in terms of existing functionality over Pandas, because the whole language was created for statistics and data science.

However, what made us choose not to use R is because none of us are familiar with the language, so development would obviously be set back as we spend time familiarizing ourselves with the language and the libraries.

Pandas, being a data science library for Python, has a lot of built-in functionality (comparable to R) that we can use to very easily fulfill some of our basic functional requirements. However, unlike R, it has an added advantage of being a Python library, a language all of us know.

Our client also requested that we use Pandas because their machine learning platform uses Pandas and Python as feature extraction pipelines, and it would be ideal if our solution was compatible with Seldon's platform.

Finally, the fact that Pandas is a Python package makes it more suitable for use with a web backend. Python has many options for web application frameworks such as Django and Flask, and it would be more convenient for development if the web application and the data processing logic were both written in the same language.

As we have chosen to use Pandas as a data processing library, which is a Python package, it does not make sense to use a non Python based web development framework.

Django is actually better than Flask in some ways, as it is a complete framework that enforces and encourages proper code structure with the MVC paradigm and provides many features that Flask does not, such as an object-oriented wrapper over databases and built-in administration tools. However, none of this additional functionality will be necessary for our backend, making Django a bloated framework that gets in the way of performance in the context of our project.

In contrast to the feature-completeness of Django, Flask advertises itself as a micro web application framework, only providing bare essentials such as routing. This makes Flask extremely lightweight and since our backend only needs the functionality Flask already provides, it is the perfect choice.

Backbone.js does not compare to Ember.js or Angular.js in terms of functionality. For example, it lacks routing, dynamic data binding and a build int templating engine. Its popularity has dwindled since its release due to the growth in popularity of more feature-filled frameworks such as Angular.js and Ember.js.

React.js is a new library that is gaining popularity among developers, but it is not a complete MVC framework like Angular.js or Ember.js, only providing the V/'view' aspect of the paradigm. Therefore, using React.js would require us spend additional time defining the MVC structure ourselves, or end up with a poorly structured app.

Ember.js, along with Angular.js, were the only suitable choices for a front-end framework for our project. They are nearly identicla with regard to feature-set, but what made us choose Angular.js over Ember.js is that some of us have had experience using the former, so choosing Angular.js should streamline the development process. Furthermore, Angular.js is the more popular framework between the two and has a much larger user base in the developer community, meaning that it would be easier for us to get support if we run into problems.

It would be a waste of time re-inventing the wheel when there are readily available open-source libraries that we can integrate into our project. Too much development time will be wasted dealing with the complexity of implementing a secure and robust Python execution engine as well as communication protocol.

Jupyter + iPython kernel is the de-facto standard notebook software for Python. It is fair to say that there are no competing solutions worth considering, and the fact that it is also included in Seldon's machine learning platform makes it a great choice.

This was a tough decision to make as the REST architecture has become so prevalent among Web services that it is the de facto standard. Providing a RESTful service would make it easy for advanced users and developers to create their own custom frontends to communicate with our backend.

However, one of the architectural constraints of REST is that the web service does not store the state of the client, and the nature of our project forces us to break that constraint. For our backend to be efficient, it must store the i.e. data of the client between requests, instead of forcing the client to comply with the stateless constraint and send its data with every request. d

A RESTful API is also based around HTTP, and HTTP does not allow servers to push data to clients. Our architecture must handle the situation where a client requests a data processing job that takes a long period of time, and it does not make sense to use a RESTful API, as requests will hang and might even timeout before the backend completes the data processing task.

Our previous assessment concluded that we ultimately need a protocol that supports pushing of updates to the client and does not enforce the constraint of statelessness, which is an inappropriate constraint for our application.

Server push will most likely be a critical capability to enable synchronization of changes made using the notebook interface to the graphical/spreadsheet interface. It would also be required for the live collaboration functionality, although that is a "would have" requirement.

A WebSocket API would fulfill all these requirements.

It would be a waste of time re-inventing the wheel when there are readily available open-source libraries that we can integrate into our project.

SocketIO seems to be the only WebSocket library that has an implementation for Flask, our backend. Team members have prior experience using SocketIO. Lastly, SocketIO is backwards compatible with browsers that don't support the WebSocket protocol introduced in HTML5, so using it as a library also adds a layer of robustness and usability to our application.

Although this option would make for a simpler architecture, it is definitely not good practice for the web app to be directly calling intensive data cleaning functions. It is also an approach that does not scale and performs poorly when the backend is under heavy load. Even in a production setting when Flask is deployed with a WSGI server with multiple server processes, a single blocking data processing operation would render an entire worker process unresponsive, and if enough users cause this scenario, the entire backend would become unresponsive.

This approach will provide scalability + robustness in the end product, overcoming the shortcomings of running the tasks synchronously within Flask. It should also boost performance as multiple cores and even multiple CPUs can be utilized to perform tasks concurrently.

Running the tasks outside Flask asynchronously also enables the infrastructure to support features such as canceling operations before they complete.

Although this looked promising initially, testing with Flask revealed that a web app which spawns new child processes and/or uses multiple threads does not work behind a server due to restrictions in WSGI. The application only behaves as expected when run in development mode as a single threaded single process server, but not when behind any WSGI server such as Gunicorn. The idea of using multiple threads or spawning child processes within a Flask app is strongly discouraged by the developer community. Furthermore the Socket.IO library we have chosen to use with Flask does not support multithreaded web apps.

This approach does adhere to best practices unlike using multithreading/multiprocessing, but is quite complicated to implement. With this approach, Flask would spawn a new service that performs all the data processing with every user that connects to the backend, and then communicate with that service using some form of IPC in the server OS such as a socket. There are libraries that could help with messaging such as ZeroMQ, but this approach does not compare to the simplicity of using an existing task manager solution i.e. Celery.

After looking through developer forums, we saw that Celery is consistently the recommended solution for asynchronous background tasks in Python. Celery handles all aspects of background tasks, such as creating, managing and communicating with worker processes.

Using Celery would allow us to queue all data processing tasks and not worry about the implementation of asynchronous background tasks. Its task management features will also simplify the development of features such as cancelling operations.

In a production environment, the Flask web app will be served by a WSGI server that uses multiple different server worker processes. That means it is not guaranteed that the user’s frontend gets served by the same server process. Every server process has an independent memory stack, so storing the user data in the memory of the Flask web app as some sort of global variable, would not be a working solution.

The only other option is to cache the user's data on disk in a HDF file format (our research revealed that the HDF file format provides the fastest read and write speeds with Pandas data objects).