Monolithic vs Microservices [4]

- Monolithic Approach: Integrates all functionalities into a single codebase, simplifying deployment but potentially complicating scalability and maintenance.

- Microservices Approach: Breaks down functionalities into independent services, enhancing scalability and fault isolation but introducing complexity in communication and deployment.

Choice and Justification: We opted for a microservices architecture. We want our project to be highly extensible, so Cisco’s engineers can add new functionality with ease. Our aim is to make each component of our project work independently, without much reliance on each other. This would allow for independent testing of our software, and easy integration of new tooling.

Device Comparison [5]

As our tools are designed to be integrated into Cisco's existing web-based platforms, the primary "device" consideration pertains to server environments:

- On-Premises Servers: Offer control over hardware and security but require significant maintenance and upfront costs.

- Cloud-Based Servers: Provide scalability and reduced maintenance but may raise concerns regarding data sovereignty and recurring costs.

Choice and Justification: We chose to develop our solution for on-premises deployment within Cisco's infrastructure. This ensures compliance with internal security protocols and leverages existing hardware, aligning with Cisco's operational preferences.​

Algorithm Comparison [6]

In developing our log analysis tools, we prioritized efficient string-searching mechanisms to enhance performance during log parsing and filtering operations. We evaluated various algorithms to identify the most suitable approaches:

- Naive String Search: Compares the pattern to every possible substring in the text sequentially. Exhibits a time complexity of O(mn), where m is the length of the pattern and n is the length of the text, leading to inefficiencies, especially with large datasets.

- Boyer–Moore–Horspool (BMH) Algorithm: A simplification of the Boyer–Moore algorithm, BMH preprocesses the pattern to create a shift table, allowing the algorithm to skip sections of the text, thereby reducing the number of comparisons. Offers an average-case complexity of O(n) and is particularly efficient for longer patterns.

Choice and Justification: For Plain Text Searches we utilized JavaScript's .includes() method, which is optimized for substring searches. While the exact implementation details of .includes() can vary across JavaScript engines, it is generally optimized for performance, often employing the Boyer–Moore algorithm [7]. For Regular Expression Searches we leveraged JavaScript's native RegExp [8] capabilities, which are highly optimized for pattern matching tasks, generally employing an NFA (Non-deterministic Finite Automata) building and NFA matching algorithm.

Front-End Library & Language Comparison

Adhering to Cisco's guidelines and aiming for seamless integration, we evaluated the following technologies:​

JavaScript & jQuery: Vanilla JS does not require a build system or additional dependencies. It is ideal for small projects like this as it does not involve complex state management like user authentication. However, its simplicity comes at the cost of not having reusable components, which can lead to slow development.

React: React is a component based JS library. One core feature is that updates to the components only re-render the changed parts of the UI rather than the entire page leading to improved responsiveness. However, React evolves frequently, and due to build processes, introduces a lot of new dependencies into the project. This may cause issues in compatibility with Cisco’s existing codebase.

Choice and Justification: We utilized jQuery, vanilla JavaScript, Sass, and Bootstrap. This combination ensures compatibility with Cisco's existing systems, minimizes additional dependencies, and provides a balance between functionality and performance.​

NoSQL vs SQL Comparison

Our first consideration for the backend design was between a SQL database and a NoSQL database.

SQL: A SQL database works well for highly structured data. Due to the predefined schema of SQL databases, they are easy to maintain, and queries can be highly optimised based on the schema due to features like column indexing. Moreover, SQL databases maintain the ACID properties, which ensures that the data always remains in a consistent state, and transactions are executed in their entirety or none at all.

NoSQL: A NoSQL database on the other hand offers a lot more flexibility than a SQL database. The data is unstructured/semi-structured. While they may not have the powerful query capabilities of SQL databases, NoSQL has lower latency for simple, repetitive queries. NoSQL databases also have high horizontal scalability, making it easier to distribute data across multiple servers or databases.

Choice and Justification: We opted to go with a NoSQL database as our project is focused on data analytics. By nature, this tool has a high read rate, and the queries that are executed are simple but frequent. We felt the performance of a NoSQL database in fetch queries is essential as the website must be responsive. Furthermore, our platform should work for all log files regardless of their format, and this is where NoSQL databases shine as a predefined schema is not needed. Finally, should the scale of data grow extensively, NoSQL databases tend to automatically manage horizontal scaling, whereas this can be difficult to implement in SQL databases.

Database Options

Since we have decided to go with a NoSQL database, we considered several NoSQL options.

Google Firebase: This is a cloud hosted NoSQL database. It offers several built in tools such as real time notifications to the users, and easy authentication. Firebase offers real time databases, which has enhanced fetch speeds than other NoSQL databases, thus ensuring high performance for the log-viewer. However, since this is cloud hosted, and this log viewer app will be handling internal data, this option was immediately ruled out. Furthermore, Firebase is predominantly used for small scale web apps, where the size of the data is limited. The 50GB storage requirement further drives Firebase away from our goals for the project.

ElasticSearch: This is a RESTful search engine used to store and retrieve data in nearly real time. ElasticSearch holds data in JSON format which is the primary format of the log files. Furthermore, ElasticSearch is highly scalable - it automatically creates and distributes the workload across multiple nodes as data size grows. However, ElasticSearch is resource intensive due to the vast range of functionalities it offers, which can result in high costs.

MongoDB: MongoDB excels in storing structured, semi-structured, and unstructured data in flexible JSON-like documents. Similar to ElasticSearch, MongoDB automatically shards large documents to multiple servers. However, MongoDB does have memory limitations, where each document can only take up a maximum of 16MB.

Choice and Justification: We decided to go with ElasticSearch database because it is primarily used for log file and data analytics. In summary, our technology choices were guided by the principles of compatibility, performance, and minimalism. Since ElasticSearch is already used within Cisco we aligned with Cisco's existing infrastructure.

Backend Framwork Comparison

Django: Django offers a very comprehensive and extensive backend framework. Django has a higher versatility than other python backend frameworks as it allows you to build websites using HTML templates or build a JSON API. Django has a strong emphasis on security.

FastAPI: FastAPI is a more limited framework that allows users to quickly create an internal endpoint. It has the highest performance due to async processing.

Flask: Flask offers a middle ground between the 2. It is used heavily for building small scale applications like the log-viewer app, and has moderate response times.

Choice and Justification: We selected FastAPI due to its high performance, ease of setup, and asynchronous capabilities, which are advantageous for handling concurrent log processing tasks. FastAPI's design facilitates rapid development and integrates seamlessly with Python's rich ecosystem of AI and data processing libraries.