For this project, the primary programming languages employed include Python, particularly Django for the back end functionality, JavaScript in conjunction with the React framework for the front end development, and HTML/CSS/JS for crafting this documentation website (you're reading!).

2.1 File Structure

At the top level of the Django project, we have the chroma, scraper, server, and generation directories, each representing a specific functionality or domain within the application. This modular approach allows for better organization, maintainability, and scalability of the codebase. Within each module, we can observe a consistent structure. The endpoints.py and urls.py files handle the API endpoints and URL routing, respectively, while the modules directory contains the core logic and functionality of the module.

The tests directory within each module houses the corresponding unit tests, ensuring that the individual components of the application are thoroughly tested and validated.

Leveraging the Google's (Custom Search Engine) API to get the top results of a search, we employ a scraper to extract the web content from these results. The scraper is equipped with the capability to handle various types of web content, including PDFs, HTML pages, and plaintext documents.

Once the relevant data is obtained online, the lengthy textual information undergoes segmentation into shorter, more manageable snippets to facilitate subsequent processing. Employing the Recursive Character Text Splitter from the Langchain library, this division process is executed with precision and efficiency. It's important to note that PDF contents and plaintext contents are handled distinctively to ensure optimal splitting. Moreover, pertinent metadata such as page numbers are meticulously retained for future reference and contextual understanding.

Utilising the segmented text chunks obtained from the previous stage, we proceed to upload them into the Chroma Vector Database, accompanied by various metadata attributes. These metadata elements encompass the document's title, page number, actual and base URLs. Notably, the base URL is specifically logged to streamline potential future actions, enabling swift disqualification of entire web sources if necessary.

Chroma was chosen for its robust search capabilities, enabling efficient retrieval of relevant documents based on user queries. Additionally, its local hosting ensures that the database is readily accessible and responsive, enhancing the overall user experience.

Once the database is established, upon receiving a user query, we initiate a search within the database to retrieve the ten most pertinent documents. Subsequently, we format these documents for seamless integration into subsequent processing stages.

In the final phase of the pipeline, we feed the selected texts into an OpenAI completion model tailored to specific templates aligned with user requirements. This model serves to enhance the coherence and alignment of the content with the designated template. By leveraging the completion model, we ensure that the contents are logically structured and seamlessly fit into the intended template, thereby enriching the overall user experience.

3.1 Frontend File Structure

1. api: The directory that contain the api calls to the backend API
2. assets: Dedicated to storing the application's assets like images
3. components: Contains the reusable components used throughout the application
4. context: Contains the context providers, which manage application state and provides a centralized ways to share data across components.
5. languages: Responsible for the internatiolization (i18n) of the application, housing the (English, Spanish and French) translation files
6. pages: Contains the main pages of the application
7. styles: Contains the component specific stylings
8. utils: Contains functionality modules such as firebase

Upon initial entry to our website, users are greeted with a captivating set of flip-page images showcasing real situations on the field. Below this engaging visual display, visitors will discover an overview of our main tools and technologies. As they scroll further, a curated collection of articles delving into the research behind our project awaits them. These resources are thoughtfully provided to cater to users seeking a deeper understanding of the principles underpinning our endeavor.

To ensure inclusivity and accessibility for individuals from diverse linguistic backgrounds, our website offers seamless language translation capabilities. Users can easily switch between languages, including popular options such as French and Spanish, enabling a broader audience to navigate and engage with our content effectively.

To meet our clients' requirements and ensure the seamless storage and presentation of historical conversations, we have implemented a login page. Leveraging Firebase, a robust authentication system is established, allowing users to log in securely using their Google or Microsoft credentials.

By integrating Firebase authentication, we not only enhance the security of user accounts but also streamline the login process, providing a seamless experience for our users. This ensures that historical conversations are accurately stored and presented, maintaining the integrity and continuity of user interactions.

We also enabled multiple login options inlcuding conventional email, Google and Microsoft login options. This flexibility caters to a wider audience, ensuring that users can access our platform using their preferred authentication method.

Following successful login, users gain access to the chatbox functionality, enabling them to input their queries. The chatbot processes these queries, generating responses within approximately one minute. Each response is accompanied by corresponding sources for further reference, fostering transparency and credibility.

Should users desire additional information or clarification on the points provided, they have the option to pose follow-up questions. This interactive feature ensures that users can delve deeper into specific topics of interest.

Adjacent to the chatbox, on the left side of the page, resides a dedicated history section. Here, users can conveniently access and search past questions, facilitating easy retrieval of previous interactions for reference. This feature enhances user experience by promoting continuity and accessibility across multiple sessions.

A dedicated data management page has been meticulously designed to empower logged-in users with the ability to oversee and manipulate approved data sources utilized within the chatbot system. Users are granted the authority to add or delete specific data sources and to modify the status of existing sources, which includes designations such as "approved," "under review," or "rejected."

During the process of adding new sources, a robust alert mechanism is implemented. In the event that a new link or organization name already exists within the system, users are promptly notified through an alert. This preventive measure ensures that duplicate entries are avoided, safeguarding the integrity and efficiency of the data management process.