2.2 Preprocessing

To enhance the performance and capabilites of our chatbot, we build upon the RAG model by incorporating additional techniques. Specifically, we implemented documnet chunking mechanism that divides the source materials (e.g. websites, PDF files) into smaller, semantically coherent units. This chunking process helps to concentrate the relevant information and improve the precision of the retrieval process, as the chatbot can focus on the most pertinent sections of the sources.

In particular, we explored the langchain framework [11] due to its in-built chunking functions, which saves development time and thus, allows us to iteratively experiment more.

2.2.1 CharacterTextSplitter

The CharacterTextSplitter is a simple yet effective method for dividing text into chunks based on character-level patterns. The heuristics used in this approach are designed to identify natural breaks in the text, such as paragraphs, headings, or bullet points, to create meaningful chunks. It is best for content that are relatively uniform such as news articles.

2.2.2 RecursiveCharacterTextSplitter

The RecursiveTextSplitter is a more sophisticated method that recursively splits text into smaller chunks based on syntactic and semantic cues. This approach is more suitable for complex documents with multiple sections and sub-sections, as it can identify and extract relevant information at different levels of granularity. In particular, it is highly versatile as it allows you to dictate which characters to preferably split at.

2.2.3 NLTKTextSplitter

Langchain also integrates with the Natural Language Toolkit (NLTK) through the NLTKTextSplitter, which leverages NLTK's sentence and paragraph splitting capabilities. This text splitter can be useful for sources where the logical breaks between content units are more clearly defined, such as academic papers or technical manuals.

2.2.4 spaCy Text Splitting

In addition to the Langchain based text splitting, we also explored the use of spaCy's text splitting capabilities [12]. spaCy provides advanced NLP fetures like tokenization, part-of-speech (POS) tagging and named entity recognition (NER), which can be leveraged to identify breaks and transitions between source documents.

2.2.5 Content based Chunking

Something that we didn't thoroughly explore was the idea of content based chunking, where the chunks are based on the content of the text. This is useful for formatted languages like LaTex or Markdown, however, for our case, the input sources were always in plain text, hence redundant for our case. However, this would be interesting to serve as a future work for format specific applications.

2.3 Find Relevancy

With the sourcing and chunking mechanism in place, we then needed a way to effectively retrieve the relevant chunks relative to the user's query. For this, we explored different types of distanc metrics [13], namely cosine similarity, euclidean distance, and Jaccard similarity, to determine the most suitable approach for our chatbot. These similarity measures extend beyond simple keyword/ syntax matching and instead aim to capture the semantic and contextual relationship between queries and candidate chunks. This enables the system to identify the most informative and appropriate chunks to include in the chatbot's response.

2.3.1 Cosine Similarity

Cosine similarity [14] is a measure of the cosine of the angle between two non-zero vectors. It is a widely used metric in information retrieval and text mining tasks, as it can effectively capture the semantic similarity between two text representations, even if they do not share any common words. The key idea behind cosine similarity is to measure the orientation of the two vectors, rather than their magnitude. This means that two vectors can be considered similar if they point in a similar direction, regardless of their absolute lengths. Mathematically, the cosine similarity between two vectors x and y is calculated as:


Where A and B are the two vectors being compared, and \(\|A\|\) and \(\|B\|\) are the magnitudes of the vectors. The resulting value ranges from -1 to 1, with 1 indicating perfect similarity, 0 indicating orthogonality, and -1 indicating complete dissimilarity.

2.3.2 Euclidean Distance

Euclidean distance is a straightforward metric that measures the straight-line distance between two points in a multi-dimensional space. In the context of text embeddings, Euclidean distance can be used to quantify the proximity between the vector representations of the user's query and the candidate chunks. The key idea behind Euclidean distance is to measure the absolute difference between the corresponding elements of the two vectors. Mathematically, the Euclidean distance between two vectors x and y is calculated as:


Where x and y are the two vectors being compared, and n is the number of dimensions in the vector space.

2.3.3 Jaccard Similarity

Jaccard similarity [15] is a measure of the overlap between two sets, defined as the size of the intersection divided by the size of the union of the sets. In the context of text, Jaccard similarity can be used to compare the overlap between the words or n-grams present in the user's query and the candidate chunks. The key idea behind Jaccard similarity is to focus on the shared and unique elements between the two sets, rather than their absolute magnitudes. Mathematically, the Jaccard similarity between two sets A and B is calculated as:


Jaccard similarity can be useful for identifying chunks that share a high degree of lexical overlap with the user's query, which may indicate a strong topical relevance. This approach is particularly effective when the user's query and the relevant chunks can be well-represented by the presence or absence of specific words or n-grams.

2.4 Prompting

A crucial aspect of our chatbot is the way in which we leverage the LLM through prompting. Prompt engineering [16] refers to the process of providing input to a LLM in a specific format to elicit the desired output or behaviour. In the context of our project, prompting plays a vital role in ensuring the generated responses are well-aligned with the user's query and the overarching goals of our project. One of the core challenges we hope to address is prompt alignment - the process of designing prompts that elicit reponses fromm the LLM that are coherent, infromative and tailored to the specific context of the user's query. This goes beyond simply providing the LLM with just the user's query; instead we have explored various prompt engineering approaches to ebtter align the model's generation capabilities with our desired output characteristics.

2.4.1 Zero-Shot Prompting

In a zero-shot setting, the LLM is provided with a prompt that doesn't include any task specific examples or demonstrations. Instead, the prompt relies on the model's general language understanding and reasoning abilites to generate the desired output. This output can be cost-effective, as it does not require extensive fine-tuning or the collection of labelled training data. However, the accuracy of zero-shot prompting may be lower relative to other methods as it is dependent on the model's ability to generalize and infer the task requirements solely from the prompt.

2.4.2 Few-Shot Prompting

In contrast to zero-shot, as the name suggests, few-shot prompting involves providing the LLM with a small number of task-specfic examples within the prompt. This allows the model to better understand the desired output format and type of information that should be included in the response. Few shot prompting strikes the balance between cost and accuracy, as it requires less resources than full fine-tuning while providing task-specfic guidance to the model.

2.4.3 Chain of Thought Prompting

This approach [17] allows the iteration of prompts by encourging the LLM to engage in step-by-step reasoning and articulate its though process explicity. This can be useful for complex queries that require multiple steps of reasoning or for generating detailed explanations. Past research has shown that chaining prompts can lead to more coherent and informative responses from the LLM.

2.4.4 LLM based Prompting

Something new our team explored was the idea of using the LLM itself to generate the prompts. This is done by providing the LLM with a seed prompt and allowing it to generate the subsequent prompts based on the context and user query. We iterate the process, at each time, testing and tuning the prompt by evaluating the generated responses and using the LLM to adjust the prompt.

2.5 Large Language Models

Large language models (LLMs) are at the core of our chatbot's capabilities, providing the foundation for natural language understanding and generation. We explored several prominent LLM options to determine the most suitable approach for our project.

2.5.1 OpenAI's GPT-3.5 / GPT-4

OpenAI's GPT-3.5 and GPT-4 are among the most advanced and widely-used large language models. These models have demonstrated impressive performance across a wide range of natural language tasks, including text generation, question answering, and language understanding.

2.5.2 Gemini

Google's newly released LLM which replaced the BERT model, Gemini is a powerful language model that has been optimized for a wide range of natural language processing tasks.

2.5.3 Offline LLMs

While the cloud-hosted, API-accessible LLMs like GPT-3.5 and GPT-4 offer convenience and scalability, we also explored the potential of using offline LLM solutions. Offline LLMs can provide increased privacy, lower latency, and the ability to run on resource-constrained devices, which may be beneficial for certain deployment scenarios.

The promising offline LLM options include Llama 2 [18] and Mistal8x7b [19], which provides a wide range of pre-trained models that can be fine-tuned and deployed locally. Additionally, specialized offline LLM solutions, such as those developed by companies like Anthropic or Cohere, may offer optimized performance and integration for our chatbot use case.