IMPLEMENTATION DETAILS
This section provides a comprehensive overview of the implementation details for our systematic review generation system. Below you'll find detailed explanations of each component and the technical decisions behind them.
Upserting Text Chunks to Pinecone
Before any processing of text is done, the user must upload their own PDFs, which are saved locally with JavaScript in the front end. The ID that is associated with the files and systematic review is posted to the upsert API endpoint ('/api/upsert') in Flask.
1. Process the PDFs into chunks
The files that have been uploaded by the user must first be fetched using their associated ID because the input files are in a directory named by its ID. A dictionary is created that contains each file name without its extension (i.e. '.pdf') alongside its file path.
files = {}
for filename in os.listdir(path):
full_path = os.path.join(path, filename)
if os.path.isfile(full_path):
# Extract the filename without extension
name_without_extension = os.path.splitext(filename)[0]
files[name_without_extension] = full_path
PyMuPDF is used to open and read the contents of each PDF. This text is cleaned by removing unnecessary whitespaces and empty lines. For each PDF content, the text is split into smaller chunks of size 1500 characters with a 300-character overlap using Langchain's text splitter class (RecursiveCharacterTextSplitter).
def split_text_into_chunks(text, chunk_size=1500, overlap=300):
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=chunk_size,
chunk_overlap=overlap,
separators=['\n\n', '\n', '.', '?', '!']
)
chunks = text_splitter.split_text(text)
return chunks
2. Classification of text chunks
Each text chunk is categorised into one of the five sections of a systematic review, including introduction, methods, results, discussion and conclusion. Our approach to categorising each chunk was by inputting them into a prompt and using OpenAI's GPT model to assign it to the most relevant section.
prompt = f'''
You are an AI assistant classifying research paper sections.
The following text is an excerpt from a scientific paper.
Determine whether it belongs to one of these sections:
- Background
- Methods
- Results
- Discussion
- Conclusion
If uncertain, choose the most relevant section.
---TEXT---
{chunk_text}
------------------
Output only the section name:
'''
3. Upserting chunks to Pinecone
Following categorisation, each chunk is embedded into vectors using OpenAI's embedding model integrated using Langchain. They are then uploaded with metadata containing the text content, source and section into a specified namespace within a Pinecone index.
namespace = f'systematic_review/{paper_id}/{section}'
index.upsert([
(
f'{paper_id}-chunk-{i}',
vector,
{
'text': chunk,
'source': paper_id,
'section': section
}
)
], namespace=namespace)
When upserting text chunks, the ThreadPoolExecutor class from the 'concurrent.futures' inbuilt Python module is used to upsert text concurrently using multiple threads.
with ThreadPoolExecutor() as executor:
for i, chunk in enumerate(text_chunks):
executor.submit(upsert_chunk, i, chunk, paper_id, stored_sections, index)
4. Validation
Before moving forward to generating the sections of a systematic review, a validation process is done to ensure all text chunks have been successfully uploaded into Pinecone and all data can be retrieved for review generation.
def check_all_upserted_chunks(files, chunks_count, max_retries=10, delay=2):
index = pinecone.Index(PINECONE_INDEX_NAME)
retries = 0
while retries < max_retries:
print(f'Retries: {retries}')
total_vector_count = 0
index_stats = index.describe_index_stats()
namespaces = index_stats.get('namespaces', {})
for namespace, stats in namespaces.items():
for paper_id in files:
if namespace.startswith(f'systematic_review/{paper_id}'):
total_vector_count += stats.get('vector_count', 0)
print(f'Vector Count: {total_vector_count}')
if total_vector_count == chunks_count:
print(f'All vectors are fully indexed!')
return
retries += 1
sleep(delay)
print(f'Vectors may not be fully indexed yet. Proceeding with querying.')
Generating Systematic Review
The systematic review is generated by individually generating each section sequentially (background, methods, results, discussion, conclusion) using the OpenAI model. Within the prompts, the model is given information on the markdown format it should use and the topic points it should include.
1. Retrieve from the vector database
When the user is redirected to another page after uploading their PDFs and the research topic, an HTTP request containing the ID and the research topic is sent from an async fetch function in Next.js and is received by the Flask endpoint '/api/generate'.
export async function generateText(prompt, id) {
const response = await fetch('http://127.0.0.1:5000/api/generate', {
method: 'POST',
headers: {
'Content-Type': 'application/json'
},
body: JSON.stringify({ prompt, id })
});
const data = await response.json();
return data;
@app.route('/api/generate', methods=['POST'])
def generate_full_systematic_review():
data = request.json
query = data.get('prompt')
id = data.get('id')
paper_ids = [paper for paper in get_files(id)]
The research topic is first embedded into a vector with OpenAI's embedding model and is used to query a namespace of an index. The namespace is determined using the file name and series of vectors that are related to a review section (e.g. background).
query_vector = embeddings.embed_query(query)
def query_with_namespace(paper_id, section, query_vector, top_k, index):
index = pinecone.Index(PINECONE_INDEX_NAME)
namespace = f'systematic_review/{paper_id}/{section}'
print(f'Querying Pinecone in namespace: "{namespace}"')
results = index.query(
vector=query_vector,
top_k=top_k,
namespace=namespace,
include_metadata=True
)
if results['matches']:
print(f'Found {len(results[\'matches\'])} results in {namespace}')
return results
else:
print(f'No relevant results found in {namespace}')
Similarly to our upsert process, multithreading is utilised when searching for the most relevant text chunks in the database through concurrent.futures' ThreadPoolExecutor class.
result = []
with ThreadPoolExecutor() as executor:
future_to_queries = {
executor.submit(
query_with_namespace,
paper_id,
section,
query_vector,
top_k,
index
): paper_id for paper_id in paper_ids
}
for future in as_completed(future_to_queries):
try:
response = future.result()
result.extend([match['metadata']['text'] for match in response['matches']])
except Exception as e:
print(f'Error querying namespace {future_to_queries[future]}: {e}')
return result
2. Generate the section of systematic review
A scaffold prompt is used for each section of the systematic review, which sets the role the LLM should play alongside writing guidelines and output constraints.
prompt = textwrap.dedent(f'''
# 📚 Systematic Review Writing Task: {section_title}
You are an expert researcher conducting a Systematic Review following PRISMA guidelines.
Your task is to generate the {section_title} section in a structured, evidence-based, and academic manner.
## 🔍 Key Writing Guidelines
- Follow PRISMA guidelines for transparency & reproducibility
Clarity & Coherence: Ensure smooth readability
- Maintain logical flow with clear transition sentences
- Avoid redundancy—summarize rather than repeat ideas
## 📌 Context (Original Query & Research Data)
Below is the original query that sets the topic:
## 🛠 Instructions for Writing This Section
{section_prompt}
- Use this data as a reference to ensure consistency
- If methodology or prior findings are referenced, align your section accordingly
## 🎯 Output Constraints
- Format: Use markdown headings (## for main, ### for sub)
- Word Limit: Aim for {max_length} words
- Clarity & Coherence: Ensure smooth readability
- MUST NOT: Do not include in-text citations
## 📝 Now, generate the full {section_title} section:
''')
The method used to ensure each section of the systematic review contains relevant content; distinct additional prompts are embedded into the previous prompt to highlight the required contents and guidelines for the section.
section_prompt = textwrap.dedent(f'''
{section_title}
Generate the {section_title} section for the Systematic Review using relevant information retrieved through a RAG system. Structure the content logically and focus on summarizing the key findings from the retrieved information.
🔹 Required Content:
1. Relevant Information Summary:
• Provide a general overview of the information retrieved
• Focus on summarizing the main themes and findings
2. Study Types and Methodological Insights (If Available):
• Briefly describe any study types or methodological details
• Only include if explicitly mentioned in the content
3. Key Findings and Themes:
• Present a clear, concise summary of major findings
• Highlight recurring themes or notable differences
4. General Observations and Gaps:
• Mention any apparent gaps in the retrieved information
• Avoid making unsupported assumptions
🔹 Writing Style:
✅ Focused on clear, unbiased summaries
✅ Only present information directly supported by content
✅ Flexible structure, prioritizing relevance
''').strip()
Once the response has been generated by the LLM for a particular section, it undergoes a cleaning process to remove any similar sentences.
filtered_sentences = []
for sentence in unique_sentences:
if not filtered_sentences: # First sentence is always added
filtered_sentences.append(sentence)
continue
# Compute similarity between new sentence and existing filtered ones
embeddings1 = bert_model.encode([sentence], convert_to_tensor=True)
embeddings2 = bert_model.encode(filtered_sentences, convert_to_tensor=True)
similarity_scores = util.pytorch_cos_sim(embeddings1, embeddings2).mean().item()
# Only add sentence if it is not too similar to previous ones
if similarity_scores < similarity_threshold:
filtered_sentences.append(sentence)
3. Store text as PDF
Each generated section is combined together to form the whole systematic review. Before it is returned to be saved into the MySQL database, the text is locally stored as a PDF using the 'markdown-pdf' module.
def store_pdf(text, id):
pdf = MarkdownPdf(toc_level=3)
pdf.add_section(Section(text, toc=False))
path = os.path.abspath(os.path.join(
os.path.dirname(__file__),
'..', '..', 'frontend', 'public',
'output', str(id), 'systematic_review.pdf'
))
os.makedirs(os.path.dirname(path), exist_ok=True)
pdf.save(path)
The systematic review is returned in JSON format back to the Next.js frontend. The review, user ID, review ID and research question are then embedded into a POST request that is sent to the '/api/save' endpoint.
export async function saveSystematicReview(user_id, prompt_id, prompt, systematic_review) {
try {
const response = await fetch('http://127.0.0.1:5000/api/save', {
method: 'POST',
headers: {
'Content-Type': 'application/json'
},
body: JSON.stringify({ user_id, prompt_id, prompt, systematic_review })
});
const data = await response.json();
return data;
} catch (error) {
console.error('Error saving systematic review:', error);
}
}
@app.route('/api/save', methods=['POST'])
def save_history():
data = request.json
user_id = data.get('user_id')
user_id = user_id[0]
prompt_id = data.get('prompt_id')
prompt = data.get('prompt')
systematic_review = data.get('systematic_review')
conn = connect_to_database()
cursor = conn.cursor()
try:
cursor.execute(
'INSERT INTO history (user_id, prompt_id, user_input, model_output) VALUES (%s, %s, %s, %s)',
(user_id, prompt_id, prompt, systematic_review)
)
conn.commit()
return jsonify({'message': 'Systematic review has been stored successfully'})
except mysql.connector.IntegrityError:
return jsonify({'error': 'Systematic review already exists'}), 409
finally:
cursor.close()
conn.close()
Database Functionality
1. Connect to database
Whenever the Flask backend wants to interact with the MySQL database (e.g. log in the system), it must establish a connection between itself and the database using the connect function from the 'mysql.connector' module.
def connect_to_database():
return mysql.connector.connect(
host=MYSQL_HOST,
user=MYSQL_USER,
password=MYSQL_PASSWORD,
database=DATABASE_NAME
)
User System
1. Signing up
Users can sign up by entering a username, password and a confirmation password. Before this data is sent to an API endpoint for database storage, the strength of the password is checked using the 'zod' library in the frontend.
export const SignupSchema = z.object({
username: z
.string()
.min(3, { message: 'Username must be at least 3 characters long.' })
.trim(),
password: z
.string()
.min(8, { message: 'Be at least 8 characters long.' })
.regex(/[A-Z]/, { message: 'Contain at least one capital letter.' })
.regex(/[0-9]/, { message: 'Contain at least one number.' })
.regex(/[^a-zA-Z0-9]/, {
message: 'Contain at least one special character.',
})
.trim(),
confirmPassword: z
.string()
.trim(),
}).refine((data) => data.password === data.confirmPassword, {
message: 'Passwords much match',
path: ['confirmPassword'],
});
To compare the user's form inputs to the schema, the safeParse() method is used. If all fields meet the requirements of the schema, it will return true in a success attribute.
const validatedFields = SignupSchema.safeParse({
username: username,
password: password,
confirmPassword: confirmPassword
})
if (!validatedFields.success) {
return {
errors: validatedFields.error.flatten().fieldErrors,
}
}
If the user's inputs are valid, a POST request is sent to the '/api/register' endpoint with the username and password in the body. In the backend, the username and password are received for saving into the MySQL users table.
@app.route('/api/register', methods=['POST'])
def register():
data = request.json
username = data.get('username')
password = data.get('password')
if not username or not password:
return jsonify({'error': 'Missing username or password'}), 400
password_hash = bcrypt.hashpw(password.encode('utf-8'), bcrypt.gensalt())
conn = connect_to_database()
cursor = conn.cursor()
cursor.execute(
'INSERT INTO users (username, password_hash) VALUES (%s, %s)',
(username, password_hash)
)
conn.commit()
2. Logging in
The login system is similar to the signup process, with the distinction that validation occurs with the backend interacting with the MySQL database.
@app.route('/api/login', methods=['POST'])
def login():
data = request.json
username = data.get('username')
password = data.get('password')
conn = connect_to_database()
cursor = conn.cursor()
cursor.execute('SELECT id, password_hash FROM users WHERE username = %s', (username,))
user = cursor.fetchone()
cursor.close()
conn.close()
if user and bcrypt.checkpw(password.encode('utf-8'), user[1].encode('utf-8')):
return jsonify({'message': 'User logged in successfully'}), 200
else:
return jsonify({'error': 'Invalid username or password'}), 401
3. User login tracking
The user's details are stored in the local storage of the browser, which allows the user state to be maintained when the user enters different pages of the website or leaves it.
const AuthContext = createContext();
export const AuthProvider = ({ children }) => {
const [user, setUser] = useState(null);
...
return (
{children}
);
};
export default function App() {
const { user } = useAuth();
return (
<AuthProvider>
<MainContent user={user} />
</AuthProvider>
);
}When the user successfully logs in, the login function within 'AuthProvider' is executed, which sets the user state to the username.
const login = (username) => {
setUser({ username });
localStorage.setItem('user', JSON.stringify({ username }));
};
When the user logs out, the opposite occurs, such that the user state is set to null to indicate the user is not signed in.
const logout = () => {
setUser(null);
localStorage.removeItem('user');
};
Whenever the user enters the website, there is a check for the stored user inside the browser's local storage using the 'useEffect' hook.
useEffect(() => {
const storedUser = localStorage.getItem('user');
if (storedUser) {
setUser(JSON.parse(storedUser));
}
}, []);
File & Query Input
1. Uploading and deleting files
Uploading and deleting files is done with the 'fs' module from Node.js on the frontend. When the user uploads the files, there is an API call containing the ID and the files to the '/API/upload_files' endpoint in the 'API' frontend directory.
for (const file of files) {
const arrayBuffer = await file.arrayBuffer();
const buffer = new Uint8Array(arrayBuffer);
const filePath = path.join(uploadDir, file.name);
await fs.writeFile(filePath, buffer);
}
Deleting a file works similarly, where the file name is retrieved from a POST request to the '/api/delete_file'.
const deleteDir = path.join(process.cwd(), 'public', 'files', id);
const filePath = path.join(deleteDir, file);
await fs.unlink(filePath);
const files = await fs.readdir(deleteDir);
if (files.length === 0) {
await fs.rmdir(deleteDir);
}
2. Sending the prompt
The prompt is put into a URL parameter of the redirected systematic review page URL. It is retrieved from the 'useSearchParams' function from Next.js.
router.push(`/${id}?prompt=${encodeURIComponent(currentPrompt)}`);
const searchParams = useSearchParams();
const prompt = searchParams.get('prompt');
User History
1. Retrieving and Deleting History
When the website is rendered, the user history is retrieved. This is done by first checking if there is a non-null user in the user state, followed by getting the user ID and then getting the user history.
useEffect(() => {
const fetchHistory = async () => {
if (user) {
const user_id = (await queryID(user.username)).user_id[0];
const response = (await queryUsersHistory(user_id)).result;
setHistory(response);
}
};
fetchHistory();
}, [refresh, user]);
To get the user ID, a POST request is sent to the '/api/query_user' endpoint containing the username.
cursor.execute(
'SELECT id FROM users WHERE username = %s',
(username,)
)
result = cursor.fetchone()
return jsonify({'message': 'Found user successfully',
'user_id': result}), 200
Getting the user's history deleted involves sending a POST request containing the user ID to the '/api/query_user_history' endpoint.
cursor.execute(
'SELECT prompt_id, user_input FROM history WHERE user_id = %s ORDER BY created_at DESC',
(user_id,)
)
result = cursor.fetchall()
return jsonify({'message': 'Found user history successfully',
'result': result}), 200
When a user hovers over a previously generated systematic review, they have the option to delete it from their history. If the user chooses to delete, a request containing the ID is sent to the '/api/delete_user_history' Flask endpoint.
cursor.execute(
'DELETE FROM history WHERE prompt_id = %s',
(prompt_id,)
)
conn.commit()
return jsonify({'message': 'Systematic review successfully'}), 200
2. Displaying User History
Recall that the history of the user is fetched if there is a valid user state. If the contents length of the history is greater than 0, each systematic review is shown as a list item displaying the user's input and contains the key of the review ID.
{history.length > 0 && (
<ul>
{history?.map((item) => {
const [prompt_id, user_input] = item || [];
if (!prompt_id) return null;
return (
<li key={prompt_id}>
<div
onMouseEnter={() => toggleHover(prompt_id)}
onMouseLeave={() => {
toggleHover(null);
setEditOpenPromptId(null);
}}
onClick={() => router.push(`/${prompt_id}`)}
>
{user_input}
{hoveredPrompt === prompt_id && (
<div className="edit-menu">
{/* edit menu */}
</div>
)}
</div>
</li>
);
})}
</ul>
)}When the user hovers over a certain history list item, the 'toggleHover' function sets the 'hoveredPrompt' state to the ID linked with the list item.
function refreshHistory() {
setRefresh(!refresh);
}
const toggleHover = (prompt_id) => {
setHoveredPrompt(prompt_id);
};
const toggleEditMenu = (prompt_id, event) => {
event.stopPropagation();
setEditOpenPromptId(editOpenPromptId === prompt_id ? null : prompt_id);
};
{hoveredPrompt === prompt_id && (
<div
onClick={(event) => toggleEditMenu(prompt_id, event)}
>
<img src='ellipsis.svg' alt='Close Icon' height='20' width='20'/>
</div>
{editOpenPromptId === prompt_id && (
<div
onMouseLeave={() => setEditOpenPromptId(null)}
>
<button
onClick={(event) => delete_history(prompt_id, refreshHistory, event)}
>
Delete
</button>
</div>
)}
)}Systematic Review Display Page
1. Check if there is a systematic review
The URL used to view a systematic review is given by its ID in the following format: 'localhost:3000/[id]'. This is implemented by using a 'page.js' file in the '/app/[id]' directory of the frontend.
const response = await queryHistory(prompt_id);
if (response.error) {
return {
error: response.error,
success: false
}
}
return {
message: 'Systematic review was found successfully',
prompt: response.prompt,
systematic_review: response.systematic_review,
success: true
@app.route('/api/query', methods=['POST'])
def query():
data = request.json
prompt_id = data.get('prompt_id')
conn = connect_to_database()
cursor = conn.cursor()
try:
cursor.execute(
'SELECT user_input, model_output FROM history WHERE prompt_id = %s',
(prompt_id,)
)
result = cursor.fetchone()
user_input, model_output = result
return jsonify({'message': 'Found systematic review successfully',
'prompt': user_input,
'systematic_review': model_output}), 200
except:
return jsonify({'error': 'No systematic review found'}), 404
finally:
cursor.close()
conn.close()
2. Systematic Review Pipeline
If a systematic review is not found, it will go through the systematic review generation pipeline. The 'useState' hooks are to keep track of what is happening during the pipeline so progress can be displayed to the user.
setUpsertLoading(true);
try {
await upsert(id);
} catch (error) {
console.error('Upsert failed:', error);
}
setUpsertLoading(false);
@app.route('/api/upsert', methods=['POST'])
def init_pinecone():
data = request.json
id = data.get('id')
try:
initialise_pinecone()
text_chunks_count, files = process_and_store_all_pdfs(id)
check_all_upserted_chunks(files=files, chunks_count=text_chunks_count)
Before dealing with the PDFs, the vector database in Pinecone must be initialised. The names of indexes associated with the Pinecone API key given are checked, and if the index name is not found, a new index is created.
if PINECONE_INDEX_NAME not in pinecone.list_indexes().names():
print(f'Creating Pinecone index: {PINECONE_INDEX_NAME}...')
pinecone.create_index(
name=PINECONE_INDEX_NAME,
dimension=VECTOR_DIMENSION,
metric=SEARCH_METRIC,
spec=ServerlessSpec(
cloud=SPEC_CLOUD,
region=SPEC_REGION
)
)
print(f'Index "{PINECONE_INDEX_NAME}" created successfully!')
else:
print(f'Index "{PINECONE_INDEX_NAME}" already exists.')
After the vector database has been initialised, the text is extracted from the PDFs and split into chunks. These text chunks are classified and then uploaded to the vector database.
setGenerateLoading(true);
try {
const generateRes = await generate(prompt, id);
const systematic_review = generateRes.systematic_review
const username = JSON.parse(localStorage.getItem('user')).username
const user_id = (await queryID(username)).user_id;
await save(user_id, id, prompt, systematic_review);
} catch (error) {
console.error('Generation failed:', error);
}
setGenerateLoading(false);
Finally, the quality of the generated systematic review is checked by generating various graphs with an API request.
setQualityLoading(true);
try {
await check_quality(id);
setQualityLoading(false);
setGenerate(false);
toggleUpdate();
} catch (error) {
console.error('Quality checking failed:', error);
}
3. Fetch systematic review
If there is a systematic review found using the given ID, a success attribute set to true is returned. This 'displayText' state is to be set to true, which allows the systematic review React element to be displayed.
const response = await query(id);
if (response?.success) {
setDisplayText(true);
const data = await query(id);
setInput(data.prompt);
setOutput(data.systematic_review);
return (
{displayText && }
)
The ReactMarkdown component from the 'react-markdown' React module is used to display markdown text in standard HTML.
<ReactMarkdown
remarkPlugins={[remarkBreaks]}
components={{
h2: ({ children }) => <h2 className="text-2xl font-semibold">{children}</h2>,
h3: ({ children }) => <h3 className="text-xl font-semibold">{children}</h3>,
p: ({ children }) => <p>{children}</p>
}}
>
{preprocessText(text)}
</ReactMarkdown>