Offline Baseline Evaluation
Benchmark model behaviour before connecting tools or external systems.
Introduction and Background
Since the public release of ChatGPT in late 2022, LLMs have moved rapidly from research labs into mainstream products, internal tooling, and enterprise workflows. As these systems become embedded in decision-making, content generation, retrieval, and tool execution, their security properties matter just as much as their usefulness.
Prompt injection is a particularly important risk because it exploits how language models fundamentally work. Unlike traditional software, LLMs process trusted instructions and untrusted content through the same natural-language channel. This creates ambiguity between “instructions” and “data”, allowing attackers to manipulate behaviour through direct prompts, retrieved documents, tool outputs, or even multimodal inputs [12].
This matters even more in enterprise environments. Once a model is connected to retrieval pipelines, APIs, internal documents, or tool-calling systems, prompt injection is no longer just a chatbot failure: it becomes a route to data leakage, unsafe automation, and loss of control over downstream actions.
Because this is a wide and multi-layered project, we first needed substantial research to understand the true scope and depth of prompt injection risks before defining architecture, features, and evaluation.
Research Report
To ground the project in evidence rather than assumptions, we produced a dedicated research report on prompt injection protection. This report became the basis of our internal knowledge base and informed how we framed the problem, compared existing approaches, and prioritised the features and evaluation criteria of the platform.
What the research covered
- What prompt injection is, why it is a major security risk, and why it matters in enterprise AI systems.
- Main vulnerability types: direct and indirect injection, obfuscation, tool-use risks, memory/long-horizon issues, multimodal attacks, and workflow contamination.
- Main mitigation strategies with defence in depth: input validation, prompt structuring, output filtering, sandboxing, and context sanitisation.
- Testing methods and frameworks for continuous evaluation across prompts, tools, pipelines, and deployments.
How we used it
- It shaped the platform structure: vulnerabilities, mitigations, and testing are separated but connected in a natural and structured way.
- It helped us frame prompt injection as a system-level security issue, not just a bad prompt.
- It guided the knowledge base content, especially the vulnerability and mitigation sections; and also serves as the primary reference for our internal chatbot.
- It strengthened our focus on testing and comparison as continuous security practices.
Research report: the full report is embedded below for in-page reading, with dedicated actions to open it in a separate tab or download a local copy.
Vulnerability Taxonomy
Prompt injection is not one single attack pattern. It is a family of vulnerabilities caused by how LLM systems combine natural-language instructions, external context, memory, and tool access. Our taxonomy groups the main classes of risk that appear repeatedly in literature and real deployment scenarios [1].
Direct prompt injection
A user explicitly attempts to override intended behaviour with adversarial instructions, for example by asking the model to ignore system guidance, reveal hidden instructions, or behave outside policy.
Indirect prompt injection
Malicious instructions are hidden inside external content such as emails, webpages, database entries, or documents that the model later reads and mistakenly treats as actionable.
Instruction–data boundary confusion
Because system prompts, user input, retrieved text, and tool outputs are processed in a shared text context, the model may fail to maintain a strict distinction between “content” and “instructions”.
Obfuscation and encoding attacks
Attackers may disguise malicious instructions through Base64, Unicode homoglyphs, multilingual phrasing, emoji patterns, or fragmented prompts to bypass simple filters.
Tool-use and action injection
If an LLM can call tools or APIs, an injection attack may push it to generate unsafe arguments, perform unintended actions, or trust poisoned tool outputs.
Memory and long-horizon vulnerabilities
Seemingly harmless prompts can accumulate across multiple turns, gradually steering the model’s behaviour and causing safety drift over time.
Multimodal injection
Hidden instructions can be embedded in images or other non-text modalities, creating attack paths that bypass text-only validation or filtering strategies.
Multi-agent / workflow contamination
In agentic systems, one compromised step can contaminate shared memory, planning outputs, or downstream agents, turning a local failure into a broader workflow failure.
Mitigation Strategies
Our research consistently showed that there is no single fix for prompt injection. Effective protection requires defence in depth: reducing exposure, checking behaviour, constraining actions, and continuously testing the system as it evolves. This layered approach is also reflected in OWASP guidance [1].
Defence in depth: the goal is not to assume that one layer will stop every attack, but to combine preventative, detective, and containment-focused safeguards.
Input Validation & Sanitisation
Treating all user input and external content as untrusted, applying filtering, validation, schema checks, or auxiliary classification before the model sees it.
Robust Prompt Design & Context Separation
Structuring prompts so trusted instructions are clearly separated from untrusted data, rather than concatenating everything into one ambiguous block.
Output Filtering, Guardrails & Oversight
Validating outputs before they are shown to users or used in downstream actions, and adding secondary checks or human review for high-risk operations.
Multi-Agent Defence Pipelines
Splitting responsibilities across specialised stages such as review, sanitisation, execution, and output checking to reduce single-point compromise.
Context Sanitisation for Long or External Inputs
Analysing retrieved or long-context content before it enters the prompt, especially in RAG systems, browsers, and pipelines that consume untrusted documents.
Sandboxing & Permission Control
Restricting what tools the model can call, validating arguments, and requiring approval for sensitive actions so successful injections have limited real-world impact.
Model-Level Alignment
Alignment, adversarial fine-tuning, and policy training improve baseline robustness, but should be treated as one layer of defence rather than a full solution.
Testing Methodologies
Testing was one of the most important parts of our research because prompt injection cannot be understood from theory alone. The report showed that secure evaluation has to go beyond simple prompt examples and cover the full lifecycle of an LLM system: baseline benchmarking before deployment, integration testing once tools and external components are connected, automated adversarial testing to catch regressions, sandboxed testing for realistic risky scenarios, and continued monitoring as the system evolves.
Research takeaway: no single framework gives full coverage. The most effective approach is a layered testing strategy combining different approaches.
Testing Approaches We Identified
Integration Testing
Test tools, APIs, retrieval pipelines, and agent workflows in realistic combinations.
Automated Adversarial Regression
Run repeatable hostile prompts continuously to catch regressions early.
Sandboxed Scenario Testing
Simulate risky multi-step interactions without exposing production systems.
Post-Deployment Monitoring
Track safety drift and newly emerging threats as the system evolves.
Why This Matters
Context Changes Risk
A model that looks safe in isolated tests can fail once tools and external data are connected.
Single Benchmarks Are Not Enough
One-off results do not represent multi-turn workflows, chained actions, or real deployment pressure.
Testing Must Be Continuous
Reliable security comes from broad, repeated evaluation tied to real system behaviour.
Platform Research and Technical Overview
Tooling review, implementation choices, and technical trade-offs behind the delivered system.
Existing Solutions
Before deciding what our platform should do, we reviewed both practical security tooling and broader enterprise-facing AI platforms. Our goal was not to copy an existing product, but to understand how current solutions approach prompt injection, red-teaming, evaluation, and operational safety [1], [2], [8], and [9].
Foundry
Azure AI Foundry is a managed platform for accessing and orchestrating foundation models, provider integrations, and enterprise AI workflows. Our industry partner, Josh McDonald, introduced it early in the project as a practical enterprise reference point [3].
Pros
- Supports multi-model and managed-provider workflows in one enterprise environment.
- Strong credential and API integration model for operational use.
Cons
- Can add platform-level complexity compared with single-provider setups.
- Enterprise licensing and Azure-centric patterns reduce portability for ad hoc experiments.
NVIDIA Garak
Garak was the most directly reflected external testing framework in the implemented platform. It demonstrated the value of repeatable adversarial probing and informed the framework-scan capability exposed through the Testing page [2].
Pros
- Broad automated adversarial probing across multiple risk categories.
- Good fit for repeatable and regression-oriented security scans.
Cons
- More effective for broad scanning than deep root-cause explanation.
- Tuning and interpreting results still expects a technically confident operator.
Microsoft PyRIT
PyRIT was relevant from a research and architecture perspective because it frames prompt injection testing as structured red-teaming. We ship a PyRIT runner in the backend framework layer alongside Garak; in practice users run Garak from the Testing UI as the default automated scan, with PyRIT available in the same architectural family for extension [8].
Pros
- Strong structured red-teaming model for enterprise-style testing pipelines.
- Useful for governance and repeatable security workflow design.
Cons
- Less prominently surfaced in the UI than Garak; operators still lean on Garak for routine scans.
- Heavier pipeline-oriented setup if you push it beyond our lightweight integration.
PromptBench
PromptBench informed the importance of adversarial benchmarking and comparing behaviour across prompt variants rather than relying on one or two examples [10].
Pros
- Strong for systematic adversarial robustness benchmarking at scale.
- Encourages structured comparison across prompt variants and scenarios.
Cons
- More benchmark-oriented than user-facing for interactive teaching experiences.
- Less emphasis on guided mitigation workflows for non-expert users.
OWASP PromptMe
PromptMe was useful because it connects testing to recognised OWASP threat scenarios and reinforces a developer-facing, application-security view of prompt injection [1].
Pros
- Directly aligns prompt injection testing with recognised OWASP threat framing.
- Developer-friendly perspective for communicating application-security risks.
Cons
- Not directly integrated as an active runner in the current platform.
- Primarily patterns and references rather than a drop-in harness for our stack.
OpenAI Evals and benchmark suites
OpenAI Evals, together with benchmark-oriented resources such as AgentHarm, AdvBench, and HarmBench, influenced the idea that testing should be reproducible, extensible, and suitable for regression workflows rather than one-off manual checks [9], [11].
Pros
- Supports reproducible, extensible, and regression-friendly evaluation workflows.
- Strong benchmark coverage through complementary safety datasets.
Cons
- Requires additional adaptation to serve as a standalone educational UI layer.
- Often assumes OpenAI-centric flows; multi-provider use needs extra glue.
Key Inspirations We Applied
The following ideas most directly shaped the platform that was actually delivered.
Foundry
We used Foundry as the provider layer for model access and credential-managed API integration, which reinforced our multi-provider design instead of locking the system to a single model stack [3].
NVIDIA Garak
It shaped our framework-assisted testing workflow so evaluation is not only manual prompt demonstrations, but also repeatable scan-based security checks [2].
OWASP PromptMe
Although not directly integrated, it guided how we present risk in a developer-facing way and aligns with our OWASP Top 10 references and security-learning resources across the report [1].
Frontend
We focused this comparison on major frontend framework options and then documented the stack we actually applied in the final implementation.
React
Pros
- Large ecosystem and strong community support.
- Component model suits complex UIs and incremental, maintainable growth.
Cons
- Requires choosing and integrating supporting libraries.
- State and architecture decisions can become complex over time.
Vue
Pros
- Lower learning curve and clear single-file component structure.
- Good built-in conventions for rapid development.
Cons
- Smaller enterprise footprint in our target context.
- Less aligned with the existing team familiarity and repo direction.
Angular
Pros
- Comprehensive framework with strong structure and conventions.
- Good for large-scale enterprise applications with strict patterns.
Cons
- Heavier setup and steeper learning curve for this project scope.
- More framework overhead than needed for a coursework platform.
Svelte
Pros
- Very lightweight runtime and concise component syntax.
- Strong performance characteristics for interactive UI work.
Cons
- Smaller ecosystem for some advanced integration needs.
- Less consistent with the project’s existing implementation path.
Applied Frontend Decision
Backend
For the backend framework choice, we focused on a direct comparison between FastAPI and Django [6], [7].
FastAPI
Pros
- High development speed for API-first systems with automatic OpenAPI docs and typed validation.
- Strong async support and good performance for I/O-heavy model and streaming endpoints.
Cons
- Fewer built-in batteries than full-stack frameworks.
- Some architecture pieces must be assembled by the team.
Django
Pros
- Mature full-stack framework with strong conventions and tooling.
- Includes admin, auth, and ORM out of the box.
Cons
- Heavier framework footprint for an API-centric project.
- Can introduce extra complexity for this project scope.
Backend Decision
FastAPI
We selected FastAPI because the platform is primarily API-driven and benefits from fast iteration, typed schemas, and straightforward integration with model orchestration and streaming endpoints [6].
Platform Architecture Summary
The platform combines a React SPA frontend with a Python FastAPI backend, emphasizing flexibility, multi-provider support, and security-focused architecture [4], [5], [6].
Frontend
Stack: React 18 + TypeScript, bundled with Vite, styled with Tailwind CSS v4 and shadcn/ui components. Includes mandatory dark mode, client-side routing, smooth animations, and PDF export capabilities.
Why: supports interactive multi-view workflows (Testing, Mitigations, Prompt Enhancer, Security Assistant) with responsive, accessible components and consistent design.
Backend
Stack: Python with FastAPI REST API. SQLite for persistence with SQLAlchemy ORM. httpx for HTTP calls to multiple model providers (OpenAI, Anthropic, OpenAI-compatible). Provider router abstraction keeps vendor-specific logic centralised.
Why: prioritises flexibility and integration speed. Multi-provider design avoids vendor lock-in. Lightweight database suitable for coursework scale.
Testing Framework
Frameworks: Azure AI Foundry and NVIDIA Garak. Foundry supports managed model access and credential-based provider integration, while Garak provides repeatable adversarial probing through framework-style scan workflows [2], [3].
Why: together they support both operational model access and structured security evaluation, making testing more consistent than manual prompt checks alone.
References
The following sources were the main references used in our research report and knowledge base, numbered in IEEE style.
Note For references to information contained within the report, refer to content in this same report.
- OWASP GenAI Security Project, “LLM01:2025 Prompt Injection,” OWASP Foundation, 2025. Accessed: Mar. 22, 2026. [Online]. Available: https://genai.owasp.org/llmrisk/llm01-prompt-injection/
- NVIDIA, “garak,” GitHub repository. Accessed: Mar. 24, 2026. [Online]. Available: https://github.com/NVIDIA/garak
- Microsoft, “Azure AI Foundry documentation,” Microsoft Learn. Accessed: Mar. 20, 2026. [Online]. Available: https://learn.microsoft.com/azure/ai-foundry/
- Meta Platforms, Inc., “React documentation,” react.dev. Accessed: Mar. 25, 2026. [Online]. Available: https://react.dev/
- Vite, “Getting started with Vite,” vite.dev. Accessed: Mar. 23, 2026. [Online]. Available: https://vite.dev/guide/
- Sebastián Ramírez, “FastAPI documentation,” fastapi.tiangolo.com. Accessed: Mar. 24, 2026. [Online]. Available: https://fastapi.tiangolo.com/
- Django Software Foundation, “Django documentation,” Django Project. Accessed: Mar. 22, 2026. [Online]. Available: https://docs.djangoproject.com/
- Microsoft, “PyRIT,” GitHub repository. Accessed: Mar. 24, 2026. [Online]. Available: https://github.com/Azure/PyRIT
- OpenAI, “evals,” GitHub repository. Accessed: Mar. 20, 2026. [Online]. Available: https://github.com/openai/evals
- Microsoft, “PromptBench,” GitHub repository. Accessed: Mar. 25, 2026. [Online]. Available: https://github.com/microsoft/promptbench
- M. Mazeika et al., “HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal,” arXiv:2402.04249, 2024. Accessed: Mar. 21, 2026. [Online]. Available: https://arxiv.org/abs/2402.04249
- Darktrace, “What is Prompt Injection?” Darktrace Cyber AI Glossary. Accessed: Mar. 23, 2026. [Online]. Available: https://www.darktrace.com/cyber-ai-glossary/prompt-injections