Testing strategy
We used several types of testing to make sure the platform was reliable both technically and from a user point of view. Our goal was to check the parts of the product that matter most: learning about prompt injection, exploring vulnerabilities, and running tests in the app.
The backbone of technical regression is an automated backend suite in
pytest
pytest
A widely used Python testing framework: you write plain test functions with assertions, discover and run them with one command, and extend behaviour with plugins (for example pytest-asyncio for async code).
pytest documentation
— hundreds of unit and integration checks that we can re-run after every change; manual work sits in the browser, with partners, and in short user sessions.
What this included
- Automated unit and integration testing (pytest): fast, repeatable checks on isolated modules and on multi-step HTTP/API flows through the real app wiring (mostly mocked LLMs; optional live-provider tests when keys are set).
- Manual browser testing: checking that the main screens work across common browsers and window sizes.
- Partner feedback: Avanade gave regular feedback on layout, navigation, and feature placement.
- Light user testing: seven people outside the team completed core tasks and told us what felt clear or confusing.
Backend pytest
Backend pytest
We ran the backend test suite locally from backend/ after installing the development dependencies
(pip install -e ".[dev]", then PYTHONPATH=src pytest). Figures below are from one recorded run.
Results from one recorded run
Unit and integration testing
The backend test suite uses pytest, with pytest-asyncio for async parts of the system. Here, integration testing means automated tests written in pytest, not manual testing. These tests send requests to the API, check the status codes and JSON responses, and make sure different parts of the backend work properly together.
Most tests use mocks instead of real model calls, which keeps the suite fast and practical to run. A smaller set of tests can call live APIs when the right keys are available. If those keys are not present, the tests are skipped.
Unit tests
Unit tests check individual parts of the backend on their own. This includes things like mitigations, prompt helpers, scoring logic, API routes, storage, and runners such as the Garak wrapper. These tests make it easier to change the backend without worrying that small changes will quietly break core behaviour.
Example of the kind of behaviour covered:
def test_flagged_threshold_respects_score():
"""Risk at or above the threshold should mark output as flagged."""
analyser = PromptTestAnalyser()
low = AnalysisResult(risk_score=0.35, lines=[])
high = AnalysisResult(risk_score=0.45, lines=[])
assert analyser.is_flagged(low) is False
assert analyser.is_flagged(high) is TrueIntegration tests
Integration tests check larger flows through the application, for example whether an HTTP request passes through the system correctly and returns the expected result. Some of these tests can also use live providers when API keys are present.
Example of this kind of pattern:
@pytest.mark.asyncio
async def test_create_named_test_returns_id(async_client):
payload = {"name": "demo", "prompt": "Hello", "expect_pass": True}
response = await async_client.post("/api/tests", json=payload)
assert response.status_code == 201
body = response.json()
assert "id" in body
assert body["name"] == "demo"What we covered in automated tests
The automated backend test suite is where we check that core behaviour stays reliable as the codebase changes. Most tests run quickly because model calls are mocked, while a smaller number can use live APIs when keys are available.
API and application flow
- Creating, listing, and running tests
- Security Assistant and streaming paths were covered
- Error handling for invalid input or missing resources
Analysis, scoring, and tools
- Prompt injection analysers and risk scoring
- Garak runner behaviour with mocked subprocess calls
- Mitigation handling and score-related edge cases
Data and configuration
- SQLite-backed repositories and migrations used in testing
- Threshold and weight logic used by scoring
Optional live model checks
- OpenAI, Anthropic, and compatible endpoints when keys are available
- These tests are skipped automatically if the required keys are missing
Compatibility & Responsive design testing
We manually tested the main parts of the platform across Chrome, Safari, Firefox, and Edge to make sure the experience was consistent in normal desktop use. This included browsing the catalogue, reading vulnerability and knowledge pages, using the testing area, and interacting with the Security Assistant. Across these checks, the core flows worked reliably and we did not see inconsistent behaviour between browsers.
For responsiveness, we tested the platform at a range of window sizes rather than only in a full desktop view. We checked that the layout still worked well on narrower screens, smaller laptops, and tablet-sized widths, and that navigation, panels, and key actions remained easy to find and use. Content adjusted cleanly without awkward horizontal scrolling or broken layout, and the platform remained comfortable to use as the available space changed.
We also checked both light and dark modes to make sure text, controls, and spacing remained clear across themes. In normal use, chat and test runs felt responsive on the devices we used, and these checks did not reveal any functional issues in the main user journeys.
Performance/stress testing
We lined up the most demanding Garak configurations the product offers and ran them back to back on the same stack a consultant would use: ordinary hardware, ordinary network, the testing screen left open while the work ran underneath. Somewhere in the middle of a long pass you stop watching the clock and start watching whether progress, labels, and partial results still mean something.
They did. Even when a scan stretched well beyond a quick smoke test, the run completed with summaries we could return to the next day without guesswork. We repeated the exercise on separate evenings until the last rough edges were gone—not because we were chasing a benchmark on a chart, but because we wanted to be sure a serious pass would spend attention on the findings, not on babysitting the shell around them.
This was a confidence exercise, not a published performance study. It tells you what we did with our own time; it does not pretend to replace industry load harnesses or fixed SLA numbers.
Testing prompts for the enhancer and Security Assistant
As well as the automatic backend tests, we checked the written instructions (prompts) for two features people use a lot: the prompt enhancer and the Security Assistant. We wanted those instructions to be easy to understand, stay on topic, and fit what the user was trying to do.
Prompt enhancer
We tried short text, long pasted blocks, and vague questions. We read the rewritten text next to what the user had typed and asked: did this really help, or just add noise? If the result was too stiff, changed what the user meant, or added things they did not ask for, we changed the prompt and tried again.
Security Assistant
We looked at follow-up questions, messages with very little context, and people trying to move the chat outside the knowledge base. When answers were too vague, too sure of themselves, or did not match what is actually in the app, we updated the prompts. Keeping answers close to the real content mattered most here, so users can trust what they see.
The automatic tests cover the code and how the system behaves. This extra manual work helped the enhancer and the assistant feel dependable in day-to-day use.
User acceptance testing
Alongside automated testing, we also wanted to understand how clear and easy the platform was to use in practice. We kept these sessions short and focused. Participants completed a small set of core tasks and shared what they found clear or confusing.
Feedback from Avanade
Our partner at Avanade met with us regularly throughout the project, usually through short demos every two weeks, with more detailed reviews at key milestones. They also used the platform in their own time outside meetings, which made their feedback more practical and grounded in real use rather than just reactions during live calls.
A consistent theme in their feedback was clarity. They wanted users to be able to tell immediately where they were in the platform and what each part was for. In particular, they felt it was important to make a clearer distinction between the reference side of the product, such as the vulnerability and knowledge pages, and the interactive testing side. They also encouraged us to make navigation between these areas more direct, so users could move between reading and testing without unnecessary steps.
They also commented on the testing experience itself. Their view was that the main flow should be as clear as possible: start a test, review the outcome, and then decide what to do next. Based on this, we moved less important actions lower down the page and simplified some labels so they felt more consistent. They also gave useful feedback on presentation, especially around dark mode and contrast, to make sure the interface remained clear in different settings, including meetings and shared screens.
Overall, their feedback helped us improve the structure and usability of the platform in a practical way. In normal use during these reviews, neither we nor Avanade encountered any crashes or blocking issues.
Independent testers
We also invited seven people outside the project to try the platform. None had seen it before. We asked each person to complete the same small set of tasks: open a vulnerability page, read the explanation, create a simple named test, and interpret the result. Each session lasted around twenty minutes, which gave us a consistent basis for comparing feedback.
The group included people from two age ranges, 19 to 22 and 40 to 55, with a mix of students and working adults in both technical and non-technical roles. Overall, the feedback was positive. Several people said the interface felt clear and easy to follow, and a few mentioned that the chat-style testing flow felt familiar, which made the process easier to understand. Most were able to understand the basic meaning of pass and fail after going through one example, although some said that the risk score could use a little more explanation when seen for the first time.
The feedback was mainly about small improvements rather than major problems. A few people wanted a clearer label for the button that starts a new test. Others suggested adding a short piece of helper text next to the risk score. We also received useful feedback on the prompt enhancer. While the feature itself made sense once explained, some testers were not immediately sure what it was for, whether they needed to use it, or how much it would change what they had written. This showed that users could understand the feature once it was explained, but that its purpose was not immediately obvious at first glance.
All seven testers were able to complete the core task successfully and rated it as easy on a short anonymous scale. We treated these sessions as practical feedback rather than as a formal usability study. Across these sessions, and in our own testing and partner reviews, we did not encounter functional issues or instability in normal use.