Testing

Testing Strategy

Why We Tested This Way

SmolPC 2.0 spans a shared Rust inference engine, a Tauri desktop shell, a Svelte frontend, and host-application bridges for GIMP, Blender, Writer, and Slides. A regression in any layer — a renamed JSON field, a broken backend selection path, a misconfigured DLL search order — can silently break the entire platform even when the UI still appears to load. Automated testing was therefore not optional: it was the primary mechanism for catching these cross-boundary failures before they reached users.

Two-Tier Strategy

Scale: ~302 Test Functions

The test suite covers the full stack from low-level tensor extraction to high-level provisioning workflows. The following table summarises the distribution across all components.

Tools and Commands

# Engine and connector tests
cargo test -p smolpc-engine-core -p smolpc-engine-client -p smolpc-engine-host
cargo test -p smolpc-connector-common -p smolpc-connector-blender
cargo test -p smolpc-connector-gimp -p smolpc-connector-libreoffice

# Desktop app tests
cargo test -p smolpc-desktop

# Frontend checks
npm run check
npm run boundary:check

# Security audit
cargo audit

Test Isolation Patterns

Rust's cargo test runs tests in parallel by default, and environment variables are process-global. Without isolation, env-mutating tests would produce non-deterministic failures. The project uses four isolation patterns to prevent this:

CI Pipeline: 7 Jobs on Every Push and PR

Every push to main and every pull request triggers seven CI jobs on windows-latest runners. This provides fast feedback on regressions across the frontend, the engine, the desktop app, architectural boundaries, code style, and dependency security. The pipeline is defined in .github/workflows/ci.yml.

Boundary Enforcement: 8 Rules

The boundary enforcement script enforces eight rules that prevent architectural regression. Five are file-absence rules (certain legacy paths must not exist); three are content rules (certain patterns must not appear in specific files). These rules exist because SmolPC migrated from an Ollama-backed architecture to a self-contained engine, and the engine-host boundary is the most safety-critical: if the desktop app imports smolpc-engine-host directly instead of going through smolpc-engine-client over HTTP, it defeats process isolation and means a C FFI crash in the engine takes down the entire UI.

Unit Testing

Unit tests lock down the most failure-prone logic across every component. SmolPC is particularly vulnerable to subtle regressions at internal boundaries: a renamed JSON field breaks the Svelte frontend silently, a changed default in backend selection sends every user to the wrong inference path, and a broken DLL search order prevents startup on machines without a GPU. The unit test suite covers all of these failure modes automatically and runs on every CI push.

Engine Host Tests (49 tests)

The largest single test file, covering the engine's core server-side logic. These tests verify that the engine starts, selects backends, parses requests, classifies errors, and handles audio correctly — all without requiring real hardware.

Engine Core Tests (40 tests across 7 files)

Engine core tests live as inline #[cfg(test)] modules inside the source files they verify. This keeps tests close to the code they protect and makes it obvious when a logic change needs a corresponding test update.

Engine Client Tests (18 tests)

The engine client is the typed interface between the desktop app and the engine host. Its tests verify that the contract between these two processes is stable, that environment overrides parse correctly, and that version compatibility gates work as expected.

Desktop App Tests (76 tests across 19 modules)

The Tauri desktop app tests cover the boundary between the frontend and the Rust backend: supervisor handle operations, engine client adaptation, inference commands, lifecycle state machines, path resolution, audio processing, provisioning, and setup flows.

Connector Tests (102 tests across 3 connectors + 16 shared)

Connector crates bridge the unified shell to external desktop applications. Their tests use #[cfg(test)] mock implementations for bridge, runtime, and transport layers so that they run without real Blender, GIMP, or LibreOffice processes. This keeps them deterministic and fast enough for CI while still exercising the full provider and executor logic.

Blender Connector (32 tests across 7 modules)

GIMP Connector (28 tests across 6 modules)

LibreOffice Connector (26 tests across 5 modules)

Connector-Common (16 tests across 5 modules)

Type Contract, Security, and Benchmark Tests (17 tests)

Analysis: What Passing Unit Tests Guarantee

When all ~302 unit tests pass, the following properties are verified automatically: the engine starts on any machine regardless of which DLLs are present, backend selection follows the correct preference and fallback chain, backend decisions persist and recover from corruption, all JSON wire formats match the frontend's expectations, ChatML is rendered correctly for each model family, memory pressure thresholds and model selection heuristics are correct, the engine-host process isolation boundary is not violated by any source file, and all connector providers degrade gracefully when their host application is unavailable. These guarantees are enforced on every push without requiring hardware or manual intervention.

Integration Testing

Integration testing was essential because the platform is only useful when its parts work together: the shared engine must start correctly, the unified app must display the right state, each mode must route to the correct provider, and host-application bridges must behave predictably. Integration tests require real hardware and are not automated in CI because there is no way to simulate a real NPU, discrete GPU, or running GIMP/Blender process in a CI container.

Hardware Coverage

Three physical test machines cover the backend matrix, ensuring that every supported inference path is exercised on real hardware:

The Core Ultra machine is the primary NPU test target. The RTX workstation validates DirectML with a discrete NVIDIA GPU. The Intel CPU machine validates the pure fallback path — the lowest-capability configuration the platform supports.

Integration Test Protocol

After any change to generation config, backend selection, or model loading, the following protocol is executed manually on each relevant machine:

1. Start the engine with a forced backend: $env:SMOLPC_FORCE_EP = "cpu" (or "directml", "openvino_npu")
2. Send a streaming chat completion via curl to localhost:19432/v1/chat/completions
3. Verify: response streams tokens (not a single dump), output is coherent code (not garbage or repetition), generation stops at a reasonable length, TTFT and tokens/sec are within expected range, no crash or "unknown exception" errors

End-to-End Scenarios

Automated Integration Checks

In addition to the live scenarios above, the repository includes automated checks around engine-status mapping, runtime-mode restoration, setup preparation, memory-pressure classification, connector execution, and audio request contracts. These tests cover the boundaries between the engine, the Tauri commands, and the mode providers where regression risk is highest.

# Commands exercised during integration verification
cargo test -p smolpc-engine-client
cargo test -p smolpc-engine-host
cargo test -p smolpc-desktop
cargo test -p smolpc-connector-gimp
cargo test -p smolpc-connector-blender
cargo test -p smolpc-connector-libreoffice

Prompt Testing

Prompt testing was necessary because SmolPC runs offline quantised models, and their behaviour varies more across quantisation levels and inference backends than cloud models. A prompt that works reliably with Qwen2.5-1.5B via CPU might produce a different tool-call format when the same model runs on the NPU via OpenVINO, or might generate malformed JSON that the executor must recover from. Prompt testing checks that the assistants remain understandable, reliable, and correctly routed in realistic user-facing scenarios across these conditions.

Compatibility Testing

Compatibility testing focused on the environments and conditions that matter most for local deployment: packaged desktop behaviour on Windows, hardware-specific runtime paths, and graceful fallback when a preferred dependency or host application is missing. Because the project bundles its own inference engine and runtime DLLs, compatibility is as much about setup and dependency handling as it is about raw operating-system support.

Hardware and OS Coverage

Testing was conducted on three physical machines running Windows 11 (see the Integration Testing section for the full hardware matrix). Degradation was verified by removing specific DLLs from the resource directory and confirming that the engine starts with reduced capability rather than crashing. The startup degradation unit tests (4 tests using tempdir()) automate this verification for ORT, OpenVINO, and NPU plugin bundles.

Performance and Stress Testing

Performance matters because SmolPC runs local models on varied hardware. A machine with 16 GB RAM and an NPU has a fundamentally different performance profile than an 8 GB CPU-only laptop, and both must feel usable. The project includes a dedicated benchmark infrastructure for systematic measurement, plus automated tests for the heuristics that drive runtime behaviour (memory pressure, model selection, streaming).

Benchmark Infrastructure: smolpc-benchmark

The smolpc-benchmark crate is a standalone CLI tool that measures inference performance across backends using fixed prompts, greedy decoding (temperature=0.0), and process-specific resource monitoring. This produces reproducible, comparable results across machines and across code changes.

Metrics Collected

Statistical Analysis

The Stats struct computes population statistics with linear interpolation for percentiles. The benchmark stats module is itself covered by 5 unit tests verifying empty input, single value, known values, percentile interpolation, and the two-value edge case.

pub struct Stats {
    pub mean: f64,
    pub median: f64,
    pub p90: f64,
    pub p95: f64,
    pub std_dev: f64,
    pub min: f64,
    pub max: f64,
}

Performance-Related Automated Tests

In addition to the benchmark tool, several automated tests verify the heuristics that drive runtime performance decisions:

Benchmark Method

The benchmark runs used the project's dedicated benchmarking tool with a fixed prompt corpus and deterministic decoding settings. This kept the prompt inputs aligned across all three machines, but the configurations were not perfectly identical: two machines used 10 runs per prompt with 2 warmups, while the Core Ultra Vivobook used 3 runs per prompt with 1 warmup for a quicker backend comparison pass. The results below should therefore be read as representative measurements that support the hardware-aware runtime story, rather than as one fully normalised laboratory benchmark matrix.

We focused the main report on time to first token, tokens per second, and time per output token, because these most directly affect perceived responsiveness in a local assistant. Full raw JSON artefacts, including per-prompt breakdowns and skipped combinations, are linked in the Appendix benchmark section.

Test Machines

Summary Results

Peak memory is intentionally omitted from the main comparison table because the benchmark tracks engine-process RSS rather than full GPU VRAM use. Throughput and latency are therefore the more reliable headline metrics for the report body.

Key Findings

• CPU fallback remained viable on every machine where the model assets were present, which supports the project's hardware-aware fallback strategy.
• On the strongest laptop, DirectML delivered the best sustained throughput for both tested models, reaching 108.62 tok/s for qwen2.5-1.5b-instruct and 52.46 tok/s for qwen3-4b.
• CPU often produced the lowest TTFT on the strongest laptop, but DirectML dominated longer generation throughput, which is more noticeable in sustained chat use.
• OpenVINO NPU support was functional on the Core Ultra laptop, but in this benchmark set it did not outperform the CPU or DirectML paths on latency or sustained decoding speed.
• The older 7.7 GB laptop completed only the smaller CPU benchmark path successfully; the 4B model was not available there and the attempted NPU path did not expose a usable device through OpenVINO.
• All executed benchmark combinations completed without failed runs, so the main differences observed were performance and availability rather than stability.

User Acceptance Testing

User acceptance testing was planned around representative target users rather than a single generic tester. This matched the product itself: Code mode targets coding support, GIMP and Blender target creative workflows, and the LibreOffice modes target classroom and document-based tasks. Feedback was gathered informally and anonymously from representative users and peers across these workflows to check whether the product felt useful, understandable, and realistic to use as an offline assistant platform.

Tester Profiles

Test Cases

Tester Feedback

Project Partner Feedback