The SmolPC Engine

Overview and Motivation

The SmolPC inference engine is a standalone local HTTP server that runs large language models on student hardware. It is a separate process from the Tauri desktop app — it survives app restarts, isolates native FFI crashes from the UI, and can be shared across multiple consumers. The engine listens on localhost:19432 with Bearer token authentication and exposes an OpenAI-compatible API surface with Server-Sent Events (SSE) streaming.

The constraint that drives every design decision on this page: 8 GB of RAM minimum, fully offline, no cloud dependency, no data collection. The LLM itself consumes 1.5–4 GB of RAM depending on model size. Every megabyte the infrastructure consumes is a megabyte unavailable for inference.

Why a Separate Process?

The most important architectural decision in the engine is that inference runs in a separate OS process rather than inside the Tauri application via in-process FFI calls. OpenVINO and ONNX Runtime are C libraries loaded at runtime. A segfault inside the FFI layer kills the process. If that process is the Tauri app, the student loses their entire UI — every open conversation, every unsaved context, gone. With a separate engine process, the supervisor in the Tauri backend detects the crash and auto-restarts the engine while the UI remains fully responsive. The student sees a brief "reconnecting" state, not a crash dialog.

Alternatives we evaluated and rejected:

• In-process FFI — faster IPC, but a single C-level segfault takes down the entire application. Unacceptable for a tool students rely on during lessons.
• Unix domain sockets / named pipes — marginal latency improvement over TCP loopback, but loses HTTP debuggability. No benefit for localhost communication.
• gRPC — adds a protobuf compiler build dependency, protocol complexity, and a binary wire format that cannot be inspected with curl. No benefit over HTTP + SSE for a localhost-only server.

The HTTP approach also provides practical benefits beyond crash isolation:

• Debuggable. Any HTTP client works. A developer can probe the engine with curl without compiling Rust.
• Survives app restarts. The engine process persists after the app closes. Reopening the app reconnects instantly — no model reload, no 30-second startup delay.
• Shareable. Multiple consumers can hit the same engine process. This enables the future launcher architecture where the GIMP connector, Blender connector, and Code Helper all share one loaded model.
• Standard protocol. The OpenAI-compatible wire format means any client that speaks that protocol can use the engine as a drop-in local backend.

Crate Architecture

The engine is split across five Rust crates with clear separation of concerns. The dependency flow is deliberately one-directional: smolpc-engine-core has no workspace dependencies, smolpc-engine-host depends on core, smolpc-engine-client is fully independent, and the TTS sidecar is isolated in its own workspace. This means core is testable without a network, host is testable without the desktop app, and client is testable without a running engine.

Hardware Detection and Backend Selection

The engine must run on whatever hardware a secondary school provides: high-end Intel Core Ultra laptops with discrete GPUs and NPUs, budget machines with integrated graphics only, or anything in between. The hardware detection and backend selection system is a multi-stage state machine that probes the hardware, validates each backend through a gate chain, and caches successful decisions for instant startup on subsequent launches.

GPU Detection via DXGI

GPU enumeration uses the Windows DXGI API (IDXGIFactory6) directly, not WMI. This was a hard-won decision: the original implementation used Get-WmiObject queries, which hang for 60+ seconds on some Windows machines with unusual driver configurations or pending updates. The hang occurs inside the WMI service itself and cannot be timed out from the caller. DXGI, by contrast, completes in ~14 ms consistently — it is a direct COM call to the same API that DirectX uses, so it reports exactly what DirectML will see.

The DXGI probe follows this sequence:

1. Create an IDXGIFactory1, then attempt to cast to IDXGIFactory6 for high-performance GPU preference ordering.
2. Enumerate all adapters, extracting per-adapter: name, vendor ID, device ID, dedicated VRAM, and software/hardware classification.
3. Filter out software adapters (the Windows WARP rasteriser).
4. Classify as discrete if dedicated VRAM exceeds 512 MB and the adapter is not software. This threshold is a conservative midpoint — discrete GPUs typically start at 1-2 GB of dedicated VRAM.
5. Select the best candidate: discrete GPUs first, then by VRAM descending.

Critical: integrated GPU rejection. Intel UHD integrated graphics technically supports DirectML, and the DXGI probe reports it as a valid adapter. However, DirectML inference on these GPUs produces garbage logits — the output is numerical noise with no coherent text. This is detected by the preflight validation (described in Section 4), which generates a single token and inspects the logits tensor for NaN and Infinity values. When non-finite logits are detected, the backend is rejected and the engine falls through to OpenVINO CPU.

NPU and RAM Detection

NPU detection is not done through DXGI. Instead, the engine uses the OpenVINO startup probe (probe_openvino_startup), which checks whether the Intel NPU device is visible to the OpenVINO runtime and whether the driver supports the required operations. This runs asynchronously in the background during engine startup.

System RAM is queried via the Windows GlobalMemoryStatusEx API, returning total physical memory in gigabytes. This drives automatic model selection: 16 GB or more selects Qwen3 4B (if provisioned), 3–15 GB selects Qwen2.5 1.5B Instruct, and less than 3 GB falls to the smallest available model. The algorithm filters models by min_ram_gb, sorts descending, and selects the first whose model directory actually exists on disk.

Backend Selection Algorithm

Backend selection is a multi-stage decision process, evaluated in priority order:

Gate Chains

Each backend must pass every gate in its chain before it can be selected. If any gate fails, the backend is skipped and the next candidate in priority order is evaluated.

Temporary Fallback vs. Permanent Demotion

A critical invariant in the backend selection system: a preflight timeout is never persisted as a negative decision. When a preflight times out, the result is classified as TemporaryFallback — it is not written to backend_decisions.v2.json, and the backend will be retried on the next startup. This is essential because NPU first-time compilation is expected to take 3–5 minutes. On the first launch, the 300-second preflight budget may not be enough. On the second launch, the compiled blob cache is warm and the preflight completes in seconds. If the timeout were persisted as a permanent failure, the machine would skip the NPU forever — a working piece of hardware would never be used.

Only explicit failures (init errors, runtime crashes) are persisted. Timeouts are always temporary.

Fingerprint-Based Decision Cache

Successful backend decisions are cached in backend_decisions.v2.json using a composite fingerprint as the cache key. The RuntimeBundleFingerprint is computed from: runtime family (ORT or OpenVINO), canonical file paths, file sizes, modification timestamps, and version metadata strings. It is formatted as a 16-character hex string via DefaultHasher. The cache is stored as an OnceLock<Mutex<HashMap>> — each bundle is loaded exactly once per process, and attempting to load a second bundle with a different fingerprint returns an error rather than silently mixing DLL versions.

The file is written atomically (temporary file then rename) to prevent corruption from power loss or process termination during the write.

Inference Backends

This section covers the lowest and most technically demanding layer of the engine: the custom FFI wrappers that bridge Rust to the native C inference libraries. No published Rust crate bindings exist for the OpenVINO GenAI C API or the ONNX Runtime GenAI C API. The existing openvino Rust crate covers the OpenVINO core but not the GenAI extension (LLM pipeline, streaming, chat history). The ort Rust crate covers ONNX Runtime but not the GenAI extension that provides the OgaGenerator step-by-step decoding API needed for DirectML streaming. We wrote our own FFI bindings from scratch — loading DLLs at runtime, resolving function pointers by name, and managing opaque C handles with RAII guards.

Centralised DLL Loading

All native library loading is confined to a single file: runtime_loading.rs. A CI test (runtime_loading_is_centralized) scans every .rs file in the engine workspace and fails if Library::new() or load_with_flags() appears anywhere else. This enforcement exists for three reasons:

1. Windows DLL dependency order is fragile. Loading openvino_genai_c.dll before tbb12.dll causes a silent load failure because the GenAI DLL has an implicit dependency on TBB. The error message names the DLL you tried to load (openvino_genai_c.dll), not the actual missing transitive dependency (tbb12.dll), making debugging without this knowledge extremely difficult.
2. Load flags matter. We use LOAD_LIBRARY_SEARCH_DLL_LOAD_DIR | LOAD_LIBRARY_SEARCH_SYSTEM32 to restrict dependency resolution to the DLL's own directory plus System32. Without these flags, Windows searches PATH and the working directory, which can silently pick up wrong-version DLLs from the user's environment.
3. Bundle fingerprinting prevents version skew. Each bundle gets a hash-based fingerprint. If the engine is already initialised with one bundle, attempting to load a different one returns an error rather than silently mixing DLL versions — preventing ABI mismatches that would manifest as memory corruption or undefined behaviour.

The OpenVINO DLL load sequence requires 13 DLLs for CPU-only operation, or 15 when NPU support is included. All components are pinned to OpenVINO 2026.0.0 — the runtime, GenAI extension, and tokenizers must all match. Mixing versions breaks ABI compatibility. The order is strict — reordering any entry can produce confusing "DLL not found" errors that name the wrong library.

Steps 9–10 are conditionally included only when the NPU is the target device. CPU-only initialisation skips them, which avoids loading NPU driver dependencies on machines without an NPU. ONNX Runtime is simpler: 2 support DLLs (onnxruntime.dll, onnxruntime_providers_shared.dll), with the GenAI and DirectML DLLs loaded later during API struct construction.

OpenVINO GenAI Backend (CPU + NPU)

The OpenVINO wrapper loads ~28 C API symbols from openvino_c.dll (error reporting) and openvino_genai_c.dll (pipeline lifecycle, generation config, chat history, streaming, performance metrics). All functions return an OvStatus integer where 0 means success. Errors are retrieved via ov_get_error_info(status) and ov_get_last_err_msg(), which return C strings converted to Rust Result types by the check_status helper. Every _create function has a corresponding _free destructor — these are the RAII targets for OvOwned<T>.

RAII for C handles. Every opaque C pointer is wrapped in OvOwned<T>, a generic RAII guard that calls the stored destroy function on drop. The critical design detail is the ownership chain: OvOwned holds an Arc<OpenVinoGenAiApi>, which holds RetainedLibrary handles, which hold Arc<Library> references to the loaded DLLs. This ensures that function pointers are never called after their DLL has been unloaded — the destroy function is always valid when Drop::drop executes.

Pipeline creation differs between CPU and NPU. The CPU path calls ov_genai_llm_pipeline_create(model_dir, "CPU", 0, &pipeline) where 0 means no extra properties. The NPU path uses the variadic C API with 6 string arguments: ov_genai_llm_pipeline_create(model_dir, "NPU", 6, &pipeline, "CACHE_DIR", cache_path, "MAX_PROMPT_LEN", "2048", "MIN_RESPONSE_LEN", "1024"). The count parameter (6) tells the API how many variadic string arguments follow. Getting this count wrong causes undefined behaviour — an incorrect count reads garbage from the stack.

Streaming uses a C callback mechanism. The StreamerCallback struct passes a function pointer to the C library. On each generated token, the callback receives the token as a C string and returns a status: Running (0) to continue, Stop (1) to finish gracefully, or Cancel (2) to abort immediately.

The Rust-side callback function implements multiple safety nets beyond the C library's own stop logic:

• Cancellation check. An AtomicBool flag set by the user or by a timeout.
• Token limit enforcement. A counter that matches max_new_tokens, catching cases where the C-level limit is not honoured due to pipeline bugs or ABI mismatches.
• Stop string accumulation. Checks for <|im_end|> and <|endoftext|> in accumulated text. This is a second line of defence — the NPU StaticLLMPipeline does not always honour stop_token_ids reliably.
• Degenerate output detection. If 64 or more consecutive dots are generated, stops generation immediately. This is a sign of model collapse.

The async wrapper runs the blocking generation call in tokio::task::spawn_blocking and reads tokens from an mpsc::unbounded_channel with a 90-second watchdog timer. If no token arrives within 90 seconds (which can happen during NPU compilation on first load), the wrapper sets the cancelled flag and returns a timeout error. The watchdog resets after each token.

DirectML Backend (Discrete GPU)

The DirectML wrapper uses the ONNX Runtime GenAI C API with a fundamentally different generation model. Where OpenVINO uses a callback-based streaming API, ORT GenAI uses a step-by-step token generation loop: OgaGenerator_GenerateNextToken generates one token, OgaGenerator_GetNextTokens reads the token ID, OgaGenerator_IsDone checks for completion, and OgaTokenizerStreamDecode converts the token to text. Each iteration of this loop produces one token and sends it through the channel.

Key difference in calling convention: ORT GenAI uses extern "system" (Microsoft x64 calling convention on 64-bit Windows), while OpenVINO uses extern "C". The error pattern also differs — ORT GenAI functions return *mut OgaResult where null means success and non-null means an error with an embedded message string.

Model loading creates an OgaConfig, clears default providers, appends "dml" as the execution provider, and optionally sets a specific GPU adapter index via OgaConfigSetDecoderProviderOptionsHardwareDeviceId. This is how the engine targets a specific discrete GPU when multiple GPUs are present.

Preflight validation is the mechanism that catches garbage output on integrated GPUs. The preflight generates a single token and inspects the logits tensor:

1. Encode a short prompt and generate one token with max_length = prompt_tokens + 1.
2. Call OgaGenerator_GetLogits to access the logits tensor.
3. Read the tensor's element type (Float32 or Float16), shape, and data pointer.
4. Compute the bounds of the last row of logits.
5. Scan the last row for non-finite values (NaN, Infinity, negative Infinity).
6. If any non-finite values are found, return an error with the count and first offending value.

This logits-level check is more reliable than testing the decoded text, because a single non-finite logit can propagate through softmax and produce plausible-looking but semantically wrong tokens. The check catches the problem at source.

Hung driver protection. DirectML on some GPU/driver combinations can hang indefinitely during OgaGenerator_GenerateNextToken. A 45-second watchdog sets a hung flag if the timer fires. All subsequent calls to generate_stream or run_preflight immediately return an error. Recovery requires destroying and recreating the entire model.

Backend Comparison

NPU Constraints

The NPU uses OpenVINO's StaticLLMPipeline, which differs from the CPU LLMPipeline in several architecturally significant ways. These constraints are the source of the most complex workarounds in the engine.

• Fixed context window. MAX_PROMPT_LEN (2048 tokens) + MIN_RESPONSE_LEN (1024 tokens) are set at pipeline creation time and cannot be changed. Exceeding MAX_PROMPT_LEN crashes the pipeline with "unknown exception" — no graceful error, no recovery. The engine host humanises this to "conversation too long, try starting a new chat."
• Greedy decoding only. do_sample must be false. Temperature, top_k, and top_p are ignored. presence_penalty is incompatible with greedy decoding.
• Qwen3 INT4 crashes the NPU. Only INT8_SYM (symmetric per-channel quantisation via nncf.compress_weights) works. INT4 is used only on CPU.
• No extra_context API. The CPU path disables Qwen3's thinking mode via ov_genai_chat_history_set_extra_context({"enable_thinking": false}). The NPU StaticLLMPipeline does not support this API. Instead, the engine injects /nothink into the system message content.
• Template patching required. Qwen3 ships with a Jinja chat template that enables thinking by default when enable_thinking is undefined. On the NPU, where we cannot set extra_context, this causes runaway generation (thinking tokens consume the entire response budget). The ensure_qwen3_nothink_template function patches the Jinja condition from enable_thinking is defined and enable_thinking is false to not enable_thinking is defined or enable_thinking is false. Template patch failure is a hard error that aborts NPU model loading — an un-patched template produces runaway generation on every request.
• First-run compilation. The first load compiles the model to an NPU-specific blob, taking 3–5 minutes for Qwen at MAX_PROMPT_LEN=2048. The CACHE_DIR property stores this blob so subsequent loads take seconds. Changing MAX_PROMPT_LEN or MIN_RESPONSE_LEN invalidates the cache.

OpenVINO 2026.0.0 bug: setting min_new_tokens >= 1 permanently suppresses EOS token detection, causing generation to run until max_new_tokens is exhausted. The wrapper explicitly skips the set_min_new_tokens call and logs a warning if a caller attempts to set it.

Unified Runtime Adapter

The InferenceRuntimeAdapter enum unifies both backends behind a single interface with two generation methods: generate_stream() takes a raw string prompt (pre-formatted ChatML) and works with both backends; generate_stream_messages() takes structured chat messages and is only supported by OpenVINO GenAI (which has native structured chat via ov_genai_llm_pipeline_generate_with_history). DirectML returns an error for structured messages because ORT GenAI does not have a native structured chat API. The engine host checks is_openvino_genai() to decide which path to use.

Engine Lifecycle

The engine process is not managed by the operating system or a generic service framework. A purpose-built supervisor actor inside the Tauri app owns the entire lifecycle: spawning, health monitoring, crash recovery, and shutdown. This section traces that lifecycle from idle to running, through failure, and back.

State Machine

The engine lifecycle is modelled as an explicit state machine with six states and validated transitions. Invalid transitions (such as Idle directly to Running) are rejected with an error log, which makes impossible states unrepresentable at runtime.

Idle --Start--> Starting --spawn--> WaitingForHealth --healthy--> Running
                                                                      |
                                                           health loop (10s)
                                                           PID alive + HTTP
                                                                      |
                                                         3 failures --> Crashed
                                                                          |
                                                              backoff restart
                                                              1s -> 2s -> 4s
                                                                          |
                                                        max 3 per 5min --> Failed (terminal)

The state graph includes several non-obvious cycles that are important for correctness. Running can transition back to Starting when the user changes runtime mode (the engine must restart with different DLL paths). Running can transition to Running when a status refresh updates the backend or model information. Failed can transition to Starting when the user explicitly clicks Retry. These cycles are validated by the can_transition_to() method and covered by unit tests in mod.rs.

Supervisor Architecture

The EngineSupervisor is a single Tokio background task spawned during Tauri app initialisation. It runs a select! loop that listens to three concurrent event sources, processing whichever fires first.

Watch Channels and Frontend Integration

External code reads engine state through watch channels rather than locks. This means there is no contention between the supervisor and any number of concurrent readers.

Every state transition does two things: it broadcasts the new state via the watch channel, and it emits a Tauri event "engine-state-changed" with the serialised state. The Svelte frontend listens for this event and updates inferenceStore.lifecycleState, which drives loading screens, error states, and ready indicators.

Spawn Sequence

When the supervisor receives a Start command, the do_spawn_sequence() method executes a 12-step procedure that takes the engine from nothing to healthy.

1. Resolve paths -- port, data directory, resource directory, app version. Portable mode uses exe-relative paths; installed mode uses %LOCALAPPDATA%.
2. Create directories -- ensure runtime and data directories exist.
3. Generate auth token -- delete the old token file, generate a fresh 48-character alphanumeric string via the OS cryptographic random generator (OsRng), and write it to engine-token.txt.
4. Create HTTP client -- construct an EngineClient with the fresh token and http://127.0.0.1:19432.
5. Kill stale processes -- run taskkill /F /IM smolpc-engine-host.exe with CREATE_NO_WINDOW to clean up orphaned engines.
6. Spawn detached process -- launch smolpc-engine-host.exe with Windows flags DETACHED_PROCESS | CREATE_NEW_PROCESS_GROUP | CREATE_NO_WINDOW. Stderr is redirected to engine-spawn.log. The auth token is passed via the SMOLPC_ENGINE_TOKEN environment variable, not the command line, to prevent exposure through process listing.
7. Write PID file -- record the child PID to engine.pid in the shared runtime directory.
8. Transition to WaitingForHealth.
9. Poll health -- call GET /engine/health every 100ms with a 30-second timeout.
10. Transition to Running -- reset the failure counter and broadcast the client via the watch channel.
11. Refresh status -- read GET /engine/status to populate the current backend and model info.
12. Restore desired model -- if a model was previously set, call load_model(). On first failure, retry once after a 2-second delay. On second failure, log the error but remain in Running without a loaded model.

Spawn Lock

Multiple app instances (or future consumers like separate Blender/GIMP connectors) could try to spawn the engine simultaneously. A filesystem-based lock prevents this race condition.

Health Check Loop

While in the Running state, the supervisor runs a two-phase health check every 10 seconds.

1. PID liveness check -- call OpenProcess (Windows) to verify the engine process is still alive. This catches cases where the process has exited but the port has not been released yet.
2. HTTP health check -- call GET /engine/health. This catches cases where the process is alive but hung (deadlocked FFI call, infinite loop in the runtime).
3. On success -- reset the failure counter to 0 and optionally refresh backend/model status.
4. On failure -- increment the failure counter. After 3 consecutive failures, transition to Crashed.

PID death triggers an immediate crash transition (process exit is definitive), while HTTP failures require three consecutive misses before the engine is considered crashed. This prevents a transient network hiccup or a slow model load from triggering unnecessary restarts.

Automatic Restart and Backoff

After each successful restart, the supervisor automatically attempts to restore the previously loaded model (with one retry on failure). This means a student whose engine crashed mid-conversation will return to a working state within seconds, with their model already loaded.

Graceful Shutdown

The shutdown path is designed with nested timeouts to guarantee cleanup even if the engine hangs.

1. Stop audio -- halt recording and playback synchronously to prevent Windows WASAPI session corruption.
2. Snapshot PID -- capture the current engine PID before shutdown clears it.
3. Send shutdown command -- the supervisor calls POST /engine/shutdown with a 5-second timeout. The engine receives the request, notifies its internal shutdown channel, and stops accepting new connections.
4. Clear state -- the supervisor sets the client and PID to None and broadcasts via watch channels.
5. Force-kill fallback -- if the 5-second HTTP shutdown fails, the 8-second outer timeout fires. The supervisor verifies the PID still belongs to smolpc-engine-host.exe via tasklist (PIDs are aggressively reused by Windows), then runs taskkill /F /PID <pid>.

The 8-second outer timeout intentionally exceeds the 5-second inner timeout. This layering ensures the force-kill path only activates if the HTTP shutdown genuinely failed, not due to a race.

TTS Sidecar Lifecycle

The TTS sidecar (smolpc-tts-server) is managed by the engine host, not the supervisor. It runs as a separate process on port 19433 with the same detached/windowless flags. Health is checked at 200ms intervals during startup with a 15-second total budget. The engine proxies /v1/audio/speech requests to localhost:19433/synthesize. If a speech request finds the sidecar unhealthy, the engine attempts one respawn before returning 503.

API Surface

The engine exposes 13 HTTP endpoints on localhost:19432. Every request must include a Bearer token in the Authorization header. Token comparison uses constant-time equality to prevent timing side-channel attacks.

Authentication

The token is a 48-character alphanumeric string generated by OsRng and passed to the engine via the SMOLPC_ENGINE_TOKEN environment variable. It is stored on disk in engine-token.txt inside %LOCALAPPDATA%\SmolPC 2.0\engine-runtime\. The token is rotated on every engine spawn -- restarting the engine invalidates all previous tokens. Verification uses a hand-rolled XOR-based comparison that accumulates byte differences and returns equality only if the accumulated diff is zero, taking the same time regardless of where a mismatch occurs.

Health and Status Endpoints

Startup and Model Endpoints

Generation Endpoints

Audio Endpoints

Lifecycle Endpoints

Concurrency Model

The engine enforces strict concurrency limits through Tokio semaphores to prevent out-of-memory crashes on 8GB hardware.

An atomic generating flag provides a fast-path check before acquiring the semaphore. An atomic model_transition_in_progress flag prevents model unload during active transitions. Both flags use RAII guards (GenerationPermit and TransitionGuard) for automatic cleanup.

SSE Streaming Format

The /v1/chat/completions endpoint streams responses as Server-Sent Events in the OpenAI wire format. The event sequence is:

1. Role announcement -- data: {"choices":[{"delta":{"role":"assistant"}}]}
2. Token chunks (repeated) -- data: {"choices":[{"delta":{"content":"token"}}]}
3. Metrics event -- includes total_tokens, time_to_first_token_ms, tokens_per_second, total_time_ms, and truncated
4. Finish chunk -- data: {"choices":[{"delta":{},"finish_reason":"stop"}]}
5. Done signal -- data: [DONE]

Error events (such as INFERENCE_GENERATION_CANCELLED or ENGINE_STREAM_ERROR) replace the metrics event if generation fails.

Error Response Format

All errors follow a consistent shape: {"error": "message"}. The engine uses a small set of HTTP status codes, each with a specific meaning in the engine context.

Voice and Audio Pipeline

Voice features are part of the shared platform rather than a separate assistant mode. The same microphone and read-aloud controls are available in Code, GIMP, Blender, Writer, and Slides. This section traces the data path for both speech-to-text and text-to-speech.

Speech-to-Text Path

1. Capture -- the Tauri app captures device-native audio using cpal (cross-platform audio library).
2. Resample -- rubato converts the captured audio to 16kHz mono, the format expected by Whisper.
3. Transcribe -- raw f32 PCM samples are sent as a binary body to POST /v1/audio/transcriptions. The engine runs Whisper base.en with INT8 quantisation on OpenVINO CPU. The model is lazy-loaded on the first transcription request.
4. Two-call buffer pattern -- the Tauri app uses a two-call buffer to smooth out partial transcriptions before inserting the final text into the composer.
5. Compose -- the transcribed text is placed in the composer bar so the student can review and edit it before sending.

Text-to-Speech Path

1. Request -- when the user presses read-aloud on an assistant message, the frontend calls POST /v1/audio/speech with the message text.
2. Proxy -- the engine host proxies the request to smolpc-tts-server on localhost:19433/synthesize.
3. Synthesise -- KittenTTS generates WAV audio.
4. Playback -- the Tauri app plays the returned WAV data through rodio.

Why a Separate Sidecar

The TTS server runs in its own workspace and process because of an ort (ONNX Runtime Rust crate) version conflict. The TTS server depends on one version of ort; the engine depends on another. Cargo does not allow two versions of the same crate with conflicting native library requirements in one workspace. The sidecar also provides process isolation: a TTS crash does not affect inference, and a model loading hang in the engine does not block speech synthesis.

Thinking Filter

Qwen3 models may emit chain-of-thought reasoning wrapped in <think>...</think> tags, even when the /nothink directive is present. The ThinkingFilter in chat.rs acts as a safety net that strips these blocks from the SSE token stream before they reach the client.

The filter is a state machine that buffers incoming tokens and tracks whether it is currently inside a thinking block. It handles partial tags at token boundaries (for example, one token ending with <thi and the next starting with nk>) by computing the longest suffix of the buffer that could be a prefix of the target tag. At end-of-stream, any remaining buffered text outside a thinking block is flushed. This makes the filter robust against arbitrary tokenisation splits.

Graceful Degradation

If the microphone is missing or the TTS sidecar fails to start, voice controls are disabled but text chat remains fully functional. The voice semaphore (capacity 1) serialises STT and TTS operations so that a hung transcription cannot block a concurrent speech request indefinitely -- the 30-second semaphore timeout returns a 504 and frees the resource.

Challenges and Design Decisions

This section documents the most significant engineering challenges encountered during development. Each is framed as a problem, what was tried, why it was hard, and how it was solved. Together they illustrate the gap between "local inference works in a demo" and "local inference works reliably on 8GB school laptops."