Implementation

Overview

The implementation moved from separate assistant prototypes toward one unified desktop product built around one shared engine, one shipped application shell, and host-specific connector crates. The most important implementation story is therefore not four isolated assistants, but the way the Svelte frontend, Tauri backend, engine supervisor, and mode providers work together to deliver multiple workflows through one offline desktop application.

Shared Infrastructure

The shared infrastructure is what turns SmolPC 2.0 into a platform instead of a collection of demos. The live app is the unified shell in app/, while runtime ownership sits in the shared engine and host-specific tool integrations sit in connector crates. Root workspace scripts proxy common development commands into the app workspace, while packaging and release preparation live under app-local PowerShell scripts.

Project Structure

.
app/
  src/
  src-tauri/
connectors/
  blender/
  gimp/
  libreoffice/
engine/
  crates/
    smolpc-engine-core
    smolpc-engine-host
    smolpc-engine-client
docs/
scripts/

Shared Engine Integration

The shared engine is a standalone inference server responsible for startup, readiness, backend selection, model loading, generation, cancellation, and audio endpoints. Rather than embedding inference logic in the Tauri backend, the app communicates with the engine through a typed client library (smolpc-engine-client) and the documented localhost HTTP API. This separation is one of the most important architectural decisions in the project — see the dedicated Engine page for the complete design and implementation story.

EngineSupervisor Lifecycle Control

The EngineSupervisor manages the engine process lifecycle through an actor-pattern design using message passing: spawn, health check every 10 seconds, crash recovery with exponential backoff (1s, 2s, 4s — max 3 per 5-minute window), and desired-model auto-restoration. This replaced a fragile shared-mutex design that produced deadlocks and race conditions. Full lifecycle details including the state machine, PID management, spawn locks, and shutdown flow are covered on the Engine page.

Runtime and Model Selection

The engine supports three inference backends — DirectML (discrete GPU via ONNX Runtime GenAI), OpenVINO (NPU and CPU via custom GenAI C API wrappers), and llama.cpp (CPU fallback) — selected automatically through a multi-stage gate chain with hardware detection, preflight validation, and fingerprint-based decision caching. The complete selection algorithm, NPU constraints, and FFI architecture are documented on the Engine page.

Setup and Provisioning

The implementation includes setup preparation for bundled Python, runtime dependencies, host-app availability, and local model provisioning. Root scripts such as npm run runtime:setup:openvino, npm run runtime:setup:python, npm run model:setup:qwen25-instruct, and npm run model:setup:qwen3-4b prepare the common environment. App-local scripts then handle DirectML runtime setup, engine staging, and offline bundle packaging.

Communication Between Components

The communication path is intentionally layered. The user interacts with the Svelte frontend, which calls a Tauri command. The Tauri backend either routes the request to the shared engine or to a mode-specific provider. The provider may then call the engine, a host-app bridge, or both. This layered path is the core implementation pattern that ties the whole platform together.

Unified App Shell

The current integrated application lives in app/. Although Code mode remains the clearest baseline workflow, the shipped shell now supports Code, GIMP, Blender, Writer, and Slides inside one consistent interface. The same component tree handles chat, setup, diagnostics, host-app launching, and voice controls across all of these modes.

Mode Configuration and Switching

Modes are configured as structured frontend objects with labels, suggestions, and availability state. On the backend, the mode registry maps each mode to a provider kind: Code uses a local path, GIMP uses MCP, Blender uses a hybrid path, and Writer plus Slides share the LibreOffice MCP family. This was the key step that turned a single assistant UI into a mode-driven platform shell.

Shared UI Components

The live shell is built from reusable components such as Sidebar, WorkspaceHeader, WorkspaceControls, AppModeDropdown, ConversationView, ComposerBar, SetupBanner, SetupPanel, SetupWizard, ModeHelpDrawer, HardwarePanel, ModelInfoPanel, and a startup loading screen. This gives every mode the same visual language even when the underlying execution path is very different.

Streaming Chat Flow

Assistant requests are sent through a unified frontend API using Tauri channels for streaming events. This lets the interface display partial tokens, completion, cancellation, and error events in one consistent conversation view. Streaming is particularly important for local generation because it keeps the product responsive even when heavier models are running on modest hardware.

Local Persistence

The frontend stores chats, current mode chat IDs, and settings in localStorage. Chats are organised per mode rather than as one global list, allowing the user to switch between workflows without losing context. Settings such as runtime preference, selected model, and theme are also persisted locally.

Setup and Availability Gating

The UI does not assume that every host application is installed or ready. Startup and setup checks map host availability into per-mode availability, so GIMP, Blender, Writer, and Slides can be shown as unavailable with a clear reason rather than appearing broken. The same setup state also drives the setup wizard and the startup loading flow.

Assistant Routing and Providers

A major implementation feature is the provider registry in the Rust backend. The assistant request layer receives one unified request and then dispatches it to the correct provider family based on the selected mode. This keeps the UI simple while still allowing each workflow to have a different execution model.

Provider Family	Modes	Provider Kind	Implementation Role
Code provider	Code	Local	Direct text generation through the shared engine and chat streaming path
GIMP provider	GIMP	MCP	Bridge management, tool discovery, tool execution, and undo support
Blender provider	Blender	Hybrid	Scene bridge management, local retrieval, and scene-aware assistance
LibreOffice provider	Writer and Slides	LibreOffice MCP	Shared office runtime handling, tool filtering, planner-assisted execution, and summaries

Code Mode and Core Chat Experience

Code mode is the simplest implementation path and the foundation of the unified shell. It uses the shared engine for local text generation and presents the result through the same conversation UI that other modes reuse. It is also the clearest place where the shared shell's voice controls are exposed to the user.

Key Implementation Features

Blocking engine startup handshake: startup is mediated by the supervisor and the readiness contract rather than direct process assumptions, which makes the interface safer on slow or partially configured machines.

Streaming generation: responses are streamed token by token through the Tauri command boundary, allowing the frontend to update the active assistant message incrementally.

Memory and runtime awareness: the inference layer exposes backend status, active model, and memory-pressure information so the UI can warn the user when a selected model is likely to perform badly for tool-backed modes.

Voice Input and Read-Aloud

Voice features were implemented as an extension of the existing SmolPC chat workflow rather than as a separate subsystem from the user's point of view. In the current unified shell, these features appear as an optional microphone button in the composer and a read-aloud control on assistant messages.

Frameworks and Libraries

Framework / Library	Purpose	How It Was Used
`cpal`	Microphone capture	Used in the Tauri layer to record audio from the user's input device at its native sample rate
`rubato`	Audio resampling	Used to convert recorded audio into the 16kHz mono format expected by the local Whisper path
Whisper base.en INT8 via OpenVINO	Local speech-to-text	Used through the engine's `/v1/audio/transcriptions` endpoint on CPU
`rodio`	Audio playback	Used to play returned WAV data locally when the user requests read-aloud on an assistant message
`smolpc-tts-server` and KittenTTS	Local text-to-speech	Used as a detached sidecar behind `/v1/audio/speech` so TTS stays offline without complicating the main inference process

Key Features Implementation

Microphone capture and transcription insertion: the app records locally, resamples the audio, sends it to the engine, and inserts the resulting text back into the composer so the user can review it before sending.

Per-message read-aloud: assistant responses can be synthesised on demand and played back locally, with the UI tracking loading, playing, and stop states through a dedicated voice store.

Detached TTS boundary: text-to-speech is isolated behind a sidecar process and the engine API so voice-output problems degrade to an unavailable control instead of destabilising the main chat or runtime-selection path.

GIMP Mode

GIMP mode is implemented as a host-assisted workflow rather than plain text chat. The provider is responsible for preparing the required runtime, detecting GIMP, establishing the MCP session, discovering tools, and executing image-editing actions.

Frameworks and Libraries

Framework / Library	Purpose	How It Was Used
Tauri + Rust	Desktop app backend and process orchestration	Used to manage the GIMP bridge runtime, tool execution, and setup flow
MCP client layer	Structured tool communication	Used to connect to the GIMP bridge, list tools, and execute actions
Shared engine	Local planning and summarisation	Used when natural-language requests need planning or text support before or after a tool action

Key Features Implementation

Bundled runtime preparation: setup logic prepares or validates the required GIMP-side runtime resources before trying to connect, reducing manual setup for the user.

Connection and tool discovery: the provider connects to the bridge, stores discovered tools, and exposes mode state back to the frontend.

Undo support: GIMP mode is one of the few workflows in the unified app that explicitly exposes undo through the provider interface.

Blender Mode

Blender mode combines a host bridge with local retrieval. This makes it different from both Code mode and GIMP mode: it is not just a tool executor, and it is not just a plain text assistant. Instead, it is designed to answer scene-aware questions and support Blender workflows using both bridge state and local reference material.

Frameworks and Libraries

Framework / Library	Purpose	How It Was Used
Tauri + Rust	Backend orchestration and desktop integration	Used to manage the Blender scene bridge and assistant routing
Bridge runtime and addon	Scene-aware host communication	Used to retrieve live scene information and expose Blender-specific tooling
Local RAG index	Reference retrieval	Used to load local Blender reference documents and enrich responses with relevant context

Key Features Implementation

Scene bridge startup: the provider starts or connects to a local scene bridge and stores runtime state so Blender mode can remain scene-aware across requests.

RAG-backed assistance: the provider loads a local retrieval index and combines it with live scene information, allowing responses to use both static reference material and current workspace context.

Hybrid execution path: Blender mode can gather scene state, retrieve context, and use the shared engine for generated explanations within one request flow.

LibreOffice Modes

The LibreOffice workflows are implemented through one shared provider family. This reflects the system design decision that office features should share runtime management and tool orchestration instead of being built as completely separate products.

Frameworks and Libraries

Framework / Library	Purpose	How It Was Used
Tauri + Rust	Backend orchestration	Used to manage LibreOffice runtime setup, provider state, and assistant execution
MCP-backed office runtime	Structured office tool access	Used to connect Writer and Slides workflows to office actions through an allowlisted tool layer
Shared engine planner and streamer	Planning and summarisation	Used to decide whether a tool call is needed and to produce short user-facing summaries after a successful office action

Key Features Implementation

Mode-specific tool filtering: the provider filters the raw office tool list based on the selected mode so that each office workflow only exposes the tools appropriate to that task.

Planner then tool execution: the office executor first uses the engine to decide whether a tool call is needed, then executes the chosen office action, and finally generates a short summary for the user.

Shared provider family: Writer and Slides are grouped under one provider family, which reduces duplication and fits the unified platform architecture.

Packaging, Launch, and Setup Features

The project also includes substantial implementation work for setup preparation, provisioning, and packaging. These pieces matter because the project goal is not only local AI, but local AI that is practical to deploy on school or lab machines.

Root development commands: the workspace exposes npm run check, npm run boundary:check, npm run tauri:dev, npm run runtime:setup:openvino, npm run runtime:setup:python, and the model setup scripts as shared entry points.

App-local packaging scripts: the app workspace adds runtime:setup:dml, engine:build:release, package:offline, and package:offline:bundle for release preparation and offline deployment artefacts.

Offline-first provisioning direction: the current packaging design supports an online installer, offline USB bundles, and a portable-style path, with model discovery and provisioning intended to happen locally rather than through a permanently online dependency.