Evaluation

Summary of Achievements

This evaluation measures the delivered system against the functional requirements defined in the Requirements page. The strongest project outcome is not just one assistant mode in isolation, but the successful pivot toward a unified offline platform with a shared local engine, a common app shell, and assistant-specific workflows for coding, creative tools, and office tasks.

Must-Have Requirements

ID Requirement Priority Outcome Notes
FR-M1 Natural-language text input across assistant modes Must have Achieved Text-based workflows are central to the unified app and all major assistant flows.
FR-M2 Unified desktop shell for Code mode, LibreOffice, GIMP, and Blender workflows Must have Achieved The shipped app/ shell now hosts Code, GIMP, Blender, Writer, and Slides inside one shared desktop workspace.
FR-M3 Code mode supports coding questions, explanation, and debugging help Must have Achieved Code mode is the flagship workflow and provides a simple local coding assistant experience inside SmolPC 2.0.
FR-M4 LibreOffice assistant supports document and presentation workflows Must have Achieved Writer and Slides are supported through one shared LibreOffice provider family, with room for more workflow breadth and polish.
FR-M5 GIMP assistant supports image-editing workflows Must have Achieved The assistant can translate natural-language edits into tool-based image workflows through the GIMP bridge.
FR-M6 Blender assistant supports scene-aware 3D help and workflow guidance Must have Achieved Blender mode combines a local bridge with scene context and documentation retrieval.
FR-M7 Local on-device inference during normal use Must have Achieved The final architecture prioritises local inference rather than cloud-hosted AI services.
FR-M8 SmolPC 2.0 Home provides a unified launcher for supported tools Must have Achieved The shared shell, mode selector, setup flow, and host-launch controls give users one entry point into the platform instead of separate assistant launches.

Should-Have Requirements

ID Requirement Priority Outcome Notes
FR-S1 Visible loading and generation feedback Should have Achieved Streaming and loading feedback help the app feel responsive even on local hardware.
FR-S2 Per-tool chat persistence across sessions Should have Achieved Chat state is preserved so users can continue tool-specific conversations later.
FR-S3 Host-application detection and setup guidance Should have Achieved Setup gating, startup checks, and per-mode availability messaging are built into the unified app flow.
FR-S4 Choose the best supported local runtime path and fall back gracefully Should have Achieved The shared engine recognises CPU, DirectML, and OpenVINO NPU paths and falls back when needed.
FR-S5 Open or focus the relevant host application from the workflow Should have Achieved This supports the transition between the assistant shell and the external creative or office tool.
FR-S6 Recover from missing prerequisites or runtime failures without losing chat state Should have Achieved Error handling and setup flows were designed to recover cleanly rather than force users to restart from scratch.
FR-S7 Optional local voice input and read-aloud in SmolPC 2.0 Should have Achieved The current app shell supports local dictation and per-message read-aloud through the engine's audio endpoints while keeping voice traffic on-device.

Could-Have Requirements

ID Requirement Priority Outcome Notes
FR-C1 Local benchmark or diagnostics features Could have Achieved Diagnostics and benchmark-related UI are included to support runtime comparison across devices.
FR-C2 Local retrieval-augmented support for documentation or help content Could have Achieved Blender mode includes local retrieval so scene-aware guidance can be combined with relevant documentation.

Overall Completion

Category Completion
Must-have requirements 8 out of 8 achieved
Should-have requirements 7 out of 7 achieved
Could-have requirements 2 out of 2 achieved
Won't-have requirements Intentionally excluded rather than missed

Known Bugs

The remaining issues are less about missing platform architecture and more about the current limits of local-model tool calling, packaging variability, and host-application coordination. These are the areas that would most improve reliability and deployment confidence if refined before final submission.

ID Area Issue Priority Planned Direction
BUG-01 Setup and packaging Bundled prerequisites and host-application setup can still vary across machines, especially when local runtimes or tool plugins are not in the expected state. High Tighten packaged setup flows, improve validation, and expand final compatibility testing.
BUG-02 GIMP workflow reliability GIMP is currently the least reliable specialist mode because successful edits depend on bridge state, plugin availability, and consistent interpretation of tool-heavy edit requests. High Prioritise GIMP bridge hardening, clearer recovery messaging, and targeted testing for common local editing flows.
BUG-03 Complex multi-step prompts Small local models can still be less predictable on long, ambiguous, or tool-heavy requests than on focused single-purpose prompts. Medium Refine planner prompts, recovery strategies, and scenario testing for tool-heavy flows.
BUG-04 Voice I/O packaging and hardware assumptions Voice features depend on microphone access, audio output, and correctly bundled Whisper and TTS resources, so some environment-specific setup issues are still possible. Low Complete final packaged validation for voice resources and improve recovery messaging when audio features are unavailable.

Critical Evaluation

The project should be judged not only by feature count, but by whether the final direction solves the real deployment problem it set out to address. In that sense, the most important success was the shift from disconnected or harder-to-deploy local prototypes toward a coherent shared-engine platform that better fits an educational offline use case. The main current limitations are therefore not missing platform foundations, but the practical limits of local AI, especially around small-model tool calling in more complex workflows.

Functionality

Functionally, the system now covers all major intended tool families: coding assistance, office workflows, image editing, and Blender support. It also now includes optional local voice input and read-aloud in the unified app. The unified shell is a major improvement over earlier separate prototypes because it gives the project a single identity and a common interaction model. The key point is that the scaffolding for the whole platform is now in place; the main remaining limitation is that current local models are not equally reliable across every tool-calling scenario, especially in longer or more ambiguous requests.

Code Mode

Code mode succeeds as a simple local coding assistant and is the clearest expression of the platform's offline-first goal. Its current simplicity is also part of its strength: it is a reliable, deployable local interface for coding help. Richer project-context integration remains useful future work, but it is not the main current limitation of the platform. The addition of local voice input and read-aloud also strengthens the accessibility story of the wider SmolPC 2.0 platform.

LibreOffice Assistant

The office modes are valuable because they preserve the original educational productivity motivation of SmolPC, and they show that structured document workflows can be driven locally. Their current ceiling is mainly the planning depth that can be expected from compact local models in more complex multi-step actions, rather than a gap in the shared platform itself.

GIMP Assistant

GIMP mode shows the value of tool-aware editing rather than generic chat, but it is currently the clearest shortcoming in the platform. The combination of bridge reliability, host-state dependence, and tool-heavy edit planning makes it the most sensitive to both environment issues and the present limits of small local models.

Blender Assistant

Blender mode is particularly strong conceptually because it combines scene awareness with local retrieval. It demonstrates how a specialist assistant can do more than answer generic questions, though ambitious scene-aware workflows are still bounded by the reliability of local tool orchestration and model planning.

Stability

Stability improved significantly once common responsibilities were moved into the shared engine and unified shell. Automated tests now cover many of the failure-prone boundaries, including runtime selection, engine-state mapping, setup preparation, provider logic, and compatibility gates. The main remaining stability issues are concentrated in tool-heavy host-integrated flows, especially GIMP, rather than in the shared platform foundation itself.

Shared Engine and App Shell

Centralising startup, runtime choice, and model orchestration made the platform easier to reason about and test. Keeping speech-to-text inside the engine while isolating text-to-speech in a detached sidecar also reduces the blast radius of voice-specific failures. This architectural consolidation is one of the strongest technical decisions in the whole project.

Host-Integrated Modes

The host-integrated modes are naturally more sensitive to external application state and bridge setup than pure text chat. In practice, the main stability concern is not the unified shell but the reliability of tool execution and recovery in app-specific workflows, with GIMP currently the strongest example.

Efficiency

Efficiency should be understood in context: this is a local-first AI platform, so acceptable performance depends on startup behaviour, streaming responsiveness, memory pressure handling, and choosing the best supported runtime for the current device. The architecture is well aligned with that goal, and the more important result is that the platform can choose an appropriate local runtime path and remain usable across supported devices.

Runtime Selection

The move to hardware-aware runtime selection is a major improvement over a fixed local stack. It allows the system to aim for the best supported local performance rather than assuming one narrow hardware profile.

Model Trade-Offs

The platform demonstrates a realistic trade-off between model quality, responsiveness, and device capability. The main current ceiling is not whether the architecture works, but how much tool-calling reliability can be extracted from small local models. Stronger local baselines improve consistency across the wider set of assistant modes, while smaller models remain useful as fallback options.

Compatibility

Compatibility remains strongest on the Windows-first packaged path that shaped most of the runtime and host-application work. The shared engine improves consistency across different local hardware by choosing between CPU, DirectML, and OpenVINO NPU where supported, while host-based modes still depend on the relevant external application being installed and reachable. Voice features add another compatibility layer because microphone access, audio output, and bundled voice resources must all be present, but the current design degrades these features more safely than a monolithic in-process approach would. These are deployment constraints, not signs that the platform direction itself is incomplete.

Maintainability

Maintainability improved substantially after the architecture pivot. Separating responsibilities across the shared engine, the app shell, and the mode-provider layer makes the system easier to extend than the earlier prototype path. The remaining challenge is coordination: because several layers interact, changes to contracts or setup behaviour still need careful documentation and regression testing.

UI/UX

The unified interface is a strong usability improvement because it gives the project a single consistent entry point instead of multiple disconnected tools. Code mode currently offers the clearest and simplest user journey, and the added mic and read-aloud controls strengthen its accessibility story. The more specialised modes benefit from the same shell, and the main UX challenge now is explaining tool-aware workflows and recovery when local-model planning or tool execution reaches its limits, especially in GIMP.

Project Management

Project management was shaped by several major technical pivots. Although this increased complexity, the pivots were not arbitrary: they were responses to real deployment, packaging, and hardware constraints. In hindsight, that adaptability was a strength, because it led to a more coherent and deployable final direction. The main cost was that more time had to be spent establishing robust platform foundations before the team could fully focus on mode-specific polish and optimisation.

Individual Contributions

These tables provide the agreed contribution breakdowns for the system artefacts and the website report. Each row sums to 100% so the split is clear across all major work packages.

For the system artefacts table, the work packages below are grouped around the actual project subsystems and integrations rather than a generic template. This reflects how the work was really split across the unified shell, shared backend, deployment flow, and tool-specific modes.

System Artefacts

Work Package Stefanos Siddhant Mathias Aishah Rabiyah
Research and Experiments 25% 20% 25% 15% 15%
UI/UX and Frontend Shell 25% 15% 35% 10% 15%
Shared Backend / Engine / Runtime Infrastructure 50% 15% 15% 10% 10%
Setup, Provisioning, and Deployment 45% 35% 10% 5% 5%
Code Mode and Core Chat Experience 60% 10% 20% 5% 5%
LibreOffice Integration 10% 35% 35% 10% 10%
GIMP Integration 10% 5% 5% 40% 40%
Blender Integration 15% 50% 15% 10% 10%
Testing and Validation 40% 25% 15% 10% 10%
Overall Contribution 31% 23% 19% 13% 14%

Website

Work Package Stefanos Siddhant Mathias Aishah Rabiyah
Website Template and Setup 10% 10% 50% 15% 15%
Home 15% 15% 20% 25% 25%
Video 10% 10% 20% 30% 30%
Requirements 25% 25% 20% 15% 15%
Research 20% 20% 20% 20% 20%
UI Design 5% 5% 50% 20% 20%
System Design 25% 10% 40% 10% 15%
Engine 75% 5% 10% 5% 5%
Implementation 15% 15% 40% 15% 15%
Testing 10% 30% 20% 20% 20%
Evaluation and Future Work 15% 15% 20% 25% 25%
User and Deployment Manuals 10% 10% 20% 30% 30%
Legal Issues 20% 20% 30% 15% 15%
Blog and Monthly Video 10% 10% 40% 20% 20%
Overall Contribution 19% 17% 26% 19% 19%

Future Work

The next stage of work should focus less on adding brand-new ideas and more on turning the current platform into a more polished, better-measured, and easier-to-deploy system. The architecture is now strong enough that future effort can concentrate on reliability, workflow depth, and accessibility.

Short-Term Improvements

Code Mode

Deepen context injection, project awareness, and support for richer coding interactions while preserving the simple offline model.

LibreOffice Assistant

Expand the breadth and robustness of office workflows, especially around structured tool usage, clearer summaries of actions taken, and smoother recovery from missing prerequisites.

GIMP Assistant

Reduce setup friction, broaden the range of reliable image-editing operations, and improve the clarity of edit summaries and undo behaviour.

Blender Assistant

Deepen scene introspection, strengthen documentation retrieval quality, and improve the flow between natural-language guidance and concrete Blender actions.

Long-Term Vision

Additional Assistants

The connector-and-provider architecture means the platform could support additional local desktop tools without rewriting the shared shell or the engine contract. Plausible next integrations include vector-graphics workflows such as Inkscape, video or timeline tools such as Kdenlive, or other classroom-relevant creative applications that can expose a structured bridge. The important point is that future assistants can plug into the existing mode system, setup flow, and offline runtime rather than starting from scratch.

Deeper Integration

Over time, the platform could move from separate assistant modes toward stronger cross-tool continuity, shared context, and a more seamless workflow between coding, documents, graphics, and 3D tasks.

Performance, Accessibility, and Deployment

A future release should expand hardware validation, add more accessibility options, broaden voice support beyond the flagship chat flow, and improve packaging so that local deployment is as smooth as the underlying architecture now promises.