Evaluation
Summary of Achievements
This evaluation measures the delivered system against the functional requirements defined in the Requirements page. The strongest project outcome is not just one assistant mode in isolation, but the successful pivot toward a unified offline platform with a shared local engine, a common app shell, and assistant-specific workflows for coding, creative tools, and office tasks.
Must-Have Requirements
| ID | Requirement | Priority | Outcome | Notes |
|---|---|---|---|---|
| FR-M1 | Natural-language text input across assistant modes | Must have | Achieved | Text-based workflows are central to the unified app and all major assistant flows. |
| FR-M2 | Unified desktop shell for Code mode, LibreOffice, GIMP, and Blender workflows | Must have | Achieved | The shipped app/ shell now hosts Code, GIMP, Blender, Writer, and Slides inside one shared desktop workspace. |
| FR-M3 | Code mode supports coding questions, explanation, and debugging help | Must have | Achieved | Code mode is the flagship workflow and provides a simple local coding assistant experience inside SmolPC 2.0. |
| FR-M4 | LibreOffice assistant supports document and presentation workflows | Must have | Achieved | Writer and Slides are supported through one shared LibreOffice provider family, with room for more workflow breadth and polish. |
| FR-M5 | GIMP assistant supports image-editing workflows | Must have | Achieved | The assistant can translate natural-language edits into tool-based image workflows through the GIMP bridge. |
| FR-M6 | Blender assistant supports scene-aware 3D help and workflow guidance | Must have | Achieved | Blender mode combines a local bridge with scene context and documentation retrieval. |
| FR-M7 | Local on-device inference during normal use | Must have | Achieved | The final architecture prioritises local inference rather than cloud-hosted AI services. |
| FR-M8 | SmolPC 2.0 Home provides a unified launcher for supported tools | Must have | Achieved | The shared shell, mode selector, setup flow, and host-launch controls give users one entry point into the platform instead of separate assistant launches. |
Should-Have Requirements
| ID | Requirement | Priority | Outcome | Notes |
|---|---|---|---|---|
| FR-S1 | Visible loading and generation feedback | Should have | Achieved | Streaming and loading feedback help the app feel responsive even on local hardware. |
| FR-S2 | Per-tool chat persistence across sessions | Should have | Achieved | Chat state is preserved so users can continue tool-specific conversations later. |
| FR-S3 | Host-application detection and setup guidance | Should have | Achieved | Setup gating, startup checks, and per-mode availability messaging are built into the unified app flow. |
| FR-S4 | Choose the best supported local runtime path and fall back gracefully | Should have | Achieved | The shared engine recognises CPU, DirectML, and OpenVINO NPU paths and falls back when needed. |
| FR-S5 | Open or focus the relevant host application from the workflow | Should have | Achieved | This supports the transition between the assistant shell and the external creative or office tool. |
| FR-S6 | Recover from missing prerequisites or runtime failures without losing chat state | Should have | Achieved | Error handling and setup flows were designed to recover cleanly rather than force users to restart from scratch. |
| FR-S7 | Optional local voice input and read-aloud in SmolPC 2.0 | Should have | Achieved | The current app shell supports local dictation and per-message read-aloud through the engine's audio endpoints while keeping voice traffic on-device. |
Could-Have Requirements
| ID | Requirement | Priority | Outcome | Notes |
|---|---|---|---|---|
| FR-C1 | Local benchmark or diagnostics features | Could have | Achieved | Diagnostics and benchmark-related UI are included to support runtime comparison across devices. |
| FR-C2 | Local retrieval-augmented support for documentation or help content | Could have | Achieved | Blender mode includes local retrieval so scene-aware guidance can be combined with relevant documentation. |
Overall Completion
| Category | Completion |
|---|---|
| Must-have requirements | 8 out of 8 achieved |
| Should-have requirements | 7 out of 7 achieved |
| Could-have requirements | 2 out of 2 achieved |
| Won't-have requirements | Intentionally excluded rather than missed |
Known Bugs
The remaining issues are less about missing platform architecture and more about the current limits of local-model tool calling, packaging variability, and host-application coordination. These are the areas that would most improve reliability and deployment confidence if refined before final submission.
| ID | Area | Issue | Priority | Planned Direction |
|---|---|---|---|---|
| BUG-01 | Setup and packaging | Bundled prerequisites and host-application setup can still vary across machines, especially when local runtimes or tool plugins are not in the expected state. | High | Tighten packaged setup flows, improve validation, and expand final compatibility testing. |
| BUG-02 | GIMP workflow reliability | GIMP is currently the least reliable specialist mode because successful edits depend on bridge state, plugin availability, and consistent interpretation of tool-heavy edit requests. | High | Prioritise GIMP bridge hardening, clearer recovery messaging, and targeted testing for common local editing flows. |
| BUG-03 | Complex multi-step prompts | Small local models can still be less predictable on long, ambiguous, or tool-heavy requests than on focused single-purpose prompts. | Medium | Refine planner prompts, recovery strategies, and scenario testing for tool-heavy flows. |
| BUG-04 | Voice I/O packaging and hardware assumptions | Voice features depend on microphone access, audio output, and correctly bundled Whisper and TTS resources, so some environment-specific setup issues are still possible. | Low | Complete final packaged validation for voice resources and improve recovery messaging when audio features are unavailable. |
Critical Evaluation
The project should be judged not only by feature count, but by whether the final direction solves the real deployment problem it set out to address. In that sense, the most important success was the shift from disconnected or harder-to-deploy local prototypes toward a coherent shared-engine platform that better fits an educational offline use case. The main current limitations are therefore not missing platform foundations, but the practical limits of local AI, especially around small-model tool calling in more complex workflows.
Functionality
Functionally, the system now covers all major intended tool families: coding assistance, office workflows, image editing, and Blender support. It also now includes optional local voice input and read-aloud in the unified app. The unified shell is a major improvement over earlier separate prototypes because it gives the project a single identity and a common interaction model. The key point is that the scaffolding for the whole platform is now in place; the main remaining limitation is that current local models are not equally reliable across every tool-calling scenario, especially in longer or more ambiguous requests.
Code Mode
Code mode succeeds as a simple local coding assistant and is the clearest expression of the platform's offline-first goal. Its current simplicity is also part of its strength: it is a reliable, deployable local interface for coding help. Richer project-context integration remains useful future work, but it is not the main current limitation of the platform. The addition of local voice input and read-aloud also strengthens the accessibility story of the wider SmolPC 2.0 platform.
LibreOffice Assistant
The office modes are valuable because they preserve the original educational productivity motivation of SmolPC, and they show that structured document workflows can be driven locally. Their current ceiling is mainly the planning depth that can be expected from compact local models in more complex multi-step actions, rather than a gap in the shared platform itself.
GIMP Assistant
GIMP mode shows the value of tool-aware editing rather than generic chat, but it is currently the clearest shortcoming in the platform. The combination of bridge reliability, host-state dependence, and tool-heavy edit planning makes it the most sensitive to both environment issues and the present limits of small local models.
Blender Assistant
Blender mode is particularly strong conceptually because it combines scene awareness with local retrieval. It demonstrates how a specialist assistant can do more than answer generic questions, though ambitious scene-aware workflows are still bounded by the reliability of local tool orchestration and model planning.
Stability
Stability improved significantly once common responsibilities were moved into the shared engine and unified shell. Automated tests now cover many of the failure-prone boundaries, including runtime selection, engine-state mapping, setup preparation, provider logic, and compatibility gates. The main remaining stability issues are concentrated in tool-heavy host-integrated flows, especially GIMP, rather than in the shared platform foundation itself.
Shared Engine and App Shell
Centralising startup, runtime choice, and model orchestration made the platform easier to reason about and test. Keeping speech-to-text inside the engine while isolating text-to-speech in a detached sidecar also reduces the blast radius of voice-specific failures. This architectural consolidation is one of the strongest technical decisions in the whole project.
Host-Integrated Modes
The host-integrated modes are naturally more sensitive to external application state and bridge setup than pure text chat. In practice, the main stability concern is not the unified shell but the reliability of tool execution and recovery in app-specific workflows, with GIMP currently the strongest example.
Efficiency
Efficiency should be understood in context: this is a local-first AI platform, so acceptable performance depends on startup behaviour, streaming responsiveness, memory pressure handling, and choosing the best supported runtime for the current device. The architecture is well aligned with that goal, and the more important result is that the platform can choose an appropriate local runtime path and remain usable across supported devices.
Runtime Selection
The move to hardware-aware runtime selection is a major improvement over a fixed local stack. It allows the system to aim for the best supported local performance rather than assuming one narrow hardware profile.
Model Trade-Offs
The platform demonstrates a realistic trade-off between model quality, responsiveness, and device capability. The main current ceiling is not whether the architecture works, but how much tool-calling reliability can be extracted from small local models. Stronger local baselines improve consistency across the wider set of assistant modes, while smaller models remain useful as fallback options.
Compatibility
Compatibility remains strongest on the Windows-first packaged path that shaped most of the runtime and host-application work. The shared engine improves consistency across different local hardware by choosing between CPU, DirectML, and OpenVINO NPU where supported, while host-based modes still depend on the relevant external application being installed and reachable. Voice features add another compatibility layer because microphone access, audio output, and bundled voice resources must all be present, but the current design degrades these features more safely than a monolithic in-process approach would. These are deployment constraints, not signs that the platform direction itself is incomplete.
Maintainability
Maintainability improved substantially after the architecture pivot. Separating responsibilities across the shared engine, the app shell, and the mode-provider layer makes the system easier to extend than the earlier prototype path. The remaining challenge is coordination: because several layers interact, changes to contracts or setup behaviour still need careful documentation and regression testing.
UI/UX
The unified interface is a strong usability improvement because it gives the project a single consistent entry point instead of multiple disconnected tools. Code mode currently offers the clearest and simplest user journey, and the added mic and read-aloud controls strengthen its accessibility story. The more specialised modes benefit from the same shell, and the main UX challenge now is explaining tool-aware workflows and recovery when local-model planning or tool execution reaches its limits, especially in GIMP.
Project Management
Project management was shaped by several major technical pivots. Although this increased complexity, the pivots were not arbitrary: they were responses to real deployment, packaging, and hardware constraints. In hindsight, that adaptability was a strength, because it led to a more coherent and deployable final direction. The main cost was that more time had to be spent establishing robust platform foundations before the team could fully focus on mode-specific polish and optimisation.
Individual Contributions
These tables provide the agreed contribution breakdowns for the system artefacts and the website report. Each row sums to 100% so the split is clear across all major work packages.
For the system artefacts table, the work packages below are grouped around the actual project subsystems and integrations rather than a generic template. This reflects how the work was really split across the unified shell, shared backend, deployment flow, and tool-specific modes.
System Artefacts
| Work Package | Stefanos | Siddhant | Mathias | Aishah | Rabiyah |
|---|---|---|---|---|---|
| Research and Experiments | 25% | 20% | 25% | 15% | 15% |
| UI/UX and Frontend Shell | 25% | 15% | 35% | 10% | 15% |
| Shared Backend / Engine / Runtime Infrastructure | 50% | 15% | 15% | 10% | 10% |
| Setup, Provisioning, and Deployment | 45% | 35% | 10% | 5% | 5% |
| Code Mode and Core Chat Experience | 60% | 10% | 20% | 5% | 5% |
| LibreOffice Integration | 10% | 35% | 35% | 10% | 10% |
| GIMP Integration | 10% | 5% | 5% | 40% | 40% |
| Blender Integration | 15% | 50% | 15% | 10% | 10% |
| Testing and Validation | 40% | 25% | 15% | 10% | 10% |
| Overall Contribution | 31% | 23% | 19% | 13% | 14% |
Website
| Work Package | Stefanos | Siddhant | Mathias | Aishah | Rabiyah |
|---|---|---|---|---|---|
| Website Template and Setup | 10% | 10% | 50% | 15% | 15% |
| Home | 15% | 15% | 20% | 25% | 25% |
| Video | 10% | 10% | 20% | 30% | 30% |
| Requirements | 25% | 25% | 20% | 15% | 15% |
| Research | 20% | 20% | 20% | 20% | 20% |
| UI Design | 5% | 5% | 50% | 20% | 20% |
| System Design | 25% | 10% | 40% | 10% | 15% |
| Engine | 75% | 5% | 10% | 5% | 5% |
| Implementation | 15% | 15% | 40% | 15% | 15% |
| Testing | 10% | 30% | 20% | 20% | 20% |
| Evaluation and Future Work | 15% | 15% | 20% | 25% | 25% |
| User and Deployment Manuals | 10% | 10% | 20% | 30% | 30% |
| Legal Issues | 20% | 20% | 30% | 15% | 15% |
| Blog and Monthly Video | 10% | 10% | 40% | 20% | 20% |
| Overall Contribution | 19% | 17% | 26% | 19% | 19% |
Future Work
The next stage of work should focus less on adding brand-new ideas and more on turning the current platform into a more polished, better-measured, and easier-to-deploy system. The architecture is now strong enough that future effort can concentrate on reliability, workflow depth, and accessibility.
Short-Term Improvements
Code Mode
Deepen context injection, project awareness, and support for richer coding interactions while preserving the simple offline model.
LibreOffice Assistant
Expand the breadth and robustness of office workflows, especially around structured tool usage, clearer summaries of actions taken, and smoother recovery from missing prerequisites.
GIMP Assistant
Reduce setup friction, broaden the range of reliable image-editing operations, and improve the clarity of edit summaries and undo behaviour.
Blender Assistant
Deepen scene introspection, strengthen documentation retrieval quality, and improve the flow between natural-language guidance and concrete Blender actions.
Long-Term Vision
Additional Assistants
The connector-and-provider architecture means the platform could support additional local desktop tools without rewriting the shared shell or the engine contract. Plausible next integrations include vector-graphics workflows such as Inkscape, video or timeline tools such as Kdenlive, or other classroom-relevant creative applications that can expose a structured bridge. The important point is that future assistants can plug into the existing mode system, setup flow, and offline runtime rather than starting from scratch.
Deeper Integration
Over time, the platform could move from separate assistant modes toward stronger cross-tool continuity, shared context, and a more seamless workflow between coding, documents, graphics, and 3D tasks.
Performance, Accessibility, and Deployment
A future release should expand hardware validation, add more accessibility options, broaden voice support beyond the flagship chat flow, and improve packaging so that local deployment is as smooth as the underlying architecture now promises.