System Design

Architectural design and technical specifications of the Portalt platform

System Architecture

Portalt's architecture implements a layered, service-oriented design that integrates virtual reality experiences with AI capabilities. The system architecture consists of four primary layers that handle distinct responsibilities while maintaining clear interfaces for inter-layer communication:

  • Frontend Layer: The user interface components that different types of users interact with
  • Authentication Layer: Handles user identity and access control
  • Storage Layer: Manages data persistence and file storage
  • Compute Layer: Powers real-time interactions and AI processing

This architecture enforces separation of concerns while facilitating efficient inter-layer communication through well-defined interfaces and protocols. Each layer operates independently while providing essential services to adjacent layers through standardized APIs.

System Architecture Diagram

The system architecture diagram shows how different components interact across layers

Component Description

Frontend Layer

  • NextJS Admin Platform: A comprehensive administrative dashboard for managing VR experiences and content.
    • Scene Management:
      • • Create and manage VR activities and scenes with drag-and-drop interface for 3D models
    • Activity Management:
      • • Create and organize VR activities
      • • Set up interactive elements and triggers
    • Asset Management:
      • • Upload and manage 3D models and assets
      • • Preview 3D models before use
      • • Track asset usage across activities
    • Organization Management:
      • • Create and manage multiple organizations
      • • Generate secure pairing codes for organization activities editing access
      • • Set up organization-specific content and settings
      • • Manage user roles and permissions
    • Document Management:
      • • Upload and organize supporting documents
      • • Link documents to specific activities
  • VR Activity Editor App: A Unity application for editing VR activities.
    • • Integrates with local SQLite storage
    • • Supports scene object configuration
    • • Features asset management system
  • VR Activity Visitor App: The end-user facing application for experiencing the VR activities.
    • • Connects to Ubiq server for multiplayer
    • • Implements WebRTC-based communication
    • • Features real-time scene synchronization
    • • Supports voice interaction capabilities

Authentication Layer

  • Clerk Auth Service: Handles user authentication, authorization, and organization management across the platform.
    • • Implements organization-based access control
    • • Features pairing code system for organization access
    • • Provides secure token management
    • • Supports multi-organization structure
    • • Implements role-based permissions
    • • Features secure session management
  • Pairing System: Secure organization access management
    • • Generates unique 6-character pairing codes
    • • Implements 24-hour code expiration
    • • Features active/inactive code status
    • • Provides organization-specific access
    • • Supports secure code validation

Storage Layer

  • Local SQLite DB: Stores activities and assets metadata along with configuration data locally.
    • • Implements MongoDB-like Collection interface
    • • Features JSON-based document storage
    • • Supports complex query operations
    • • Implements efficient indexing system
    • • Features automatic table creation
    • • Provides transaction support
    • • Implements WAL journaling mode
  • IBM Cloud Object Storage: Cloud storage and retrieval solution for assets and documents files.
    • • Implements secure file upload/download
    • • Features asset versioning
    • • Supports large file handling
    • • Provides CDN integration
    • • Implements efficient caching
    • • Features access control integration
  • Data Management Features:
    • • Implements Repository pattern for data access
    • • Features automatic schema validation
    • • Supports complex data relationships
    • • Provides efficient query optimization
    • • Implements data integrity checks
    • • Features automatic backup support

Compute Layer

The Compute Layer serves as the core processing engine of the system, handling real-time communication, AI services, and data processing. It's built on a modular architecture using several design patterns to ensure scalability, maintainability, and extensibility.

1. Ubiq Server

Central multiplayer server for Unity communication implemented with NodeJS.

  • • Manages WebRTC connections
  • • Handles real-time player synchronization
  • • Implements Mediator pattern for message routing
  • • Provides room management and peer connection handling
  • • Supports real-time data streaming
  • • Uses event-based communication
2. Genie Services Framework

Modular service architecture for AI capabilities, built on the ServiceController base class.

  • RAG Service:
    • • Retrieval-Augmented Generation
    • • Context-aware response generation
    • • Document retrieval and processing
    • • Integration with vector store
  • STT Service:
    • • Speech-to-Text conversion
    • • Real-time audio processing
    • • Wake word detection
    • • Multi-user support
  • TTS Service:
    • • Text-to-Speech synthesis
    • • Voice customization
    • • Streaming audio output
    • • WebSocket-based communication
3. Whisper Processing Service

Advanced speech processing service with standalone integration.

  • • Efficient local processing
  • • High-speed speech recognition
  • • Multi-language support
  • • Real-time transcription using OpenAI Whisper
  • • Voice synthesis using Whisper Speech
  • • WebSocket-based communication
4. Granite LLM Service

Quantized local LLM processing (3.2B parameters) with REST API interface.

  • • Efficient local processing
  • • Low latency responses
  • • Context-aware generation
  • • Resource-optimized operation
Integration & Performance
Unity Integration
  • • WebRTC-based communication
  • • Real-time audio streaming
  • • Event-based message passing
  • • State synchronization
Performance Optimization
  • • Local processing for low latency
  • • Optimized audio streaming
  • • Efficient message routing
  • • Resource management
Security & Monitoring
Security Measures
  • • WebSocket encryption
  • • JWT-based authentication
  • • Process isolation
  • • Rate limiting
Monitoring
  • • Service health checks
  • • Structured logging

Site Map

Site Map Diagram

Sequence Diagrams

Scene Creation Flow

Scene Creation Flow Diagram

Purpose: Illustrates the process of creating and configuring a new scene in the system.

Flow Description:

  • • Client initiates scene creation with activity and organization IDs
  • • Scene Service creates a new scene record in the database
  • • Client adds scene configuration which is stored in the database
  • • Scene objects are added and validated by the Asset Service
  • • Complete scene data is returned to the client

Pairing Code Generation and Validation Flow

Pairing Code Flow Diagram

Purpose: Illustrates the process of generating and validating pairing codes for organization access.

Flow Description:

Pairing Code Generation:
  • • Client requests new pairing code
  • • PairingService generates 6-character code
  • • Code is stored with organization ID
  • • 24-hour expiration time is set
  • • Code is returned to client
Pairing Code Validation:
  • • Client submits code
  • • PairingService queries database
  • • System checks validity and expiration
  • • Validation result returned

Ubiq-Genie Interaction Flow

Ubiq-Genie Flow Diagram

Purpose: Demonstrates the flow of voice interaction and AI response generation in the system.

Components and Flow:

Main Components:
  • • Unity Scene (Client)
  • • MediaReceiver (Audio)
  • • Speech-to-Text Service
  • • RAG Service
  • • Local LLM Server
  • • Text-to-Speech Service
  • • NetworkScene (Server)
Sequence of Events:
  • • User speaks into microphone
  • • Audio data processed
  • • Speech detected and transcribed
  • • RAG service queries LLM
  • • Response converted to speech
  • • Audio played to user
  • • Agent performs gestures

Design Patterns

Ubiq-Genie Patterns

The Ubiq-Genie framework implements several design patterns to create a flexible and maintainable system. Design patterns are proven solutions to common software design problems, helping us write more organized and scalable code. Let's explore each pattern and understand how it contributes to our system:

1. Observer Pattern

What is it? The Observer pattern is like a subscription system - components can subscribe to events and get notified when those events occur.

How we use it: In our system, this pattern enables real-time communication between different parts of the application. For example, when a user speaks, multiple components need to know about it:

  • • The Speech-to-Text service needs to process the audio
  • • The virtual agent needs to show listening indicators
  • • The UI needs to update its state

Benefits:

  • • Components don't need to know about each other directly
  • • Easy to add new features without changing existing code
  • • Enables real-time updates across the system
Observer Pattern Diagram

The Observer pattern showing how events flow through the system

2. Template Method Pattern

What is it? Think of this pattern like a cooking recipe with some steps that must be followed exactly and others that can be customized. It defines a skeleton of operations while allowing some steps to be overridden.

How we use it: Our ApplicationController provides a standard way to initialize and run services while allowing each service to define its own specific behavior:

  • • Every service follows the same lifecycle (initialize → configure → start)
  • • Each service can customize how it handles these steps
  • • Ensures consistent behavior across different services
Template Method Pattern Diagram

The Template Method pattern showing standard service lifecycle

3. Factory Pattern

What is it? The Factory pattern is like a specialized workshop that knows how to create different types of components. Instead of creating objects directly, we ask the factory to create them for us.

How we use it: When the ConversationalAgent needs different services (speech, text, etc.), it uses factories to create them:

  • • Centralizes component creation logic
  • • Makes it easy to switch between different implementations
  • • Handles complex initialization requirements
Factory Pattern Diagram

The Factory pattern showing how components are created

4. Strategy Pattern

What is it? The Strategy pattern is like having interchangeable tools for the same job. It allows us to switch between different algorithms or approaches at runtime.

How we use it: Our AI services use this pattern to support different implementation strategies:

  • • Speech-to-Text can switch between local and cloud processing
  • • Text-to-Speech can use different voice synthesis methods
  • • RAG service can use different embedding approaches
Strategy Pattern Diagram

The Strategy pattern showing interchangeable service implementations

5. Mediator Pattern

What is it? Think of the Mediator pattern like an air traffic controller - instead of planes communicating directly with each other, they all communicate through the controller.

How we use it: Our NetworkScene acts as a mediator for all network communication:

  • • Handles message routing between components
  • • Manages peer connections and room state
  • • Coordinates real-time updates
Mediator Pattern Diagram

The Mediator pattern showing centralized communication control

How These Patterns Work Together

These patterns complement each other to create a robust and flexible system:

  • • The Observer pattern enables event-driven communication
  • • The Template Method ensures consistent service behavior
  • • The Factory creates the right components when needed
  • • The Strategy pattern allows for flexible implementations
  • • The Mediator coordinates all these interactions
Patterns Relationships Diagram

How different design patterns interact in our system

Admin App Patterns

1. Repository Pattern

Purpose: Abstracts data persistence operations

Implementation: Collection class in db.ts

class Collection<T extends { _id?: string }> {
  private tableName: string;
  
  constructor(tableName: string) {
    this.tableName = tableName;
    this.ensureTableExists();
    this.ensureIndexes();
  }

  // CRUD Operations
  findOne(query: Partial<T>): T | null
  find(query: Partial<T> = {}): T[]
  insertOne(doc: T): { insertedId: string }
  updateOne(filter: Partial<T>, update: UpdateOptions): UpdateResult
  deleteOne(filter: Partial<T>): { deletedCount: number }
}

2. Singleton Pattern

Purpose: Ensures single database connection instance

Implementation: SQLite database connection in sqlite.ts

// Development mode singleton
if (process.env.NODE_ENV === "development") {
  let globalWithSqlite = global as typeof globalThis & {
    _sqliteDatabase?: BetterSqlite3.Database;
  };

  if (!globalWithSqlite._sqliteDatabase) {
    globalWithSqlite._sqliteDatabase = new BetterSqlite3(DB_PATH);
    initDatabase(globalWithSqlite._sqliteDatabase);
  }
  db = globalWithSqlite._sqliteDatabase;
}

3. Factory Pattern

Purpose: Creates database client instances

Implementation: getDbClient function in db.ts

export function getDbClient() {
  return {
    db: function(dbName: string) {
      return {
        collection: function<T extends { _id?: string }>(collectionName: string) {
          return new Collection<T>(collectionName);
        }
      };
    }
  };
}

4. Strategy Pattern

Purpose: Handles different types of database operations

Implementation: Relations class for handling relationships

export class Relations {
  static async linkDocumentToActivity(documentId: string, activityId: string): Promise<void>
  static async unlinkDocumentFromActivity(documentId: string, activityId: string): Promise<void>
  static getDocumentsByActivityId(activityId: string): string[]
  static getActivitiesByDocumentId(documentId: string): string[]
}

5. Adapter Pattern

Purpose: Adapts MongoDB-like operations to SQLite

Implementation: Collection class methods mimicking MongoDB operations

// MongoDB-like operations
updateOne(
  filter: Partial<T>,
  update: { 
    $set?: Partial<T>, 
    $addToSet?: Record<string, any>, 
    $pull?: Record<string, any> 
  }
): { matchedCount: number, modifiedCount: number }

Benefits of Used Patterns

  • Repository Pattern:
    • • Centralizes data access logic
    • • Provides consistent interface
    • • Makes testing easier
  • Singleton Pattern:
    • • Ensures single database connection
    • • Prevents resource waste
    • • Maintains connection state
  • Factory Pattern:
    • • Flexible object creation
    • • Encapsulates creation logic
    • • Easy to extend
  • Adapter Pattern:
    • • Makes SQLite work like MongoDB
    • • Reduces learning curve
    • • Maintains consistent API

Data Storage

Data Schema

Our database schema is designed to support the complex relationships between different entities in the system while maintaining flexibility for future extensions. Let's explore how the data is organized:

ER Diagram

Entity Relationship diagram showing how different data entities are connected

Database Structure Overview

The database implements a hybrid approach combining SQLite's ACID compliance with document-based storage flexibility, enabling both structured relationships and dynamic data storage.

Base Collection Structure
  • • Universal fields: id, data, created_at, updated_at
  • • JSON document storage in data field
  • • Timestamp tracking for all records
Main Collections
  • • Activities: Activity information
  • • Documents: Document metadata
  • • Assets: Asset information
  • • Organizations: Organization data
  • • Scenes: Scene information
  • • PairingCodes: Organization access codes
Relationships & Junction Tables
  • • ActivityDocuments: Many-to-many activity-document relations
  • • Scene-Activity relationships
  • • Organization-Scene associations
  • • Configuration linkages
Configuration Collections
  • • ScenesConfiguration: Scene-level settings
  • • SceneConfiguration: Object-level configurations
Pairing System Structure
  • • Enables organization members to access:
  • - Organization's activities in Unity editor app
  • - IBM Cloud Object Storage resources
  • • Features:
  • - Unique 6-character pairing codes
  • - 24-hour code expiration
  • - Organization-specific access control
  • - Active status tracking
Index Structure
  • • idx_scenes_activity: Activity indexing
  • • idx_scenes_org: Organization indexing
  • • idx_scene_config: Configuration indexing
  • • Universal updated_at indexing
Key Features
  • • Flexible document storage
  • • Strong entity relationships
  • • Organization-based access control
  • • Comprehensive asset management
  • • Efficient document handling
  • • Secure pairing system

Packages and APIs

NextJS Admin App

Core Dependencies

Framework and Runtime
  • • Next.js (v15.1.3)
  • • React (v19.0.0)
  • • Electron (v35.0.0)
  • • TypeScript (v5)
Database
  • • better-sqlite3 (v11.8.1)
  • • sqlite3 (v5.1.7)
  • • mongodb (v6.12.0)
Authentication & Security
  • • @clerk/nextjs (v6.9.6)
  • • jose (v5.9.6)
  • • zod (v3.24.2)
Cloud Storage
  • • ibm-cos-sdk (v1.14.0)
  • • formidable (v3.5.2)
UI Components
  • • @radix-ui/react-* (Alert Dialog, Checkbox, Dialog, etc.)
  • • lucide-react (v0.469.0)
  • • tailwindcss (v3.4.1)
  • • tailwind-merge (v2.6.0)
  • • class-variance-authority (v0.7.1)

API Endpoints

Authentication & Organization
  • • POST /api/pairing/generate: Generate organization access code
  • • POST /api/pairing/validate: Validate code and grant access to:
  • - Unity editor app for activity editing
  • - IBM Cloud Object Storage resources
Asset Management
  • • GET/PATCH/DELETE /api/assets/[assetId]
  • • GET /api/assets/[assetId]/access
Document Management
  • • GET/POST /api/documents
  • • DELETE /api/documents/[documentId]
  • • POST /api/documents/[documentId]/unlink
Activity Management
  • • GET/POST /api/activities
  • • GET/PATCH/DELETE /api/activities/[activityId]

Ubiq Genie Framework

Core Framework Dependencies

Ubiq Framework
  • • ubiq-server: Server-side networking
  • • ubiq: Client-side networking
  • • Real-time communication
  • • Room & peer management
Node.js Dependencies
  • • ws: WebSocket client/server
  • • nconf: Configuration management
  • • dotenv: Environment variables
  • • EventEmitter: Event handling

API Endpoints

Room Management
  • • /ws/room: Room management
  • • /ws/peer: Peer connections
  • • /ws/media: Media streaming

Performance Requirements

API Latency
  • • WebSocket: < 50ms
  • • REST: < 100ms
  • • LLM: < 200ms
  • • STT/TTS: < 150ms
Resource Usage
  • • Memory: < 2GB per service
  • • CPU: < 50% per core
  • • Network: < 10MB/s per connection
  • • Storage: < 1GB per service

AI Services

API Endpoints

WebSocket Endpoints
  • • /ws/stt: Speech-to-Text streaming
  • • /ws/tts: Text-to-Speech streaming
REST Endpoints
  • • POST /stt: Speech-to-Text conversion
  • • POST /tts: Text-to-Speech synthesis
  • • POST /query: RAG queries
  • • POST /generate: LLM generation

LLM Server (Granite)

Core Components
  • • llama.cpp: Core inference engine
  • • cpp-httplib: HTTP server
  • • Model: 3.2B parameters (quantized)
Configuration
  • • GPU Layers: 99
  • • Context Size: 8192 tokens
  • • CPU Threads: 4
  • • Temperature: 0.8

Speech Processing Server

Key Packages
  • • FastAPI (v0.115.11)
  • • faster-whisper (v1.1.1)
  • • WhisperSpeech (v0.8.9)
  • • torchaudio (v2.6.0)
Models
  • • STT: Whisper "base" model
  • • TTS: WhisperSpeech "s2a-q4-tiny-en+pl"
  • • Audio: 24kHz to 48kHz resampling