Implementation

Implementation

This section provides a detailed explanation of the key features implemented in the AR Learner project. Each subsection covers a specific feature across the Android application, web interface, and LLM server components.

AR Visualization (Android App)

Technology Stack

  • ARCore SDK for Android
  • SceneView library
  • Android Camera2 API
  • Google Maps SDK

Implementation Details

AR visualization is the core feature of our Android application, displaying virtual location pins in the real world when users are within 15 meters of a reported location. We implemented this using Google's ARCore SDK, which provides motion tracking and environmental understanding capabilities.

The AR visualization process follows these steps:

1. Location Tracking: We use FusedLocationProviderClient to track the user's current location with high accuracy:

fusedLocationClient.requestLocationUpdates(
  LocationRequest.Builder(5000).setPriority(Priority.PRIORITY_HIGH_ACCURACY).build(),
  locationCallback,
  Looper.getMainLooper()
)

2. Nearby Reports Detection: When a location update is received, we query the database for reports within a 15-meter radius:

fun getNearbyReports(currentLocation: Location, radiusMeters: Float): List<DataItem> {
  return reportsRepository.getAllReports().filter { report ->
    val reportLocation = Location("").apply {
      latitude = report.latitude
      longitude = report.longitude
    }
    currentLocation.distanceTo(reportLocation) <= radiusMeters
  }
}

3. AR Pin Placement: For each nearby report, we calculate its position in 3D space and render a 3D pin model using SceneView, which simplifies the ARCore implementation. The pin's position is determined by converting GPS coordinates to AR world coordinates.

4. Continuous Updates: As the user moves, we continuously update the positions of AR pins, adding new pins for reports that come within range and removing pins for reports that go out of range.

This implementation creates an immersive experience where users can discover location-based reports by physically exploring their environment.

Map View (Android App)

Technology Stack

  • Google Maps Compose
  • Jetpack Compose
  • Kotlin Coroutines and Flow

Implementation Details

The Map View provides a 2D overview of all reports in the area using Google Maps Compose, a Jetpack Compose wrapper for the Google Maps SDK.

Key implementation aspects include:

1. Map Configuration: We initialize the Google Map with appropriate settings for location tracking:

GoogleMap(
  modifier = Modifier.fillMaxSize(),
  properties = MapProperties(isMyLocationEnabled = true),
  uiSettings = MapUiSettings(zoomControlsEnabled = true),
  onMapLoaded = { /* Map loaded callback */ }
) {
  // Map content, including markers
  MapMarkers(reports)
}

2. Report Markers: We display reports as markers on the map, with each marker showing the report title and description in an info window when tapped.

3. Search Functionality: We implemented a search feature using Compose state management that filters reports based on title or category:

fun filterReports(reports: List<DataItem>, searchText: String): List<DataItem> {
  if (searchText.isEmpty()) return reports
  return reports.filter { report ->
    report.title.contains(searchText, ignoreCase = true) ||
    report.category.contains(searchText, ignoreCase = true)
  }
}

4. Location Centering: We provide a button to center the map on the user's current location, helping users find reports near them.

The Map View complements the AR view, allowing users to see all reports regardless of distance and offering a familiar map-based interface for navigation.

Report Creation (Android App)

Technology Stack

  • Jetpack Compose Navigation
  • CameraX API
  • Room Database
  • Kotlin Flows

Implementation Details

The Report Creation feature allows users to create new location-based reports through a multi-step workflow using Jetpack Compose Navigation.

The workflow consists of three main steps:

1. Image Capture/Selection: Users can take a new photo using the device camera or select an existing image from their gallery. We use the CameraX API for camera access and MediaStore for gallery access.

2. Report Details: Users enter a title, description, and category for the report using Compose input fields:

OutlinedTextField(
  value = title,
  onValueChange = { title = it },
  label = { Text("Title") },
  modifier = Modifier.fillMaxWidth()
)

3. Location Selection: Users select the precise location for the report on a map. They can either place a marker manually or use their current location:

GoogleMap(
  onMapClick = { latLng -> selectedLocation = latLng },
  modifier = Modifier.weight(1f)
) {
  selectedLocation?.let { location ->
    Marker(state = MarkerState(position = location))
  }
}

4. Data Storage: The completed report is saved to the local Room database, which triggers UI updates through Kotlin Flow.

This structured approach ensures all necessary information is collected while providing a smooth user experience.

LLM Server Implementation

Technology Stack

  • Azure Container Instances
  • llama.cpp
  • GGUF model format
  • Docker

Implementation Details

The LLM server provides AI-powered chatbot capabilities that both the Android app and web interface can access. We implemented this using llama.cpp to serve a quantized language model through a REST API.

Implementation steps:

1. Model Selection: We selected a Llama 3 model in GGUF format for optimal performance and resource efficiency. The model is stored in Azure Blob Storage and loaded when the container starts.

2. Server Configuration: We configured the llama.cpp server to provide OpenAI-compatible API endpoints:

./bin/server -m "/app/models/$LLM_FILE_NAME" --host 0.0.0.0 --port 8000 --context-size 2048

3. Docker Containerization: We packaged the server in a Docker container for deployment on Azure Container Instances. The container includes llama.cpp built with CUDA support for GPU acceleration when available.

4. API Endpoints: The server exposes standard endpoints compatible with OpenAI's Chat API, including /v1/chat/completions for processing chat requests and /v1/models for listing available models.

5. Prompt Engineering: We designed system prompts specifically for location-based queries to ensure relevant and helpful responses about local areas.

This LLM server architecture provides an efficient way to deploy AI capabilities that can be accessed by both the Android app and web interface.

The LLM server can be accessed in the link http://llama-server-new.westeurope.azurecontainer.io:8000/

Android Chatbot Integration

Technology Stack

  • Retrofit for API requests
  • Kotlin Coroutines
  • Jetpack Compose UI

Implementation Details

The Android app integrates with the LLM server to provide an in-app chatbot experience for location-based queries.

Key implementation aspects:

1. API Client: We use Retrofit to create a type-safe client for the LLM server API:

interface LlmApiService {
  @POST("v1/chat/completions")
  suspend fun getChatCompletion(@Body request: ChatRequest): ChatResponse
}

2. Chat UI: We implemented a clean chat interface using Jetpack Compose, with separate message bubbles for user and assistant messages.

3. Location Context: To make responses more relevant, we enhance user queries with location data:

val contextPrompt = "User is at ${location.latitude}, ${location.longitude}. " +
  "Nearby: ${nearbyReports.joinToString { it.title }}. Query: $userMessage"

4. Error Handling: We implemented proper error handling to provide feedback when the LLM server is unavailable or returns an error.

This chatbot integration provides users with an AI assistant that can answer questions about their location and nearby points of interest.

Web Interface Implementation

Technology Stack

  • Node.js
  • Express.js
  • MySQL (Azure Database)
  • HTML/CSS/JavaScript
  • Fetch API

Implementation Details

The web interface provides browser-based access to the AR Learner system without requiring the mobile app.

The web interface can be accessed in the link below:

https://n13website-e6e8hrcvcaama0ew.uksouth-01.azurewebsites.net/

Implementation highlights:

1. Server Setup: We used Express.js to create a web server that handles API requests and serves static content:

const express = require("express");
const app = express();
app.use(cors());
app.use(express.json({ limit: '16mb' }));
app.listen(5000, () => console.log("Server running on port 5000"));

2. Database Connection: We connect to Azure MySQL using the mysql2 library with proper security settings.

3. API Endpoints: We implemented RESTful endpoints for report management:

app.get("/locations", (req, res) => {
  const page = parseInt(req.query.page) || 1;
  const limit = parseInt(req.query.limit) || 10;
  connection.query(
    "SELECT * FROM reports ORDER BY report_id DESC LIMIT ? OFFSET ?",
    [limit, (page - 1) * limit],
    (err, results) => {
      if (err) res.status(500).json({ error: err.message });
      else res.json(results);
    }
  );
});

4. Map Integration: We implemented an interactive map using Leaflet.js to display reports as markers.

5. Chatbot Frontend: We created a web-based chatbot interface that communicates with the LLM server using the Fetch API.

6. Responsive Design: We used CSS media queries to ensure the interface works well on both desktop and mobile browsers.

The web interface shares the same database and LLM backend as the Android app, ensuring consistent functionality across platforms.

Database Architecture

Technology Stack

  • Room Persistence Library (Android)
  • Azure MySQL (Web)
  • Repository Pattern
  • Kotlin Coroutines and Flow

Implementation Details

The database architecture follows the Repository pattern, providing a clean API for data access and abstracting the data sources.

Key components:

1. Entity Definition: We defined the DataItem entity to represent location reports with fields for title, description, category, image, location coordinates, and timestamp.

2. Data Access Objects: We created DAOs for database operations, using suspend functions for asynchronous operations:

@Dao
interface DataDao {
  @Insert
  suspend fun insertReport(report: DataItem)
  @Query("SELECT * FROM data_items ORDER BY timestamp DESC")
  fun getAllReports(): Flow<List<DataItem>>
}

3. Repository Layer: The repository mediates between data sources and the rest of the application, providing a clean API for data access.

4. Cross-Platform Data Access: The Android app uses Room for local storage, while the web interface connects directly to Azure MySQL. Both implementations provide the same functional interface.

5. Reactive Updates: We use Kotlin Flow on Android to reactively update the UI when the underlying data changes.

This architecture ensures efficient data storage and retrieval with a clean separation of concerns that makes the codebase more maintainable.

Integration Between Components

The AR Learner system integrates three main components:

  1. Android App: Provides AR visualization, map view, report creation, and chatbot features
  2. Web Interface: Offers browser-based access to reports and chatbot
  3. LLM Server: Delivers AI capabilities to both the Android app and web interface

Data flows between these components as follows:

  • Reports created in either the Android app or web interface are stored in the database
  • The Android app can display reports from both sources in AR and map views
  • Both interfaces access the same LLM server for chatbot functionality
  • Location data from the device enhances chatbot queries for more relevant responses

This integrated approach ensures a consistent experience across platforms while leveraging the unique capabilities of each component.