Widget Implementation

This section gives a brief overview of the implementation of our features on the frontend widget. As a high-level overview, the widget uses an event-driven architecture with reactive front-end components that communicates with our database via our custom API. It manages state by handling user interactions and form submissions.

Frameworks and APIs Used

Our system's whole purpose is to make mapping technology more accessible. That is why we decided to use a variety of already existing mapping technologies and frameworks as well as established frontend designs for a clean final look. Note that all the required API keys should be defined in the .env.local file.

Maps and Location Services

Google Maps API

The main component in our widget is a GoogleMaps map component with functionalities such as street view and satellite view. In each datapoint we also provide a link to Google Maps navigation for easy way-finding. For our clustering feature (where we cluster markers for better view), we also use the @googlemaps/markerclusterer library.

what3words API

Our system is integrated with what3words functionality, allowing users to get the what3words tag for any data point location, for easy memorization and also in case of emergency, quick communication. We also use the w3w API to help us overlay the gird on our map, which we then use to showcase spatial information about a data point.

APIs

Azure Speech Services

We use this API for our text-to-speech and speech-to-text functionality, allowing navigation to data points hands-free.

Frameworks and UI Libraries

React

Used to build a responsive and scalable widget with nice reusable components.

Location and Mapping Functionality

This section describes how our widget handles the display of the map and user on the widget.

User Location and Map Display

When the widget first loads, it tries using the browser's API to get the initial position and updates the user location. This happens in the following useEffect hook:

useEffect(() => {
  if (navigator.geolocation) {
    navigator.geolocation.getCurrentPosition(
      (position) => {
        const { latitude, longitude } = position.coords;
        setUserLocation({ lat: latitude, lng: longitude });
        fetchNearbyLocations(latitude, longitude, 10);
      },
      (error) => {
        console.error("Error getting user location:", error);
        setUserLocation(DEFAULT_LOCATION);
        fetchNearbyLocations(
          DEFAULT_LOCATION.lat,
          DEFAULT_LOCATION.lng,
          10
        );
      }
    );
  } else {
    console.error("Geolocation is not supported by this browser.");
    setUserLocation(DEFAULT_LOCATION);
    fetchNearbyLocations(
      DEFAULT_LOCATION.lat,
      DEFAULT_LOCATION.lng,
      10
    );
  }
}, []);

It updates the userLocation to this new location and proceeds to call the fetchNearbyLocations function which uses our API to get all data points in a 10 mile radius from our website. If this is unsuccessful, the system defaults to our DEFAULT_LOCATION in central London.
The user's location is then displayed by creating a marker with a pulsating effect so it can be easily found on the screen. The marker is created with the help of the Google Maps JavaScript API feature Marker. The marker is created with a pulsating effect for maximum visibility.

// Create a pulsating marker for the user's location
const userMarker = new google.maps.Marker({
  position: userLocation,
  map: mapRef.current,
  icon: {
    path: google.maps.SymbolPath.CIRCLE,
    scale: 10,
    fillColor: "#4285F4",
    fillOpacity: 1,
    strokeColor: "#FFFFFF",
    strokeWeight: 2,
  },
  title: "Your Location",
  zIndex: 1000, // Ensure user marker is on top
});

// Add pulsating effect
const pulsate = (marker) => {
  let opacity = 1;
  let increasing = false;
  
  setInterval(() => {
    if (opacity <= 0.5) {
      increasing = true;
    } else if (opacity >= 1) {
      increasing = false;
    }
    
    opacity = increasing ? opacity + 0.01 : opacity - 0.01;
    
    marker.setIcon({
      ...marker.getIcon(),
      fillOpacity: opacity,
    });
  }, 50);
};

Note that the marker is displayed on the GoogleMaps map component of which useEffect also gets triggered upon initialization. The map is, similarly to the user marker, initialized through the Google Maps API feature new google.maps.Map. To the left of the map component, a sidebar with accessibility features is initialized. These features are described in more detail in the sections below.

Data Layers Functionality

Our system organizes accessibility information into intuitive data layers (such as wheelchair services, zebra crossings) that can be toggled to view on the map and clicked for extra information.

Data Layers Initialization

When the useEffect that navigates the user runs, it calls the fetchNearbyLocations. This function is responsible for retrieving the data points from our database, using our API. Upon successful receivement of the data via the API, the data is stored in the local data structure variable nearbyLocations. Before storing into the nearbyLocations state variable, we also conduct filtering of the data into respective data layers. This is done by checking whether the data-point's location.data_layers field contains the zebra_crossings (or any other data layer) string.

const fetchNearbyLocations = async (lat, lng, radius) => {
  setIsLoading(true);
  try {
    const response = await fetch(
      `${API_ENDPOINTS.NEARBY_LOCATIONS}?lat=${lat}&lng=${lng}&radius=${radius}`
    );
    const data = await response.json();
    
    // Process and categorize locations
    const locations = data.locations || [];
    
    // Filter locations into different categories
    const zebraCrossings = locations.filter(location =>
      location.data_layers.some(layer => layer.name === "zebra_crossings")
    );
    
    const wheelchairServices = locations.filter(location =>
      location.data_layers.some(layer => layer.name === "wheelchair_services")
    );
    
    // Update state with all locations and categorized ones
    setNearbyLocations(locations);
    setZebraCrossingLocations(zebraCrossings);
    setWheelchairServiceLocations(wheelchairServices);
    
    // If locations found, center map on first one
    if (locations.length > 0) {
      // Center map logic...
    }
  } catch (error) {
    console.error("Error fetching nearby locations:", error);
    setErrorMessage("Failed to load location data. Please try again later.");
  } finally {
    setIsLoading(false);
  }
};

When the specific data layer is toggled on, the filtering functionality is again conducted on the nearbyLocations so the correct layer is displayed. The markers for this toggled-on data layer are displayed with similar functionality as the user marker, using the Google Maps API with a custom marker icon that we made in Figma. If clicked on, the markers call the createInfoWindowContent function that displays more precise information about that datapoint (also using the what3words API request ti translate that specific location into a what3words address.)

Marker Clustering

An important accessibility - related feature in our system. It's purpose is to prevent displaying all markers all on top of each when the map is zoomed-out. To achieve a more elegant and efficient solution that prevents having to render hundreds of markers a once, we integrated with the MarkerClusterer library, which displays markers in clusters as a function of how zoomed the user is into the map.

// Create marker clusterer to group markers
if (markerClusterer) {
  markerClusterer.clearMarkers();
}

// Add all markers to the clusterer
const markers = [...zebraCrossingMarkers, ...wheelchairServiceMarkers];
markerClusterer = new MarkerClusterer({
  map: mapRef.current,
  markers: markers,
  algorithm: new SuperClusterAlgorithm({
    radius: 100,
    maxZoom: 15,
  }),
  renderer: {
    render: ({ count, position }) => {
      return new google.maps.Marker({
        position,
        label: { text: String(count), color: "#FFFFFF" },
        icon: {
          path: google.maps.SymbolPath.CIRCLE,
          fillColor: "#4285F4",
          fillOpacity: 0.8,
          strokeWeight: 1,
          strokeColor: "#FFFFFF",
          scale: Math.log10(count) * 10 + 15,
        },
        zIndex: Number(google.maps.Marker.MAX_ZINDEX) - count,
      });
    },
  },
});

Accessibility Features

We've built our widget with accessible design principles from the beginning which ensures that the widget can be set to be tailored to a variety of accessibilities. This section will outline the most important accessibility features in more depth. These have been voted by our testers to be most important for the application, and are the following: high-contrast mode, focus mode, and speech interface.
I will also briefly explain other features such as different font styles and sizes.

High-contrast theme (Dark Mode) and yellow theme (Yellow Mode)

We use a colorMode variable in our code to track the current theme, where we allow 3 different color modes to exist, namely type ColorMode = "light" | "dark" | "yellow"; . Note that dark mode is what we call the black-background yellow-text theme. The 3 different modes can be toggled simply by dedicated buttons with clean icons, for example

<button
  onClick={() => setColorMode("dark")}
  className={`flex-1 px-3 py-2 border-x ${
    colorMode === "dark" ? "bg-gray-300 text-black" : ""
  }`}
>
  <MoonIcon className="h-4 w-4 mx-auto" />
</button>

Note that the specific styles for each of the color modes are defined in the mapStyles section in the code, where essentially the original Google Maps map color schemes are manually set to specific values. Specific designs for these were found on online opensource design pages. Note that the different color themes do not only enforce a change in the map style, but also in all other Ui components. To achieve this, each UI component has a conditional code snippet that applies different styles based on the currently-selected theme, see example below.

<div className={`p-4 space-y-4 h-full ${
  colorMode === "dark" ? "bg-gray-800 text-yellow-300 border-r-2 border-gray-700" 
  : colorMode === "yellow" ? "bg-yellow-100 border-yellow-100" 
  : "bg-white border-white"
}`}

Voice Control Integration

Under the map component, the widget has a Siri-lookalike microphone button. In order to attract attention, the button pulsates and turns red upon being pressed. When the button is pressed, we invoke the Microsoft Cognitive Services Speech SDK via their API, where we also set our language preferences. We receive an initialized voice recognizer and synthesizer.

useEffect(() => {
  const subscriptionKey = ${process.env.NEXT_PUBLIC_AZURE_SPEECH_API_KEY};
  const serviceRegion = "uksouth";
  
  try {
    const speechConfig = SpeechSDK.SpeechConfig.fromSubscription(
      subscriptionKey,
      serviceRegion
    );
    speechConfig.speechRecognitionLanguage = "en-GB";
    speechConfig.speechSynthesisLanguage = "en-GB";
    
    const audioConfig = SpeechSDK.AudioConfig.fromDefaultMicrophoneInput();
    const recognizer = new SpeechSDK.SpeechRecognizer(speechConfig, audioConfig);
    const synthesizer = new SpeechSDK.SpeechSynthesizer(speechConfig);
    // ...

The recognizer and synthesizer objects are then used to interact with the user. First, the speak library function is invoked on the syntehsizer which prompts the user - speak("Please say the data layer you want to select."); . Then, the user's voice commands are processed by the recognizer - first put into lowercase and saved as a string. Then, the string gets checked for any occurences of any data layer words, such as zebra crossing, before toggling that data layer on. This functionality is implemented in the processSpeechCommand function. When listening for specific words, such is the case in the listenForNavigationConfirmation function, a custom function is created that again processes the spoken text by putting it into lowercase and then searching for an occurance of a specific word. In the case of waiting for user's confirmation, for example, we allow the words "yes" and "yeah", see code snippet below. We essentially implement the whole speech functionality with these described functions.

const listenForNavigationConfirmation = (destination: LatLng) => {
  //...
  recognizer.recognizeOnceAsync(
    (result) => {
      const transcript = result.text.toLowerCase();
      
      if (transcript.includes("yes") || transcript.includes("yeah")) {
        speak("Opening Google Maps navigation.");
        window.open(createDirectionsLink(destination, userLocation), "_blank");
      } else {
        speak("Navigation cancelled. The location is still visible on your map.");
    // ...

Focus Mode

A very important (as per claims of the members of the visually impaired community) accessibility feature is the focus mode. What focus mode does is essentially dim the screen apart from a small circle around the cursor which is light. If hovering the "focus" over the map component, the component also grows in size for a magnifying glass effect.
More specifically, we first define the stylistic behavior of this feature in our CSS styling file, where parameters such as the --focus-mode-zoom-factor are set.
After toggling the feature on, we have a useEffect that takes care of mouse tracking with the help of the handleMouseMove function.

const handleMouseMove = (e) => {
  // Store cursor coordinates as CSS variables
  document.documentElement.style.setProperty("--mouse-x", `${e.clientX}px`);
  document.documentElement.style.setProperty("--mouse-y", `${e.clientY}px`);
  // ...
}

Then, the code compares whether the mouse coordinates are above the map container space on the screen and magnifies, centred from the location of the mouse. The map is then scaled up by the CSS transform function - mapElement.style.transform = `scale(var(--focus-mode-zoom-factor))`; .

Other Accessibility Features

Similarly to the different themes, the widget has functionality to toggle between different font styles and sizes. Similarly to the theme functionality, the application maintains a global state and then conditionally checks in UI components whether it is necessary to transform the text to be large / dyslexic font. The toggling buttons are similarly to theme changes, implemented in the sidebar.

What3words and Grid Functionality

Our integration with what3words (addressing system that divides the world into 3x3m squares, each with 3 words associated to it) allows users to easily remember any datapoint location as well as talk about it quickly in case of emergency. The grid functionality allows users to see the datapoint displayed spatially.

What3Words Grid Implementation

We have implemented both the standard w3w 3x3m grid and an enhanced 0.5x0.5m grid for more precise spatial information (for potential smaller data points that will be added later).
The widget provides three grid visualization modes: none, 3x3m (standard what3words grid), and 0.5x0.5m (high-precision grid). When a user activates the 3x3m grid toggle, we fetch data from the what3words API to display the lines. The 0.5x0.5m grid functionality then subdivides each what3words square into smaller 0.5m cells for more detailed info, provided the object is stored in our database under 0.5mx0.5m resolution.
When fetching the grid from what3words, we first get the current map boundaries and make an API request. Then, using the Google Maps API, we draw Polyline objects, overlayed on top of the map.

const fetchAndDisplayW3WGrid = () => {
//...
  
  // We format bounding box for API call
  const boundingBox = `${sw.lat()},${sw.lng()},${ne.lat()},${ne.lng()}`;
  
  fetch(`https://api.what3words.com/v3/grid-section?key=${API_KEY}&bounding-box=${boundingBox}`)
    .then(response => response.json())
    .then(data => {
      // Draw grid lines on the map
      data.lines.forEach(line => {
        const polyline = new google.maps.Polyline({
// ...
}

For the high-precision 0.5x0.5m grid, we subdivide each 3x3m square into a 6×6 grid of smaller cells. When a datapoint marker is pressed, additional information, including the coloured grid, is shown. Because what3words API does not support grid coloring, we build our custom highlightLocationGridCells function, as shown below. Again, we use a Google Maps API Rectangle object to place the object in the exact correct place on the Map object. Note that we found a bug within this functionality, which is described in more detailed in Evaluation - Known Bugs.

const highlightLocationGridCells = (location, color) => {
  // Create a rectangle for the grid cell
  const rectangle = new google.maps.Rectangle({
    bounds: new google.maps.LatLngBounds(
      { lat: location.bottom_left_latitude, lng: location.bottom_left_longitude },
      { lat: location.top_right_latitude, lng: location.top_right_longitude }
    ),
    map: mapRef.current,
    fillColor: cellColor,
    // colors, etc...
  });
  return rectangle;
}

Reporting Accessibility Features

Our widget enables users to report accessibility features at specific locations, creating a community-run crowdsourced database of information. The reporting system works through a modal interface where users can select features present at a location and add additional information.

When a user reports accessibility features, the system updates both the UI immediately and sends the new information in the form of reportData to our API. This functionality happens in the handleSubmitFeatures function, which gets invoked after the user clicks on the "Submit" button on the modal. For good UX practices, we also show a "Thank you message" to confirm the update to the user.

const handleSubmitFeatures = async () => {
  // Show thank you message immediately for better UX
  setShowThankYouMessage(true);

  try {
    // Create report data from user selections
    const reportData = {
      wheelchair_ramp: selectedAccessibilityFeatures.includes("wheelchairRamp"),
      sound_zebra_crossing: selectedAccessibilityFeatures.includes("soundZebraCrossing"),
      // Other features...
    };
    setNearbyLocations(prev => {
      return prev.map(location => {
        if (location.id === currentReportLocationId) {
        // ...
        return location;
      });
    });

    // Send to API asynchronously
    fetch(`/v1/locations/${currentReportLocationId}/reports`, {
      method: "PATCH",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify(reportData)
    });
  } // .. Error Handling
}

In order to try and fix the Report bug (Bug 1), which originates from the createInfoWindowContent function (more indepthly described below), we directly modify the infoWindow to display the new features immediately. This essentially just dynamically changes the html in the infoWindow.
When displaying location information, we render accessibility features with intuitive icons and organize them in a clean list format that is displayed in our infoWindow. The information is generated based on the current state of our data points in the nearbyLocations array. For locations without any reported features, we also show a prompt encouraging users to be the first to report.
The infoWindow is created as follows. The location information and the information about the accessibility features at that data point are fetched from the internal data structure (nearbyLocations ) and not fetched from our database API to improve efficiency.This does mean, that if somebody does report some accessibility features during the time we are using the product, we will not see the updates before reloading the page. In the meantime, the what3words API is also invoked (similarly as in the grid section) and provides the what3words tag for that specific location.

const createInfoWindowContent = (location, userLocation) => {
  // Calculate center of the location, caluclate distance form user...
  const center = {
    lat: (location.bottom_left_latitude + location.top_right_latitude) / 2,
    lng: (location.bottom_left_longitude + location.top_right_longitude) / 2,
  };
  // Calculate distance from user
  // ...

  // Create HTML for accessibility features
  let accessibilityFeaturesHTML = "";
  if (location.total_reports) {
    accessibilityFeaturesHTML = `
      // set up HTML...
    `;
    
    // Add icons for each reported feature
    if (location.wheelchair_ramp_reports) 
      accessibilityFeaturesHTML += "
♿ Wheelchair Ramp
";
    // Add other features...
    
  } else {
    accessibilityFeaturesHTML = `
        
No features reported yet. Be the first...
  }

  // Return the complete HTML content
  return `
    // ...
}