Implementation

Section 1 - Libraries and Tools

Babylon.js

An open-source 3D engine built using WebGL and JavaScript. Our project is based around its 3D scene rendering capabilities.

Google Blockly

A block-based programming library, which is used to modify the Babylon.js scene.

Next.js

We chose Next.js as the framework for our application due to its powerful routing features and performance optimization. It builds on top of React, allowing us to create a fast, scalable, application with minimal configuration.

OpenAI GPT4-o

We chose the GPT4-o model, created OpenAI that has approximately 8 billion parameters. It was benchmarked against four other models (see Research) and demonstrated the best balance between response speed and accuracy.

React Markdown

A React library that converts markdown into a React tree, used to format GPT4-o's responses for the user.

React Resizable Panels

A React library for building resizable, collapsible panels, useful for our complex layout.

Hugging Face Transformers

A JavaScript library for using models from Hugging Face, which we used for our comparisions with GPT, as well as our color name transformer model.

SyntaxHighlighter

A tool for displaying our formatted code with syntax highlighting, making code easier to read.

Section 2 - Babylon.js initialisation

To initialise Babylon.js, we declare an Engine, which handles the rendering, as well as a SceneContext class that handles our scene model (for the Model-View-Controller architecture).

let Eng: Engine;
export let Sc: SceneContext;

export const setupScene = () => {
    Sc = new SceneContext(Eng);
    Sc.attachCamera();
};

We then create a React component, SceneComponent, and integrate it with Babylon.js. We ensure that the engine is only created once when the component mounts so that we can resize the panel components without forcing a reload of the entire Babylon Engine.

export const SceneComponent = ({ ...rest }) => {
    const reactCanvas = useRef(null);

    // Initialise engine and scene only once on component mount
    useLayoutEffect(() => {
        const { current: canvas } = reactCanvas;
        if (!canvas) return;

        Eng = new Engine(canvas, true, {}, true);
        setupScene();
        Eng.runRenderLoop(() => Sc.render());

        // Cleanup on component unmount
        return Eng.dispose;
    }, []);
    
    // Adjust canvas size on window resize
    useLayoutEffect(() => Eng.resize());

    return (
        <canvas style={{ width: "100%", height: "100%" }} ref={reactCanvas} {...rest} />
    );
};

As a default scene, we provide the user with a sun, sky, and pre-compile the Babylon.js material to reduce stuttering:

export class SceneContext {
    scene: Scene;
    reflection: ReflectionProbe;
    sky: SkyMaterial;
    sun: DirectionalLight;

    constructor(engine: AbstractEngine) {
        this.scene = new Scene(engine, {});

        // Set up the sun and sky
        this.sun = new DirectionalLight("__sun", new Vector3(0.0, -1.0, 0.0), this.scene);
        this.sun.intensity = 5.0;

        this.sky = new SkyMaterial("skyMaterial", this.scene);
        this.sky.useSunPosition = true;
        this.sky.sunPosition = this.sun.direction.scale(-1.0);
        const skybox = MeshBuilder.CreateBox("__skyBox", { size: 1000.0, sideOrientation: Mesh.BACKSIDE });
        skybox.material = this.sky;

        // Configure reflection for the skybox
        this.reflection = new ReflectionProbe('ref', 512, this.scene);
        this.reflection.renderList = [skybox];

        // Force shader compilation to reduce first-object stutter
        let mesh = MeshBuilder.CreateBox("forcePipelineCompile", { size: 1 }, this.scene);
        let mat = new PBRMaterial(`${mesh.name}-material`, this.scene);
        mat.albedoColor = new Color3(0.3, 0.3, 0.3);
        mat.ambientColor = Color3.Black();
        mat.emissiveColor = new Color3(0.0, 0.0, 0.0);
        mat.emissiveIntensity = 1.0;
        mat.metallic = 0;
        mat.roughness = 0;
        mat.reflectionTexture = this.reflection.cubeTexture;
        mat.forceCompilation(mesh);
        mesh.material = mat;
    }
}

Finally, attachCamera sets up a free camera with user controls, enabling smooth navigation and interactive features like zooming via the mouse wheel. It also applies tonemapping and various other post-processing effects for a more accurate and filmic look.

attachCamera() {
    let cam = new FreeCamera("camera1", new Vector3(0, 5, -10), this.scene);
    cam.setTarget(Vector3.Zero());
    const canvas = this.scene.getEngine().getRenderingCanvas()!;
    cam.speed = 0.2;

    // Attach control with custom key bindings (WASD for movement, E/Q for vertical)
    cam.attachControl(canvas, true);
    cam.keysUp.push(87);    // "W"
    cam.keysDown.push(83);  // "S"
    cam.keysLeft.push(65);  // "A"
    cam.keysRight.push(68); // "D"
    cam.keysUpward.push(69); // "E"
    cam.keysDownward.push(81); // "Q"

    // Implement zoom functionality via mouse wheel
    const zoomSpeed = 1;
    const handleWheel = (event: WheelEvent) => {
        event.preventDefault();
        if (event.deltaY < 0) {
            const forward = cam.getForwardRay().direction.scale(zoomSpeed);
            cam.position = cam.position.add(forward);
        } else {
            const backward = cam.getForwardRay().direction.scale(zoomSpeed);
            cam.position = cam.position.subtract(backward);
        }
    };
    canvas.addEventListener("wheel", handleWheel, { passive: false });
    this.scene.onDispose = () => canvas.removeEventListener("wheel", handleWheel);

    // Apply post-processing effects for a filmic look
    const pipeline = new DefaultRenderingPipeline("pipe", true, this.scene, [cam]);
    pipeline.imageProcessingEnabled = true;
    pipeline.imageProcessing.toneMappingEnabled = true;
    pipeline.imageProcessing.toneMappingType = TonemappingOperator.Photographic;
    pipeline.bloomEnabled = true;

    // Add a subtle glow effect
    let gl = new GlowLayer("glow", this.scene, {
        mainTextureFixedSize: 1024,
        blurKernelSize: 64,
    });
    gl.intensity = 0.1;
}

Section 3 - Blockly Configuration

The Blockly segment of the code allows our application to offer custom blocks and automatically generate corresponding code based on user interactions.

We define a block definition type, BlockDef, that specifies all the necessary features and properties each individual block should include.

type BlockDef = {
    type: string,
    tooltip: string,
    message0: string,
    args0: any[],
    message1?: string,
    args1?: any[],
    colour?: number,
    nextStatement?: string | string[] | null,
    previousStatement?: string | string[] | null,
    output?: string,
    extensions?: string[],
    gen: (b: Block, g: JavascriptGenerator) => [string, number] | string,
};

For example, here is the hierarchy block category:

const HIERARCHY_CATEGORY: BlockDef[] = [
    {
        type: "label",
        tooltip: "change the label of an object",
        message0: "name this %1",
        args0: [
            {
                type: "field_input",
                name: "name",
            }
        ],
        previousStatement: "top_mesh",
        nextStatement: "top_mesh",
        gen: (b, g) => {
            let name = b.getFieldValue("name")!;
            return `    mesh.name = "${name}";\n`;
        }
    },
    {
        type: "parent_to",
        tooltip: "change the parent of an object",
        message0: "set parent to %1",
        args0: [
            {
                type: "field_input",
                name: "name",
            }
        ],
        previousStatement: ["top_mesh", "mesh"],
        nextStatement: ["top_mesh", "mesh"],
        gen: (b, g) => {
            let name = b.getFieldValue("name")!;
            return `    mesh.parent = scene.getMeshByName("${name}");\n`;
        }
    },
];

Once each category array is initialized, we merge them into a single collection. Their types are prefixed by the category name, and default colours are applied if not provided. This allows us to create a TOOLBOX constant that is the foundation for category bar users see in the bottom of the Blockly workspace.

const CATEGORIES: [string, number, BlockDef[]][] = [
    ["object", 150, OBJECT_CATEGORY],
    ["light", 180, LIGHT_CATEGORY],
    ["material", 80, MATERIAL_CATEGORY],
    ["sky", 220, SKY_CATEGORY],
    ["position", 300, POSITION_CATERGORY],
    ["rotation", 320, ROTATION_CATEGORY],
    ["hierarchy", 340, HIERARCHY_CATEGORY],
    ["animation", 270, ANIMATION_CATEGORY],
    ["texture", 100, TEXTURE_CATEGORY],
];

const CUSTOM_BLOCKS = [...CUSTOM_COLOR, ...CATEGORIES.flatMap(([name, colour, blocks]) => blocks.map((block) => {
    block.type = `${name}_${block.type}`;
    if (!block.colour) block.colour = colour;
    return block;
}))];

const CUSTOM_CATEGORIES = CATEGORIES.map(([name, colour, blocks]) => ({
    kind: "category",
    name,
    colour,
    contents: blocks.map((block: any) => ({ kind: "block", type: block.type })),
}));

const TOOLBOX = {
    kind: "categoryToolbox",
    contents: [
        ...CUSTOM_CATEGORIES,
        {
            kind: "category",
            name: "colour",
            colour: 20,
            contents: [
                { kind: "block", type: "colour_picker" },
                { kind: "block", type: "colour_random" },
                { kind: "block", type: "colour_rgb" },
                { kind: "block", type: "colour_blend" },
                { kind: "block", type: "colour_ai" },
            ],
        },
    ],
};

To convert these visual blocks into executable code, we define a generateCode function, which combines all individually generated code snippets into a single code string. Additonally, we also save our workspace in Local Storage to be able to persist changes on page reloads.

export const generateCode = (workspace: Workspace) => {
	let code = `${preamble}\n\n`;

	let json = serialization.workspaces.save(workspace);
	if (!json.blocks) return code;
	let topBlocks = json.blocks.blocks;

	let headless = new Workspace();
	for (const block of topBlocks) {
		serialization.workspaces.load({ blocks: { languageVersion: 0, blocks: [block] } }, headless);
		code += `{\n${javascriptGenerator.workspaceToCode(headless)}}\n\n`;
	}

	const state = serialization.workspaces.save(workspace);
	try {
		localStorage.setItem("blocklyWorkspace", JSON.stringify(state));
	} catch (_) { }

	return code;
};

To evaluate this generated code, we have a simple runCode function which disposes of all old meshes and lights in the scene - resetting the scene so that the user's generation code can be run, and then runs the user code.

export const runCode = (sc: SceneContext, code: string) => {
	for (const mesh of sc.scene.meshes) {
		if (mesh.name !== "__skyBox")
			mesh.dispose();
	}
	for (const light of sc.scene.lights) {
		if (light.name !== "__sun")
			light.dispose();
	}
	sc.scene.onBeforeRenderObservable.clear();
	sc.sky.sunPosition = new V3(0.0, 1.0, 0.0);
	sc.sun.direction = sc.sky.sunPosition.scale(-1);
	sc.sun.position = sc.sky.sunPosition;

	const MeshBuilder = MB;
	const PointLight = PL;
	const DirectionalLight = DL;
	const Vector3 = V3;
	const Quaternion = Q;
	const Color3 = C3;
	const PBRMaterial = PBRM;
	const Texture = T;
	const Sun = sc.sun;
	const Sky = sc.sky;
	const Reflection = sc.reflection;
	const scene = sc.scene;
	const getTexture = G;

	try {
		eval(code);
	} catch (e) {
		console.log(e);
	}

};

Section 4 - Usage of AI

We used AI in two key ways. First, we integrated a Text-to-Hex model from HuggingFace to convert color names into hex color codes. To address the non-determinism of AI outputs, as well as improve performance, we implemented a cache that stores the model’s responses. This reduces model executions (since it is run locally) and ensures consistent hex values across the application.

import { pipeline, Text2TextGenerationPipeline, env } from "@huggingface/transformers";

let Pipeline: Text2TextGenerationPipeline | null = null;
let Cached: Map<string, string> = new Map();

env.allowRemoteModels = false;
env.allowLocalModels = true;
env.localModelPath = "/models/";
pipeline("text2text-generation", "oddadmix/name-to-hex", {
    device: "wasm",
    dtype: "int8",
}).then(pipe => Pipeline = pipe).catch(console.log);

export const cachePrompt = async (prompt: string) => {
    if (prompt.length == 0) return;
    if (Cached.get(prompt) === undefined && Pipeline !== null) {
        const col = (await Pipeline(prompt))[0].generated_text;
        Cached.set(prompt, col);
    }
}

export const getHexOfText = (prompt: string): string => {
    return Cached.get(prompt) ?? "#ff0000";
}

Our AI assistant follows a similar implementation, but we enhanced its precision by deploying a detailed System Prompt. This prompt enumerates all the application’s features and components, providing the AI with the necessary context to generate highly specific and fine-tuned responses.

export const AiComponentGPT = async (prompt: string) => {
  const completion = await client.chat.completions.create({
    model: "gpt-4o",
    messages: [
      {
        role: "developer",
        content: `You are a helpful assistant in an application that uses a Scratch-like interface to teach 3D programming concepts to children aged 8-12. Users will ask two types of questions:
        
1. **Conceptual Questions** — These ask about 3D modeling concepts (e.g., 'What is a texture?' or 'What does rotation mean in 3D?'). For these, provide a simple, accurate explanation using clear and friendly language suitable for children. Use analogies or examples if helpful.

2. **Instructional Questions** — These ask how to do something in the application (e.g., 'How can I make my object glow?' or 'How do I change the sky?'). These usually start with 'how can I', 'how do I', or similar.

To answer instructional questions, you must understand the application's structure. The app allows users to build 3D scenes using blocks from the following categories:

- **Object**: Create a shape (e.g., cube, sphere).
- **Light**: Create a light (e.g., point or directional light), or change the colour and intensity.
- **Material**: Change the object's appearance using options like color, texture, metallic, roughness, glow color, glow brightness, glow texture.
- **Sky**: Customize the background using options like brightness, haze, latitude, time of day, and time.
- **Position**: Move the object to a specific location.
- **Rotation**: Rotate the object.
- **Hierarchy**: Label objects and allow setting parents of objects to other, labelled objects.
- **Animation**: Apply animations and create repeating behavior with a loop.
- **Texture**: Choose a texture for the object.
- **Colour**: Create colours, or use an AI model to create one from a name.

When a user asks how to do something, identify the most relevant categories and explain which ones to use and why. Keep your response friendly, clear, and step-by-step where needed, always aligned with the block-based, beginner-friendly nature of the app also add spaces and enter characters to make it well formatted and return in MD format. do not add excessive new lines and spaces, the exact format you provide is how it will be printed.
Keep in mind that the application is declarative, so there is no 'Play' or 'Start' button, and that interaction with the scene view does not affect the block coding environment.
`
      },
      {
        role: "user",
        content: prompt
      }
    ],
  });
  return completion.choices[0].message.content;
};