Interactive Video Projection Mapping with Three.js

August 28, 2025

3

Projection mapping has lengthy fascinated audiences within the bodily world, turning buildings, sculptures, and whole cityscapes into transferring canvases. What in the event you might recreate that very same sense of spectacle instantly contained in the browser? With WebGL and Three.js, you possibly can mission video not onto partitions or monuments however onto dynamic 3D grids product of lots of of cubes, each carrying a fraction of the video like a digital mosaic.

On this tutorial we’ll discover methods to simulate video projection mapping in a purely digital surroundings, from constructing a grid of cubes, to UV-mapping video textures, to making use of masks that decide which cubes seem. The result’s a mesmerizing impact that feels each sculptural and cinematic, good for interactive installations, portfolio showcases, or just as a playground to push your inventive coding expertise additional.

What’s Video Projection Mapping within the Actual World?

When describing video projection mapping, it’s best to consider big buildings lit up with animations throughout festivals, or artwork installations the place a transferring picture is “painted” onto sculptures.

Listed here are some examples of real-world video projections:

Bringing it to our 3D World

In 3D graphics, we are able to do one thing comparable: as an alternative of shining a bodily projector, we map a video texture onto objects in a scene.

Due to this fact, let’s construct a grid of cubes utilizing a masks picture that can decide which cubes are seen. A video texture is UV-mapped so every dice reveals the precise video fragment that corresponds to its grid cell—collectively they reconstruct the video, however solely the place the masks is darkish.

Prerequesites:

Three.js r155+
A small, high-contrast masks picture (e.g. a coronary heart silhouette).
A video URL with CORS enabled.

Our Boilerplate and Beginning Level

Here’s a primary starter setup, i.e. the minimal quantity of code and construction it’s worthwhile to get a scene rendering within the browser, with out worrying concerning the particular inventive content material but.

export default class Fashions {
	constructor(gl_app) {
        ...
        this.createGrid()
    }

    createGrid() {
        const geometry = new THREE.BoxGeometry( 1, 1, 1 );
        this.materials = new THREE.MeshStandardMaterial( { shade: 0xff0000 } );
        const dice = new THREE.Mesh( geometry, this.materials );
        this.group.add( dice );
        this.is_ready = true
    }
    
    ...
}

The result’s a spinning purple dice:

Creating the Grid

A centered grid of cubes (10×10 by default). Each dice has the identical dimension and materials. The grid spacing and total scale are configurable.

export default class Fashions {
	constructor(gl_app) {
        ...

		this.gridSize = 10;
        this.spacing = 0.75;
        this.createGrid()
    }

    createGrid() {
        this.materials = new THREE.MeshStandardMaterial( { shade: 0xff0000 } );
        
        // Grid parameters
        for (let x = 0; x < this.gridSize; x++) {
            for (let y = 0; y < this.gridSize; y++) {
                const geometry = new THREE.BoxGeometry(0.5, 0.5, 0.5);
                const mesh = new THREE.Mesh(geometry, this.materials);
                mesh.place.x = (x - (this.gridSize - 1) / 2) * this.spacing;
                mesh.place.y = (y - (this.gridSize - 1) / 2) * this.spacing;
                mesh.place.z = 0;

                this.group.add(mesh);
            }
        }
        this.group.scale.setScalar(0.5)
        ...
    }   
    ...
}

Key parameters

World-space distance between dice facilities. Improve for bigger gaps, lower to pack tighter.

What number of cells per aspect. A ten×10 grid ⇒ 100 cubes

Creating the Video Texture

This operate creates a video texture in Three.js so you need to use a taking part in HTML <video> as the feel on 3D objects.

Creates an HTML <video> aspect totally in JavaScript (not added to the DOM).
We’ll feed this aspect to Three.js to make use of its frames as a texture.
loop = true → restarts routinely when it reaches the tip.
muted = true → most browsers block autoplay for unmuted movies, so muting ensures it performs with out consumer interplay.
.play() → begins playback.
⚠️ Some browsers nonetheless want a click on/contact earlier than autoplay works — you possibly can add a fallback listener if wanted.

export default class Fashions {
	constructor(gl_app) {
        ...
        this.createGrid()
    }

    createVideoTexture() {
		this.video = doc.createElement('video')
		this.video.src = 'https://commondatastorage.googleapis.com/gtv-videos-bucket/pattern/BigBuckBunny.mp4'
		this.video.crossOrigin = 'nameless'
		this.video.loop = true
		this.video.muted = true
		this.video.play()

		// Create video texture
		this.videoTexture = new THREE.VideoTexture(this.video)
		this.videoTexture.minFilter = THREE.LinearFilter
		this.videoTexture.magFilter = THREE.LinearFilter
		this.videoTexture.colorSpace = THREE.SRGBColorSpace
		this.videoTexture.wrapS = THREE.ClampToEdgeWrap
		this.videoTexture.wrapT = THREE.ClampToEdgeWrap

		// Create materials with video texture
		this.materials = new THREE.MeshBasicMaterial({ 
			map: this.videoTexture,
			aspect: THREE.FrontSide
		})
    }

    createGrid() {
        this.createVideoTexture()
        ...
    }
    ...
}

That is the video we’re utilizing: Huge Buck Bunny (with out CORS)

All of the meshes have the identical texture utilized:

Attributing Projection to the Grid

We will probably be turning the video right into a texture atlas cut up right into a gridSize × gridSize lattice.
Every dice within the grid will get its personal little UV window (sub-rectangle) of the video so, collectively, all cubes reconstruct the total body.

Why per-cube geometry? As a result of we are able to create a brand new BoxGeometry for every dice because the UVs should be distinctive per dice. If all cubes shared one geometry, they’d additionally share the identical UVs and present the identical a part of the video.

export default class Fashions {
	constructor(gl_app) {
        ...
        this.createGrid()
    }

    createGrid() {
        ...
		// Grid parameters
        for (let x = 0; x < this.gridSize; x++) {
            for (let y = 0; y < this.gridSize; y++) {
                
                const geometry = new THREE.BoxGeometry(0.5, 0.5, 0.5);
                
				// Create particular person geometry for every field to have distinctive UV mapping
				// Calculate UV coordinates for this particular field
				const uvX = x / this.gridSize
				const uvY = y / this.gridSize // Take away the flip to match appropriate orientation
				const uvWidth = 1 / this.gridSize
				const uvHeight = 1 / this.gridSize
				
				// Get the UV attribute
				const uvAttribute = geometry.attributes.uv
				const uvArray = uvAttribute.array
				
				// Map every face of the field to point out the identical portion of video
				// We'll give attention to the entrance face (face 4) for the principle projection
				for (let i = 0; i < uvArray.size; i += 2) {
					// Map all faces to the identical UV area for consistency
					uvArray[i] = uvX + (uvArray[i] * uvWidth)     // U coordinate
					uvArray[i + 1] = uvY + (uvArray[i + 1] * uvHeight) // V coordinate
				}
				
				// Mark the attribute as needing replace
				uvAttribute.needsUpdate = true
                ...
            }
        }
        ...
    }
    ...
}

The UV window for cell (x, y)
For a grid of dimension N = gridSize:

UV origin of this cell:
– uvX = x / N
– uvY = y / N
UV dimension of every cell:
– uvWidth = 1 / N
– uvHeight = 1 / N

Outcome: each face of the field now samples the identical sub-region of the video (and we famous “give attention to the entrance face”; this method maps all faces to that area for consistency).

Creating Masks

We have to create a canvas utilizing a masks that determines which cubes are seen within the grid.

Black (darkish) pixels → dice is created.
White (gentle) pixels → dice is skipped.

To do that, we have to:

Load the masks picture.
Scale it all the way down to match our grid dimension.
Learn its pixel shade information.
Move that information into the grid-building step.

export default class Fashions {
	constructor(gl_app) {
        ...
		this.createMask()
    }

	createMask() {
        // Create a canvas to learn masks pixel information
        const canvas = doc.createElement('canvas')
        const ctx = canvas.getContext('2nd')

        const maskImage = new Picture()
        maskImage.crossOrigin = 'nameless'
        maskImage.onload = () => {
            // Get unique picture dimensions to protect side ratio
            const originalWidth = maskImage.width
            const originalHeight = maskImage.top
            const aspectRatio = originalWidth / originalHeight

            // Calculate grid dimensions based mostly on side ratio
            this.gridWidth
			this.gridHeight
            if (aspectRatio > 1) {
                // Picture is wider than tall
                this.gridWidth = this.gridSize
                this.gridHeight = Math.spherical(this.gridSize / aspectRatio)
            } else {
                // Picture is taller than extensive or sq.
                this.gridHeight = this.gridSize
                this.gridWidth = Math.spherical(this.gridSize * aspectRatio)
            }

            canvas.width = this.gridWidth
            canvas.top = this.gridHeight
            ctx.drawImage(maskImage, 0, 0, this.gridWidth, this.gridHeight)

            const imageData = ctx.getImageData(0, 0, this.gridWidth, this.gridHeight)
            this.information = imageData.information
			this.createGrid()
		}

        maskImage.src = '../photographs/coronary heart.jpg'
	}
    ...
}

Match masks decision to grid

We don’t wish to stretch the masks — this retains it proportional to the grid.
gridWidth and gridHeight are what number of masks pixels we’ll pattern horizontally and vertically.
This matches the logical dice grid, so every dice can correspond to at least one pixel within the masks.

Making use of the Masks to the Grid

Let’s combines mask-based filtering with customized UV mapping to resolve the place within the grid packing containers ought to seem, and how every field maps to a bit of the projected video.
Right here’s the idea step-by-step:

Loops by means of each potential (x, y) place in a digital grid.
At every grid cell, it can resolve whether or not to position a field and, in that case, methods to texture it.
flippedY: Flips the Y-axis as a result of picture coordinates begin from the top-left, whereas the grid’s origin begins from the bottom-left.
pixelIndex: Locates the pixel within the this.information array.
Every pixel shops 4 values: purple, inexperienced, blue, alpha.
Extracts the R, G, and B values for that masks pixel.
Brightness is calculated as the typical of R, G, B.
If the pixel is darkish sufficient (brightness < 128), a dice will probably be created.
White pixels are ignored → these positions keep empty.

export default class Fashions {
	constructor(gl_app) {
        ...
		this.createMask()
    }

	createMask() {
        ...
	}

    createGrid() {
        ...
        for (let x = 0; x < this.gridSize; x++) {
            for (let y = 0; y < this.gridSize; y++) {
                
                const geometry = new THREE.BoxGeometry(0.5, 0.5, 0.5);

                // Get pixel shade from masks (pattern at grid place)
                // Flip Y coordinate to match picture orientation
                const flippedY = this.gridHeight - 1 - y
                const pixelIndex = (flippedY * this.gridWidth + x) * 4
                const r = this.information[pixelIndex]
                const g = this.information[pixelIndex + 1]
                const b = this.information[pixelIndex + 2]

                // Calculate brightness (0 = black, 255 = white)
                const brightness = (r + g + b) / 3

                // Solely create field if pixel is darkish (black reveals, white hides)
                if (brightness < 128) { // Threshold for black vs white

                    // Create particular person geometry for every field to have distinctive UV mapping
                    // Calculate UV coordinates for this particular field
                    const uvX = x / this.gridSize
                    const uvY = y / this.gridSize // Take away the flip to match appropriate orientation
                    const uvWidth = 1 / this.gridSize
                    const uvHeight = 1 / this.gridSize
                    
                    // Get the UV attribute
                    const uvAttribute = geometry.attributes.uv
                    const uvArray = uvAttribute.array
                    
                    // Map every face of the field to point out the identical portion of video
                    // We'll give attention to the entrance face (face 4) for the principle projection
                    for (let i = 0; i < uvArray.size; i += 2) {
                        // Map all faces to the identical UV area for consistency
                        uvArray[i] = uvX + (uvArray[i] * uvWidth)     // U coordinate
                        uvArray[i + 1] = uvY + (uvArray[i + 1] * uvHeight) // V coordinate
                    }
                    
                    // Mark the attribute as needing replace
                    uvAttribute.needsUpdate = true
                    
                    const mesh = new THREE.Mesh(geometry, this.materials);

                    mesh.place.x = (x - (this.gridSize - 1) / 2) * this.spacing;
                    mesh.place.y = (y - (this.gridSize - 1) / 2) * this.spacing;
                    mesh.place.z = 0;

                    this.group.add(mesh);
                }
            }
        }
        ...
    }
    ...
}

Additional steps

UV mapping is the method of mapping 2D video pixels onto 3D geometry.
Every dice will get its personal distinctive UV coordinates equivalent to its place within the grid.
uvWidth and uvHeight are how a lot of the video texture every dice covers.
Modifies the dice’s uv attribute so all faces show the very same portion of the video.

Right here is the end result with the masks utilized:

Including Some Depth and Movement to the Grid

Including refined movement alongside the Z-axis brings the in any other case static grid to life, making the projection really feel extra dynamic and dimensional.

replace() {
    if (this.is_ready) {
        this.group.kids.forEach((mannequin, index) => {
            mannequin.place.z = Math.sin(Date.now() * 0.005 + index * 0.1) * 0.6
        })
    }
}

It’s the time for A number of Grids

Up till now we’ve been working with a single masks and a single video, however the actual enjoyable begins once we begin layering a number of projections collectively. By combining totally different masks photographs with their very own video sources, we are able to create a set of unbiased grids that coexist in the identical scene. Every grid can carry its personal identification and movement, opening the door to richer compositions, transitions, and storytelling results.

1. A Playlist of Masks and Movies

export default class Fashions {
	constructor(gl_app) {
        ...
        this.grids_config = [
            {
                id: 'heart',
                mask: `heart.jpg`,
                video: `fruits_trail_squared-transcode.mp4`
            },
            {
                id: 'codrops',
                mask: `codrops.jpg`,
                video: `KinectCube_1350-transcode.mp4`
            },
            {
                id: 'smile',
                mask: `smile.jpg`,
                video: `infinte-grid_squared-transcode.mp4`
            },
        ]
        this.grids_config.forEach((config, index) => this.createMask(config, index))
        this.grids = []
    }
...
}

As an alternative of 1 masks and one video, we now have a listing of mask-video pairs.

Every object defines:

id → identify/id for every grid.
masks → the black/white picture that controls which cubes seem.
video → the feel that will probably be mapped onto these cubes.

This lets you have a number of totally different projections in the identical scene.

2. Looping Over All Grids

As soon as we now have our playlist of masks–video pairs outlined, the subsequent step is to undergo every merchandise and put together it for rendering.

For each configuration within the listing we name createMask(config, index), which takes care of loading the masks picture, studying its pixels, after which passing the info alongside to construct the corresponding grid.

On the similar time, we hold monitor of all of the grids by storing them in a this.grids array, so afterward we are able to animate them, present or disguise them, and change between them interactively.

3. createMask(config, index)

createMask(config, index) {
    ...
    maskImage.onload = () => {
        ...
        this.createGrid(config, index)
    }
    maskImage.src = `../photographs/${config.masks}`
}

Hundreds the masks picture for the present grid.
When the picture is loaded, runs the masks pixel-reading logic (as defined earlier than) after which calls createGrid() with the identical config and index.
The masks determines which cubes are seen for this particular grid.

4. createVideoTexture(config, index)

createVideoTexture(config, index) {
    this.video = doc.createElement('video')
    this.video.src = `../movies/${config.video}`
    ...
}

Creates a <video> aspect utilizing the particular video file for this grid.
The video is then transformed to a THREE.VideoTexture and assigned as the fabric for the cubes on this grid.
Every grid can have its personal unbiased video taking part in.

5. createGrid(config, index)

createGrid(config, index) {
        this.createVideoTexture(config, index)
        const grid_group = new THREE.Group()
        this.group.add(grid_group)

        for (let x = 0; x < this.gridSize; x++) {
            for (let y = 0; y < this.gridSize; y++) {
                    ...
                    grid_group.add(mesh);
            }
        }
        grid_group.identify = config.id
        this.grids.push(grid_group);
        grid_group.place.z = - 2 * index 
        ...
    }

Creates a brand new THREE.Group for this grid so all its cubes will be moved collectively.
This retains every masks/video projection remoted.
grid_group.identify: Assigns a reputation (you would possibly later use config.id right here).
this.grids.push(grid_group): Shops this grid in an array so you possibly can management it later (e.g., present/disguise, animate, change movies).
grid_group.place.z: Offsets every grid additional again in Z-space in order that they don’t overlap visually.

And right here is the end result for the a number of grids:

And eventually: Interplay & Animations

Let’s begin by making a easy UI with some buttons on our HTML:

<ul class="btns">
	<li class="btns__item">
		<button class="lively" data-id="coronary heart">
			...
		</button>
	</li>
	<li class="btns__item">
		<button data-id="codrops">
			...
		</button>
	</li>
	<li class="btns__item">
		<button data-id="smile">
			...
		</button>
	</li>
</ul>

We’ll additionally create a data-current="coronary heart" to our canvas aspect, will probably be vital to vary its background-color relying on which button was clicked.

<canvas id="sketch" data-current="coronary heart"></canvas>

Let’s not create some colours for every grid utilizing CSS:

[data-current="heart"] {
	background-color: #e19800;
}

[data-current="codrops"] {
	background-color: #00a00b
}

[data-current="smile"] {
	background-color: #b90000;
}

Time to use to create the interactions:

createGrid(config, index) {
    ...
    this.initInteractions()
}

1. this.initInteractions()

initInteractions() {
    this.present = 'coronary heart'
    this.previous = null
    this.is_animating = false
    this.length = 1

    this.DOM = {
        $btns: doc.querySelectorAll('.btns__item button'),
        $canvas: doc.querySelector('canvas')
    }
    this.grids.forEach(grid => {
        if(grid.identify != this.present) {
            grid.kids.forEach(mesh => mesh.scale.setScalar(0))
        }
    })
    this.bindEvents()
}

this.present → The at present lively grid ID. Begins as "coronary heart" so the "coronary heart" grid will probably be seen by default.
this.previous → Used to retailer the earlier grid ID when switching between grids.
this.is_animating → Boolean flag to forestall triggering a brand new transition whereas one remains to be working.
this.length → How lengthy the animation takes (in seconds).
$btns → Selects all of the buttons inside .btns__item. Every button seemingly corresponds to a grid you possibly can change to.
$canvas → Selects the principle <canvas> aspect the place the Three.js scene is rendered.

Loops by means of all of the grids within the scene.

If the grid is not the present one (grid.identify != this.present),
→ It units all of that grid’s cubes (mesh) to scale = 0 so they’re invisible firstly.
This implies solely the "coronary heart" grid will probably be seen when the scene first hundreds.

2. bindEvents()

bindEvents() {
    this.DOM.$btns.forEach(($btn, index) => {
        $btn.addEventListener('click on', () => {
            if (this.is_animating) return
            this.is_animating = true
            this.DOM.$btns.forEach(($btn, btnIndex) => {
                btnIndex === index ? $btn.classList.add('lively') : $btn.classList.take away('lively')
            })
            this.previous = this.present
            this.present = `${$btn.dataset.id}`
            this.revealGrid()
            this.hideGrid()
        })
    })
}

This bindEvents() technique wires up the UI buttons in order that clicking one will set off switching between grids within the 3D scene.

For every button, connect a click on occasion handler.
If an animation is already working, do nothing — this prevents beginning a number of transitions on the similar time.
Units is_animating to true so no different clicks are processed till the present change finishes.

Loops by means of all buttons once more:

If that is the clicked button → add the lively CSS class (spotlight it).
In any other case → take away the lively class (unhighlight).
this.previous → retains monitor of which grid was seen earlier than the clicking.
this.present → updates to the brand new grid’s ID based mostly on the button’s data-id attribute.
- Instance: if the button has data-id="coronary heart", this.present turns into "coronary heart".

Calls two separate strategies:

revealGrid() → makes the newly chosen grid seem (by scaling its cubes from 0 to full dimension).
hideGrid() → hides the earlier grid (by scaling its cubes again all the way down to 0).

3. revealGrid() & hideGrid()

revealGrid() {
    // Filter the present grid based mostly on this.present worth
    const grid = this.grids.discover(merchandise => merchandise.identify === this.present);
    
    this.DOM.$canvas.dataset.present = `${this.present}` 
    const tl = gsap.timeline({ delay: this.length * 0.25, defaults: { ease: 'power1.out', length: this.length } })
    grid.kids.forEach((youngster, index) => {
        tl
            .to(youngster.scale, { x: 1, y: 1, z: 1, ease: 'power3.inOut' }, index * 0.001)
            .to(youngster.place, { z: 0 }, '<')
    })
}

hideGrid() {
    // Filter the present grid based mostly on this.previous worth
    const grid = this.grids.discover(merchandise => merchandise.identify === this.previous);
    const tl = gsap.timeline({
        defaults: { ease: 'power1.out', length: this.length },
        onComplete: () => { this.is_animating = false }
    })
    grid.kids.forEach((youngster, index) => {
        tl
            .to(youngster.scale, { x: 0, y: 0, z: 0, ease: 'power3.inOut' }, index * 0.001)
            .to(youngster.place, {
                z: 6, onComplete: () => {
                    gsap.set(youngster.scale, { x: 0, y: 0, z: 0 })
                    gsap.set(youngster.place, { z: - 6 })
                }
            }, '<')
    })
}

And that’s it! A full animated and interactive Video Projection Slider, made with lots of of small cubes (meshes).

⚠️ Perfomance issues

The method used on this tutorial, is the best and extra digestable technique to apply the projection idea; Nonetheless, it will possibly create too many draw calls: 100–1,000 cubes would possibly fantastic; tens of hundreds will be sluggish. In the event you want extra detailed grid or extra meshes on it, think about InstancedMesh and Shaders.

Going additional

This a totally practical and versatile idea; Due to this fact, it opens so many prospects.
Which will be utilized in some actually cool methods, like scrollable story-telling, exhibition simulation, intro animations, portfolio showcase and and so forth.

Listed here are some hyperlinks so that you can get impressed:

Last Phrases

I hope you’ve loved this tutorial, and provides a attempt in your tasks or simply discover the probabilities by altering the grid parameters, masks and movies.

And speaking concerning the movies, these used on this instance are screen-recording of the Inventive Code classes contained in my Net Animations platform vwlab.io, the place you possibly can discover ways to create extra interactions and animations like this one.

Come be a part of us, you’ll be greater than welcome! ☺️❤️

Supply hyperlink