Grid Displacement Texture with RGB Shift utilizing Three.js GPGPU and Shaders

On this tutorial, you’ll learn to create a pixel/grid displacement impact utilizing Three.js, enhanced with shaders and GPGPU strategies. The information covers the appliance of a delicate RGB shift impact that dynamically responds to cursor motion. By the top, you’ll achieve a strong understanding of manipulating textures and creating interactive visible results in WebGL, increasing your inventive capabilities with Three.js.

It’s beneficial that you’ve got some fundamental understanding of Three.js and WebGL for understanding this tutorial. Let’s dive in!

The Setup

To create this impact, we are going to want two textures: the primary is the picture we wish to apply the impact to, and the second is a texture containing the info for our impact. Right here’s how the second texture will look:

First, we are going to create a fundamental Three.js airplane with a ShaderMaterial that may show our picture and add it to our Three.js scene.

createGeometry() {
    this.geometry = new THREE.PlaneGeometry(1, 1)
  }

  createMaterial() {
    this.materials = new THREE.ShaderMaterial({
      vertexShader,
      fragmentShader,
      uniforms: {
        uTexture: new THREE.Uniform(new THREE.Vector4()),
        uContainerResolution: new THREE.Uniform(new THREE.Vector2(window.innerWidth, window.innerHeight)),
        uImageResolution: new THREE.Uniform(new THREE.Vector2()),
      },
    })
  }

  setTexture() {
    this.materials.uniforms.uTexture.worth = new THREE.TextureLoader().load(this.factor.src, ({ picture }) => {
      const { naturalWidth, naturalHeight } = picture
      this.materials.uniforms.uImageResolution.worth = new THREE.Vector2(naturalWidth, naturalHeight)
    })
  }

  createMesh() {
    this.mesh = new THREE.Mesh(this.geometry, this.materials)
  }

I handed the viewport dimensions to the uContainerResolution uniform as a result of my mesh occupies your entire viewport area. If you’d like your picture to have a distinct dimension, you have to to go the width and top of the HTML factor containing the picture.

Right here is the vertex shader code, which is able to stay unchanged since we’re not going to switch the vertices.

various vec2 vUv;

void predominant()
{
    vec4 modelPosition = modelMatrix * vec4(place, 1.0);
    vec4 viewPosition = viewMatrix * modelPosition;
    vec4 projectedPosition = projectionMatrix * viewPosition;
    gl_Position = projectedPosition;    

    vUv=uv;
}

And right here is the preliminary fragment shader:

uniform sampler2D uTexture;

various vec2 vUv;
uniform vec2 uContainerResolution;
uniform vec2 uImageResolution;


vec2 coverUvs(vec2 imageRes,vec2 containerRes)
{
    float imageAspectX = imageRes.x/imageRes.y;
    float imageAspectY = imageRes.y/imageRes.x;
    
    float containerAspectX = containerRes.x/containerRes.y;
    float containerAspectY = containerRes.y/containerRes.x;

    vec2 ratio = vec2(
        min(containerAspectX / imageAspectX, 1.0),
        min(containerAspectY / imageAspectY, 1.0)
    );

    vec2 newUvs = vec2(
        vUv.x * ratio.x + (1.0 - ratio.x) * 0.5,
        vUv.y * ratio.y + (1.0 - ratio.y) * 0.5
    );

    return newUvs;
}


void predominant()
{
    vec2 newUvs = coverUvs(uImageResolution,uContainerResolution);            
    
    vec4 picture = texture2D(uTexture,newUvs);    

    gl_FragColor = picture;
}

The coverUvs perform returns a set of UVs that may make the picture texture wrap behave just like the CSS object-fit: cowl; property. Right here is the outcome:

Implementing Displacement with GPGPU

Now we’re going to implement the displacement texture in a separate shader, and there’s a motive for this: we are able to’t depend on basic Three.js shaders to use our impact.

As you noticed within the video of the displacement texture, there’s a path following the mouse motion that slowly fades out when the mouse leaves the realm. We will’t create this impact in our present shader as a result of the info shouldn’t be persistent. The shader runs at every body utilizing its preliminary inputs (uniforms and varyings), and there’s no approach to entry the earlier state.

Thankfully, Three.js gives a utility referred to as GPUComputationRenderer. It permits us to output a computed fragment shader as a texture and use this texture because the enter of our shader within the subsequent body. That is referred to as a Buffer Texture. Right here’s the way it works:

First, we’re going to initialize the GPUComputationRenderer occasion. For that, I’ll create a category referred to as GPGPU.

import fragmentShader from '../shaders/gpgpu/gpgpu.glsl' 
// the fragment shader we're going to use within the gpgpu

// ...class constructor
createGPGPURenderer() {
    this.gpgpuRenderer = new GPUComputationRenderer(
      this.dimension, //the scale of the grid we wish to create, within the instance the scale is 27
      this.dimension,
      this.renderer //the WebGLRenderer we're utilizing for our scene
    )
  }
  createDataTexture() {
    this.dataTexture = this.gpgpuRenderer.createTexture()
  }

  createVariable() {
    this.variable = this.gpgpuRenderer.addVariable('uGrid', fragmentShader, this.dataTexture)
    this.variable.materials.uniforms.uGridSize = new THREE.Uniform(this.dimension)
    this.variable.materials.uniforms.uMouse = new THREE.Uniform(new THREE.Vector2(0, 0))
    this.variable.materials.uniforms.uDeltaMouse = new THREE.Uniform(new THREE.Vector2(0, 0))
  }

  setRendererDependencies() {
    this.gpgpuRenderer.setVariableDependencies(this.variable, [this.variable])
  }

  initiateRenderer() {
    this.gpgpuRenderer.init()
  }

That is just about a generic instantiation code for a GPUComputationRenderer occasion.

1. We create the occasion in createGPGPURenderer.
2. We create a DataTexture object in createDataTexture, which can be populated with the results of the computed shader.
3. We create a “variable” in createVariable. This time period is utilized by GPUComputationRenderer to consult with the feel we’re going to output. I suppose it’s referred to as that as a result of our texture goes to fluctuate at every body in response to our computations.
4. We set the dependencies of the GPGPU.
5. We initialize our occasion.

Now we’re going to create the fragment shader that our GPGPU will use.

void predominant()
{
    vec2 uv = gl_FragCoord.xy/decision.xy;

    vec4 shade = texture(uGrid,uv);

    shade.r = 1.;
    
    gl_FragColor = shade;
}

The present texture that our GPGPU is creating is a plain purple picture. Discover that we didn’t must declare uniform sampler2D uGrid within the header of the shader as a result of we declared it as a variable of the GPUComputationRenderer occasion.

Now we’re going to retrieve the feel and apply it to our picture.

Right here is the whole code for our GPGPU class.

constructor({ renderer, scene }: Props) {
    this.scene = scene
    this.renderer = renderer

    this.params = {
      dimension: 700,
    }

    this.dimension = Math.ceil(Math.sqrt(this.params.dimension))
    this.time = 0

    this.createGPGPURenderer()
    this.createDataTexture()
    this.createVariable()
    this.setRendererDependencies()
    this.initiateRenderer()
  }

  createGPGPURenderer() {
    this.gpgpuRenderer = new GPUComputationRenderer(
      this.dimension, //the scale of the grid we wish to create, within the instance the scale is 27
      this.dimension,
      this.renderer //the WebGLRenderer we're utilizing for our scene
    )
  }
  createDataTexture() {
    this.dataTexture = this.gpgpuRenderer.createTexture()
  }

  createVariable() {
    this.variable = this.gpgpuRenderer.addVariable('uGrid', fragmentShader, this.dataTexture)
    this.variable.materials.uniforms.uGridSize = new THREE.Uniform(this.dimension)
    this.variable.materials.uniforms.uMouse = new THREE.Uniform(new THREE.Vector2(0, 0))
    this.variable.materials.uniforms.uDeltaMouse = new THREE.Uniform(new THREE.Vector2(0, 0))
  }

  setRendererDependencies() {
    this.gpgpuRenderer.setVariableDependencies(this.variable, [this.variable])
  }

  initiateRenderer() {
    this.gpgpuRenderer.init()
  }


  getTexture() {
    return this.gpgpuRenderer.getCurrentRenderTarget(this.variable).textures[0]
  }

  render() {
    this.gpgpuRenderer.compute()
  }

The render technique can be referred to as every body, and the getTexture technique will return our computed texture.

Within the materials of the primary airplane we created, we are going to add a uGrid uniform. This uniform will comprise the feel retrieved by the GPGPU.

createMaterial() {
    this.materials = new THREE.ShaderMaterial({
      vertexShader,
      fragmentShader,
      uniforms: {
        uTexture: new THREE.Uniform(new THREE.Vector4()),        
        uContainerResolution: new THREE.Uniform(new THREE.Vector2(window.innerWidth, window.innerHeight)),
        uImageResolution: new THREE.Uniform(new THREE.Vector2()),


        //add this new Uniform
        uGrid: new THREE.Uniform(new THREE.Vector4()),
      },
    })
  }

Now we’re going to replace this uniform in every body after computing the GPGPU texture,

render() {
    this.gpgpu.render()
    this.materials.uniforms.uGrid.worth = this.gpgpu.getTexture()
}

Now, contained in the fragment shader of our first picture airplane, let’s show this texture.

uniform sampler2D uGrid;

void predominant()
{
    vec2 newUvs = coverUvs(uImageResolution,uContainerResolution);            

    vec4 picture = texture2D(uTexture,newUvs);    
    vec4 displacement = texture2D(uGrid,newUvs);

    gl_FragColor = displacement;
}

It is best to see this outcome. That is precisely what we wish. Keep in mind, all our GPGPU is doing for now could be setting an empty texture to purple.

Dealing with Mouse Motion

Now we’re going to begin engaged on the displacement impact. First, we have to monitor mouse motion and go it as a uniform to the GPGPU shader.

We are going to create a Raycaster and go the mouse UVs to the GPGPU. Since we solely have one mesh in our scene for this instance, the one UVs it should return can be these of our airplane containing the picture.

createRayCaster() {
    this.raycaster = new THREE.Raycaster()
    this.mouse = new THREE.Vector2()
  }

  onMouseMove(occasion: MouseEvent) {
    this.mouse.x = (occasion.clientX / window.innerWidth) * 2 - 1
    this.mouse.y = -(occasion.clientY / window.innerHeight) * 2 + 1

    this.raycaster.setFromCamera(this.mouse, this.digicam)
    
    const intersects = this.raycaster.intersectObjects(this.scene.youngsters)
    const goal = intersects[0]
    if (goal && 'materials' in goal.object) {
      const targetMesh = intersects[0].object as THREE.Mesh
      
      if(targetMesh && goal.uv)
      {
        this.gpgpu.updateMouse(goal.uv)
      }    
    }
  }

  addEventListeners() {
    window.addEventListener('mousemove', this.onMouseMove.bind(this))
  }

Keep in mind that within the createVariable technique of the GPGPU, we assigned it a uniform uMouse. We’re going to replace this uniform within the updateMouse technique of the GPGPU class. We may also replace the uDeltaMouse uniform (we are going to want it quickly).

updateMouse(uv: THREE.Vector2) {

    const present = this.variable.materials.uniforms.uMouse.worth as THREE.Vector2

    present.subVectors(uv, present)

    this.variable.materials.uniforms.uDeltaMouse.worth = present
    this.variable.materials.uniforms.uMouse.worth = uv
  }

Now, within the GPGPU fragment shader, we are going to retrieve the mouse coordinates to calculate the space between every pixel of the feel and the mouse. We are going to then apply the mouse delta to the feel primarily based on this distance.

uniform vec2 uMouse;
uniform vec2 uDeltaMouse;


void predominant()
{
    vec2 uv = gl_FragCoord.xy/decision.xy;

    vec4 shade = texture(uGrid,uv);

    float dist = distance(uv,uMouse);
    dist = 1.-(smoothstep(0.,0.22,dist));


    shade.rg+=uDeltaMouse*dist;
    
    gl_FragColor = shade;
}

It is best to get one thing like this:

Discover that if you transfer your cursor from left to proper, it’s coloring, and if you transfer it from proper to left, you might be erasing. It is because the delta of the UVs is unfavorable if you go from proper to left and optimistic the opposite method round.

You’ll be able to type of see the place that is going. Clearly, we’re not going to show our displacement texture; we wish to apply it to our preliminary picture. The present texture we’ve got is much from good, so we gained’t use it but, however you possibly can already check it on our picture if you would like!

Do this within the fragment shader of your airplane:

void predominant()
{
    vec2 newUvs = coverUvs(uImageResolution,uContainerResolution);            

    vec4 picture = texture2D(uTexture,newUvs);    
    vec4 displacement = texture2D(uGrid,newUvs);

    vec2 finalUvs = newUvs - displacement.rg*0.01;
    
    vec4 finalImage = texture2D(uTexture,finalUvs);

    gl_FragColor = finalImage;
}

Right here’s what it is best to get:

The primary downside is that the form of the displacement shouldn’t be a sq.. It is because we’re utilizing the identical UVs for our displacement as for the picture. To repair this, we’re going to give our displacement its personal UVs utilizing our coverUvs perform.

void predominant()
{
    vec2 newUvs = coverUvs(uImageResolution,uContainerResolution);     
    vec2 squareUvs = coverUvs(vec2(1.),uContainerResolution);       

    vec4 picture = texture2D(uTexture,newUvs);    
    vec4 displacement = texture2D(uGrid,squareUvs);

    vec2 finalUvs = newUvs - displacement.rg*0.01;
    
    vec4 finalImage = texture2D(uTexture,finalUvs);

    gl_FragColor = finalImage;
}

Now it is best to have a square-shaped displacement. You’ll be able to show our texture once more since we nonetheless have to work on it. Within the gl_FragColor of the airplane shader, set the worth again to displacement.

The most important challenge you possibly can clearly see with our present texture is that it’s not fading out. To repair that, we’re going to multiply the colour by a price smaller than 1, which is able to trigger it to progressively are likely to 0.

//... gpgpu shader
shade.rg+=uDeltaMouse*dist;

float uRelaxation =  0.965;
shade.rg*=uRelaxation;
    
gl_FragColor = shade;

Now it’s a bit bit higher, however nonetheless not good. The pixels which are nearer to the cursor take much more time to fade out. It is because they’ve gathered way more shade, in order that they take longer to succeed in 0. To repair this, we’re going to add a brand new float uniform.

Add this on the backside of the createVariable technique of the GPGPU:

this.variable.materials.uniforms.uMouseMove = new THREE.Uniform(0)

Then add this on the prime of updateMouse:

updateMouse(uv: THREE.Vector2) {
  this.variable.materials.uniforms.uMouseMove.worth = 1
// ... gpgpu.updateMouse

Then, add this to the render technique of the GPGPU:

render() {
    this.variable.materials.uniforms.uMouseMove.worth *= 0.95
    this.variable.materials.uniforms.uDeltaMouse.worth.multiplyScalar(0.965)

    this.gpgpuRenderer.compute()
  }

Now you would possibly discover that the colours are very weak. It is because the worth of uDeltaMouse is fading out too shortly. We have to improve it within the updateMouse technique:

updateMouse(uv: THREE.Vector2) {
    this.variable.materials.uniforms.uMouseMove.worth = 1

    const present = this.variable.materials.uniforms.uMouse.worth as THREE.Vector2

    present.subVectors(uv, present)
    present.multiplyScalar(80)

    this.variable.materials.uniforms.uDeltaMouse.worth = present
    this.variable.materials.uniforms.uMouse.worth = uv
  }

Now we’ve got our desired displacement impact:

Creating the RGB Shift Impact

All that’s left to do is the RGB shift impact. Understanding this impact is fairly easy. You most likely know {that a} shade in GLSL is a vec3 containing the purple, inexperienced, and blue parts of a fraction. What we’re going to do is apply the displacement to every particular person shade of our picture, however with completely different intensities. This fashion, we are going to discover a shift between the colours.

Within the fragment shader of the airplane, add this code proper earlier than the gl_FragColor = finalImage;

/* 
* rgb shift 
*/

//separate set of UVs for every shade
vec2 redUvs = finalUvs;
vec2 blueUvs = finalUvs;
vec2 greenUvs = finalUvs;    

//The shift will observe the displacement course however with a diminished depth, 
//we'd like the impact to be delicate
vec2 shift = displacement.rg*0.001;

//The shift energy will rely upon the velocity of the mouse transfer, 
//for the reason that depth depend on deltaMouse we simply have to make use of the size of the (purple,inexperienced) vector    
float displacementStrength=size(displacement.rg);
displacementStrength = clamp(displacementStrength,0.,2.);

//We apply completely different strengths to every shade

float redStrength = 1.+displacementStrength*0.25;
redUvs += shift*redStrength;    

float blueStrength = 1.+displacementStrength*1.5;
blueUvs += shift*blueStrength; 

float greenStrength = 1.+displacementStrength*2.;
greenUvs += shift*greenStrength;


float purple = texture2D(uTexture,redUvs).r;
float blue = texture2D(uTexture,blueUvs).b;    
float inexperienced = texture2D(uTexture,greenUvs).g; 

//we apply the shift impact to our picture
finalImage.r =purple;
finalImage.g =inexperienced;
finalImage.b =blue;


gl_FragColor = finalImage;

And now we’ve got our impact!

Thanks for studying!