This codelab will go through an example of building an AR web application. It uses JavaScript to render 3D models that appear as if they exist in the real world.

You will use the still-in-development WebXR Device API, (the successor to the WebVR API), that combines both augmented reality (AR) and virtual reality (VR) functionality. We'll be focusing on experimental AR extensions to the WebXR Device API that are being developed in Chrome.

What is Augmented Reality?

Augmented reality (AR) is a term usually used to describe the mixing of computer-generated graphics with the real world, which, in the case of phone-based Augmented Reality, means convincingly placing computer graphics over a live camera feed. In order for this effect to remain convincing as the phone moves through the world, the AR-enabled device needs to understand the world it is moving through, which may include detecting surfaces and estimating lighting of the environment. Additionally, the device also needs to determine its "pose" (position and orientation) in 3D space.

Augmented reality usage has been increasing, with the popularity of AR usage in apps like selfie filters and AR-based games. Today, there are already hundreds of millions of AR-enabled smartphones, less than a year after the release of ARCore, Google's augmented reality platform, and ARKit by Apple. With the technology now in the hands of millions of people, initial proposals of AR extensions for the WebXR Device API can be experimentally implemented behind flags.

What you will build

In this codelab, you're going to build a web application that places a model in the real world using augmented reality. Your app will:

  1. Use your device's sensors to determine and track its position and orientation in the world.
  2. Render a 3D model composited on top of a live camera view.
  3. Execute hit tests to place objects on top of discovered surfaces in the real world.

What you'll learn

This codelab is focused on augmented reality APIs. Non-relevant concepts and code blocks are glossed over and are provided for you in the corresponding repository code.

What you'll need

This is an overview of what you'll need, and we'll go into more detail shortly.

Get Chrome with AR Features

At the time of writing, the initial AR features are only implemented in Chrome Canary builds with minimum versions of 69.0.3453.2. You can go to Settings -> About Chrome to see what version of Chrome you're using.

Ensure AR features are enabled on Chrome

At the time of writing, the initial AR features are behind both webxr and webxr-hit-test flags. To enable WebXR augmented reality support in Chrome:

  1. Verify that your Android phone is one of the supported ARCore devices
  2. Confirm Chrome version is >= 69.0.3453.2
  3. Type chrome://flags in the URL bar
  4. Type webxr in the Search flags input field
  5. Set the WebXR Device API (#webxr) flag to Enabled
  1. Set the WebXR Hit Test (#webxr-hit-test) flag to Enabled
  2. Tap RELAUNCH NOW to ensure the updated flags take effect

Visit the link below on your AR device to try the first step of this Codelab. If you get a page with a message displaying "Your browser does not have AR features", re-check the version of Chrome Canary and the WebXR flags, which might require a browser restart.

TRY IT

Download the Code

Click the following link to download all the code for this codelab on your workstation:

Download source code

Unpack the downloaded zip file. This will unpack a root folder (ar-with-webxr-master), which contains directories of several steps of this codelab, along with all the resources you need. The step-05 and step-06 folders contain the desired end state of the 5th and 6th steps of this codelab, as well as the final result. They are there for reference. We'll be doing all our coding work in a directory called work.

Install and verify web server

You're free to use your own web server, but we'll walk you through using the Chrome Web Server if you don't have one set up. If you don't have that app installed on your workstation yet, you can install it from the Chrome Web Store.

Install Web Server for Chrome

After installing the Web Server for Chrome app, go to chrome://apps and click on the Web Server icon:

You'll see this dialog next, which allows you to configure your local web server:

  1. Click the choose folder button, and select the ar-with-webxr-master folder. This will enable you to serve your work in progress via the URL highlighted in the web server dialog (in the Web Server URL(s) section).
  2. Under Options, make sure Automatically show index.html is checked.
  3. STOP and RESTART the server by sliding the toggle labeled Web Server: STARTED/STOPPED to the left, and then back to the right.
  4. Verify that at least one Web Server URL(s) appears:

Now we also want to configure our AR device such that visiting localhost:8887 on our AR device will access the same port on our workstation.

  1. On you development workstation, go to chrome://inspect and click the Port forwarding... button:

Use the Port forwarding settings dialog to forward port 8887 to localhost:8887. Ensure that Enable port forwarding is checked:

Test your connection:

  1. Connect your AR device to your workstation via a USB cable.
  2. On your AR device in Chrome Canary enter http://localhost:8887 into the URL bar.
    Your AR device should forward this request to your development workstation's web server. You should see a directory of files.
  3. On your AR device, tap on the work directory to load the work/index.html page.

You should see page that contains an ENTER AUGMENTED REALITY button...

...However, if you see an Unsupported Browser error page, go back and confirm the Chrome Canary version, chrome://flags and restart Chrome.

  1. Once the connection to your web server is working with your AR device, click the ENTER AUGMENTED REALITY button.
    You may be prompted to install ARCore:
  2. The first time you run an AR application you'll see a camera permissions prompt:

Once everything is good to go, there should be a scene of cubes overlayed on top of a camera feed. The scene understanding improves as more of the world is parsed by the camera, so some moving around can help stabilize things.

History

WebGL is a powerful graphics library enabling the rendering of 3D content on the web, but access to VR devices from the web are necessary for discovery, refresh rate synchronization and positioning. The experimental WebVR 1.1 API has been implemented in browsers as web developers explored building VR applications for the web. This provided the framework to render a stereoscopic web scene with appropriate distortions for a VR headset. Daydream- and GearVR-enabled mobile devices and full room systems like the Oculus Rift or HTC Vive are some of the platforms supported by different browsers.

As the industry and use cases evolved, so did the need to support AR on the web. Due to the technological similarities between AR and VR, the WebXR Device API was created to encompass both. While still in-development and subject to change, the core of the WebXR Device API is mostly stable, and is being developed by representatives from all major browser vendors. The API supports VR experiences, with initial AR proposals just now being prototyped and explored.

Implementation

The first implementation of the WebXR Device API were made available in Chrome 67 behind a flag (#webxr) and as an origin trial. The initial experimental AR features are in Chrome 69+ behind a flag, #webxr-hit-test. At the time of writing, all browsers with WebVR implementations have committed to supporting the WebXR Device API in the future.

The Future

The only scene understanding currently available to the browser is a "hit test" feature. This allows you to cast a ray out from the device, for example based on a user screen tap, and return any collisions with the real world, allowing us to use that information to overlay virtual scenes.

Future explorations may expand upon scene understanding, providing things like light estimation, surfaces, meshes, feature point clouds, and more.

We've provided an HTML page with CSS styling and JavaScript for enabling basic AR functionality that renders a scene of cubes in sync with your camera's position. This speeds up the setup process and allows this codelab to focus on the AR features.

The HTML page

We're building an AR experience into a traditional webpage using existing web technologies. In this specific experience, we'll use a full screen rendering canvas, so our HTML file doesn't need to have too much complexity. The CSS ensures the <canvas> injected by our graphics library is fullscreen. The HTML page loads the scripts.

AR features require a user gesture to initiate, so there are some Material Design Lite elements for displaying the "Start AR" button, and the unsupported browser message.

The index.html file that is already in your work directory should look something like this (this is a subset of the actual contents, don't copy this code into your file):

<!doctype html>
<html>
  <head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Building an augmented reality application with the WebXR Device API</title>
    <link rel="stylesheet" type="text/css" href="../shared/app.css" />
    <link rel="stylesheet" type="text/css" href="../third_party/mdl/material.min.css" />
  </head>
  <body>
    <div id="enter-ar-info" class="demo-card mdl-card mdl-shadow--4dp">
      <!-- Material Design elements for demo --> 
      <!-- ... -->
    </div>
    <script src="../third_party/three.js/three.js"></script>
    <script src="../shared/utils.js"></script>
    <script src="app.js"></script>
  </body>
</html>



Check out the key JavaScript code

Our app consists of using the 3D JavaScript library three.js, some utilities and all of our WebXR/app-specific code in app.js. Let's walk through our app's boilerplate.

Your work directory also already includes the app code (app.js), in it you'll find the App class:

class App {
  constructor() {
    ...
  }

  async init() {
    ...
  }

  async onEnterAR() {
    ...
  }

  onNoXRDevice() {
    ...
  }

  async onSessionStarted(session) {
   ...
  }

  onXRFrame(time, frame) {
    ...
  }
};

window.app = new App();

We instantiate our app and store it as window.app for convenience while using Chrome DevTools to debug.

Our constructor calls this.init() which is an async function that will start up our XRSession for working with AR. This function checks for the existence of navigator.xr, the entry point for the WebXR Device API, as well as XRSession.prototype.requestHitTest, the AR feature enabled by the webxr-hit-test Chrome flag.

If all is well, we bind a click listener on our "Enter Augmented Reality" button to attempt to create an XR session when the user clicks the button.

class App {
  ...
  async init() {
    if (navigator.xr && XRSession.prototype.requestHitTest) {
      try {
        this.device = await navigator.xr.requestDevice();
      } catch (e) {
        this.onNoXRDevice();
        return;
      }
    } else {
      this.onNoXRDevice();
      return;
    }

    document.querySelector('#enter-ar').addEventListener('click', this.onEnterAR);
  }
}

When we do find an XRDevice, we store it as this.device. In order to interact with the device, we need to request an XRSession from it. An XRDevice can have multiple XRSessions, where each session exposes the device pose, the user's environment and handles rendering to device.

We want the output of the session to be displayed on the page, so we must create an XRPresentationContext, similar to how we'd create a WebGLRenderingContext if we were rendering our own WebGL content.

class App {
  ...
  async onEnterAR() {
    const outputCanvas = document.createElement('canvas');
    const ctx = outputCanvas.getContext('xrpresent');
    
    try {
      const session = await this.device.requestSession({
        outputContext: ctx,
        environmentIntegration: true,
      });
      document.body.appendChild(outputCanvas);
      this.onSessionStarted(session);
    } catch (e) {
      this.onNoXRDevice();
    }
  }
}

Calling getContext('xrpresent') on our canvas returns an XRPresentationContext, which is the context that will be displayed on our XR device. Then we request a session via requestSession() on the XR device with our output presentation context, as well as the environmentIntegration flag indicating we want to use AR features, and then await the promise.

Once we have our XRSession, we're ready to set up the rendering with three.js and kick off our animation loop. We create a three.js WebGLRenderer, which contains our second canvas, ensuring alpha and preserveDrawingBuffer are set to true and disabling auto clear. We use the WebGLRenderingContext from three and asynchronously set the compatible XR device. Once the context is considered compatible with the device, we can create an XRWebGLLayer and set it as the XRSession's baseLayer. This tells our session that we want to use this context to draw our scene, to subsequently be displayed on the canvas created in this.init(), composited with our live camera feed.

To render a three.js scene we need three components: a WebGLRenderer to handle the rendering, a scene of objects to render, and a camera to indicate the perspective that scene is rendered from. We'll use a scene created from DemoUtils.createCubeScene() to prepopulate a scene with many cubes floating in space. If you haven't worked with three.js or WebGL before, no worries! If you run into issues rendering, compare your code with the examples.

Before we kick off our rendering loop, we'll need to get an XRFrameOfReference with a 'eye-level' value, indicating that our device is tracking position (as opposed to orientation-only VR experiences like Daydream or GearVR). Once we have our frame of reference, we can use the XRSession's requestAnimationFrame to kick off our rendering loop, similar to window.requestAnimationFrame.

class App {
  ...
  async onSessionStarted(session) {
    this.session = session;

    document.body.classList.add('ar');
    
    this.renderer = new THREE.WebGLRenderer({
      alpha: true,
      preserveDrawingBuffer: true,
    });
    this.renderer.autoClear = false;

    this.gl = this.renderer.getContext();
    
    await this.gl.setCompatibleXRDevice(this.session.device);
  
    this.session.baseLayer = new XRWebGLLayer(this.session, this.gl);

    this.scene = DemoUtils.createCubeScene();

    this.camera = new THREE.PerspectiveCamera();
    this.camera.matrixAutoUpdate = false;

    this.frameOfRef = await this.session.requestFrameOfReference('eye-level');
    this.session.requestAnimationFrame(this.onXRFrame);
  }
}

On every frame, this.onXRFrame will be called with a timestamp and an XRPresentationFrame. From our frame object, we can get an XRDevicePose, an object, which describes our position and orientation in space, and an array of XRViews, which describes every viewpoint we should render the scene from in order to properly display on the current device.

First, we must fetch the current pose and queue up the next frame's animation by calling session.requestAnimationFrame(this.onXRFrame) before we render. While stereoscopic VR has two views (one for each eye), we will only have one view since we're displaying a fullscreen AR experience. To render we need to loop through each view and set up a camera using the projection matrix it provides and view matrix it gets from the pose. This syncs up the virtual camera's position and orientation with our device's estimated physical position and orientation. Then we can tell our renderer to render our scene with the provided virtual camera.

class App {
  ...
  onXRFrame(time, frame) {
    const session = frame.session;
    const pose = frame.getDevicePose(this.frameOfRef);

    session.requestAnimationFrame(this.onXRFrame);

    this.gl.bindFramebuffer(this.gl.FRAMEBUFFER, this.session.baseLayer.framebuffer);

    if (pose) {
      for (let view of frame.views) {
        const viewport = session.baseLayer.getViewport(view);
        this.renderer.setSize(viewport.width, viewport.height);

        this.camera.projectionMatrix.fromArray(view.projectionMatrix);
        const viewMatrix = new THREE.Matrix4().fromArray(pose.getViewMatrix(view));
        this.camera.matrix.getInverse(viewMatrix);
        this.camera.updateMatrixWorld(true);

        this.renderer.clearDepth();

        this.renderer.render(this.scene, this.camera);
      }
    }
  }
}

And that's it! We've walked through the code that fetches an XRDevice, creates an XRSession, and renders a scene on each frame, updating our virtual camera's pose with our device's estimated physical pose.

Test it out

Now that you've looked through the code, let's see our boilerplate in action! We already visited this site during setup, but let's take a look now that we've walked through the code. You should see your camera feed with cubes floating in space whose perspective changes as you move your device. Tracking improves the more you move around, so explore what works for you and your device.

TRY IT

If you run into any issues running this app, check the Introduction and Getting set up.

Now that we have confirmed that we have our live video feed, our device's pose setting our camera's position and orientation, and 3D cubes rendering on top, it's time to start interacting with the real world using a hit test. We want to be able to find a surface in the real world (like on the ground, for example), and place a cube there.

What is a "hit test"?


A hit test is generally a way to cast out a straight line from a point in space in some direction and determine if it intersects with any objects we're interested in. In our case, we'll be tapping on the screen of our AR device, so imagine a ray traveling from your finger, through your device, and straight into the physical world in front of it as seen by your device's camera.

The WebXR Device API will let us know if this ray intersected any objects in the real world, determined by the underlying AR capabilities and understanding of the world. If we're looking at the world through our device, angled slightly towards the ground, and we tap on our phone such that the ray will hit the ground, we should expect a response of the location of where that collision occurred.

Set up our scene

Let's add a div to the index.html with ID stabilization to display an animation to users representing stabilization status, prompting them to move around with their device. This will be displayed once we're in AR, and hidden once the reticle finds a surface, controlled via <body> classes.

  <div id="stabilization"></div>
  <script src="../third_party/three.js/three.js"></script>
  ...
</body>
</html>

Now in our app.js, the first thing we want to do is get rid of our floating cube scene. In our onSessionStarted function, replace the DemoUtils.createCubeScene() with a fresh new THREE.Scene().

  // this.scene = DemoUtils.createCubeScene();
  this.scene = new THREE.Scene();

Right after we create the scene, we'll need to create an object to place during our collision. Renderable objects in three.js are represented as THREE.Mesh objects, which contains geometry and material. Here's the code to create the geometry and material, as well as adjusting the transform of the cube so that its origin is on its bottom face. For now, we just create the mesh, transform it once, and store it as this.model.

  const geometry = new THREE.BoxBufferGeometry(0.5, 0.5, 0.5);
  const material = new THREE.MeshNormalMaterial();
  geometry.applyMatrix(new THREE.Matrix4().makeTranslation(0, 0.25, 0));
  this.model = new THREE.Mesh(geometry, material);

We've provided a Reticle three.js object (source code can be found in shared/utils.js) that attempts to continuously track hit-tests in the center of the screen, to provide some visual feedback to the user about the device's understanding of the world. We add the constructor in the onSessionStarted function after we fetch the frame of reference. In our onXRFrame function, we call the reticle's update method, and add a class 'stabilized' to the body once our Reticle finds a surface. This hides the stabilization helper animation that prompts a user to move their device around.

class App {
  ...
  onSessionStarted(session) {
    ...
    this.reticle = new Reticle(this.session, this.camera);
    this.scene.add(this.reticle);
    
    this.frameOfRef = await this.session.requestFrameOfReference('eye-level');
    this.session.requestAnimationFrame(this.onXRFrame);
    ...
  }
  onXRFrame(time, frame) {
    let session = frame.session;
    let pose = frame.getDevicePose(this.frameOfRef);

    this.reticle.update(this.frameOfRef);
    if (this.reticle.visible && !this.stabilized) {
      this.stabilized = true;
      document.body.classList.add('stabilized');
    }
    ...
  }
}

We now want to instead perform a hit test only when we tap on the screen, so we'll need an event handler.

class App {
  constructor() {
    ...
    this.onClick = this.onClick.bind(this);
  }
  
  onSessionStarted(session) {
    ...
    this.reticle = new Reticle(this.session, this.camera);
    this.scene.add(this.reticle);

    window.addEventListener('click', this.onClick);
  }

  onClick(e) {
    console.log('click!');
  }
}

Check out your app now, and if you're using Chrome DevTools on your device, ensure that you're seeing the console statements from clicking above. If you want to setup Chrome DevTools, refer to Get Started with Remote Debugging Android Devices.

Firing the hit test

Now that we have our model and taps bound to an event, it's time to actually fire our hit test. XRSession's API requires an origin point, a direction, and our XRFrameOfReference we created earlier. Our origin point is our device's position in the WebXR coordinate system, and the direction is a vector point out of the back of our device. three.js has some handy functions for projecting a vector out from a point, so let's use those. This might seem daunting to those unfamiliar with matrices and vectors, but the most important thing here is understanding that this is casting a straight line out of the back of the device. We can leverage a raycaster from three.js to handle the math for us, which gives us an origin and vector as THREE.Vector3s. The x and y positions in setFromCamera is the position in screen space, and this function takes values in normalized device coordinates between -1 and 1. We just want to attempt to place an object in the middle of the screen, so our x and y values are just 0.

requestHitTest returns a promise that resolves to an array of XRHitResults, which store matrices of the location of where the hit occurred. If the straight line out of our device hit a wall or the floor, the hit's position will be where that straight line intersects the wall or the floor. If we found a hit, take the first hit (closest hit) and turn it into a THREE.Matrix4. Then we can take our cube we created earlier and place it at the position of collision. We also ensure that the cube is added to our scene so we can render it. Since we now need to use await in onClick, we must make onClick an async function.

class App {
  ...
  async onClick(e) {
    const x = 0;
    const y = 0;
   
    this.raycaster = this.raycaster || new THREE.Raycaster();
    this.raycaster.setFromCamera({ x, y }, this.camera);
    const ray = this.raycaster.ray;
    
    const origin = new Float32Array(ray.origin.toArray());
    const direction = new Float32Array(ray.direction.toArray());
    const hits = await this.session.requestHitTest(origin,
                                                   direction,
                                                   this.frameOfRef);

    if (hits.length) {
      const hit = hits[0];
      const hitMatrix = new THREE.Matrix4().fromArray(hit.hitMatrix);

      this.model.position.setFromMatrixPosition(hitMatrix);

      this.scene.add(this.model);
    }
  }
}

At this point, we have an event handler responding to our taps, casting out a ray from the back of our device, and if an intersection is found, placing a cube at the point of collision. The underlying AR platform powering Chrome for Android can detect horizontal and vertical surfaces. Look around your environment and see where you can place your brand new cube!

Test it out

TRY IT

We now have an app that puts cubes on real world surfaces. While it is pretty fun to be able to render things in the real world from our mobile device, our bright red cube isn't very compelling. Lets load a model in our scene from Poly, a collection of 3D assets submitted by artists, many under CC-BY license, ready to be remixed and used in your scene. We'll be using this Arctic Fox model by Naomi Chen, CC-BY.

Let's add an attribution for the model we're going to use. Create an #info div, and add a few lines for our attribution:

  <div id="stabilization"></div>
  <div id="info">
    <span>
      <a href="https://poly.google.com/view/dK08uQ8-Zm9">Model</a> by
      <a href="https://poly.google.com/user/f8cGQY15_-g">Naomi Chen</a>
      <a href="https://creativecommons.org/licenses/by/2.0/">CC-BY</a>
    </span><br />
  </div>
  <script src="../third_party/three.js/three.js"></script>
  ...
</body>
</html>

Loading OBJ and MTL files

Before we begin, we'll be using three's OBJLoader to load our model and MTLLoader to load our materials. The loaders are not included within three.js, but provided separately. We have a copy however in our third_party/three.js directory, so lets add it to our index.html. Be sure that it's loaded after three.js but before our app code!

<html>
...
<body>
  ...
  <script src="../third_party/three.js/three.js"></script>
  <script src="../third_party/three.js/OBJLoader.js"></script>
  <script src="../third_party/three.js/MTLLoader.js"></script>
  <script src="../shared/utils.js"></script>
  <script src="app.js"></script>
</body>
</html>

Our arctic fox model and texture are already included in the assets/ directory. Open up app.js and add some constants referencing their location at the top of this file.

const MODEL_OBJ_URL = '../assets/ArcticFox_Posed.obj';
const MODEL_MTL_URL = '../assets/ArcticFox_Posed.mtl';
const MODEL_SCALE = 0.1;

Next, we go back to our onSessionStarted function. We need to add lights to our scene and load our model. Use DemoUtils.createLitScene() which will create a THREE.Scene for us with some lights already included. Then, we want to use another utility, DemoUtils.loadModel(), which uses the OBJLoader and MTLLoader we've included previously. This returns a promise resolving to the model as a three.js object, indicating when the loading is done, but note that we do not want to await on it -- we want to continue rendering while the model is loading.

Once our model is loaded, we store it as this.model. Finally, models come in all shapes and sizes. We'll need to scale down our model proportionally, 0.1 is a good value for our arctic fox. For loading other models, be sure to try a variety of very different scales -- it's common to not see a model because it's 1000 times bigger or smaller than expected!

async onSessionStarted(session) {
  ...
  // this.scene = new THREE.Scene();
  this.scene = DemoUtils.createLitScene();
  
  DemoUtils.fixFramebuffer(this);
  
  // We no longer need our cube model.
  // Sorry, cube!
  /*
  const geometry = new THREE.BoxBufferGeometry(0.5, 0.5, 0.5);
  const material = new THREE.MeshBasicMaterial();
  geometry.applyMatrix(new THREE.Matrix4().makeTranslation(0, 0.25, 0));
  this.model = new THREE.Mesh(geometry, material);
  */

  DemoUtils.loadModel(MODEL_OBJ_URL, MODEL_MTL_URL).then(model => {
    this.model = model;
    this.model.scale.set(MODEL_SCALE, MODEL_SCALE, MODEL_SCALE);
  });

  ...
}

Placing our new model

We're almost done! At this point, our hit test function will place this.model object, which is now an arctic fox rather than a cube, but there's a few more things we can do to polish this up. In our onClick handler, since our model is being loaded asynchronously, we could attempt to place the model before it's loaded. Add a quick check at the top to return early if this.model does not yet exist. Now, in our conditional if we have a hit, we want to rotate the fox to face us. We made a utility that handles some math to get the angle from the model's position, as well as our position, and rotate the fox's Y axis so that the fox faces us when placed.

async onClick(e) {
  if (!this.model) {
    return;
  }
  ...

  if (hits.length) {
    const hit = hits[0];

    const hitMatrix = new THREE.Matrix4().fromArray(hit.hitMatrix);

    this.model.position.setFromMatrixPosition(hitMatrix);

    DemoUtils.lookAtOnY(this.model, this.camera);

    this.scene.add(this.model);
  }
}

Test it out

Now we should be able to tap on our AR device once our reticle has stabilized and see our arctic wolf model on a surface! Try the link below on your AR device to see a completed version of this step.

TRY IT

While we are using a stylized, low-poly fox model, things like proper lighting and shadows on our digital objects adds a large amount of realism and immersion in our scene. Lighting and shadows are handled by three.js, and we specify which lights should cast shadows, which materials should receive and render these shadows, and which meshes can cast shadows. Luckily, our scene contains a light that casts a shadow, and a flat surface in which to render only shadows. The completed work to enable shadows can be found in final/app.js.

First, we must enable shadows on our three.js WebGLRenderer. After we create the renderer, set the following values on its shadowMap:

async onSessionStarted(session) {
  ...
  this.renderer = new THREE.WebGLRenderer(...);
  ...
  this.renderer.shadowMap.enabled = true;
  this.renderer.shadowMap.type = THREE.PCFSoftShadowMap;
  ...
}

We need to call a small utility DemoUtils.fixFramebuffer() to fix a framebuffer issue with three.js -- more info in the source if you're interested, but for now, just know that it fixes an issue with shadows in WebXR. Next, we need to set our fox's meshes to cast shadows. Right before we set the model scale, iterate over the children mesh and flip the castShadow boolean to true.

async onSessionStarted(session) {
  ...

  // Don't forget to call this helper function to
  // use shadows with WebXR!
  DemoUtils.fixFramebuffer(this);

  DemoUtils.loadModel(MODEL_OBJ_URL, MODEL_MTL_URL).then(model => {
    this.model = model;

    // Set all meshes contained in the model to cast a shadow
    this.model.children.forEach(mesh => mesh.castShadow = true);

    this.model.scale.set(MODEL_SCALE, MODEL_SCALE, MODEL_SCALE);
  });
  ...
}

Last but not least, our current scene has an object called shadowMesh that is a flat, horizontal surface that only renders shadows. Initially, this surface has a Y position of 10000 units, and once we find a surface via our hit test, we'll want to move our shadowMesh to be the same height as our real world surface, such that the shadow of the fox is rendered on top of the real world ground. In our hit test function after we position this.model, find the shadowMesh in the scene's children, and place the shadowMesh's Y position to that of the model.

async onClick(e) {
  ...
  if (hits.length) {
    ...
    const shadowMesh = this.scene.children.find(c => c.name === 'shadowMesh');
    shadowMesh.position.y = this.model.position.y;
    this.scene.add(this.model);
  }
}

Test it out

This should be very similar to the previous step, except now we should have some soft shadows near our fox casting on the ground.

TRY IT

We now have our arctic fox model from Poly loaded and placed on a surface in the real world, casting a shadow. The requestHitTest function from WebXR is the first AR scene understanding component that's implemented, but in the future, there may be more capabilities of measuring the light in a scene, surfaces and meshes, depth information and point clouds. These capabilities will allow us to better integrate our digital objects with the real world. For example, light estimation would match the shadow of a model with the real world, and depth data would give us the ability to occlude digital objects when real world objects are in front of them. Still being defined for the WebXR Device API are Anchors, the ability to track real world objects and more accurately position objects in a scene.

Extra credit: pick your own model from Poly

Find another model from Poly and load it into your scene. Be sure to fiddle with the scaling if you're unable to see the model!

Resources