Create an AR game using Unity's AR Foundation

1. Overview

ARCore is Google's framework for building augmented reality experiences on smartphones. You can use Unity's AR Foundation to build cross-platform AR applications.

What you'll build

In this codelab, you'll build a simple game using AR Foundation. The goal of the game is to collect packages using a car that you control using your handheld device.

However, this won't happen in an entirely virtual world! You'll mix physical atoms and digital bits to create a new type of player experience by creating a game that understands the environment around the player.

By the end of this codelab, your game will be able to:

• Detect real-world planes and draw a playing field over it.
• Cast rays from the camera's view and detect intersections with planes.
• React to real-world lighting conditions to give your game extra realism.

What you'll learn

• How to set up a project that uses Unity's AR Foundation.
• How to use ARPlaneManager to subscribe to new planes.
• How to use Raycast to find intersections with virtual geometry
• How to use ARLightEstimationData to light your scene.

What you'll need

• A supported ARCore device, connected using a USB cable to your development machine.
• Google Play Services for AR 1.23 or later.
• An installation of Unity Hub or Unity 2020 LTS.

2. Set up your development environment

In this step, you will get your environment ready for development with Unity's AR Foundation.

Ensure your device is compatible with AR

AR experiences on Android devices are driven by ARCore, which is available on ARCore supported devices. Ensure that your development device is compatible with AR. Alternatively, you can use a correctly configured AR-compatible Android Emulator instance.

Setup USB debugging on your device

You will need to enable Developer options on your device to run debug apps. If you haven't done so yet, refer to Android documentation on Enable developer options and USB debugging.

Install Unity (2020.3 LTS)

On your workstation, install Unity 2020 LTS. In this codelab, screenshots are shown of Unity's UI in the 2020.3 (LTS) version. Other versions of Unity may work, but may require additional steps. It may look different than the screenshots shown here.

Create a new project

Create a new project using the Universal Render Pipeline template. Give it a descriptive name and an appropriate location, and press CREATE.

Install the required frameworks

Unity's AR Foundation can be found in the Unity Package Manager.

1. Open it by clicking on Window > Package Manager.

2. In this window, install the packages you'll be using in this codelab. View the latest versions of these frameworks by expanding its entry using the ► icon. Install the latest versions for each of these frameworks:
   • AR Foundation
   • ARCore XR Plugin

When you're done, your Package Manager should look something like this:

Install the starter package

For this codelab, we've provided a starter package that contains prefabs and scripts that will expedite some parts of the codelab so you can focus on how to use AR Foundation.

1. Download the starter package
2. Install the starter package by opening Assets > Import Package > Custom Package... and opening starter-package.unitypackage.
3. In the window that pops up, ensure everything is selected.
4. Click Import.

Change Build Settings

Since the application will run on Android, change the build platform to Android:

1. Open File > Build Settings.
2. In the Platform pane, select Android.
3. Optionally, enable Development Build and Script Debugging in order to retain debugging information while your app runs.
4. Click Switch Platform.

Change Project Settings

AR Foundation needs to be configured to initialize XR systems on startup.

1. Open Edit > Project Settings... and click on the XR Plug-in Management section.
2. In the Android tab, enable ARCore.

3. In the left-hand pane, click the Player section.
4. In the Android tab, under Other Settings, remove Vulkan from Graphics APIs.

5. AR Required apps using ARCore require a minimum API level of 24. Scroll down and find Minimum API Level. Set the minimum API level to 24.

Add the required scene elements

The Universal Render Pipeline template comes with some game objects you won't be using in this tutorial.

1. Delete all game objects in the SampleScene.

2. Add AR Foundation objects. Right-click in the Hierarchy pane. Use this menu to add:

• XR > AR Session: This object controls the lifecycle of an AR experience.
• XR > AR Session Origin: This object transforms AR coordinates into Unity world coordinates.
• Light > Directional Light: This provides a light source to illuminate game objects.

Your hierarchy should look like this:

3. Expand the AR Session Origin you created in the hierarchy, and select the AR Camera object. In the inspector, change its tag to MainCamera.

Set up rendering

Unity's Universal Render Pipeline needs one change to be compatible with AR Foundation.

1. In the Project pane, navigate through Assets > Settings to find the ForwardRenderer asset.

2. Select the ForwardRenderer.
3. In the Inspector pane, use Add Renderer Feature to add an AR Background Renderer Feature. This component will render the camera feed in your scene.

Verify the setup

1. Ensure that your device is plugged in, and that ADB debugging is on.
2. Click File > Build And Run... This will upload the application to your device and start it when it has been installed.
3. You should see the camera feed on your device's screen.

In the next step, you'll start adding functionality to your app.

3. Detect planes in the real world

Now that a basic scene has been set up, you can start on developing the game. In this step, you will detect planes and draw them to the scene.

Add an ARPlaneManager component

An ARPlaneManager detects ARPlanes and creates, updates, and removes game objects when the device's understanding of the environment changes.

1. Using the Hierarchy pane, create an empty GameObject.
2. Rename it to Driving Surface Manager. This component will display planes until one is selected by the player.
3. Select the new game object. Within the Inspector pane, click Add Component to add a AR Plane Manager.

4. Configure the ARPlaneManager by setting the Plane Prefab field:
   1. Click the button next to None to bring up the Select GameObject window.
   2. Select the Assets tab and search for Driving Surface Plane.

This prefab from the starter package provides a gritty floor texture that will be used as the plane decoration.

5. Change the Detection Mode to Horizontal. This configures the ARPlaneManager to only provide horizontal planes, ideal for driving on.

Add an ARRaycastManager component

An ARRaycastManager exposes raycast functionality. In the next step, we'll use this object to provide the controls for the user.

1. Ensure the object called Driving Surface Manager is selected in the Hierarchy pane.
2. In the Inspector, click Add Component to add an ARRaycastManager component to your game object.

No further configuration is needed for this component.

Add a DrivingSurfaceManager component

A DrivingSurfaceManager is a helper script from the Starter Package that allows for an ARPlane to be selected. Once a ARPlane is selected, all other planes will be hidden, and new planes will be disabled.

1. Ensure the object called Driving Surface Manager is selected in the Hierarchy pane.
2. In the Inspector, click Add Component to add a DrivingSurfaceManager component to your game object.

No further configuration is needed for this component.

Run the app

1. Click File > Build And Run... to test your changes.
2. Point your device at a horizontal real-world surface and move your device around to improve ARCore's understanding of the world.

3. When ARCore has detected a plane, you should see a dirt texture cover real-world surfaces. The ARPlaneManager instantiates the given Plane Prefab for each detected plane. The Driving Surface Plane prefab has a ARPlaneMeshVisualizer component which creates a mesh for a given ARPlane.

In the next step, you will use a detected plane as a playing field.

4. Perform a hit test against detected planes

In the previous step, you programmed an application that can detect planes. These planes are reflected in your game's scene. Now, you'll add interactivity with these planes by creating an aiming reticle and a car that will drive on the detected plane's surface.

Create an aiming reticle

The control scheme for this app involves the player pointing their phone at a surface. In order to give clear visual feedback for the designated location, you'll use an aiming reticle.

In order to "stick" this reticle to an AR plane, use a hit test. A hit test is a technique that calculates intersections when casting a ray in a given direction. You will use a hit test to detect an intersection in the direction of the camera's view.

Add the reticle

1. In the Project pane near the bottom of the screen, navigate to Assets > Starter Package.
2. Place the Reticle Prefab into the scene by dragging it into the project's Hierarchy pane.
3. Select the reticle in the hierarchy.
4. In the inspector, click Add Component. Add the ReticleBehaviour script from the Starter Package. This script contains some boilerplate for controlling the reticle.
5. The ReticleBehaviour script is dependent on the Driving Surface Manager you created before, so add the dependency by clicking on the Driving Surface Manager chooser. Select the Scene tab and pick the Driving Surface Manager.

Edit the ReticleBehaviour

The ReticleBehavior script will position the reticle on the plane that's in the center of the device's viewport.

1. Open the ReticleBehaviour.cs script by double-clicking the Script field.
2. Determine the center of the screen using Camera's ViewToScreenPoint. Edit the Update() method to add the following:

var screenCenter = Camera.main.ViewportToScreenPoint(new Vector3(0.5f, 0.5f));

3. Use this point to conduct a raycast. Add the following:

var hits = new List<ARRaycastHit>();
DrivingSurfaceManager.RaycastManager.Raycast(screenCenter, hits, TrackableType.PlaneWithinBounds);

If you encounter a NullReferenceException when using Camera.main, ensure that your scene's AR Camera in Hierarchy > AR Session Origin is tagged MainCamera.

The variable hits will contain ARRaycastHits which describe points on trackables which are intersected by ray.

TrackableType.PlaneWithinBounds is a value in TrackableType. This specifies which kinds of Trackables can be detected with this ray cast. Check out Unity's documentation on TrackableType for more information.

4. Determine the intersection point of interest by querying the hits list. Prioritize the locked plane contained in DrivingSurfaceManager, and if it does not exist, use the first plane hit. Add the following to the end of Update():

CurrentPlane = null;
ARRaycastHit? hit = null;
if (hits.Length > 0)
{
    // If you don't have a locked plane already...
    var lockedPlane = DrivingSurfaceManager.LockedPlane;
    hit = lockedPlane == null
        // ... use the first hit in `hits`.
        ? hits[0]
        // Otherwise use the locked plane, if it's there.
        : hits.SingleOrDefault(x => x.trackableId == lockedPlane.trackableId);
}

5. If hit contains a result, move this GameObject's transform to the hit position.

if (hit.HasValue)
{
    CurrentPlane = DrivingSurfaceManager.PlaneManager.GetPlane(hit.Value.trackableId);
    // Move this reticle to the location of the hit.
    transform.position = hit.Value.pose.position;
}
Child.SetActive(CurrentPlane != null);

Test the reticle

1. Click File > Build And Run... to test your changes.
2. When you point your device at a plane, you should see the reticle follow your camera's movements.

Create a car

The player will control a toy car that will drive towards the location of the reticle. A model and behavior for this car is provided in the Starter Package.

Add a CarManager to your scene

1. In the Hierarchy, create a new empty GameObject.
2. Rename it to Car Spawner.
3. Select the object you created. In the Hierarchy pane, click Add Component to add the CarManager component.
4. Set up CarManager's dependencies by clicking on the chooser for each field:
   • Car Prefab: In Assets, select Car Prefab.
   • Reticle: In Scene, select Reticle Prefab.
   • Driving Surface Manager: In Scene, select Driving Surface Manager.

This CarManager behavior spawns a toy car on the plane that the reticle is on. If you'd like, check out the CarBehaviour script to learn how the car is programmed.

Test driving

1. Click File > Build And Run to test your changes.
2. When you tap on a plane, you should see a small car appear at that location. This car will follow the reticle.

Add the game element

Now that the player can control an entity in the scene, give the player a destination to drive towards.

1. Create a new empty GameObject in the Hierarchy.
2. Rename it to Package Spawner.
3. Select the object you created. In the Hierarchy pane, click Add Component to add the PackageSpawner component to it.
4. Set up PackageSpawner's dependencies by clicking on the chooser for each field:
   • Package Prefab: In Assets, select Package Prefab.
   • Driving Surface Manager In Scene, select Driving Surface Manager.

This PackageSpawner behavior spawns a new package at a random location on a locked ARPlane if there isn't a package already.

Test the game

1. Click File > Build And Run to test your changes. 2, After you create a car, a package should spawn.
2. Drive your car to the package.
3. A new one will appear at a random location.

5. Set up Lighting Estimation

Now that the basic game has been completed, add a touch of realism to your AR scene. In this step, you will use ARCore's Lighting Estimation API to detect the lighting present in the real world based on incoming camera frames. This information will be used to adapt your scene's lighting to match the real-world lighting.

Enable Lighting Estimation

1. In Hierarchy, expand the AR Session Origin and select the AR Camera object.
2. In the Inspector, expand the AR Camera Manager script.
3. Change the Lighting Estimation field to Everything.

Modify the directional light

1. In Hierarchy, select the Directional Light object.
2. Add the LightEstimation component to it. This component from the Starter Package provides some boilerplate for subscribing to lighting changes.
3. In the FrameReceived() function, add:

ARLightEstimationData lightEstimation = args.lightEstimation;

if (lightEstimation.averageBrightness.HasValue)
    Light.intensity = lightEstimation.averageBrightness.Value;

if (lightEstimation.averageColorTemperature.HasValue)
    Light.colorTemperature = lightEstimation.averageColorTemperature.Value;

if (lightEstimation.colorCorrection.HasValue)
    Light.color = lightEstimation.colorCorrection.Value;

if (lightEstimation.mainLightDirection.HasValue)
    Light.transform.rotation = Quaternion.LookRotation(lightEstimation.mainLightDirection.Value);

if (lightEstimation.mainLightColor.HasValue)
    Light.color = lightEstimation.mainLightColor.Value;

if (lightEstimation.mainLightIntensityLumens.HasValue)
    Light.intensity = lightEstimation.averageMainLightBrightness.Value;

if (lightEstimation.ambientSphericalHarmonics.HasValue)
{
    RenderSettings.ambientMode = AmbientMode.Skybox;
    RenderSettings.ambientProbe = lightEstimation.ambientSphericalHarmonics.Value;
}

Check out the AR Foundation documentation on ARLightEstimationData for more details on each parameter.

Test your changes

1. Click File > Build And Run to test your changes.
2. When looking at the objects in the scene, you may notice that they are colored depending on the environment's lighting.
3. If possible, try modifying your lighting. For example, try turning off the lights in the room you're in. You should see the lighting on the objects adapt to the change in the real-world lighting.

6. Wrap up

Congratulations! You've made it to the end of this codelab on Unity AR Foundation.

What you've covered

• How to set up a basic project using Unity's AR Foundation and the Universal Rendering Pipeline.
• How to use ARPlaneManager to subscribe to new planes.
• How to use Raycast to find intersections with virtual geometry.
• How to use ARLightEstimationData to light your scene.

Next steps

• Check out Unity's AR Foundation samples.
• Browse AR Foundation documentation.
• Check out Google's documentation on ARCore Extensions for Unity's AR Foundation.

Bonus assignments

If you want to expand on the game you've created here, here are some ideas you could pursue:

• Add a score counter to the your game by modifying a TextMeshPro when a PackageManager spawns a new package.
• Check out performance information when your game is running by enabling the Performance Overlay.
• Use Persistent Raycasts to place new objects in your scene first. When a plane is detected in that area, that object will update to snap to that plane.