Google's ARCore Extensions for AR Foundation are additions to Unity's AR Foundation cross platform API to develop Augmented Reality applications. ARCore Extensions provide access to features not, or not yet, available in the base API. Some of these features will only work on ARCore enabled Android devices, while others are designed to work on both Android and iOS.

This codelab guides you through the process of building a simple AR enabled app that uses Google's ARCore Cloud Anchor service to create a shared environment across multiple devices.

Google's ARCore uses the terms Anchor, to represent a tracked point in the scene, and Cloud Anchor, to represent a tracked point synchronized with data from the ARCore Cloud Anchor service. Since AR Foundation uses the term Reference Point, Google's ARCore Extensions have adopted the terms Cloud Reference Point and Cloud Reference Id.

What you'll build

In this codelab you will build an app that allows the user to detect planes using AR Foundation, and then, by tapping on the screen, place and host a Cloud Reference Point (blue sphere.) A subsequent tap will attempt to resolve that same point (red cylinder.) When a new point is hosted, the unique id for that point is displayed on screen, another user running the same app on another device can enter that id to resolve the same point.

(The yellow cubes represent local reference points and are optional in this codelab.)

Note: If you run into issues along the way, jump to the last section for some troubleshooting tips.

You'll need specific hardware and software to complete this codelab.

Hardware Requirements

Software Requirements

Unity Game Engine

It is highly recommended that you use Unity Hub to install and manage your Unity versions. (It makes adding Android build support and other required components easier to identify.)

ARCore Extensions for AR Foundation

Create a new Unity 3D project and change the target platform to Android.

  1. Open the Build Settings window (File > Build Settings).
  2. Select Android and click Switch Platform.

Player settings for ARCore Extensions

Click Player Settings... to configure the Android specific player settings.

At the top, under Player:

  1. Specify your Name or Company Name.
  2. Set the Product Name to ARCodelab (this is the name of the app that will be installed on your device.)

Under the Other Settings heading:

  1. In the Graphics APIs section, select and remove Vulkan.
  2. Uncheck Multithreaded Rendering
  3. Select an appropriate Package name (ex. com.<yourname>.arcodelab)
  4. Select a Minimum API Level of Android 7.0 ‘Nougat' (API Level 24).
  5. Select the Target API Level to Automatic (highest installed).
  6. Close the Project Settings window.

Depending on the version of Unity you are using, there are a few different ways to install the ARCore Extensions package into your project. In this codelab you will download and extract the arcore-unity-extensions-*.tgz archive as this method works in Unity 2019.2 and later.

  1. Download and extract the arcore-unity-extensions-*.tgz archive to a folder on your disk. For this codelab, unzipping to an easy location such as your desktop is sufficient.
  2. Open the Package Manager (Window > Package Manager).
  1. Close the Package Manager window.

Note: The package manager supports dependency management. ARCore Extensions currently requires the specific packages and versions listed below. Compatibility with newer versions of these packages is not guaranteed.

In this step, you will set up a basic AR Foundation scene that detects and displays feature points and detected planes. After importing the packages in the previous step you will find a new category of Game Objects under the GameObject > XR submenu (also available in the Hierarchy view from the right-click context menu).

Create AR Foundation game objects

  1. Using the GameObject menu (or right-clicking in the Hierarchy view), create one of each of the following objects:

Instead of the Main Camera, you are going to use the default AR Camera, which was created automatically as a child game object of the AR Session Origin game object.

  1. Delete the Main Camera.

Create Prefabs

Convert the AR Default Point Cloud and AR Default Plane to prefab objects as follows:

  1. In your Project view, select the Assets folder.
  2. Create a Prefabs folder:
  1. One at a time, drag the AR Default Point Cloud and AR Default Plane game objects from the Hierarchy view into the new Prefabs folder in the Project view. The original game objects will turn blue, indicating they are now prefab instances.
  2. In the Hierarchy view, delete the AR Default Point Cloud and AR Default Plane game objects as they're no longer needed.

Add components to AR Session Origin

Add needed components to the AR Session Origin.

  1. Select the AR Session Origin object.
  2. In the AR Session Origin object's Inspector, click the Add Component button and add an instance of AR Point Cloud Manager.
    Then, drag the AR Default Point Cloud prefab object onto the Point Cloud Prefab property.
  3. Click the Add Component button, and add an instance of AR Plane Manager.
    Then, drag the AR Default Plane prefab object onto the Plane Prefab property.
  4. Click the Add Component button, and add an instance of AR Reference Point Manager.
  5. Click the Add Component button, and add an instance of AR Raycast Manager.

Add Scene to project Scenes In Build

Make sure the sample scene is included in the build and active in your project, and it's the first build

  1. Save your SampleScene (File > Save).

  1. Choose File > Build Settings.
  2. Click the Add Open Scenes button.

Congratulations!

You have now created a basic AR app using AR Foundation! If you build and run to your USB connected Android device now, you will be able to move your device around and detect feature points and planes.

Note: Additional information regarding AR Foundation is documented on Unity's website.

ARCore Extensions requires an ARCore Extension Config asset, to select which features you intend to use. Create a Configs folder inside your Assets folder to hold this configuration.

  1. Select the Assets folder in your project.
  2. Choose Assets > Create > Folder (or right click on the folder and use the context menu)
  3. Name the folder Configs.
  4. Select the new Configs folder.
  5. Choose Assets > Create > ARCore Extensions > ARCore Extensions Config (or use the right click context menu).
  6. Select the new ARCoreExtensionsConfig asset you just created.
  7. In the ARCoreExtensionsConfig inspector, check the Enable Cloud Anchors checkbox.

To use ARCore Extensions you must add exactly one instance of the ARCore Extensions game object to your scene and configure it as follows:

  1. Choose GameObject > XR > ARCore Extensions.
  2. Select the new ARCore Extensions object in the Hierarchy view.
  3. Drag the AR Session game object onto the Session property in the ARCore Extensions inspector.
  4. Drag the AR Session Origin game object onto the Session Origin property in the ARCore Extensions inspector.
  5. Drag the ARCoreExtensionsConfig from the previous step onto the AR Core Extensions Config property.

Your project is now set up to use ARCore Extensions. However, ARCore Cloud Anchors won't work just yet.

Proceed to the next step to learn how to create your ARCore Cloud Anchor service API key.

Creating a Cloud Anchor API Key

In order to use Cloud Anchors, you must enable the ARCore Cloud Anchor API within a Google Cloud Platform project. Using your Google account, enable the ARCore Cloud Anchor API.

The first time you sign in to Google's Cloud Platform Console, you will need to accept the Google Cloud Platform Terms of Service. If you wish to continue, you will need to agree to these terms.

Select the drop down Select a project.

In the top right corner select NEW PROJECT.

In the Project name field enter ARCodelab. Then click the CREATE button.

Wait a moment for your project to become available. Again use the Select a project drop down, (you may need to navigate back to the Home screen) this time select your new ARCodelab project.

Use the menu icon in the top left to navigate to APIs & Services > Credentials.

After selecting your project, you will see a message informing you that you need to create credentials to access the APIs. Click Create credentials and select API key.

This generates a new API key named API key 1 and an alphanumeric string.

Copy the alphanumeric string to your clipboard, and return to Unity.

Adding your Cloud Anchor API Key to your project

Back in Unity, follow these steps to add your Cloud Anchor API key to your project:

  1. Choose Edit > Project settings.
  2. Expand the category XR > ARCore Extensions
  3. Paste your API key into the Android field under Cloud Anchor API Keys.
  4. Close the Project Settings window.

Your app is now fully configured to use the ARCore Cloud Anchor service.

Before continuing to the code that enables interaction with ARCore Cloud Anchors, create some elements to help visualize the process.

3D Assets

Materials

Create a Materials folder in your Assets folder.

  1. Select the Assets folder in the Project tab.
  2. Assets > Create > Folder
  3. Name the new folder Materials.

  1. Assets > Create > Material
  2. Name the new material YellowMaterial.
  3. In the Inspector for YellowMaterial, click the color for Albedo and choose a bright yellow color.

Repeat this process to create a BlueMaterial and a RedMaterial.

Next you will create 3D assets using the materials you just created.

  1. Choose GameObject > 3D Object > Cube.
  2. Rename the Cube to LocalReferencePoint.
  3. In the Inspector for LocalReferencePoint, click the gear icon to Reset the transform.
    This ensures your new object is located at (0, 0, 0) with no rotation or scaling.
  4. Set Scale to (0.1, 0.1, 0.1)
  5. Drag the YellowMaterial from the Materials folder in the Project tab onto the LocalReferencePoint in the Hierarchy.
  6. Drag the LocalReferencePoint into the Assets/Prefabs folder in the Project tab.
  7. Delete the LocalReferencePoint from the Hierarchy.

Repeat this process to create a blue sphere prefab:

Repeat one more time to create a red cylinder prefab:

UI Assets

Next you will create two UI elements: one for displaying text, and another for entering text.

Create an InputField.

  1. Choose GameObject > UI > InputField.

This adds game objects to your Hierarchy: an EventSystem and a Canvas with a child object InputField.

  1. In the Hierarchy view, select the InputField.
  2. In the Canvas > InputField inspector, look for the Rect Transform:

  1. In the Hierarchy, expand the InputField to reveal the child game objects.
  2. Select the Placeholder game object.

  1. In the inspector for the Text (Script) component for the Canvas > InputField > Placeholder, set the following:
  1. Apply the same Text (Script) component settings (Font Size, Horizontal & Vertical Overflow) to the Canvas > InputField > Text game object.

Create a Text, with a background Panel.

  1. Choose GameObject > UI > Panel.

This adds a new Panel game object to your Canvas.

  1. In the Hierarchy view, select the Panel.

  1. In the Canvas > Panel inspector, look for the Rect Transform:
  1. Choose GameObject > UI > Text.

This adds a Text game object to your Canvas.

  1. In the Hierarchy, drag the new Text object onto the Panel to make it a child object.

  1. In the Canvas > Panel > Text inspector, look for the Rect Transform:
  1. In the Inspector in the Text (Script) component, change the following settings:

You are now ready to add a controller script and write the code to create and resolve Cloud Reference Points.

Create a folder for Scripts

  1. In your Project view, select the Assets folder.
  2. Choose Assets > Create > Folder.
  3. Name the new folder Scripts.

Create a controller script and game object

  1. Choose GameObject > Create Empty.
  2. Rename the game object App Controller.
  3. In the Inspector, click the gear icon and select Reset to reset the transform.
  4. Click the Add Component button.
  5. Type New Script to filter the options, and select New Script.
  6. Type AppController, and click Create and add.
  7. In the Project view, select the Assets folder.
  8. In the Project view, drag the new AppController into the new Scripts folder.

Edit the AppController Script

Open the AppController script in your preferred editor. If one is configured to work with Unity, you can double click the name of the script in the component in the Inspector.

Add using directives

Update your script to use the following assemblies.

using System.Collections.Generic;
using Google.XR.ARCoreExtensions;
using UnityEngine;
using UnityEngine.EventSystems;
using UnityEngine.UI;
using UnityEngine.XR.ARFoundation;

Add public properties

The following properties will be connected in the Unity editor and used in this script. Add them to your script class.

public class AppController : MonoBehaviour
{
    public GameObject HostedPointPrefab;
    public GameObject ResolvedPointPrefab;
    public ARReferencePointManager ReferencePointManager;
    public ARRaycastManager RaycastManager;
    public InputField InputField;
    public Text OutputText;
}

Define app modes

Define a new enumeration called AppMode that describes the current mode of the app, and also creates an m_AppMode variable to hold the current app mode. These will be described in more detail shortly.

{
    ...

    private enum AppMode
    {
        // Wait for user to tap screen to begin hosting a point.
        TouchToHostCloudReferencePoint,

        // Poll hosted point state until it is ready to use.
        WaitingForHostedReferencePoint,

        // Wait for user to tap screen to begin resolving the point.
        TouchToResolveCloudReferencePoint,

        // Poll resolving point state until it is ready to use.
        WaitingForResolvedReferencePoint,
    }

    private AppMode m_AppMode = AppMode.TouchToHostCloudReferencePoint;
    private ARCloudReferencePoint m_CloudReferencePoint;
    private string m_CloudReferenceId;

Add to the Unity Update() Method

The Unity Update() method is called once per frame. This gives your app an opportunity to update the state over time. For this codelab, you will use this method and the AppMode you just defined to operate in four distinct phases. Add the following code to your Update() method:

TouchToHostCloudReferencePoint - The app will wait for a screen touch. When one is detected, it will raycast into the scene to find a point in the environment. That will be used to host a new Cloud Reference Point.

    void Update()
    {
        if (m_AppMode == AppMode.TouchToHostCloudReferencePoint)
        {
            OutputText.text = m_AppMode.ToString();

            if (Input.touchCount >= 1
                && Input.GetTouch(0).phase == TouchPhase.Began
                && !EventSystem.current.IsPointerOverGameObject(
                        Input.GetTouch(0).fingerId))
            {
                List<ARRaycastHit> hitResults = new List<ARRaycastHit>();
                RaycastManager.Raycast(Input.GetTouch(0).position, hitResults);
                if (hitResults.Count > 0)
                {
                    Pose pose = hitResults[0].pose;

                    // Create a reference point at the touch.
                    ARReferencePoint referencePoint =
                        ReferencePointManager.AddReferencePoint(
                            hitResults[0].pose);

                    // Create Cloud Reference Point.
                    m_CloudReferencePoint =
                        ReferencePointManager.AddCloudReferencePoint(
                            referencePoint);
                    if (m_CloudReferencePoint == null)
                    {
                        OutputText.text = "Create Failed!";
                        return;
                    }

                    // Wait for the reference point to be ready.
                    m_AppMode = AppMode.WaitingForHostedReferencePoint;
                }
            }
        }

   ...

WaitingForHostedReferencePoint - After a newly hosted point has been created, the app must wait until it is ready. This mode polls the state of the point, and when ready, reports the Cloud Reference Id for the point.

        ...

        else if (m_AppMode == AppMode.WaitingForHostedReferencePoint)
        {
            OutputText.text = m_AppMode.ToString();

            CloudReferenceState cloudReferenceState =
                m_CloudReferencePoint.cloudReferenceState;
            OutputText.text += " - " + cloudReferenceState.ToString();

            if (cloudReferenceState == CloudReferenceState.Success)
            {
                GameObject cloudAnchor = Instantiate(
                                             HostedPointPrefab,
                                             Vector3.zero,
                                             Quaternion.identity);
                cloudAnchor.transform.SetParent(
                    m_CloudReferencePoint.transform, false);

                m_CloudReferenceId = m_CloudReferencePoint.cloudReferenceId;
                m_CloudReferencePoint = null;

                m_AppMode = AppMode.TouchToResolveCloudReferencePoint;
            }
        }

        ...

TouchToResolveCloudReferencePoint - The app waits for a screen touch. When one is detected, it attempts to resolve an existing Cloud Reference Point (created locally in this case.)

        ...

        else if (m_AppMode == AppMode.TouchToResolveCloudReferencePoint)
        {
            OutputText.text = m_CloudReferenceId;

            if (Input.touchCount >= 1
                && Input.GetTouch(0).phase == TouchPhase.Began
                && !EventSystem.current.IsPointerOverGameObject(
                        Input.GetTouch(0).fingerId))
            {
                m_CloudReferencePoint =
                    ReferencePointManager.ResolveCloudReferenceId(
                        m_CloudReferenceId);
                if (m_CloudReferencePoint == null)
                {
                    OutputText.text = "Resolve Failed!";
                    m_CloudReferenceId = string.Empty;
                    m_AppMode = AppMode.TouchToHostCloudReferencePoint;
                    return;
                }

                m_CloudReferenceId = string.Empty;

                // Wait for the reference point to be ready.
                m_AppMode = AppMode.WaitingForResolvedReferencePoint;
            }
        }

        ...

WaitingForResolvedReferencePoint - As with the Hosted Reference Point, the app must wait until the point is ready before use. As before, this mode polls the state of the point until ready.

        ...

        else if (m_AppMode == AppMode.WaitingForResolvedReferencePoint)
        {
            OutputText.text = m_AppMode.ToString();

            CloudReferenceState cloudReferenceState =
                m_CloudReferencePoint.cloudReferenceState;
            OutputText.text += " - " + cloudReferenceState.ToString();

            if (cloudReferenceState == CloudReferenceState.Success)
            {
                GameObject cloudAnchor = Instantiate(
                                             ResolvedPointPrefab,
                                             Vector3.zero,
                                             Quaternion.identity);
                cloudAnchor.transform.SetParent(
                    m_CloudReferencePoint.transform, false);

                m_CloudReferencePoint = null;

                m_AppMode = AppMode.TouchToHostCloudReferencePoint;
            }
        }

Enable InputField interactions

Finally, add a method OnInputEndEdit, and add a listener in the Unity Start() method. This allows the user to enter a Cloud Anchor Id manually, obtained from another client (running this or an equivalent app using the same ARCore Cloud Anchor service API key).

    void Start()
    {
        InputField.onEndEdit.AddListener(OnInputEndEdit);
    }

    private void OnInputEndEdit(string text)
    {
        m_CloudReferenceId = string.Empty;

        m_CloudReferencePoint =
            ReferencePointManager.ResolveCloudReferenceId(text);
        if (m_CloudReferencePoint == null)
        {
            OutputText.text = "Resolve Failed!";
            m_AppMode = AppMode.TouchToHostCloudReferencePoint;
            return;
        }

        // Wait for the reference point to be ready.
        m_AppMode = AppMode.WaitingForResolvedReferencePoint;
    }

Configure Controller Script

Save the script and return to Unity. If there were no errors, the script component on your App Controller will update to look like this:

To fill in these fields, drag the appropriate game objects or asset prefabs onto the corresponding field:

  1. Drag HostedPoint prefab onto the Hosted Point Prefab field.
  2. Drag ResolvedPoint prefab onto the Resolved Point Prefab field.
  3. Drag AR Session Origin onto the Reference Point Manager field.
  4. Drag AR Session Origin onto the Raycast Manager field.
  5. Drag Canvas > InputField onto the Input Text field.
  6. Drag Canvas > Panel > Text onto the Output Text field.

Build and Run your app

Follow these steps to build and run your app:

  1. Plug in an Android device via USB.
  2. Choose File > Build and Run.
  3. Save As: ARCodeLab.apk.
  4. Wait for the app to build and deploy to your device.

The first time you attempt to deploy the app to your device you will need to Allow USB debugging on the device, simply select OK to continue.

The first time you run your app on the device you will be asked if the app has permission to use your device camera, you must allow access to continue using AR functionality.

Testing your app

When you run your app you can test the basic behavior as follows:

  1. Holding your device, move around your space, slowly scanning an area you would like to host a new Cloud Reference Point from. Try to collect at least 10 seconds of data, and scan the area from several directions before moving to the next step. You should see feature points and detected planes render on your screen.
  2. Tap on the screen to create a new Cloud Reference Point. Selecting a point near interesting features, as opposed to empty space or blank walls will generally yield better results. (If you connected the LocalReferencePoint prefab to your AR Reference Point Manager, you will see it appear at the location.)
  3. Wait for the new Cloud Reference Point to be ready. The HostedPoint prefab (a blue sphere) will appear at the location when it is ready to use, and the Cloud Reference Id will be displayed in the text field at the bottom of your screen.
  4. Again scan around the area with the reference point for at least 10 seconds, then tap the screen again to resolve the current point. (This will ultimately make two copies of the same point, but it demonstrates the flow in code you will use when you receive the Cloud Reference Id from another client using your app.)
  5. Wait for the new Cloud Reference Point to be ready. When it is resolved and ready, another LocalReferencePoint prefab (yellow cube) will appear coinciding with the ResolvedPoint prefab (red cylinder). This second local Reference Point will not appear in future versions of AR Foundation.
  6. You may repeat the process to create another Cloud Reference Point in step 1, or...
  7. Obtain a Cloud Reference Id from another device (run the app on a second device) and enter the Id into the text input field on the first device.
  8. Wait as you did in step 5 until the corresponding Cloud Reference Point is created and ready to use.

Congratulations, you've successfully built and run your first Cloud Anchor-based Augmented Reality app using Google's ARCore Extensions for Unity's AR Foundation!

Setting up your Android device for deveopment

  1. Connect your device to your development machine with a USB cable. If you developed on Windows, you might need to install the appropriate USB driver for your device.
  2. Perform the following steps to enable USB debugging in the Developer options window:

You can find more detailed information about this process at Google's android developer website.

Build failure related to licenses

If you encounter a build failure related to licenses (Failed to install the following Android SDK packages as some licences have not been accepted), you can use the following commands to review and accept these licenses:

cd <path to Android SDK>

tools/bin/sdkmanager --licenses