Unfold your camera experience

1. Before you begin

What's special about foldables?

Foldables are once-in-a-generation innovations. They provide unique experiences, and with them come unique opportunities to delight your users with differentiated features like tabletop UI for hands-free usage.

Prerequisites

• Basic knowledge of developing Android apps
• Basic knowledge of Hilt Dependency Injection framework

What you'll build

In this codelab, you build a camera app with optimized layouts for foldable devices.

You start with a basic camera app that does not react to any device posture or take advantage of the better rear camera for enhanced selfies. You update the source code in order to move the preview to the smaller display when the device is unfolded and react to the phone being set in tabletop mode.

While the camera app is the most convenient use case for this API, both of the features you learn in this codelab can be applied to any app.

What you'll learn

• How to use Jetpack Window Manager to react to posture changing
• How to move your app to the smaller display of a foldable

What you'll need

• A recent version of Android Studio
• A foldable device or foldable emulator

Note: As you develop camera apps for foldable devices, you need to take several things into account, such as supporting resizable surfaces and handling orientation and aspect ratio. To learn more, go to https://developer.android.com/training/camera2/camera-preview.

For this codelab, you use Camera ViewFinder, a lightweight library, fully compatible with your existing Camera2 codebase, that handles most of the nuances for you. To learn more, go to

https://developer.android.com/reference/androidx/camera/viewfinder/package-summary.

2. Get set up

Get the starting code

1. If you have Git installed, you can simply run the command below. To check whether Git is installed, type git --version in the terminal or command line and verify that it executes correctly.

git clone https://github.com/android/large-screen-codelabs.git

2. Optional: If you do not have Git, you can click the following button to download all the code for this codelab: Download source code

Open the first module

• In Android Studio, open the first module under /step1.

Note: You may want to follow the recommendations for updating the IDE and SDK tools if you encounter any problems.

If you're asked to use the latest Gradle version, go ahead and update it.

3. Run and observe

1. Run the code on module step1.

As you can see, this is a simple camera app. You can toggle between front and back camera and you can adjust the aspect ratio. However, the first button from the left currently does nothing—but it's going to be the entry point for the Rear Selfie mode.

2. Now, try to put the device in a half-opened position, in which the hinge is not completely flat or closed but forms a 90-degree angle.

As you can see, the app does not respond to different device postures and so layout does not change, leaving the hinge in the middle of the viewfinder.

4. Learn about the Jetpack WindowManager

The Jetpack WindowManager library helps app developers build optimized experiences for foldable devices. It contains the FoldingFeature class that describes a fold in a flexible display or a hinge between two physical display panels. Its API provides access to important information related to the device:

• state() returns FLAT if the hinge is opened at 180 degrees or HALF_OPENED otherwise.
• orientation() returns FoldingFeature.Orientation.HORIZONTAL if the FoldingFeature width is greater than the height; otherwise, returns FoldingFeature.Orientation.VERTICAL.
• bounds() provides the boundaries of the FoldingFeature in a Rect format.

The FoldingFeature class contains additional information, like occlusionType() or isSeparating(), but this codelabe doesn't explore those in depth.

Starting from version 1.1.0-beta01, the library uses the WindowAreaController, an API that enables Rear Display Mode to move the current window to the display that is aligned with the rear camera, which is great for taking selfies with the rear camera and many other use cases!

Add dependencies

• In order to use Jetpack WindowManager in your app, you need to add the following dependencies to your module-level build.gradle file:

step1/build.gradle

def work_version = '1.1.0-beta01'
implementation "androidx.window:window:$work_version"
implementation "androidx.window:window-java:$work_version"
implementation "androidx.window:window-core:$work_version"

Now you can access both the FoldingFeature and WindowAreaController classes in your app. You use them to build the ultimate foldable camera experience!

5. Implement the Rear Selfie mode

Start with Rear Display mode. The API that allows this mode is the WindowAreaControllerJavaAdapter, which requires an Executor and returns a WindowAreaSession that stores the current state. This WindowAreaSession should be retained when your Activity is destroyed and re-created, so you store it inside a ViewModel in order to safely store it across configuration changes.

1. Declare these variables in your MainActivity:

step1/MainActivity.kt

private lateinit var windowAreaController: WindowAreaControllerJavaAdapter
private lateinit var displayExecutor: Executor

2. And initialize them in the onCreate() method:

step1/MainActivity.kt

windowInfoTracker = WindowInfoTracker.getOrCreate(this)
displayExecutor = ContextCompat.getMainExecutor(this)
windowAreaController = WindowAreaControllerJavaAdapter(WindowAreaController.getOrCreate())

Now your Activity is ready to move the content on the smaller display, but you need to store the session.

3. To store the session, open the CameraViewModel and declare this variable inside it:

step1/CameraViewModel.kt

var rearDisplaySession: WindowAreaSession? = null
        private set

You need the rearDisplaySession to be a variable, as it changes every time you create one, but you want to make sure it cannot be updated from the outside since you now create a method that updates it whenever it's needed.

4. Paste this code inside the CameraViewModel:

step1/CameraViewModel.kt

fun updateSession(newSession: WindowAreaSession? = null) {
        rearDisplaySession = newSession
}

This method is invoked every time your code needs to update the session, and it's helpful to encapsulate it in a single access point.

The Rear Display API works with a listener approach: when you request to move the content to the smaller display, you initiate a session that is returned through the listener's onSessionStarted() method. When you instead want to return to the inner (and bigger) display, you close the session, and you get a confirmation in the onSessionEnded() method. You leverage these methods in order to update the rearDisplaySession inside the CameraViewModel.To create such a listener, you need to implement the WindowAreaSessionCallback interface.

5. Modify the MainActivity declaration so that it implements the WindowAreaSessionCallback interface:

step1/MainActivity.kt

class MainActivity : AppCompatActivity(), WindowAreaSessionCallback

Now, implement the onSessionStarted and onSessionEnded methods inside the MainActivity.For the first one, you want to save the WindowAreaSession, and in the second you reset it to null. This is particularly useful as the presence of the WindowAreaSession lets you decide whether to start a session or close an existing one.

step1/MainActivity.kt

override fun onSessionEnded() {
    viewModel.updateSession(null)
}

override fun onSessionStarted(session: WindowAreaSession) {
    viewModel.updateSession(session)
}

6. In the MainActivity.kt file, write the last piece of code needed for this API to work:

step1/MainActivity.kt

private fun startRearDisplayMode() {
   if (viewModel.rearDisplaySession != null) {
      viewModel.rearDisplaySession?.close()
   } else {
      windowAreaController.startRearDisplayModeSession(
         this,
         displayExecutor,
         this
      )
   }
}

As mentioned earlier, in order to understand what action to take, you need to check the presence of the rearDisplaySession inside the CameraViewModel: if it is not null, then the session is already taking place, so it closes. On the other hand, if it is null, you use the windowAreaController to start a new session, passing the Activity twice. The first time is used as a Context and the second as a WindowAreaSessionCallback listener.

7. Now, build and run the app. If you then unfold your device and tap on the rear display button, you are prompted with a message like this:

8. Click Switch screens now and see your content moved to the outer display!

6. Implement the Tabletop mode

Now it's time to make your app fold-aware: you move your content on the side or above the hinge of the device based on the orientation of the fold. To do so, you'll be acting inside the FoldingStateActor so that your code is decoupled from the Activity for easier readability.

Note: For this part, Hilt was implemented so that you could get all the dependencies right in the constructor. If you want to learn more about Hilt and dependency injection on Android, check out the Using Hilt in your Android app codelab.

The core part of this API consists in the WindowInfoTracker interface, which is created with a static method that requires an Activity:

step1/CameraCodelabDependencies.kt

@Provides
fun provideWindowInfoTracker(activity: Activity) =
        WindowInfoTracker.getOrCreate(activity)

You don't need to write this code as it's already present, but it is useful to understand how the WindowInfoTracker is built.

1. To listen for any window change, listen to these changes in the onResume() method of your Activity:

step1/MainActivity.kt

lifecycleScope.launch {
    foldingStateActor.checkFoldingState(
         this@MainActivity, 
         binding.viewFinder
    )
}

2. Now, open the FoldingStateActor file, as it's time to fill in the checkFoldingState() method.

As you already saw, it runs in the RESUMED phase of your Activity, and it leverages the WindowInfoTracker in order to listen to any layout change.

step1/FoldingStateActor.kt

windowInfoTracker.windowLayoutInfo(activity)
      .collect { newLayoutInfo ->
         activeWindowLayoutInfo = newLayoutInfo
         updateLayoutByFoldingState(cameraViewfinder)
      }

By using the WindowInfoTracker interface, you can call windowLayoutInfo() in order to collect a Flow of WindowLayoutInfo that contains all the available information in DisplayFeature.

The last step is to react to these changes and move the content accordingly. You do this inside the updateLayoutByFoldingState() method, one step at a time.

3. Make sure the activityLayoutInfo contains some DisplayFeature properties, and that at least one of them is a FoldingFeature, otherwise you don't want to do anything:

step1/FoldingStateActor.kt

val foldingFeature = activeWindowLayoutInfo?.displayFeatures
            ?.firstOrNull { it is FoldingFeature } as FoldingFeature?
            ?: return

4. Calculate the position of the fold in order to make sure the device position is impacting your layout and is not outside the bounds of your hierarchy:

step1/FoldingStateActor.kt

val foldPosition = FoldableUtils.getFeaturePositionInViewRect(
            foldingFeature,
            cameraViewfinder.parent as View
        ) ?: return

Now, you're sure that you have a FoldingFeature that impacts your layout, so you need to move your content.

5. Check if the FoldingFeature is HALF_OPEN or else you just restore the position of your content. If it is HALF_OPEN, you need to run another check and act differently based on the orientation of the fold:

step1/FoldingStateActor.kt

if (foldingFeature.state == FoldingFeature.State.HALF_OPENED) {
    when (foldingFeature.orientation) {
        FoldingFeature.Orientation.VERTICAL -> {
            cameraViewfinder.moveToRightOf(foldPosition)
        }
        FoldingFeature.Orientation.HORIZONTAL -> {
            cameraViewfinder.moveToTopOf(foldPosition)
        }
    }
} else {
    cameraViewfinder.restore()
}

If the fold is VERTICAL, you move your content to the right, otherwise you move it on top of the fold position.

6. Build and run your app, and then unfold your device and place it in tabletop mode to see the content move accordingly!

7. Congratulations

In this codelab you learned what's special about foldables, posture changes, and the Rear Display API.

Further reading

• Get started with large screens
• Learn about foldables

Reference

• Camera preview