Automated Accessibility Testing using Espresso

1. Introduction

Last Updated: 2020-04-21

What you'll learn

There are several high-level approaches to testing that you can leverage as a developer to ensure you're creating accessible experiences within your apps:

1. Automated tests for accessibility can run alongside your existing unit, UI, or integration tests as part of a presubmit check or continuous integration (CI) solution.
2. Testing with tools such as Accessibility Scanner, which you can do at every stage of the development and release process.
3. Manual testing using Android's built-in accessibility services to ensure that your app works end-to-end for users with accessibility needs in real-world conditions.
4. User testing that includes people with accessibility needs, who can provide you with valuable feedback about using your app.

This codelab focuses on writing automated tests for accessibility; specifically, it shows you how to integrate accessibility testing into a test suite that uses the Espresso test framework. By writing automated tests, you can identify opportunities to improve your app's accessibility at development time, including in your app's regression tests.

What you'll build

In this codelab, you'll be working with an existing app, Counter. This app allows users to track, increment, and decrement a numerical count. The UI for the app is very basic, and it consists of just a few views:

1. A TextView heading
2. The count as a TextView
3. Two ImageButton to increment and decrement the count

You won't write this app from scratch; that's already done. And you won't write the Espresso tests; those are already written. Instead, you'll focus on augmenting your Espresso tests so that they also check for opportunities to improve the app's accessibility.

The codelab consists of the following steps:

1. Getting the source code.
2. Understanding the existing Espresso tests.
3. Learning how to add checks for accessibility.
4. Understanding opportunities to improve the app's accessibility that you discover on the way.
5. Applying the accessibility improvements, and then seeing your tests pass.
6. Understanding how to configure your accessibility test environment when using Espresso.

What you'll need

• A rudimentary knowledge of UI testing. To view information specific to UI testing on Android, visit the Espresso basics page.
• Access to an Android device running Lollipop (API level 21) or higher.

2. Getting set up

Download the code

You can get the source code for the starting version of the app from GitHub. Clone the repo, and open Counter in Android Studio.

You'll work in the master branch throughout this codelab. If you get stuck (or just want to peek ahead), you can see the finished version of the app on the accessible branch.

Using the demo app

Run the app and you should be able to launch the Counter app.

Play with the demo app a little bit, using the increment ("+") and decrement ("-") buttons to change the count.

For many users, this is an easy app to use. But for some users, the app presents formidable challenges. In this codelab, you'll focus on how automated testing can help you discover accessibility issues in the app.

Understanding the tests

The Espresso tests are located in the CounterInstrumentedTest class.

For the purpose of this codelab, there is only a single test, which checks that the code to increment the count works correctly (for brevity, the test for decrementing the count is omitted):

class CounterInstrumentedTest {
    ...    
    @Test
    fun testIncrement() {
        Espresso.onView(withId(R.id.add_button))
            .perform(ViewActions.click())
        Espresso.onView(withId(R.id.countTV))
            .check(matches(withText("1")))
    }
}

Notice that the test doesn't have anything to do with accessibility. This test follows the common testing logic for testing a UI on Android:

1. Locate one or more views in the view hierarchy: the add button
2. Interact with the views in some manner: perform a click.
3. Make an assertion about the state of the UI: assert that the updated count is displayed.

Run the tests

First, make sure that your computer is connected to a device with USB debugging enabled.

Now run the tests by clicking on the green arrow button immediately to the left of @Test fun testIncrement(). If you're using a physical device that's connected over USB, make sure the device is unlocked with the display on. Note that pressing Ctrl+Shift+F10 (Control+Shift+R on a Mac) runs the tests in the currently-opened file.

The test should run to completion and it should pass, confirming that incrementing the count functions as expected.

In the next section, you'll modify the test to additionally check for accessibility.

3. The Accessibility Test Framework

Automated accessibility testing in Android uses the Accessibility Testing Framework for Android (ATF). ATF is a Java-based library that uses a check-based system. Each check assesses a specific aspect of an Android UI to identify opportunities for improving an app's accessibility.

ATF is open source and is available on GitHub.

Accessibility Checks with ATF

So, what checks does the Accessibility Test Framework perform?

• Presence of labels, which allow users of screen readers to understand content within your app.
• Touch target size; larger touch targets make interaction easier for those with limited manual dexterity.
• Color contrast between text and images, which can impact the legibility of your app's content.
• Other properties of UI elements to test whether Android's accessibility services can properly convey your app's semantics to end users.

4. ATF + Espresso

You can integrate ATF checks into a testing framework like Espresso. ATF is available as an optional component, which allows you to leverage your existing Espresso tests to assess your app's accessibility.

You don't need to write explicit tests for accessibility. Instead, after you've integrated ATF into your Espresso tests, you just run your existing tests and the accessibility checks run additionally automatically.

How does this work? ATF runs checks as you interact with a View using a ViewAction (more on that below). So if you, say, click a button as part of the test, ATF looks at the button, and potentially the UI around the button, and it performs an accessibility check.

Each issue reported by ATF has an associated AccessibilityCheckResultType; primarily ERROR, WARNING, and INFO. By default, the ATF + Espresso integration triggers a failure for results of type ERROR and throws an exception, which causes the test to fail.

Enabling accessibility checks

For Espresso, you can enable accessibility checks by calling AccessibilityChecks.enable() from a setup method. Adding this one line of code allows you to test your UI for accessibility, making it straightforward to integrate accessibility checking into your test suite.

You'll need to add the correct androidTestImplementation. Note that the following has been added for you in app/build.gradle:

...
androidTestImplementation 'androidx.test.espresso:espresso-accessibility:3.3.0-alpha05'
...

And the relevant import has been added to CounterInstrumentedTest.kt:

import androidx.test.espresso.accessibility.AccessibilityChecks

Now, find the companion object in CounterInstrumentedTest.kt and uncomment the AccessibilityChecks.enable() line. Your code should look like this:

companion object {
    @BeforeClass @JvmStatic
    fun enableAccessibilityChecks() {
        AccessibilityChecks.enable()
    }
}

Run the tests

Now run the test again. This time, you'll notice that the test fails. Here is the (truncated) output of the test failure:

There were 2 accessibility errors:
AppCompatImageButton{id=2131165210, ...}: 
View is missing speakable text needed for a screen reader,
AppCompatImageButton{id=2131165210,...}: View falls below the minimum recommended size for touch targets. Minimum touch target size is 48x48dp. Actual size is 24.0x24.0dp (screen density is 2.6).
...

Understanding the test failure

The test failed because ATF found two opportunities to improve the app's accessibility:

1. The add ("+") ImageButton contains an image but no label. It needs a label, so that a screenreader user can understand the purpose of the button.
2. The ImageButton also needs a larger touch target so that users with impaired manual dexterity can interact with the button more easily.

ViewActions

You will notice that both errors are limited to the add button. The test also refers to the count TextView, which it turns out also has opportunities for accessibility improvement (you'll explore those later in this codelab), but the test failure doesn't mention that view. Why? Why just the add button?

Let's look at the test again to try to understand exactly what is going on:

class CounterInstrumentedTest {
    ...    
    @Test
    fun testIncrement() {
        Espresso.onView(withId(R.id.add_button))
            .perform(ViewActions.click())
        Espresso.onView(withId(R.id.countTV))
            .check(matches(withText("1")))
    }
}

The add button is evaluated for accessibility because you performed a ViewAction on that view; the counter TextView is bypassed because you did not perform a ViewAction on it.

There are a couple of rules to keep in mind when enabling accessibility checks:

1. A ViewAction from the ViewActions class is required for ATF checks to run. If you don't interact with a view using a ViewAction, ATF checks are bypassed.
2. Performing an action like click() without a ViewAction also bypasses ATF checks.

Accessibility checks are skipped for this test:

class CounterInstrumentedTest {
    ...    
    @Test
    fun testIncrement() {
        <view>.performClick()) // No accessibility checks
        ...
    }
}

And accessibility checks run for this test:

class CounterInstrumentedTest {
    ...    
    @Test
    fun testIncrement() {
        <view>.perform(ViewActions.click()) // Accessibility checks
        ...
    }
}

You'll see later in this codelab how to expand accessibility checks to views in your layout hierarchy without performing ViewActions on every view.

Improving your UI

In this step, you'll make changes in res/layout/activity_main.xml to address the suggestions from ATF that cause your tests to fail (remember, ATF found two opportunities to improve accessibility, including a label, and increasing the touch target size):

There were 2 accessibility errors:
AppCompatImageButton{id=2131165210, ...}: 
View is missing speakable text needed for a screen reader,
AppCompatImageButton{id=2131165210,...}: View falls below the minimum recommended size for touch targets. Minimum touch target size is 48x48dp. Actual size is 24.0x24.0dp (screen density is 2.6).
...

Adding a label

First, you'll add a label to the add button.

Open res/layout/activity_main.xml and look at the code for the first ImageButton (you'll notice a lint warning about a missing contentDescription):

Add the missing contentDescription to the ImageButton layout definition:

<ImageButton
    android:id="@+id/add_button"
    ...
    android:contentDescription="@string/increment"
    ... />

The string has already been defined in strings.xml for you. The contentDescription should always be a localized string so that it can be properly translated for the user.

Run the test again, and you'll no longer see a test failure related to the button's label.

Expanding the touch target

Now you'll address ATF's other recommendation, which relates to the touch target size of the button. The touch size for the button is 24x24dp, and the failing test message indicates that the recommended minimum touch target size is 48x48dp.

You have several options for increasing the touchable area of the buttons. For example, you can do either of the following:

• Add padding around the icons.
• Add a minWidth and/or a minHeight (this will make the icons larger).
• Register a TouchDelegate.

Let's increase the area for both buttons.

Looking at res/layout/activity_main.xml, you see the following definitions for the two buttons:

<ImageButton
   android:id="@+id/add_button"
   android:layout_width="wrap_content"
   android:layout_height="wrap_content"
   ... />


<ImageButton
   android:id="@+id/subtract_button"
   android:layout_width="wrap_content"
   android:layout_height="wrap_content"
  ... />

Add some padding to each view:

<ImageButton
   ...
   android:padding="@dimen/icon_padding"
   ... />


<ImageButton
   ...
   android:padding="@dimen/icon_padding"
  ... />

The value of @dimen/icon_padding is set to 12dp (see res/dimens.xml). When this padding is applied, the touchable area of the control becomes 48dp X 48dp (24dp + 12dp in each direction).

Run the test again. The test failure related to the touch targets no longer occurs, so the test passes.

5. Working with the AccessibilityValidator object

Congratulations! You have successfully integrated accessibility checks with Espresso, and you've already made your app more accessible.

In this section, you'll learn how to configure the ATF integration with Espresso.

Remember that you integrated ATF checks by invoking AccessibilityChecks.enable() before running your test. Here's the code again:

companion object {
    @BeforeClass @JvmStatic
    fun enableAccessibilityChecks() {
        AccessibilityChecks.enable()
    }
}

Calling AccessibilityChecks.enable() returns an AccessibilityValidator object. You can use this AccessibilityValidator to customize ATF's behavior inside your tests.

Perform checks from a window's root view

All the accessibility checks you encountered so far were associated with the add button, which is the view on which you performed a ViewAction. You'll now configure your tests to examine other views in the hierarchy, without having to perform additional ViewActions on those views.

Using the AccessibilityValidator object, you can call setRunChecksFromRootView(true) to get expanded accessibility coverage by checking the entire view hierarchy on each ViewAction.

Modify your enableAccessibilityChecks() method as shown here:

companion object {
    @BeforeClass @JvmStatic
    fun enableAccessibilityChecks() {
        AccessibilityChecks.enable().setRunChecksFromRootView(true)
    }
}

Run your tests again.

This time, the test fails with the following (truncated) output:

...
There was 1 accessibility error:
AppCompatTextView{id=-1, ...}: TextView does not have required contrast of 3.000000. Actual contrast is 2.455571
at ...

The text and the view's background should have a larger color contrast. By setting setRunChecksFromRootView(true), you found more opportunities to improve your app's accessibility.

For now, leave the test failure as is. You'll address it a bit later in the codelab.

Suppressing test failures

Sometimes, you may have failing tests (as you do here), but you cannot immediately address those test failures. For this reason, you might want to temporarily suppress test failures. There are two options for doing this:

1. Instead of failing tests, you can simply log the failures, or,
2. you can ignore specific controls or categories of failures.

Logging the failure

By default, if your Espresso test runs into opportunities for accessibility improvement, ATF throws an exception, which causes your test to fail. But you can use the AccessibilityValidator to simply log the test failure using the setThrowExceptionForErrors option.

Modify your enableAccessibilityChecks() method as shown here:

companion object {
    @BeforeClass @JvmStatic
    fun enableAccessibilityChecks() {
        AccessibilityChecks.enable() // AccessibilityValidator.setRunChecksFromRootView(true)
           .setThrowExceptionForErrors(false)
    }
}

Run your tests.

The tests pass, but you'll see the suggestions from ATF in the logcat output (truncated version shown here):

Testing started at 10:51 ...

E/AccessibilityValidator: AppCompatTextView{id=2131165231, res-name=countTV, ...}: TextView does not have required contrast of 3.000000. Actual contrast is 2.455571

While logging can help keep your build green, it should be only a temporary solution, because actual test failures encourage you to make the recommended improvements to your app's accessibility.

Suppressing subsets of results

As an alternative to logging the messages from ATF, you can use setSuppressingResultMatcher to suppress specific checks that you don't immediately have time to address, keeping your build green as you address ATF's suggestions.

Test failures can be suppressed in several ways. For example, if it is known that a particular view could be more accessible, you can create a matcher to suppress that view.

Modify the code in the enableAccessibilityChecks() method to suppress errors associated with the counter TextView:

companion object {
    @BeforeClass
    @JvmStatic
    fun enableAccessibilityChecks() {
        AccessibilityChecks.enable().setRunChecksFromRootView(true)
                .setSuppressingResultMatcher(matchesViews(anyOf(withId(
                 R.id.countTV))))
    }
}

Run your tests.

You will notice that ATF doesn't make any accessibility suggestions, and the test now passes.

The preceding approach suppresses all suggestions for improving a particular view's accessibility.

You can, however, create a matcher to suppress certain types of suggestions:

companion object {
    @BeforeClass
    @JvmStatic
    fun enableAccessibilityChecks() {

        AccessibilityChecks.enable()
                .setRunChecksFromRootView(true)
                .setSuppressingResultMatcher(
                 matchesCheckNames(`is`("TextContrastViewCheck")))
    }
}

This code suppresses all suggestions related to ATF's TextContrastViewCheck in your test.

Suppressing results by matching views or checking names are only two of multiple options for suppressing known test failures; look at **ATF'**s AccessibilityCheckResultUtils.java for more options.

Note that creating an expansive list of issues to ignore can suppress more tests than intended. For this reason, be judicious when choosing issues to ignore. Here are some specific recommendations when suppressing results:

1. Make the matchers very specific, so they don't accidentally mask unrelated problems.
2. Include comments in your code that describe the problem being suppressed.
3. Remember to remove the supressions when the issues have been fixed, so they don't accidentally mask a future regression.

6. Fixing the tests

Remove all code for suppressing test failures. The enableAccessibilityChecks() method should now look like this:

companion object {
       @BeforeClass
       @JvmStatic
       fun enableAccessibilityChecks() {
           AccessibilityChecks.enable()
               .setRunChecksFromRootView(true)
       }
   }

Run your test again.

This time, the test fails with the following familiar suggestion (truncated output):

...
There was 1 accessibility error:
AppCompatTextView{id=-1, ...}: TextView does not have required contrast of 3.000000. Actual contrast is 2.455571
at ...

Improve the color contrast for the counter TextView by changing the text color from @color/grey to @color/darkGrey (these colors have already been defined in colors.xml):

<TextView
    android:id="@+id/countTV"
    ...
    android:textColor="@color/darkGrey"
    ... />

The contrast ratio is now 4.94:1, which is considerably better than what you had before:

	Background	Text Color	Contrast ratio
Before	#EEEEEE	Light gray (#999999)	2.45:1
After	#EEEEEE	Dark gray (#666666)	4.94:1

So, what constitutes adequate contrast? The Web Content Accessibility Guidelines recommend a minimum contrast ratio of 4.5:1 for all text, with a contrast ratio of 3.0:1 considered acceptable for large or bold text. You should try to meet or exceed these contrast ratios in your applications.

ATF will help you spot color contrast issues once you have a working app. But you should consider contrast issues while designing the color palette of your app. You can use one of the many contrast checkers available online to check your contrast ratios.

Now run the test again.

With the darker grey, the counter text has more color contrast with the background, so ATF shouldn't make any accessibility suggestions at all.. The test should successfully run to completion.

7. Links and Resources

We've covered a lot of topics related to Android accessibility in this codelab. Here are some links and resources you can explore:

• To learn more about accessibility testing, visit the accessibility testing page on developer.android.com.
• Get the source for the Accessibility Test Framework by visiting its GitHub repository.
• Download Accessibility Scanner from the Play Store. Accessibility Scanner is an app that suggests accessibility improvements for Android apps. Accessibility Scanner also uses the Accessibility Test Framework to identify opportunities to improve an app's accessibility, and it does not require access to an app's source code.