Web Capabilities codelab

1. Introduction and setup

Web Capabilities

We want to close the capability gap between the web and native and make it easy for developers to build great experiences on the open web. We strongly believe that every developer should have access to the capabilities they need to make a great web experience, and we are committed to a more capable web.

There are, however, some capabilities—like file system access and idle detection—that are available to native but aren't available on the web. These missing capabilities mean that some types of apps can't be delivered on the web, or are less useful.

We will design and develop these new capabilities in an open and transparent way, using the existing open web platform standards processes while getting early feedback from developers and other browser vendors as we iterate on the design, to ensure an interoperable design.

Have a look at all of the in-flight capabilities we're currently working on!

What you'll build

In this codelab, you'll play around with a number of web APIs that are brand new or only available behind a flag. So, this codelab focuses on the APIs themselves and on use cases that these APIs unlock, rather than on building a specific final product.

What you'll learn

This codelab will teach you the basic mechanics of several bleeding-edge APIs. Note that these mechanics aren't set in stone quite yet, and we very much appreciate your feedback on the developer flow.

What you'll need

As the APIs featured in this codelab are really on the bleeding edge, requirements for each API vary. Please make sure to carefully read the compatibility information at the beginning of each section.

How to approach the codelab

The codelab is not necessarily meant to be worked through sequentially. Each section represents an independent API, so feel free to cherry-pick what interests you the most.

2. Badging API

Note: We're working on this API as part of the new Capabilities project. This codelab is based on the current design, and the API may change before the feature is standardized. See the Badging API post for the latest details.

Compatibility: The Badging API is available behind the #enable-experimental-web-platform-features flag, and as an Origin Trial. At present, it is only available on Windows and macOS, and it only works for PWAs that have been installed.

The goal of the Badging API is to bring users' attention to things that happen in the background. For the sake of simplicity of the demo in this codelab, let's use the API to bring users' attention to something that's happening in the foreground. You can then make the mental transfer to things that happen in the background.

Install Airhorner

For this API to work, you need a PWA that is installed to the home screen, so the first step is to install a PWA, such as the infamous, world-famous airhorner.com. Hit the Install button in the top-right corner or use the three-dots menu to install manually.

This will show a confirmation prompt, click Install.

You now have a new icon in your operating system's dock. Click it to launch the PWA. It will have its own app window and run in standalone mode.

Setting a badge

Now that you have a PWA installed, you need some numeric data (badges can only contain numbers) to display on a badge. A straightforward thing to count in The Air Horner is, sigh, the number of times it has been horned. Actually, with the installed Airhorner app, try horning the horn and check the badge. It counts one up whenever you horn.

So, how does this work? Essentially, the code is this:

let hornCounter = 0;
const horn = document.querySelector('.horn');
horn.addEventListener('click', () => {
  navigator.setExperimentalAppBadge(++hornCounter);
});

Note: If you get an Uncaught ReferenceError, remember to set the #enable-experimental-web-platform-features flag.

Sound the airhorn a couple of times and check the PWA's icon: it will update every. single. time. the airhorn sounds. As easy as that.

Clearing a badge

The counter goes up to 99 and then starts over. You can also manually reset it. Open the DevTools Console tab, paste the line below, and press Enter.

navigator.setExperimentalAppBadge(0);

Alternatively, you can also get rid of the badge by explicitly clearing it as shown in the following snippet. Your PWA's icon should now look again like at the beginning, clear and without a badge.

navigator.clearExperimentalAppBadge();

Feedback

What did you think of this API? Please help us by briefly responding to this survey:

Was this API intuitive to use?

Yes No

Did you get the example to run?

Yes No

Got more to say? Were there missing features? Please provide quick feedback in this survey. Thank you!

3. Native File System API

Note: We're working on this API as part of the new Capabilities project. This codelab is based on the current design, and the API may change before the feature is standardized. See the Native File System API post for the latest details.

Compatibility: The Native File System API is available behind the #native-file-system-api flag, and as an Origin Trial.

The Native File System API enables developers to build powerful web apps that interact with files on the user's local device. After a user grants a web app access, this API allows web apps to read or save changes directly to files and folders on the user's device.

Reading a file

The "Hello, world" of the Native File System API is to read a local file and get the file contents. Create a plain .txt file and enter some text. Next, navigate to any secure site (that is, a site served over HTTPS) like example.com and open the DevTools console. Paste the code snippet below in the console. Because the Native File System API requires a user gesture, we attach a double-click handler on the document. We will need the file handle later on, so we just make it a global variable.

document.ondblclick = async () => {
  window.handle = await window.chooseFileSystemEntries();
  const file = await handle.getFile();
  document.body.textContent = await file.text();
};

When you then double-click anywhere in the example.com page, a file picker shows up.

Note: If you get an Uncaught ReferenceError, remember to set the #native-file-system-api flag.

Select the .txt file that you have created before. The file contents will then replace the actual body contents of example.com.

Saving a file

Next, we want to make some changes. Therefore, let's make the body editable by pasting in the code snippet below. Now, you can edit the text as if the browser were a text editor.

document.body.contentEditable = true;

Now, we want to write these changes back to the original file. Therefore, we need a writer on the file handle, which we can obtain by pasting the snippet below in the console. Again we need a user gesture, so this time we wait for a click on the main document.

document.onclick = async () => {
  const writer = await handle.createWriter();
  await writer.truncate(0);
  await writer.write(0, document.body.textContent);
  await writer.close();
};

When you now click (not double-click) the document, a permission prompt shows up. When you grant permission, the contents of the file will be whatever you have edited in the body before. Verify the changes by opening the file in a different editor (or start the process again by double-clicking the document again and re-opening your file).

Congratulations! You've just created the smallest text editor in the world [citation needed].

Feedback

What did you think of this API? Please help us by briefly responding to this survey:

Was this API intuitive to use?

Yes No

Did you get the example to run?

Yes No

Got more to say? Were there missing features? Please provide quick feedback in this survey. Thank you!

4. Shape Detection API

Note: We're working on this API as part of the new Capabilities project. This codelab is based on the current design, and the API may change before the feature is standardized. See the Shape Detection API post for the latest details.

Compatibility: The Shape Detection API is available behind the #enable-experimental-web-platform-features flag. Support for this API is dependent on hardware acceleration features that vary by operating system:

• FaceDetector: Android, macOS, Windows
• BarcodeDetector: Android, macOS
• TextDetector: Android, macOS (bounding boxes only), Windows

The Shape Detection API provides access to accelerated shape detectors (e.g., for human faces) and works on still images and/or live image feeds. Operating systems have performant and highly optimized feature detectors such as the Android FaceDetector. The Shape Detection API opens up these native implementations and exposes them through a set of JavaScript interfaces.

Currently, the supported features are face detection through the FaceDetector interface, barcode detection through the BarcodeDetector interface, and text detection (optical character recognition) through the TextDetector interface.

Face Detection

A fascinating feature of the Shape Detection API is face detection. To test it, we need a page with faces. This page with the author's face is a good start. It will look something like in the screenshot below. On a supported browser, the boundary box of the face and the face landmarks will be recognized.

You may notice that currently the face landmarks look off in the screenshot. This is because of crbug.com/914348

¯_(ツ)_/¯.

You can see how little code was required to make this happen by remixing or editing the Glitch project, especially the script.js file.

If you want to go fully dynamic and not just work with the author's face, then go to this Google Search results page full of faces in a private tab or in guest mode. Now on that page, open the Chrome Developer Tools by right-clicking anywhere and then clicking Inspect. Next, on the Console tab, paste in the snippet below. The code will highlight detected faces with a semi-transparent red box.

Note that in general it's a bad idea to just copy and paste code into production websites, and Google will rightly warn you not to do so. In this case, it's fine, though, as we're not logged in (hence the private tab or guest mode).

Note: If you get an Uncaught ReferenceError, remember to set the #enable-experimental-web-platform-features flag.

document.querySelectorAll('img[alt]:not([alt=""])').forEach(async (img) => {
  try {
    const faces = await new FaceDetector().detect(img);
    faces.forEach(face => {
      const div = document.createElement('div');
      const box = face.boundingBox;
      const computedStyle = getComputedStyle(img);
      const [top, right, bottom, left] = [
        computedStyle.marginTop,
        computedStyle.marginRight,
        computedStyle.marginBottom,
        computedStyle.marginLeft
      ].map(m => parseInt(m, 10));
      const scaleX = img.width / img.naturalWidth;
      const scaleY = img.height / img.naturalHeight;
      div.style.backgroundColor = 'rgba(255, 0, 0, 0.5)';
      div.style.position = 'absolute';
      div.style.top = `${scaleY * box.top + top}px`;
      div.style.left = `${scaleX * box.left + left}px`;
      div.style.width = `${scaleX * box.width}px`;
      div.style.height = `${scaleY * box.height}px`;
      img.before(div);
    });
  } catch(e) {
    console.error(e);
  }
});

You will note that there are some DOMException mesages, and not all images are being processed. This is because the above-the-fold images are inlined as data URIs and can thus be accessed, whereas the below-the-fold images come from a different domain that isn't configured to support CORS. For the sake of the demo, we don't need to worry about this.

Face landmark detection

In addition to just faces, per se, macOS also supports the detection of face landmarks. To test the detection of face landmarks, paste the following snippet into the Console. Reminder: the line-up of the landmarks isn't perfect at all because of crbug.com/914348, but you can see where this is headed and how powerful this feature can be.

document.querySelectorAll('img[alt]:not([alt=""])').forEach(async (img) => {
  try {
    const faces = await new FaceDetector().detect(img);
    faces.forEach(face => {
      const div = document.createElement('div');
      const box = face.boundingBox;
      const computedStyle = getComputedStyle(img);
      const [top, right, bottom, left] = [
        computedStyle.marginTop,
        computedStyle.marginRight,
        computedStyle.marginBottom,
        computedStyle.marginLeft
      ].map(m => parseInt(m, 10));
      const scaleX = img.width / img.naturalWidth;
      const scaleY = img.height / img.naturalHeight;
      div.style.backgroundColor = 'rgba(255, 0, 0, 0.5)';
      div.style.position = 'absolute';
      div.style.top = `${scaleY * box.top + top}px`;
      div.style.left = `${scaleX * box.left + left}px`;
      div.style.width = `${scaleX * box.width}px`;
      div.style.height = `${scaleY * box.height}px`;
      img.before(div);

      const landmarkSVG = document.createElementNS('http://www.w3.org/2000/svg', 'svg');
      landmarkSVG.style.position = 'absolute';
      landmarkSVG.classList.add('landmarks');
      landmarkSVG.setAttribute('viewBox', `0 0 ${img.width} ${img.height}`);
      landmarkSVG.style.width = `${img.width}px`;
      landmarkSVG.style.height = `${img.height}px`;
      face.landmarks.map((landmark) => {                    
        landmarkSVG.innerHTML += `<polygon class="landmark-${landmark.type}" points="${
        landmark.locations.map((point) => {          
          return `${scaleX * point.x},${scaleY * point.y} `;
        }).join(' ')
      }" /></svg>`;          
      });
      div.before(landmarkSVG);
    });
  } catch(e) {
    console.error(e);
  }
});

Barcode detection

The second feature of the Shape Detection API is barcode detection. Similar to before, we need a page with barcodes, such as this one. When you open it in a browser, you will see the various QR codes deciphered. Remix or edit the Glitch project, especially the script.js file to see how it's done.

If you want something more dynamic, we can again use Google Image Search. This time, in your browser navigate to this Google Search results page in a private tab or in guest mode. Now paste the snippet below in the Chrome DevTools Console tab. After a short moment, the recognized barcodes will be annotated with the raw value and the barcode type.

document.querySelectorAll('img[alt]:not([alt=""])').forEach(async (img) => {
  try {
    const barcodes = await new BarcodeDetector().detect(img);
    barcodes.forEach(barcode => {
      const div = document.createElement('div');
      const box = barcode.boundingBox;
      const computedStyle = getComputedStyle(img);
      const [top, right, bottom, left] = [
        computedStyle.marginTop,
        computedStyle.marginRight,
        computedStyle.marginBottom,
        computedStyle.marginLeft
      ].map(m => parseInt(m, 10));
      const scaleX = img.width / img.naturalWidth;
      const scaleY = img.height / img.naturalHeight;
      div.style.backgroundColor = 'rgba(255, 255, 255, 0.75)';
      div.style.position = 'absolute';
      div.style.top = `${scaleY * box.top + top}px`;
      div.style.left = `${scaleX * box.left - left}px`;
      div.style.width = `${scaleX * box.width}px`;
      div.style.height = `${scaleY * box.height}px`;
      div.style.color = 'black';
      div.style.fontSize = '14px';      
      div.textContent = `${barcode.rawValue}`;
      img.before(div);
    });
  } catch(e) {
    console.error(e);
  }
});

Text detection

The final feature of the Shape Detection API is text detection. By now you know the drill: We need a page with images that contain text, like this one with Google Books scan results. On supported browsers, you will see the text recognized and a bounding box drawn around text passages. Remix or edit the Glitch project, especially the script.js file to see how it's done.

For testing this dynamically, head over to this Search results page in a private tab or in guest mode. Now paste the snippet below in the Chrome DevTools Console tab. With a little bit of waiting, some of the text will be recognized.

document.querySelectorAll('img[alt]:not([alt=""])').forEach(async (img) => {
  try {
    const texts = await new TextDetector().detect(img);
    texts.forEach(text => {
      const div = document.createElement('div');
      const box = text.boundingBox;
      const computedStyle = getComputedStyle(img);
      const [top, right, bottom, left] = [
        computedStyle.marginTop,
        computedStyle.marginRight,
        computedStyle.marginBottom,
        computedStyle.marginLeft
      ].map(m => parseInt(m, 10));
      const scaleX = img.width / img.naturalWidth;
      const scaleY = img.height / img.naturalHeight;
      div.style.backgroundColor = 'rgba(255, 255, 255, 0.75)';
      div.style.position = 'absolute';
      div.style.top = `${scaleY * box.top + top}px`;
      div.style.left = `${scaleX * box.left - left}px`;
      div.style.width = `${scaleX * box.width}px`;
      div.style.height = `${scaleY * box.height}px`;
      div.style.color = 'black';
      div.style.fontSize = '14px';      
      div.innerHTML = text.rawValue;
      img.before(div);
    });
  } catch(e) {
    console.error(e);
  }
});

Feedback

What did you think of this API? Please help us by briefly responding to this survey:

Was this API intuitive to use?

Yes No

Did you get the example to run?

Yes No

Got more to say? Were there missing features? Please provide quick feedback in this survey. Thank you!

5. Web Share Target API

Note: We were working on this API as part of the new Capabilities project. The API has now launched 🎉 and this codelab is based on the current stable design. See the Web Share Target API post for the latest details.

Compatibility: The Web Share Target API is available on Android, Windows, Linux, and Chrome OS.

The Web Share Target API allows installed web apps to register with the underlying operating system as a share target to receive shared content from either the Web Share API or system events, like the operating-system-level share button.

Install a PWA to share to

As a first step, you need a PWA that you can share to. This time, Airhorner (luckily) won't do the job, but the Web Share Target demo app has your back. Install the app to your device's home screen.

Share something to the PWA

Next, you need something to share, such as a photo from Google Photos. Use the Share button and select the Scrapbook PWA as a share target.

When you tap the app icon, you will then land straight in the Scrapbook PWA, and the photo is right there.

So, how does this work? To find out, explore the Scrapbook PWA's web app manifest. The configuration to make the Web Share Target API work is located in the "share_target" property of the manifest that in its "action" field points to a URL that gets decorated with parameters as listed in "params".

The sharing side then populates this URL template accordingly (either facilitated through a share action, or controlled programmatically by the developer using the Web Share API), so that the receiving side can then extract the parameters and do something with them, such as display them.

{
  "action": "/_share-target",
  "enctype": "multipart/form-data",
  "method": "POST",
  "params": {
    "files": [{
      "name": "media",
      "accept": ["audio/*", "image/*", "video/*"]
    }]
  }
}

Feedback

What did you think of this API? Please help us by briefly responding to this survey:

Was this API intuitive to use?

Yes No

Did you get the example to run?

Yes No

Got more to say? Were there missing features? Please provide quick feedback in this survey. Thank you!

6. Wake Lock API

Note: We're working on this API as part of the new Capabilities project. This codelab is based on the current design, and the API may change before the feature is standardized. See the Wake Lock API post for the latest details.

Compatibility: The Wake Lock API is available behind the #enable-experimental-web-platform-features flag on Android, Windows, Linux, and macOS.

To avoid draining the battery, most devices quickly go to sleep when left idle. While this is fine most of the time, some applications need to keep the screen or the device awake in order to complete their work. The Wake Lock API provides a way to prevent the device from dimming and locking the screen or prevent the device from going to sleep. This capability enables new experiences that, until now, required a native app.

Set up a screensaver

To test the Wake Lock API, you must first ensure that your device would go to sleep. Therefore, in your operating system's preference pane, activate a screensaver of your choice and make sure that it starts after 1 minute. Make sure that it works by leaving your device alone for exactly that time (yeah, I know, it's painful). The screenshots below show macOS, but you can of course try this on your mobile Android device or any supported desktop platform.

Set a screen wake lock

Now that you know that your screensaver is working, you'll use a wake lock of type "screen" to prevent the screensaver from doing its job. Head over to the Wake Lock demo app and click the Activate

screen Wake Lock checkbox.

Starting from that moment, a wake lock is active. If you're patient enough to leave your device untouched for a minute, you will now see that the screensaver indeed didn't start.

So how does this work? To find out, head over to the Glitch project for the Wake Lock demo app and check out script.js. The gist of the code is in the snippet below. Open a new tab (or use any tab that you happen to have open) and paste the code below in a Chrome Developer Tools console. When you click the window, you should then see a wake lock that's active for exactly 10 seconds (see the console logs), and your screensaver shouldn't start.

Note: If you get an Uncaught ReferenceError, remember to set the #enable-experimental-web-platform-features flag.

if ('wakeLock' in navigator && 'request' in navigator.wakeLock) {  
  let wakeLock = null;
  
  const requestWakeLock = async () => {
    try {
      wakeLock = await navigator.wakeLock.request('screen');
      wakeLock.addEventListener('release', () => {        
        console.log('Wake Lock was released');                    
      });
      console.log('Wake Lock is active');      
    } catch (e) {      
      console.error(`${e.name}, ${e.message}`);
    } 
  };

  requestWakeLock();
  window.setTimeout(() => {
    wakeLock.release();
  }, 10 * 1000);
}

Feedback

What did you think of this API? Please help us by briefly responding to this survey:

Was this API intuitive to use?

Yes No

Did you get the example to run?

Yes No

Got more to say? Were there missing features? Please provide quick feedback in this survey. Thank you!

7. Contact Picker API

Note: We're working on this API as part of the new Capabilities project. This codelab is based on the current design, and the API may change before the feature is standardized. See the Contact Picker API post for the latest details.

Compatibility: The Wake Lock API is available behind the #enable-experimental-web-platform-features flag on Android.

An API that we are very excited about is the Contact Picker API. It allows a web app to access contacts from the device's native contact manager, so your web app has access to your contacts' names, email addresses, and telephone numbers. You can specify whether you want just one or multiple contacts and whether you want all of the fields or just a subset of names, email addresses, and telephone numbers.

Privacy considerations

Once the picker opens, you can select the contacts you want to share. You will note that there is no "select all" option, which is deliberate: we want sharing to be a conscious decision. Likewise, access is not continuous, but rather a one-time decision.

Accessing contacts

Accessing contacts is a straightforward task. Before the picker opens, you can specify what fields you want (the options being name, email, and telephone), and whether you want to access multiple or just one contact. You can test this API on an Android device by opening the demo application. The relevant section of the source code is essentially the snippet below:

getContactsButton.addEventListener('click', async () => {
  const contacts = await navigator.contacts.select(
      ['name', 'email'],
      {multiple: true});
  if (!contacts.length) {
    // No contacts were selected, or picker couldn't be opened.
    return;
  }
  console.log(contacts);
});

8. Async Clipboard API

Note: We were working on this API as part of the new Capabilities project. The API has now launched 🎉 and this codelab is based on the current stable design. See the Async Clipboard API article for the latest details.

Copying and pasting text

Until now, there was no way to programmatically copy and paste images to the system's clipboard. Recently, we added image support to the Async Clipboard API,

so that now you can copy and paste images around. What's new is that you can also write images to the clipboard. The asynchronous clipboard API supported copying and pasting of text for a while now. You can copy text to the clipboard by calling navigator.clipboard.writeText() and then later on paste that text by calling navigator.clipboard.readText().

Copying and pasting images

Now you can also write images to the clipboard. For this to work, you need the image data as a blob that you then pass to the clipboard item constructor. Finally, you can then copy this clipboard item by calling navigator.clipboard.write().

// Copy: Writing image to the clipboard
try {
  const imgURL = 'https://developers.google.com/web/updates/images/generic/file.png';
  const data = await fetch(imgURL);
  const blob = await data.blob();
  await navigator.clipboard.write([
    new ClipboardItem(Object.defineProperty({}, blob.type, {
      value: blob,
      enumerable: true
    }))
  ]);
  console.log('Image copied.');
} catch(e) {
  console.error(e, e.message);
}

Pasting the image back from the clipboard looks pretty involved, but actually just consists of getting the blob back from the clipboard item. As there can be multiple, you need to loop through them until you have the one you're interested in. For security reasons, right now this is limited to PNG images, but more image formats may be supported in the future.

async function getClipboardContents() {
  try {
    const clipboardItems = await navigator.clipboard.read();
    for (const clipboardItem of clipboardItems) {
      try {
        for (const type of clipboardItem.types) {
          const blob = await clipboardItem.getType(type);
          console.log(URL.createObjectURL(blob));
        }
      } catch (e) {
        console.error(e, e.message);
      }
    }
  } catch (e) {
    console.error(e, e.message);
  }
}

You can see this API in action in a demo app, the relevant snippets from its source code are embedded above. Copying images into the clipboard can be done without permission, but you need to grant access to paste from the clipboard.

After granting access, you can then read the image from the clipboard and paste it in the application:

9. You made it!

Congratulations, you've made it to the end of the codelab. Again, this a kind reminder that most of the APIs are still in flux and actively being worked on. Therefore, the team really appreciates your feedback, as only interaction with people like you will help us get these APIs right.

If you don't want to be living on the bleeding edge, now might be a good idea to turn the #enable-experimental-web-platform-features flag back to Disabled.

We also encourage you to have a frequent look at our Capabilities landing page. We will keep it up to date, and it has pointers to all of the in-depth articles for the APIs we work on. Keep rockin'!

Tom and the entire Capabilities team 🐡