In this codelab you'll build a simple musical game using the Oboe library, a C++ library which uses the high-performance audio APIs in the Android NDK. The objective of the game is to copy the clapping pattern you hear by tapping on the screen.

What you'll learn

What you'll need

The game plays a funky four-beat backing track that continually loops. When the game starts, it also plays a clapping sound on the first three beats of the bar.

The user must try to repeat the three claps with the same timing by tapping on the screen when the second bar begins.

Each time the user taps, the game plays a clap sound. If the tap happens at the right time, the screen flashes green. If the tap is too early or too late, the screen flashes orange or purple respectively.

Clone project

Clone the Oboe repository on github and switch to the io18codelab branch.

git clone https://github.com/google/oboe 
cd oboe
git checkout io18codelab

Open the project in Android Studio

Load Android Studio and open the codelab project:

Note that this project contains all the code samples for the Oboe library. For this codelab, you are only working with the RhythmGame sample.

Run the project

Choose the RhythmGame run configuration.

Then press CTRL+R to build and run the template app - it should compile and run but it doesn't do anything except turn the screen grey. You will add functionality to the game during this codelab.

Open the RhythmGame module

The files you'll work on for this codelab are stored in the RhythmGame module. Expand this module in the Project window, making sure that the Android view is selected.

Now expand the cpp/native-lib folder. During this codelab you will edit Game.h and Game.cpp.

Comparing with the final version

During the codelab it can be useful to refer to the final version of the code which is stored in the master branch. Android Studio makes it easy to compare changes across branches.

First, enable version control integration.

Now you can compare your code with code in the master branch.

This opens a new window. Choose the "Diff" tab. A list of files with differences appears.

Click on any file to view the differences.

Here's the game architecture:

UI

The left side of the diagram shows objects associated with the UI.

The OpenGL Surface calls tick each time the screen needs to be updated, typically 60 times a second. Game then instructs any UI rendering objects to render pixels to the OpenGL surface and the screen is updated.

The UI for the game is very simple: the single method SetGLScreenColor updates the color of the screen.

Tap events

Each time the user taps on the screen the tap method is called, passing the time the event occurred.

Audio

The right side of the diagram shows objects associated with audio. Oboe provides the AudioStream class and associated objects to allow Game to send audio data to the audio output (a speaker or headphones).

Each time the AudioStream needs more data it calls AudioDataCallback::onAudioReady. This passes an array named audioData to Game which must then fill the array with numFrames of audio frames.

In order to play a "hand clap" sound you need a file containing the digital audio data and an audio stream.

Load the sound file

The project includes a file in the src/main/assets folder named CLAP.raw which contains PCM audio data in the following format:

Sample format: 16-bit integer

Channels: 2 (stereo)

Sample rate: 48,000 kHz

To load this file into the game, use the SoundRecording::loadFromAssets method. Open Game.h and declare a SoundRecording* called mClap initializing it to nullptr.

private:
    // ...existing code... 
    SoundRecording *mClap{nullptr};

Now open Game.cpp and add the following code to start():

void Game::start() {
   mClap = SoundRecording::loadFromAssets(mAssetManager, "CLAP.raw");
}

This loads PCM audio data into the SoundRecording object when the game starts.

Creating an audio stream

To create an audio stream, use an AudioStreamBuilder. This allows you to specify the desired properties of the audio stream before creating it.

Setting the stream properties

The stream's properties should match those of the source data.

Open Game.h and declare an AudioStream* called mAudioStream.

private:
    // ...existing code... 
    AudioStream *mAudioStream{nullptr};

Now add the following code to start() in Game.cpp

void Game::start() {
    // ...existing code... 

    AudioStreamBuilder builder;
    builder.setFormat(AudioFormat::I16);
    builder.setChannelCount(2);
    builder.setSampleRate(48000);
}

Setting up the callback

Use a callback to pass audio data into the stream because this approach provides the best performance.

To use the callback, define an object that implements the AudioDataCallback interface. Rather than creating a new object you can implement this interface in Game. Open Game.h and locate the following line:

class Game {

Change it to:

class Game : public AudioStreamCallback {

Now override the AudioStreamCallback::onAudioReady method:

public:
    // ...existing code... 
     
    // Inherited from oboe::AudioStreamCallback
    DataCallbackResult
    onAudioReady(AudioStream *oboeStream, void *audioData, int32_t numFrames) override;

The audioData parameter for onAudioReady is an array into which you render the audio data using mClap->renderAudio.

Add the implementation of onAudioReady to Game.cpp.

// ...existing code... 

DataCallbackResult Game::onAudioReady(AudioStream *oboeStream, void *audioData, int32_t numFrames) {
    mClap->renderAudio(static_cast<int16_t *>(audioData), numFrames);
    return DataCallbackResult::Continue;
}

The return value DataCallbackResult::Continue tells the stream that you intend to keep sending audio data so callbacks should continue. If you return DataCallbackResult::Stop the callbacks would stop and no more audio would be played through the stream.

To complete the callback setup you must tell the audio stream builder where to find the callback object using setCallback in start:

void Game::start() {
    // ...existing code... 
    builder.setSampleRate(48000);
 
    builder.setCallback(this);
}

Creating and starting the audio stream

After all that set up, creating and starting the stream is simple. First let the builder create the stream, then start the stream.

Add the following to start:

void Game::start() {
    // ...existing code... 
    builder.setCallback(this);    

    // Open the stream
    Result result = builder.openStream(&mAudioStream);
    if (result != Result::OK){
        LOGE("Failed to open stream. Error: %s", convertToText(result));
    }

    // Start the stream
    result = mAudioStream->requestStart();
    if (result != Result::OK){
       LOGE("Failed to start stream. Error: %s", convertToText(result));
    }
}

Handle the tap event

The tap method is called each time the user taps the screen. Start the clap sound by calling setPlaying. Add the following to tap:

void Game::tap(int64_t eventTimeAsUptime) {
    mClap->setPlaying(true);
}

Build and run the app. You should hear a clap sound when you tap the screen.

When you tap on the screen, you will probably notice a delay between tapping and hearing the clap sound. This delay is known as latency. It comes primarily from two sources: the touch screen and the audio stream.

We can optimize the latency of the audio stream by changing its properties to the following:

Open Game.cpp and add the following lines just after setting the callback in start:

void Game::start() {
    // ...existing code... 
    builder.setCallback(this);
    
    builder.setPerformanceMode(PerformanceMode::LowLatency);
    builder.setSharingMode(SharingMode::Exclusive);
    // ...existing code...  
}

The buffer size can only be set after the stream has been opened. Add the following code after opening the stream in start.

void Game::start() {
    // ...existing code... 
    Result result = builder.openStream(&mAudioStream);
    if (result != Result::OK){
       LOGE("Failed to open stream. Error: %s", convertToText(result));
    }
    
    // Reduce stream latency by setting the buffer size to a multiple of the burst size
    mAudioStream->setBufferSizeInFrames(mAudioStream->getFramesPerBurst() * 2);
    // ...existing code... 
}

This sets the buffer size to two "bursts", where a burst is the amount of data written during each audio callback. Choose two bursts because this gives us a good tradeoff between latency and underrun protection. This is often referred to as double buffering and is commonly used in graphics programming.

Build and run the app. You should notice that the latency between tapping and hearing the clap sound has been reduced and the game feels a lot more responsive. Good work!

Playing a single clap sound is going to get boring pretty quickly. It would be nice to also play a backing track with a beat you can clap along to.

Up until now, the game places only clap sounds into the audio stream.

Using a mixer

To play multiple sounds simultaneously you must mix them together using a Mixer.

Create the backing track and mixer

Open Game.h and declare another SoundRecording* for the backing track and a Mixer:

private:
   // ...existing code... 
   SoundRecording *mClap{nullptr};
   
   SoundRecording *mBackingTrack{nullptr};
   Mixer mMixer;

Now in Game.cpp add the following code after the clap sound has been loaded in start.

void Game::start() {
    mClap = SoundRecording::loadFromAssets(mAssetManager, "CLAP.raw");
    mBackingTrack = SoundRecording::loadFromAssets(mAssetManager, "FUNKY_HOUSE.raw" );
    mBackingTrack->setPlaying(true);
    mBackingTrack->setLooping(true);
    mMixer.addTrack(mClap);
    mMixer.addTrack(mBackingTrack);
    // ...existing code... 
}

This loads the contents of the FUNKY_HOUSE.raw asset (which contains PCM data in the same format as the clap sound asset) into a SoundRecording object. Playback starts when the game starts and loops indefinitely.

Both the clap sound and backing track are added to the mixer.

Updating the audio callback

You now need to tell the audio callback to use the mixer rather than the clap sound for rendering. Update onAudioReady to the following:

DataCallbackResult Game::onAudioReady(AudioStream *oboeStream, void *audioData, int32_t numFrames) {

    mMixer.renderAudio(static_cast<int16_t*>(audioData), numFrames);
    return DataCallbackResult::Continue;
}

Build and run the game. You should hear the backing track and the clap sound when you tap the screen. Feel free to jam for a minute!

Now things start to get interesting. Start adding the gameplay mechanics. The game plays a series of claps at specific times. This is called the clap pattern.

For this simple game, the clap pattern is just three claps that start on the first beat of the first bar of the backing track. The user must repeat the same pattern starting on the first beat of the second bar.

When should you play the claps in the clap pattern?

The backing track has a tempo of 120 beats per minute, or 1 beat every 0.5 seconds. It also has a sample rate of 48kHz (48,000 frames per second). So you need to play a clap sound at the following times in the backing track:

Beat

Time (seconds)

Frame number

1

0

0

2

0.5

24000

3

1

48000

Synchronizing clap events with the backing track

Each time onAudioReady is called audio frames from the backing track (via the mixer) are written into the audio stream. By counting the frames which have been written, you know the exact playback position of the backing track, and therefore when to play a clap.

With this in mind, here's how to play the clap events at exactly the right time:

Key point: By linking clap playback events with the onAudioReady callback you can ensure consistent and perfect synchronization with the backing track.

Cross thread communication

The game has three threads: an OpenGL thread, a UI thread (main thread) and a real-time audio thread.

Clap events are pushed onto the scheduling queue from the UI thread and popped off the queue from the audio thread.

The queue is accessed from multiple threads so it must be thread-safe. It must also be lock-free so it does not block the audio thread. This requirement is true for any object shared with the audio thread. Why? Because blocking the audio thread can cause audio glitches, and no-one wants to hear that!

Add the code

The game already includes a LockFreeQueue class template which is thread-safe when used with a single reader thread (in this case the audio thread) and a single writer thread (the UI thread).

To declare a LockFreeQueue you must supply two template parameters:

Open Game.h and add the following declarations:

private:
    // ...existing code...  
    Mixer mMixer;
    
    LockFreeQueue<int64_t, 4> mClapEvents;
    std::atomic<int64_t> mCurrentFrame { 0 };

Note that mCurrentFrame is std::atomic because it is accessed from the UI thread.

Now in Game.cpp enqueue the clap events in start:

void Game::start() {
    // ...existing code... 
    mMixer.addTrack(mBackingTrack);

    mClapEvents.push(0);
    mClapEvents.push(24000);
    mClapEvents.push(48000);
    // ...existing code... 
}

And update onAudioReady to the following:

DataCallbackResult Game::onAudioReady(AudioStream *oboeStream, void *audioData, int32_t numFrames) {

   int64_t nextClapEvent;

   for (int i = 0; i < numFrames; ++i) {

       if (mClapEvents.peek(nextClapEvent) && mCurrentFrame == nextClapEvent){
           mClap->setPlaying(true);
           mClapEvents.pop(nextClapEvent);
       }
       mMixer.renderAudio(static_cast<int16_t*>(audioData)+(kChannelCount*i), 1);
       mCurrentFrame++;
   }

   return DataCallbackResult::Continue;
}

The for loop iterates for numFrames checking whether a clap event is due, popping it off the queue if it is, then rendering a single audio frame. Pointer arithmetic is used to tell mMixer where in audioData to render the frame.

Build and run the game. Three claps should be played exactly on the beat when the game starts. Tapping on the screen still plays the clap sounds.

Feel free to experiment with different clap patterns by changing the frame values for the clap events. You can also add more clap events, remember to increase the capacity of the mClapEvents queue.

The game plays a clap pattern and expects the user to imitate the pattern. Finally, complete the game by scoring the user's taps.

Did the user tap at the right time?

After the game claps three times in the first bar, the user should tap three times starting on the first beat of the second bar.

You shouldn't expect the user to tap at exactly the right time - that would be virtually impossible! Instead, allow some tolerance before and after the expected time. This defines a time range which we call the tap window.

If the user taps during the tap window, make the screen flash green, too early: orange and too late: purple.

Storing the tap windows is easy: store the frame number at the center of each window in a queue, the same way you did for clap events. You can then pop each window off the queue when the user taps on the screen.

Comparing tap events with the tap window

When the user taps on the screen you need to know whether the tap fell within the current tap window.

The current tap window is stored as an audio frame number, and the tap event is delivered as system uptime (milliseconds since boot). To compare these two events their times must be in the same unit of reference.

Creating a reference point

To convert from a frame number to uptime you need a reference point where both the frame number and uptime are known. The current frame number is already stored in mCurrentFrame. To create the reference point store the uptime when mCurrentFrame is updated inside onAudioReady.

Updating the screen

Once you know the accuracy of a user's tap (early, late, right-on), update the screen to provide visual feedback.

To do this use another instance of LockFreeQueue class with TapResult objects. Check this queue every time the screen is updated in the tick method .

If there's an event in the queue, pop it off and set the color of the screen using SetGLScreenColor.

Add the code

In Game.h declare the following fields:

private:
    // ...existing code... 
    std::atomic<int64_t> mCurrentFrame { 0 };
    LockFreeQueue<int64_t, kMaxQueueItems> mClapWindows;
    LockFreeQueue<TapResult, kMaxQueueItems> mUiEvents;
    std::atomic<int64_t> mLastUpdateTime { 0 };

In Game.cpp add the following code to start to add the tap windows

void Game::start() {
    // ...existing code... 
    mClapEvents.push(48000);

    mClapWindows.push(96000);
    mClapWindows.push(120000);
    mClapWindows.push(144000);
    // ...existing code... 
}

Add the following code to the end of tap to pop the tap window off the queue, determine whether the tap was successful and push the result onto the UI event queue.

void Game::tap(int64_t eventTimeAsUptime) {
   mClap->setPlaying(true);

   int64_t nextClapWindowFrame;
   if (mClapWindows.pop(nextClapWindowFrame)){

       int64_t frameDelta = nextClapWindowFrame - mCurrentFrame;
       int64_t timeDelta = convertFramesToMillis(frameDelta, kSampleRateHz);
       int64_t windowTime = mLastUpdateTime + timeDelta;
       TapResult result = getTapResult(eventTimeAsUptime, windowTime);
       mUiEvents.push(result);
   }
}

Add the following code to tick to process any UI events

void Game::tick(){

   TapResult r;
   if (mUiEvents.pop(r)) {
       renderEvent(r);
   } else {
       SetGLScreenColor(kScreenBackgroundColor);
   }
}

And finally add the following code to the end of onAudioReady to reset the uptime every time you write to the audio stream..

DataCallbackResult Game::onAudioReady(AudioStream *oboeStream, void *audioData, int32_t numFrames) {

   // ...existing code... 
   mLastUpdateTime = nowUptimeMillis();

   return DataCallbackResult::Continue;
}

That's it! Build and run the game.

You should hear three claps when the game starts. If you tap exactly on the beat three times in the second bar you should see the screen flash green on each tap. If you're early it'll flash orange, if you're late it'll flash purple. Good luck!

Additional resources