This codelab will walk you through the process of building and deploying a Kotlin Spring Boot application that runs on App Engine and communicates with an Android frontend. This should help you get started with Kotlin on server side if you come from a frontend background or are a frontend/backend developer willing to try Koltin for server-side development.

We will be building an app that receives an input image and:

  1. Finds faces in the image
  2. Predicts emotions from facial expressions
  3. Overlays emojis corresponding to predicted emotions

The app is called Emojify.

Sample source & emojified image:

Picture taken at DroidCon NYC Extended.

What you will build

You will build a Kotlin backend that receives images from an Android app, emojifies faces in the input image and sends the image back to the Android app.

We will use two Google Cloud APIs: Storage and Vision.

What you will learn

You will need

The main objective of this codelab is to show you how easy and convenient it is to build Kotlin backends running in the cloud.

Have you ever considered Kotlin for Server-side development?

Not really Yes, I guess it should be possible I am a Kotlin backend developer

How would you rate your experience with Google Cloud APIs?

Novice Intermediate Proficient

Create a Google Cloud Project

Sign-in to Google Cloud Platform console (console.cloud.google.com) and click Select a project > NEW PROJECT.

Next, you'll need to enable billing in the Cloud Console in order to use Google Cloud resources. New GCP users are eligible for a $300 free trial.

Create a Service Account and download Credentials

Since we are using Google Cloud APIs, our local project needs to know which Google Cloud project to communicate with and needs to have proper credentials to do so. A Service Account Credentials file contains keys to authenticate to Google Cloud APIs.

Follow this link to setup your Service Account.

At the end of this step, you should have a key json file downloaded to your computer and GOOGLE_APPLICATION_CREDENTIALS environment variable set up.

Add "Storage Object Admin" role to your Service Account

Our backend needs full control over Google Cloud Storage objects, including listing, creating, viewing, and deleting objects. Storage roles that satisfy this requirement are Storage Object Admin and Storage Admin. Since we don't need to read or edit bucket metadata, a Storage Object Admin role is enough.

This link walks you through how to grant roles to Service Accounts.

Enable APIs

Emojify uses 2 GCP APIs: Storage to store source and emojified images & Vision for face detection and emotion prediction.

  1. Enable Storage API.
  2. Enable Vision API.
  3. Enable App Engine Admin API.

Install Maven

Our backend app uses Maven to manage dependencies. Follow this link to install Maven.

Download the Code

Download source code

OR clone the GitHub repository

$ git clone https://github.com/GoogleCloudPlatform/kotlin-samples.git
$ cd kotlin-samples/getting-started/android-with-appengine/backend

Open Backend project in editor

The code for the backend is in the directory getting-started/android-with-appengine/backend.

The skeleton of the backend app was generated from start.spring.io. This section walks you through how to configure your pom.xml and appengine-web.xml files. The final code is complete and has all you need to get the app running.

Storage & Vision dependencies

As mentioned before, we are using two Google Cloud Client Libraries in this project: Vision and Storage. These were added to pom.xml as dependencies.

pom.xml

<dependencies>
  <!-- START Google Cloud Client Libraries dependencies -->
  <dependency>
     <groupId>com.google.cloud</groupId>
     <artifactId>google-cloud-vision</artifactId> 
  </dependency>
  <dependency>
     <groupId>com.google.cloud</groupId>
     <artifactId>google-cloud-storage</artifactId>
  </dependency>
  <!-- END Google Cloud Client Libraries dependencies -->
</dependencies>
<dependencyManagement>
  <dependencies>
     <dependency>
        <groupId>com.google.cloud</groupId>
        <artifactId>google-cloud-bom</artifactId>
        <version>0.56.0-alpha</version>
        <type>pom</type>
        <scope>import</scope>
     </dependency>
  </dependencies>
</dependencyManagement>

Maven App Engine plugin

The App Engine Maven plugin will allow us to deploy our backend to App Engine from a single command.

pom.xml

<build>
  <plugins>
     <plugin>
        <groupId>com.google.cloud.tools</groupId>
        <artifactId>appengine-maven-plugin</artifactId>
        <version>1.3.1</version>
     </plugin>
  </plugins>
</build>

App Engine descriptor

The appengine-web.xml file makes this backend runnable on App Engine Standard and allows us to configure our App Engine environment.

src/main/webapp/WEB-INF/appengine-web.xml

<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
   <threadsafe>true</threadsafe>
   <runtime>java8</runtime>
   <automatic-scaling>
       <min-instances>1</min-instances>
   </automatic-scaling>
</appengine-web-app>

Here, we specify that we are using the java8 runtime and by setting min-instances to 1, we are telling App Engine to always keep at least 1 instance of the backend up. This is to prevent the delay that occurs when the backend is called and there is no instance awake.

Code to initialize our Spring Boot app

EmojifyApplication.kt

@SpringBootApplication
class EmojifyApplication : SpringBootServletInitializer() {
   @Bean
   fun storage(): Storage = StorageOptions.getDefaultInstance().service

   @Bean
   fun vision(): ImageAnnotatorClient = ImageAnnotatorClient.create()
}

fun main(args: Array<String>) {
   runApplication<EmojifyApplication>(*args)
}

Our main function instantiates a Spring Boot application and creates instances of Storage and Vision that will be automatically injected into our Emojify controller (next step). Keep reading!

Input & output

These are our API requirements:

EmojifyController.kt

@RestController
class EmojifyController(@Value("\${storage.bucket.name}") val bucketName: String, val storage: Storage, val vision: ImageAnnotatorClient) {
  @GetMapping("/emojify")
  fun emojify(@RequestParam(value = "objectName") objectName: String): EmojifyResponse {
    ...
    val publicUrl: String =
    "https://storage.googleapis.com/$bucketName/emojified/emojified-$objectName"

    return EmojifyResponse(
       objectPath = "emojified/emojified-$objectName",
       emojifiedUrl = publicUrl
    )
  }
}

Dependency Injection

The values for bucketName, storage and vision are automatically injected by Spring Boot from the following files:

src/main/resources/application.properties

# Your bucket name must be of the format YOUR_PROJECT_ID.appspot.com for the 
# Emojify frontend to work as expected
storage.bucket.name = YOUR_PROJECT_ID.appspot.com

EmojifyApplication.kt

@SpringBootApplication
class EmojifyApplication : SpringBootServletInitializer() {
   @Bean
   fun storage(): Storage = StorageOptions.getDefaultInstance().service

   @Bean
   fun vision(): ImageAnnotatorClient = ImageAnnotatorClient.create()
}

fun main(args: Array<String>) {
   runApplication<EmojifyApplication>(*args)
}

EmojifyResponse

Our controller returns an object of type EmojifyResponse.

EmojifyController.kt

data class EmojifyResponse(
   val objectPath: String? = null,
   val emojifiedUrl: String? = null,
   val statusCode: HttpStatus = HttpStatus.OK,
   val errorCode: Int? = null,
   val errorMessage: String? = null
)

This is where all the magic happens!

Calling Vision API on source image

The code snippet below performs FACE_DETECTION on the input image using Google Cloud Vision API.

EmojifyController.kt

val source = ImageSource.newBuilder().setGcsImageUri("gs://$bucketName/$objectName").build()
val img = Image.newBuilder().setSource(source).build()
val feat = Feature.newBuilder().setType(Type.FACE_DETECTION).build()
val request = AnnotateImageRequest.newBuilder()
   .addFeatures(feat)
   .setImage(img)
   .build()

val response = vision.batchAnnotateImages(listOf(request))

Understanding Vision response

A successful FACE_DETECTION response contains a set of objects of type FaceAnnotation. A face annotation includes (among many other things):

Sample FACE_DETECTION response:

{
   ...,
   "fdBoundingPoly": {
       "vertices": [
           {
               "x": 936,
               "y": 333
           },
           {
               "x": 1017,
               "y": 333
           },
           {
               "x": 1017,
               "y": 413
           },
           {
               "x": 936,
               "y": 413
           }
       ]
   },
   ...,
   "detectionConfidence": 0.980691,
   "landmarkingConfidence": 0.57905465,
   "joyLikelihood": "VERY_LIKELY",
   "sorrowLikelihood": "VERY_UNLIKELY",
   "angerLikelihood": "VERY_UNLIKELY",
   "surpriseLikelihood": "VERY_UNLIKELY"
}

We can play with output of joyLikelihood, sorrowLikelihood, angerLikelihood and surpriseLikelihood to find the best emojis to overlay on faces detected by Vision.

Finding the best emoji

EmojifyController.kt

fun bestEmoji(annotation: FaceAnnotation): Emoji {
   val emotionsLikelihood = listOf(Likelihood.VERY_LIKELY, Likelihood.LIKELY, Likelihood.POSSIBLE)
   val emotions = mapOf(
       Emoji.JOY to annotation.joyLikelihood,
       Emoji.ANGER to annotation.angerLikelihood,
       Emoji.SURPRISE to annotation.surpriseLikelihood,
       Emoji.SORROW to annotation.sorrowLikelihood
   )
   for (likelihood in emotionsLikelihood) { // In this order: VERY_LIKELY, LIKELY, POSSIBLE
       for (emotion in emotions) { // In this order: JOY, ANGER, SURPRISE, SORROW (https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.collections/map-of.html)
           if (emotion.value == likelihood) return emotion.key // Returns emotion corresponding to likelihood
       }
   }
   return Emoji.NONE
}

Overlaying the emoji

EmojifyController.kt

val response = vision.batchAnnotateImages(listOf(request))
val imgBuff = streamFromGCS(objectName)
val gfx = imgBuff.createGraphics()

for (annotation in resp.faceAnnotationsList) {
   val imgEmoji = emojiBufferedImage[bestEmoji(annotation)]
   val poly = Polygon()
   for (vertex in annotation.fdBoundingPoly.verticesList) {
       poly.addPoint(vertex.x, vertex.y)
   }
   val height = poly.ypoints[2] - poly.ypoints[0]
   val width = poly.xpoints[1] - poly.xpoints[0]
   // Draws emoji on detected face
   gfx.drawImage(imgEmoji, poly.xpoints[0], poly.ypoints[1], height, width, null)
}

The emojified image is later uploaded to GCS and its public url is included in the final response.

EmojifyController.kt

// Writing emojified image to OutputStream
val outputStream = ByteArrayOutputStream()
ImageIO.write(imgBuff, imgType, outputStream)

// Uploading emojified image to GCS and making it public
bucket.create(
   "emojified/emojified-$objectName",
   outputStream.toByteArray(),
   Bucket.BlobTargetOption.predefinedAcl(Storage.PredefinedAcl.PUBLIC_READ)
)

return EmojifyResponse(
   objectPath = "emojified/emojified-$objectName",
   emojifiedUrl = publicUrl
)

If you completed all previous steps, your local project knows at this point which Google Cloud project the backend should be deployed into. All is left is specifying which bucket to use.

Tell backend which GCS Bucket to use

Set value of storage.bucket.name to YOUR_PROJECT_ID.appspot.com. You can find your Google Cloud project ID in the Google Cloud Console.

src/main/resources/application.properties

# Your bucket name must be of the format YOUR_PROJECT_ID.appspot.com for the 
# Emojify frontend to work as expected
storage.bucket.name = YOUR_PROJECT_ID.appspot.com

Create the App Engine app

Before you can deploy your application, you must create an App Engine application with the gcloud CLI. Without this, the maven tests will throw an error.

$ gcloud app create

Run locally

$ mvn appengine:run

The application is now running on http://localhost:8080/emojify. A request will respond with a 400 error code because no image is supplied. Test it by using the test image which was uploaded to your bucket when you built the project:

http://localhost:8080/emojify?objectName=test-emojify.png

Deploy to App Engine

$ mvn appengine:deploy

Once your app is deployed, you can test it out by using the test image which was uploaded to your bucket when you built the project:

https://YOUR_PROJECT_ID.appspot.com/emojify?objectName=test-emojify.png

If everything works as expected, you will receive a JSON response containing a Cloud Storage URI for your emojified image.

Extra Credit: Try uploading your own image by going to Google Cloud Console > Storage and upload an image to your bucket. Then call the backend and specify your image as the objectName:

https://YOUR_PROJECT_ID.appspot.com/emojify?objectName=YOUR_OBJECT_NAME

Wrapping Up

As you can see, our backend does not actually require a frontend! Having an image in your GCS bucket is enough! Now let's create an Android app to connect to our new backend!

Overview

Even though our Emojify backend can be used without a frontend, in practice you would interface your backends with nice frontends. In this case, our Android app will send pictures to the backend and display resulting images.

What you will build

You will build a single activity Android app that communicates with Emojify backend.

Key Dependencies

Download Source code (optional)

If you haven't yet downloaded the source code

Download source code

OR clone the GitHub repository

$ git clone https://github.com/GoogleCloudPlatform/kotlin-samples.git
$ cd kotlin-samples/getting-started/android-with-appengine/frontend

Open project in Android Studio

The code for the Android app is in the directory getting-started/android-with-appengine/frontend.

Create a Firebase project

  1. Go to the Firebase console
  2. Select Add Project
  3. Choose the Google Cloud project you used for the backend

Make your bucket public

The Storage Security Rule of your bucket is by default set to Private. You will need to make it Public just for the sake of this demo. Ideally, you should implement an authentication system and condition Storage access to successful user authentication.

Connect the Android app to Firebase

You can use Android Studio Firebase Assistant or link your project to Firebase manually.

If using Firebase Assistant, you can connect your app by navigating to Tools > Firebase > Storage > and clicking Connect your app. Remember to select the right Google Cloud project.

One last thing

In src/main/assets/application.properties, set value of cloud.project.id to your Google Cloud project ID. You can find it in the Google Cloud Console.

src/main/assets/application.properties

cloud.project.id = REPLACE_THIS_WITH_YOUR_PROJECT_ID

You are all set!

It's happening!

Displaying User album and retrieving selected image

This is the code that allows a user to select an existing image from his album or take a new picture using Camera2 API. All of this can happen in a single call, thanks to the Album SDK.

ImageActivity.kt

private fun selectImage() {
   Album.image(this)
       .singleChoice()
       .camera(true)
       .columnCount(2)
       .widget(
           Widget.newDarkBuilder(this)
               .title(toolbar!!.title.toString())
               .build()
       )
       .onResult { result ->
           /* do something */
       }
       .onCancel {
           finish()
       }
       .start()
}

We retrieve the selected image in .onResult callback function. Observe what happens at the 4th line of the code snippet below.

ImageActivity.kt

.onResult { result ->
  albumFiles.clear()
  albumFiles.addAll(result)
  if (result.size > 0) load(result[0].path)
}

And guess what the load function does...

Uploading selected image to Cloud Storage

ImageActivity.kt

private fun load(path: String) =
   launch(CommonPool + job) {
       uploadImage(path)
   }

The load function uses a Kotlin Coroutine. It creates a separate thread that performs the uploading in the background. This is the kind of operations you would prefer not to have running on the UI thread, otherwise you will be punished!

ImageActivity.kt

private fun uploadImage(path: String) {
   val file = Uri.fromFile(File(path))
   imageId = "${System.currentTimeMillis()}.jpg"
   val imgRef = storageRef.child(imageId)
   updateUI {
       imageView.visibility = View.GONE
       tvMessage!!.text = getString(R.string.waiting_msg_1)
   }
   imgRef.putFile(file, StorageMetadata.Builder().setContentType("image/jpg").build())
           .addOnSuccessListener { _ ->
               updateUI { tvMessage.text = getString(R.string.waiting_msg_2) }
               callEmojifyBackend()
           }
           .addOnFailureListener { err ->
               updateUI {
                   show("Cloud Storage error!")
                   tvMessage.text = getString(R.string.storage_error)
               }
               Log.e("storage", err.message)
           }
}

On success, we call Emojify backend

Now calling backend

ImageActivity.kt

private fun callEmojifyBackend() {
   val queue = Volley.newRequestQueue(this)
   val url = "${this.backendUrl}/emojify?objectName=$imageId"
   updateUI { show("Image uploaded to Storage!") }
   val request = JsonObjectRequest(Request.Method.GET, url, null,
           Response.Listener { response ->
               /* process response */
               emjojifiedUrl = response["emojifiedUrl"].toString()
               downloadAndShowImage()
               deleteSourceImage()
           },
           Response.ErrorListener { err ->
               /* Log error */
               deleteSourceImage()
           })
   request.retryPolicy = DefaultRetryPolicy(50000, 5, DefaultRetryPolicy.DEFAULT_BACKOFF_MULT)
   queue.add(request)
}

Downloading and displaying emojified image

Glide is awesome!

ImageActivity.kt

private fun downloadAndShowImage() {
   val url = emjojifiedUrl
   updateUI {
       Glide.with(this)
           .load(url)
           .apply(RequestOptions().signature(ObjectKey(System.currentTimeMillis())))
               .apply(RequestOptions()
                       .placeholder(R.drawable.placeholder)
                       .error(R.drawable.placeholder)
                       .diskCacheStrategy(DiskCacheStrategy.NONE)
                       .dontTransform()
                       .override(Target.SIZE_ORIGINAL, Target.SIZE_ORIGINAL)
                       .skipMemoryCache(true))
               .into(imageView)
   }
}

Looks like we're done!

You should have your backend already deployed in the previous steps.

Launch the Android app

  1. Activate USB Debugging on your Android device
  2. Connect your device to your laptop with a USB cable
  3. Run the Android app in Android Studio and select your device as the deployment target

This will install Emojify in your Android device

Pose with your friends

Share your emojified faces!

Picture taken at DroidCon NYC Extended.

You built a Kotlin Spring Boot app running on App Engine and that communicates with an Android frontend written in Kotlin.

You learned

If you're a Kotlin Android developer just getting started with building Kotlin backends, I hope you are convinced that Kotlin is great for Server side, for the same reasons you enjoyed using it for Android development. In fact, Kotlin shines anywhere you would use Java.

More resources

Cleanup

Running through this codelab shouldn't cost you more than a few dollars, but it could be more if you decide to use more resources or if you leave them running. See Cleanup.

License

Code demonstrated in this codelab is licensed under Apache 2.0.