AI Platform (Unified): Building a fraud detection model with AutoML

In this lab, you will use AI Platform (Unified) to train and serve a model with tabular data. This unified offering is the newest AI product offering on Google Cloud, and is currently in preview.

What you learn

You'll learn how to:

  • Upload a Managed Dataset to AI Platform (Unified)
  • Train a model with AutoML
  • Deploy your trained AutoML model to an endpoint and use that endpoint to get predictions

The total cost to run this lab on Google Cloud is about $22.

This lab uses the newest AI product offering available on Google Cloud. AI Platform (Unified) is currently in preview, and integrates the ML offerings across Google Cloud into a seamless development experience. Previously, models trained with AutoML and custom models were accessible via separate services. The new offering combines both into a unified API, along with other new products. During the preview, we encourage you to use AI Platform (Unified) for any new projects. You can also migrate existing projects to the unified platform. If you have any feedback, please see the support page.

You'll need a Google Cloud Platform project with billing enabled to run this codelab. To create a project, follow the instructions here.

Step 1: Enable the Compute Engine API

Navigate to Compute Engine and select Enable if it isn't already enabled. You'll need this to create your notebook instance.

Step 2: Enable the AI Platform API

Navigate to the AI Platform (Unified) section of your Cloud Console and click Enable AI Platform API.

AI Platform Unified dashboard

Step 3: Create an AI Platform Notebooks instance

From the AI Platform (Unified) section of your Cloud Console, click on Notebooks:

AI Platform Unified menu

From there, select New Instance. Then select the TensorFlow Enterprise 2.3 instance type without GPUs:

TFE instance

Use the default options and then click Create. Once the instance has been created, select Open JupyterLab:

Open CAIP Notebook

The data we'll use to train our model is from this Credit card fraud detection dataset. We'll use a version of this dataset made publicly available in BigQuery.

In AI Platform (Unified), you can create managed datasets for a variety of data types. You can then generate statistics on these datasets and use them to train models with AutoML or your own custom model code.

Step 1: Create a dataset

In the AI Platform (Unified) menu in your console, select Data sets:

Select Data sets

In this lab we'll be build a fraud detection model to determine whether or not a particular credit card transaction should be classified as fraudulent.

From the Data sets page, give the dataset a name, then select Tabular, and Regression/classification. Then Create the dataset:

Create dataset

Once the dataset has been created, click on it in the Data sets page:

Datasets index

There are a few options for importing data to Managed Datasets in AI Platform (Unified):

  • Uploading a local file from your computer
  • Selecting files from Cloud Storage
  • Selecting data from BigQuery

Here we'll be uploading data from a public BigQuery table.

Step 2: Import data from BigQuery

Choose "Select a table or view from BigQuery" as your import method, and then copy the following into the BigQuery table box: bigquery-public-data.ml_datasets.ulb_fraud_detection. Then select Continue:

Import BQ data

You should see something like the following after importing your dataset:

Imported data

If you'd like, you can click Generate statistics to see additional info on this dataset, but that is not required before proceeding to the next step. This dataset contains real credit card transactions. Most of the column names have been obscured, which is why they are called V1, V2, etc.

With a managed dataset uploaded, we're ready to train a model with this data. We'll be training a classification model to predict whether or not a specific transaction is fraudulent. AI Platform (Unified) gives you two options for training models:

  • AutoML: Train high-quality models with minimal effort and ML expertise.
  • Custom training: Run your custom training applications in the cloud using one of Google Cloud's pre-built containers or use your own.

In this lab, we'll use AutoML for training.

Step 1: Kick off training job

From the dataset detail page where you left off in the previous step, select Train new model on the top right. Give your model a name, or you can use the default. Select the name you used for your dataset in the Dataset dropdown. Under Target column select Class. This is an integer indicating whether or not a particular transaction was fraudulent (0 for non-fraud, 1 for fraud). Select Classification as the objective, and then click Continue:

Model training options

Under Training method, select AutoML. Then select Continue.

In this step (Choose training options), scroll down and click to expand Advanced options. Since this dataset is heavily imbalanced (less than 1% of the data contains fraudulent transactions), chooose the AUC PRC option which will maximize precision-recall for the less common class:

Advanced training options

Select Continue and then proceed to the last step (Compute and pricing). Here, enter 1 as the number of node hours for your budget and leave early stopping enabled. Training your AutoML model for 1 compute hour is typically a good start for understanding whether there is a relationship between the features and label you've selected. From there, you can modify your features and train for more time to improve model performance. Next, select Start training.

You'll get an email when your training job completes. Training will take slightly longer than an hour to account for time to spin up and tear down resources.

Duration 5:00

In this step we'll see how our model performed.

Once your model training job has completed, navigate to the Models tab in AI Platform (Unified). Click on the model you just trained and take a look at the Evaluate tab. There are many evaluation metrics here - we'll focus on two: the Confusion Matrix and Feature Importance.

Step 1: Understand the confusion matrix

A confusion matrix tell us the percentage of examples from each class in our test set that our model predicted correctly. In the case of an imbalanced dataset like the one we're dealing with, this is a better measure of our model's performance than overall accuracy.

Remember that less than 1% of the examples in our dataset were fraudulent transactions, so if our model accuracy is 99% there's a good chance it's just randomly guessing the non-fraudulent class 99% of the time. That's why looking at our model's accuracy for each class is a better metric here.

If you scroll down on the Evaluate tab, you should see a confusion matrix that looks something like this (exact percentages may vary):

Confusion matrix

The confusion matrix shows our initial model is able to classify 85% of the fraudulent examples in our test set correctly. This is pretty good, especially considering our significant dataset imbalance. Next we could try training our model for more compute hours to see if we can improve from this 85%.

Step 2: Looking at feature importance

Below the confusion matrix, you should see a feature importance chart that looks like this:

Feature importance

This shows us the features that provided the biggest signal to our model when making predictions. Feature importance is one type of Explainable AI - a field that includes various methods for getting more insight into an ML model is making predictions. The feature importance chart seen here is calculated as an aggregate by looking at all of our model's predictions on our test set. It shows us the most important features across a batch of examples.

This chart would be more exciting if most of the features in our dataset were not obscured. We might learn, for example, that the type of a transaction (transfer, deposit, etc.) was the biggest indicator of fraud.

In a real-world scenario, these feature importance values could be used to help us improve our model, and to have more confidence in it's predictions. We might decide to remove the least important features next time we train a model, or to combine two of our more significant features into a feature cross to see if this improves model performance.

We're looking at feature importance across a batch here, but we can also get feature importance for individual predictions in AI Platform. We'll see how to do that once we've deployed our model.

Now that we have a trained model, the next step is to create an Endpoint in AI Platform. A Model resource in AI Platform can have multiple endpoints associated with it, and you can split traffic between endpoints.

Step 1: Creating an endpoint

On your model page, navigate to the Deploy and test tab and click Deploy to endpoint:

Deploy and test

Give your endpoint a name, like fraud_v1, leave the traffic splitting settings as is, and then select a machine type for your model deployment. We used an n1-highcpu-2 here, but you can choose whichever machine type you'd like, including one with GPUs:

Create endpoint

Then select Done and click Continue. Leave the selected location settings as is and then click Deploy. Your endpoint will take a few minutes to deploy. When it is completed you'll see a green check mark next to it:

Deployed endpoint

You're getting close! Now you're ready to get predictions on your deployed model.

Duration 10:00

There are a few options for getting model predictions:

  • AI Platform UI
  • AI Platform API

We'll show both here.

Step 1: Get model predictions in the UI

On your model page where your endpoint is shown (where we left off in the last step), scroll down to the Test your model section:

Test model

Here, AI Platform has chosen random values for each of our model's features that we can use to get a test prediction. You are welcome to change these values if you'd like. Scroll down to the bottom of the page and select Predict.

In the Prediction result section of the page, you should see your model's predicted percentage for each class. A confidence score of 0.99 for class 0, for example, means that your model thinks this example has a 99% of being non-fraudulent.

Step 2: Get model predictions with the AI Platform API

The UI is a great way to make sure your deployed endpoint is working as expected, but chances are you'll want to get predictions dynamically via a REST API call. To show you how to get model predictions here, we'll be using the AI Platform Notebook instance you created at the beginning of this lab.

Next, open the notebook instance you created, and open a Python 3 notebook from the Launcher:

Open notebook

In your notebook, run the following in a cell to install the AI Platform (Unified) SDK:

!pip3 install https://storage.googleapis.com/google-cloud-aiplatform/libraries/python/0.2.0/google-cloud-aiplatform-0.2.0.tar.gz

In your notebook, add and run a cell with the following function. This function makes a prediction request to our deployed endpoint:

from google.cloud import aiplatform

def predict_tabular_model(endpoint: str, instance: dict, parameters_dict: dict):
    client_options = dict(api_endpoint="us-central1-prediction-aiplatform.googleapis.com")
    client = aiplatform.gapic.PredictionServiceClient(client_options=client_options)
    from google.protobuf import json_format
    from google.protobuf.struct_pb2 import Value

    # The format of the parameters must be consistent with what the model expects.
    parameters = json_format.ParseDict(parameters_dict, Value())

    # The format of the instances must be consistent with what the model expects.
    instances_list = [instance]
    instances = [json_format.ParseDict(s, Value()) for s in instances_list]
    response = client.predict(
        endpoint=endpoint, instances=instance, parameters=parameters
    )

    print("deployed_model_id:", response.deployed_model_id, '\n')
    predictions = response.predictions
    print("predictions")
    print(predictions)

Below that, add and run a new cell with the following example from our test set:

test_instance={
    'Time': 80422,
    'Amount': 17.99,
    'V1': -0.24,
    'V2': -0.027,
    'V3': 0.064,
    'V4': -0.16,
    'V5': -0.152,
    'V6': -0.3,
    'V7': -0.03,
    'V8': -0.01,
    'V9': -0.13,
    'V10': -0.18,
    'V11': -0.16,
    'V12': 0.06,
    'V13': -0.11,
    'V14': 2.1,
    'V15': -0.07,
    'V16': -0.033,
    'V17': -0.14,
    'V18': -0.08,
    'V19': -0.062,
    'V20': -0.08,
    'V21': -0.06,
    'V22': -0.088,
    'V23': -0.03,
    'V24': 0.01,
    'V25': -0.04,
    'V26': -0.99,
    'V27': -0.13,
    'V28': 0.003
}

Next, in the AI Platform UI, click on Sample Request for the endpoint you just deployed:

Sample request

Click on the Python tab, and copy the first string in the request. This contains your project and endpoint IDs:

Endpoint string

Finally, make a prediction request to your endpoint by replacing your-endpoint-str with the string you copied above:

predict_custom_model_sample(
   "your-endpoint-str",
    test_instance,
    {}
)

You should see a prediction around .99 for the 0 class, which means the model thinks there is a 99% chance this transaction is non-fraudulent.

🎉 Congratulations! 🎉

You've learned how to use unified Cloud AI Platform to:

  • Upload a managed dataset
  • Train and evaluate a model on tabular data using AutoML
  • Deploy the model to an endpoint
  • Get predictions on a model endpoint using the SDK for AI Platorm (Unified)

To learn more about different parts of the unified AI Platform, check out the documentation.

If you'd like to continue using the notebook you created in this lab, it is recommended that you turn it off when not in use. From the Notebooks UI in your Cloud Console, select the notebook and then select Stop:

Stop instance

If you'd like to delete the notebook entirely, simply click the Delete button in the top right.

To delete the endpoint you deployed, navigate to the Endpoints section of your AI Platform (Unified) console and click the delete icon:

Delete endpoint

To delete the Storage Bucket, using the Navigation menu in your Cloud Console, browse to Storage, select your bucket, and click Delete:

Delete storage