How to Use Cloud Run with Gemini Function Calling

1. Introduction

Overview

In this codelab, you'll see how you can give Gemini access to real time data by using a new feature called Function Calling. To simulate real time data, you will build a weather service endpoint that returns the current weather for 2 locations. Then you'll build a chat app, powered by Gemini, that uses function calling to retrieve the current weather.

Let's use a quick visual to understand Function Calling.

• The prompt requests for current weather locations at a given location
• This prompt + the function contract for getWeather() is sent to Gemini
• Gemini asks that the chat bot app calls "getWeather(Seattle)" on its behalf
• The app sends back the results (40 degress F and rainy)
• Gemini sends back the results to the caller

To recap, Gemini does not call the Function. You as the developer must call the function and send the results back Gemini.

What you'll learn

• How Gemini function calling works
• How to deploy a Gemini-powered chatbot app as a Cloud Run service

2. Setup and Requirements

Prerequisites

• You are logged into the Cloud Console.
• You have previously deployed a 2nd gen function. For example, you can follow the deploy a Cloud Function 2nd gen quickstart to get started.

Activate Cloud Shell

1. From the Cloud Console, click Activate Cloud Shell .

If this is your first time starting Cloud Shell, you're presented with an intermediate screen describing what it is. If you were presented with an intermediate screen, click Continue.

It should only take a few moments to provision and connect to Cloud Shell.

This virtual machine is loaded with all the development tools needed. It offers a persistent 5 GB home directory and runs in Google Cloud, greatly enhancing network performance and authentication. Much, if not all, of your work in this codelab can be done with a browser.

Once connected to Cloud Shell, you should see that you are authenticated and that the project is set to your project ID.

2. Run the following command in Cloud Shell to confirm that you are authenticated:

gcloud auth list

Command output

 Credentialed Accounts
ACTIVE  ACCOUNT
*       <my_account>@<my_domain.com>

To set the active account, run:
    $ gcloud config set account `ACCOUNT`

Note: The gcloud command-line tool is the powerful and unified command-line tool in Google Cloud. It comes preinstalled in Cloud Shell. You will notice its support for tab completion. You may be prompted to authenticate the first time you run a command. For more information, see gcloud command-line tool overview.

3. Run the following command in Cloud Shell to confirm that the gcloud command knows about your project:

gcloud config list project

Command output

[core]
project = <PROJECT_ID>

If it is not, you can set it with this command:

gcloud config set project <PROJECT_ID>

Command output

Updated property [core/project].

Note: If you're completing this tutorial outside of Cloud Shell, follow Set up Application Default Credentials.

3. Setup environment variables and enable APIs

Setup environment variables

You can set environment variables that will be used throughout this codelab.

PROJECT_ID=<YOUR_PROJECT_ID>
REGION=<YOUR_REGION, e.g. us-central1>
WEATHER_SERVICE=weatherservice
FRONTEND=frontend
SERVICE_ACCOUNT="vertex-ai-caller"
SERVICE_ACCOUNT_ADDRESS=$SERVICE_ACCOUNT@$PROJECT_ID.iam.gserviceaccount.com

Enable APIs

Before you can start using this codelab, there are several APIs you will need to enable. This codelab requires using the following APIs. You can enable those APIs by running the following command:

gcloud services enable run.googleapis.com \
    cloudbuild.googleapis.com \
    aiplatform.googleapis.com

4. Create a service account to call Vertex AI

This service account will be used by Cloud Run to call the Vertex AI Gemini API.

First, create the service account by running this command:

gcloud iam service-accounts create $SERVICE_ACCOUNT \
  --display-name="Cloud Run to access Vertex AI APIs"

Second, grant the Vertex AI User role to the service account.

gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member serviceAccount:$SERVICE_ACCOUNT_ADDRESS \
  --role=roles/aiplatform.user

5. Create the backend Cloud Run service

First, create a directory for the source code and cd into that directory.

mkdir -p gemini-function-calling/weatherservice gemini-function-calling/frontend && cd gemini-function-calling/weatherservice

Then, create a package.json file with the following content:

{
    "name": "weatherservice",
    "version": "1.0.0",
    "description": "",
    "main": "app.js",
    "scripts": {
        "test": "echo \"Error: no test specified\" && exit 1"
    },
    "keywords": [],
    "author": "",
    "license": "ISC",
    "dependencies": {
        "express": "^4.18.3"
    }
}

Next, create an app.js source file with the content below. This file contains the entry point for the service and contains the main logic for the app.

const express = require("express");
const app = express();

app.get("/getweather", (req, res) => {
    const location = req.query.location;
    let temp, conditions;

    if (location == "New Orleans") {
        temp = 99;
        conditions = "hot and humid";
    } else if (location == "Seattle") {
        temp = 40;
        conditions = "rainy and overcast";
    } else {
        res.status(400).send("there is no data for the requested location");
    }

    res.json({
        weather: temp,
        location: location,
        conditions: conditions
    });
});

const port = parseInt(process.env.PORT) || 8080;
app.listen(port, () => {
    console.log(`weather service: listening on port ${port}`);
});

app.get("/", (req, res) => {
    res.send("welcome to hard-coded weather!");
});

Deploy the Weather Service

You can use this command to deploy the weather service.

gcloud run deploy $WEATHER_SERVICE \
  --source . \
  --region $REGION \
  --allow-unauthenticated

Test the Weather Service

You can verify the weather for the 2 locations using curl:

WEATHER_SERVICE_URL=$(gcloud run services describe $WEATHER_SERVICE \
              --platform managed \
              --region=$REGION \
              --format='value(status.url)')

curl $WEATHER_SERVICE_URL/getweather?location=Seattle

curl $WEATHER_SERVICE_URL/getweather?location\=New%20Orleans

Seattle is 40 degrees F and rainy and New Orleans is 99 degrees F and humid, always.

6. Create the Frontend service

First, cd into the frontend directory.

cd gemini-function-calling/frontend

Then, create a package.json file with the following content:

{
  "name": "demo1",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "start": "node app.js",
    "nodemon": "nodemon app.js",
    "cssdev": "npx tailwindcss -i ./input.css -o ./public/output.css --watch",
    "tailwind": "npx tailwindcss -i ./input.css -o ./public/output.css",
    "dev": "npm run tailwind && npm run nodemon"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "@google-cloud/vertexai": "^0.4.0",
    "axios": "^1.6.7",
    "express": "^4.18.2",
    "express-ws": "^5.0.2",
    "htmx.org": "^1.9.10"
  },
  "devDependencies": {
    "nodemon": "^3.1.0",
    "tailwindcss": "^3.4.1"
  }
}

Next, create an app.js source file with the content below. This file contains the entry point for the service and contains the main logic for the app.

Note: note: To keep this codelab short, the chat history for this codelab is ephemeral. The history only persists for this specific instance of the Cloud Run service. If you want to persist chat history across multiple instances of a Cloud Run service, you can check out this codelab for how to use session middleware to pin chat history to a session id.

const express = require("express");
const app = express();
app.use(express.urlencoded({ extended: true }));
app.use(express.json());
const path = require("path");

const fs = require("fs");
const util = require("util");
const { spinnerSvg } = require("./spinnerSvg.js");

const expressWs = require("express-ws")(app);

app.use(express.static("public"));

const {
    VertexAI,
    FunctionDeclarationSchemaType
} = require("@google-cloud/vertexai");

// get project and location from metadata service
const metadataService = require("./metadataService.js");

// instance of Gemini model
let generativeModel;

// 1: define the function
const functionDeclarations = [
    {
        function_declarations: [
            {
                name: "getweather",
                description: "get weather for a given location",
                parameters: {
                    type: FunctionDeclarationSchemaType.OBJECT,
                    properties: {
                        location: {
                            type: FunctionDeclarationSchemaType.STRING
                        },
                        degrees: {
                            type: FunctionDeclarationSchemaType.NUMBER,
                            "description":
                                "current temperature in fahrenheit"
                        },
                        conditions: {
                            type: FunctionDeclarationSchemaType.STRING,
                            "description":
                                "how the weather feels subjectively"
                        }
                    },
                    required: ["location"]
                }
            }
        ]
    }
];

// on instance startup
const port = parseInt(process.env.PORT) || 8080;
app.listen(port, async () => {
    console.log(`demo1: listening on port ${port}`);

    const project = await metadataService.getProjectId();
    const location = await metadataService.getRegion();

    // Vertex client library instance
    const vertex_ai = new VertexAI({
        project: project,
        location: location
    });

    // Instantiate models
    generativeModel = vertex_ai.getGenerativeModel({
        model: "gemini-1.0-pro-001"
    });
});

const axios = require("axios");
const baseUrl = "https://weatherservice-k6msmyp47q-uc.a.run.app";

app.ws("/sendMessage", async function (ws, req) {

    // this chat history will be pinned to the current 
    // Cloud Run instance. Consider using Firestore &
    // Firebase anonymous auth instead.

    // start ephemeral chat session with Gemini
    const chatWithModel = generativeModel.startChat({
        tools: functionDeclarations
    });

    ws.on("message", async function (message) {
        let questionToAsk = JSON.parse(message).message;
        console.log("WebSocket message: " + questionToAsk);

        ws.send(`<div hx-swap-oob="beforeend:#toupdate"><div
                        id="questionToAsk"
                        class="text-black m-2 text-right border p-2 rounded-lg ml-24">
                        ${questionToAsk}
                    </div></div>`);

        // to simulate a natural pause in conversation
        await sleep(500);

        // get timestamp for div to replace
        const now = "fromGemini" + Date.now();

        ws.send(`<div hx-swap-oob="beforeend:#toupdate"><div
                        id=${now}
                        class=" text-blue-400 m-2 text-left border p-2 rounded-lg mr-24">
                        ${spinnerSvg}
                    </div></div>`);

        const results = await chatWithModel.sendMessage(questionToAsk);

        // Function calling demo
        let response1 = await results.response;
        let data = response1.candidates[0].content.parts[0];

        let methodToCall = data.functionCall;
        if (methodToCall === undefined) {
            console.log("Gemini says: ", data.text);
            ws.send(`<div
                        id=${now}
                        hx-swap-oob="true"
                        hx-swap="outerHTML"
                        class="text-blue-400 m-2 text-left border p-2 rounded-lg mr-24">
                        ${data.text}
                    </div>`);

            // bail out - Gemini doesn't want to return a function
            return;
        }

        // otherwise Gemini wants to call a function
        console.log(
            "Gemini wants to call: " +
                methodToCall.name +
                " with args: " +
                util.inspect(methodToCall.args, {
                    showHidden: false,
                    depth: null,
                    colors: true
                })
        );

        // make the external call
        let jsonReturned;
        try {
            const responseFunctionCalling = await axios.get(
                baseUrl + "/" + methodToCall.name,

                {
                    params: {
                        location: methodToCall.args.location
                    }
                }
            );
            jsonReturned = responseFunctionCalling.data;
        } catch (ex) {
            // in case an invalid location was provided
            jsonReturned = ex.response.data;
        }

        console.log("jsonReturned: ", jsonReturned);

        // tell the model what function we just called
        const functionResponseParts = [
            {
                functionResponse: {
                    name: methodToCall.name,
                    response: {
                        name: methodToCall.name,
                        content: { jsonReturned }
                    }
                }
            }
        ];

        // // Send a follow up message with a FunctionResponse
        const result2 = await chatWithModel.sendMessage(
            functionResponseParts
        );

        // This should include a text response from the model using the response content
        // provided above
        const response2 = await result2.response;
        let answer = response2.candidates[0].content.parts[0].text;
        console.log("answer: ", answer);

        ws.send(`<div
                        id=${now}
                        hx-swap-oob="true"
                        hx-swap="outerHTML"
                        class="text-blue-400 m-2 text-left border p-2 rounded-lg mr-24">
                        ${answer}
                    </div>`);
    });

    ws.on("close", () => {
        console.log("WebSocket was closed");
    });
});

function sleep(ms) {
    return new Promise((resolve) => {
        setTimeout(resolve, ms);
    });
}

Create a input.css file for tailwindCSS.

@tailwind base;
@tailwind components;
@tailwind utilities;

Create the tailwind.config.js file for tailwindCSS.

/** @type {import('tailwindcss').Config} */
module.exports = {
    content: ["./**/*.{html,js}"],
    theme: {
        extend: {}
    },
    plugins: []
};

Create the metadataService.js file for getting the project id and region for the deployed Cloud Run service. These values will be used to instantiate an instance of the Vertex AI client libraries.

Note: To run your app locally, you must set your project ID and your region on lines 1 and 2 below:

const your_project_id = "YOUR_PROJECT_ID";
const your_region = "YOUR_REGION";

const axios = require("axios");

module.exports = {
    getProjectId: async () => {
        let project = "";
        try {
            // Fetch the token to make a GCF to GCF call
            const response = await axios.get(
                "http://metadata.google.internal/computeMetadata/v1/project/project-id",
                {
                    headers: {
                        "Metadata-Flavor": "Google"
                    }
                }
            );

            if (response.data == "") {
                // running locally on Cloud Shell
                project = your_project_id;
            } else {
                // use project id from metadata service
                project = response.data;
            }
        } catch (ex) {
            // running locally on local terminal
            project = your_project_id;
        }

        return project;
    },

    getRegion: async () => {
        let region = "";
        try {
            // Fetch the token to make a GCF to GCF call
            const response = await axios.get(
                "http://metadata.google.internal/computeMetadata/v1/instance/region",
                {
                    headers: {
                        "Metadata-Flavor": "Google"
                    }
                }
            );

            if (response.data == "") {
                // running locally on Cloud Shell
                region = your_region;
            } else {
                // use region from metadata service
                let regionFull = response.data;
                const index = regionFull.lastIndexOf("/");
                region = regionFull.substring(index + 1);
            }
        } catch (ex) {
            // running locally on local terminal
            region = your_region;
        }
        return region;
    }
};

Create a file called spinnerSvg.js

module.exports.spinnerSvg = `<svg class="animate-spin -ml-1 mr-3 h-5 w-5 text-blue-500"
                    xmlns="http://www.w3.org/2000/svg"
                    fill="none"
                    viewBox="0 0 24 24"
                >
                    <circle
                        class="opacity-25"
                        cx="12"
                        cy="12"
                        r="10"
                        stroke="currentColor"
                        stroke-width="4"
                    ></circle>
                    <path
                        class="opacity-75"
                        fill="currentColor"
                        d="M4 12a8 8 0 018-8V0C5.373 0 0 5.373 0 12h4zm2 5.291A7.962 7.962 0 014 12H0c0 3.042 1.135 5.824 3 7.938l3-2.647z"
                    ></path></svg>`;

Create a new public directory.

mkdir public
cd public

Now create the index.html file for the front end, which will use htmx.

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="UTF-8" />
        <meta
            name="viewport"
            content="width=device-width, initial-scale=1.0"
        />
        <script
            src="https://unpkg.com/htmx.org@1.9.10"
            integrity="sha384-D1Kt99CQMDuVetoL1lrYwg5t+9QdHe7NLX/SoJYkXDFfX37iInKRy5xLSi8nO7UC"
            crossorigin="anonymous"
        ></script>

        <link href="./output.css" rel="stylesheet" />
        <script src="https://unpkg.com/htmx.org/dist/ext/ws.js"></script>

        <title>Demo 2</title>
    </head>
    <body>
        <div id="herewego" text-center>
            <!-- <div id="replaceme2" hx-swap-oob="true">Hello world</div> -->
            <div
                class="container mx-auto mt-8 text-center max-w-screen-lg"
            >
                <div
                    class="overflow-y-scroll bg-white p-2 border h-[500px] space-y-4 rounded-lg m-auto"
                >
                    <div id="toupdate"></div>
                </div>
                <form
                    hx-trigger="submit, keyup[keyCode==13] from:body"
                    hx-ext="ws"
                    ws-connect="/sendMessage"
                    ws-send=""
                    hx-on="htmx:wsAfterSend: document.getElementById('message').value = ''"
                >
                    <div class="mb-6 mt-6 flex gap-4">
                        <textarea
                            rows="2"
                            type="text"
                            id="message"
                            name="message"
                            class="block grow rounded-lg border p-6 resize-none"
                            required
                        >
What&apos;s is the current weather in Seattle?</textarea
                        >
                        <button
                            type="submit"
                            class="bg-blue-500 text-white px-4 py-2 rounded-lg text-center text-sm font-medium"
                        >
                            Send
                        </button>
                    </div>
                </form>
            </div>
        </div>
    </body>
</html>

7. Run the Frontend service locally

First, make sure you are in the directory frontend for your codelab.

cd .. && pwd

Then, install dependencies by running the following command:

npm install

Using ADC when running locally

If you are running in Cloud Shell, you are already running on a Google Compute Engine virtual machine. Your credentials associated with this virtual machine (as shown by running gcloud auth list) will automatically be used by Application Default Credentials, so it is not necessary to use the gcloud auth application-default login command. You can skip down to the section Run the app locally

However, if you are running on your local terminal (i.e. not in Cloud Shell), you will need to use Application Default Credentials to authenticate to Google APIs. You can either 1) login using your credentials (provided you have both Vertex AI User and Datastore User roles) or 2) you can login by impersonating the service account used in this codelab.

Option 1) Using your credentials for ADC

If you want to use your credentials, you can first run gcloud auth list to verify how you are authenticated in gcloud. Next, you may need to grant your identity the Vertex AI User role. If your identity has the Owner role, you already have this Vertex AI user role. If not, you can run this command to grant your identity Vertex AI user role and the Datastore User role.

USER=<YOUR_PRINCIPAL_EMAIL>

gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member user:$USER \
  --role=roles/aiplatform.user

gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member user:$USER \
  --role=roles/datastore.user

Then run the following command

gcloud auth application-default login

Option 2) Impersonating a Service Account for ADC

If you want to use the service account created in this codelab, your user account will need to have the Service Account Token Creator role. You can obtain this role by running the following command:

gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member user:$USER \
  --role=roles/iam.serviceAccountTokenCreator

Next, you'll run the following command to use ADC with the service account

gcloud auth application-default login --impersonate-service-account=$SERVICE_ACCOUNT_ADDRESS

Run the app locally

Lastly, you can start the app by running the following script. This dev script will also generate the output.css file from tailwindCSS.

npm run dev

You can preview the website by opening the Web Preview button and selecting Preview Port 8080

8. Deploy and test the Frontend service

First, run this command to start the deployment and specify the service account to be used. If a service account is not specified, the default compute service account is used.

gcloud run deploy $FRONTEND \
  --service-account $SERVICE_ACCOUNT_ADDRESS \
  --source . \
  --region $REGION \
  --allow-unauthenticated

Open the service URL for the frontend in your browser. Ask a question "What is the current weather in Seattle?" and Gemini should respond "It is currently 40 degrees and rainy." If you ask "What is the current weather in Boston?", Gemini will respond with "I cannot fulfill this request. The available weather API does not have data for Boston."

9. Congratulations!

Congratulations for completing the codelab!

We recommend reviewing the documentation Cloud Run, Vertex AI Gemini APIs, and function calling.

What we've covered

• How Gemini function calling works
• How to deploy a Gemini-powered chatbot app as a Cloud Run service

10. Clean up

To avoid inadvertent charges, (for example, if this Cloud Run servcie is inadvertently invoked more times than your monthly Cloud Run invokement allocation in the free tier), you can either delete the Cloud Run service or delete the project you created in Step 2.

To delete the Cloud Run services, go to the Cloud Run Cloud Console at https://console.cloud.google.com/functions/ and delete the $WEATHER_SERVICE and $FRONTEND services you created in this codelab.

You may also want to delete the vertex-ai-caller service account or revoke the Vertex AI User role, to avoid any inadvertent calls to Gemini.

If you choose to delete the entire project, you can go to https://console.cloud.google.com/cloud-resource-manager, select the project you created in Step 2, and choose Delete. If you delete the project, you'll need to change projects in your Cloud SDK. You can view the list of all available projects by running gcloud projects list.