Securing cross-cloud agentic enterprise deployments

1. Introduction

In this codelab, you will deploy a single agent / multi tool deployment securely using the Agent Development Kit (ADK), Agent Engine and Google Kubernetes Engine. You will learn how an AI agent initiated by a user in Gemini Enterprise securely navigates Google Cloud, relying on GKE Gateway with Service Extensions to redact sensitive data inflight from MCP tool responses.

What you learn

What you need

  • A web browser such as Chrome
  • A Google Cloud project with billing enabled
  • Basic familiarity with Terraform, Kubernetes, and Python

This codelab is for developers and security professionals who want to deploy and secure agentic workflows in enterprise environments.

2. Before you begin

Create a Google Cloud project and enable the required APIs.

  1. In the Google Cloud Console, on the project selector page, select or create a Google Cloud project .
  2. Make sure that billing is enabled for your Cloud project. Learn how to check if billing is enabled on a project.

Required IAM roles

This codelab assumes that you have the Project Owner role for your Google Cloud project.

Enable APIs

  1. In the Google Cloud console, click Activate Cloud Shell: If you've never used Cloud Shell before, a pane appears giving you the choice to start Cloud Shell in a trusted environment with or without a boost. If you are asked to authorize Cloud Shell, click Authorize.
  2. In Cloud Shell, enable all required APIs:
    gcloud services enable \
    compute.googleapis.com \
    container.googleapis.com \
    dns.googleapis.com \
    certificatemanager.googleapis.com \
    dlp.googleapis.com \
    aiplatform.googleapis.com \
    discoveryengine.googleapis.com \
    apigee.googleapis.com
    

Install dependencies

In Cloud Shell, ensure you have the required tools installed. Terraform, kubectl, and Go are typically pre-installed. You need to install uv (the Python package manager) and Skaffold:

# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh

# Install Skaffold
curl -Lo skaffold https://storage.googleapis.com/skaffold/releases/latest/skaffold-linux-amd64 && \
sudo install skaffold /usr/local/bin/

Set environment variables

Set the following environment variables that are used throughout this codelab:

export PROJECT_ID=$(gcloud config get project)
export REGION=us-central1
export LOCATION=${REGION}

Create a public DNS zone

This codelab requires a public DNS zone to pre-exist in your project before you apply the Terraform configuration. This zone is needed for nameserver delegation so that the setup can automate the creation of required records for Certificate Manager.

Run the following command in Cloud Shell to create the zone:

gcloud dns managed-zones create "inference-gateway-zone" \
    --dns-name="gateway.example.com." \
    --description="Public zone for Inference Gateway" \
    --visibility="public" \
    --project="${PROJECT_ID}"

3. Clone a GitHub repository

  1. In a terminal on your local machine, clone the cloud-networking-solutions repository:
    git clone https://github.com/googleCloudPlatform/cloud-networking-solutions.git
    
  2. Navigate to the downloaded repository directory:
    cd cloud-networking-solutions/demos/service-extensions-gke-gateway
    

4. Deploy infrastructure with Terraform

You will use Terraform to provision the foundational network, GKE cluster, and required identity configurations.

  1. Navigate to the terraform directory in the cloned repository:
    cd terraform
    
  2. Configure the Terraform backend. Create a backend.conf file for partial backend configuration. Replace with a globally unique bucket name.
    cat <<EOF > backend.conf
    bucket = "<YOUR_TERRAFORM_STATE_BUCKET>"
    prefix = "terraform"
    EOF
    
  1. Copy the example variable file and update it with your project values:
    cp example.tfvars terraform.tfvars
    
  2. Edit terraform.tfvars and replace the placeholder values.Replace the following:
    • project_id: your Google Cloud project ID.
    • organization_id: your numeric GCP organization ID.
    • dns_zone_domain: a domain you control (e.g., demo.example.com.). Must end with a dot.
    Ensure the following feature flags are enabled (they are pre-configured in the example file):
    • enable_certificate_manager = true
    • enable_model_armor = true
    • enable_agent_engine = true
    • enable_psc_interface = true
  3. Initialize and apply the Terraform configuration:
    terraform init -backend-config=backend.conf
    terraform plan -out=tfplan
    terraform apply "tfplan"
    

5. Deploy sample workloads with Skaffold

Next, deploy the MCP servers and external processing services to your GKE cluster.

  1. Connect to the GKE cluster created by Terraform:
    gcloud container clusters get-credentials gateway-cluster \
        --region=${REGION} \
        --project=${PROJECT_ID}
    
  2. Navigate back to the project root and configure the Kubernetes manifest templates. Copy the example configuration and set your project ID and base domain:Set BASE_DOMAIN to match your environment. The BASE_DOMAIN should match your dns_zone_domain Terraform variable (without the trailing dot).
    export BASE_DOMAIN=example.com
    
  3. Generate the Kubernetes manifests and skaffold.yaml from templates:
    bash k8s/generate.sh
    envsubst '${PROJECT_ID}' < skaffold.yaml.tmpl > skaffold.yaml
    
  4. Deploy all backend services:
    skaffold run -m legacy-dms,income-verification-api,corporate-email,dlp-ext-proc
    
  5. Deploy the Gateway and HTTPRoute configuration:
    kubectl apply -k k8s/gateway-internal/
    

6. Apply AI guardrails with Model Armor

You can delegate content security decisions, such as removing harmful prompts or preventing data leaks, to Model Armor.

Prerequisites: Grant IAM Roles

You must grant the required roles to the GKE Gateway service account. The service account follows the format: service-GATEWAY_PROJECT_NUMBER@gcp-sa-dep.iam.gserviceaccount.com.

Run the following commands to grant the necessary permissions:

export GATEWAY_PROJECT_NUMBER=$(gcloud projects describe $PROJECT_ID --format="value(projectNumber)")

# Grant roles in the Gateway project
gcloud projects add-iam-policy-binding $PROJECT_ID \
 --member=serviceAccount:service-${GATEWAY_PROJECT_NUMBER}@gcp-sa-dep.iam.gserviceaccount.com \
 --role=roles/modelarmor.calloutUser

gcloud projects add-iam-policy-binding $PROJECT_ID \
 --member=serviceAccount:service-${GATEWAY_PROJECT_NUMBER}@gcp-sa-dep.iam.gserviceaccount.com \
 --role=roles/serviceusage.serviceUsageConsumer

# Grant role in the project containing Model Armor templates
gcloud projects add-iam-policy-binding $PROJECT_ID \
 --member=serviceAccount:service-${GATEWAY_PROJECT_NUMBER}@gcp-sa-dep.iam.gserviceaccount.com \
 --role=roles/modelarmor.user

Create the Model Armor Authorization Extension

Define an extension that points to the Model Armor service in your region. Save this configuration as ma-content-authz-extension.yaml.

Export the Model Armor Template ID created by the Terraform.

export MA_TEMPLATE_ID=$(cd terraform && terraform output -raw model_armor_template_id)
cat >ma-content-authz-extension.yaml <<EOF
name: ma-ext
service: modelarmor.$LOCATION.rep.googleapis.com
metadata:
  model_armor_settings: '[
  {
  "response_template_id": "projects/${PROJECT_ID}/locations/$LOCATION/templates/${MA_TEMPLATE_ID}",
  "request_template_id": "projects/${PROJECT_ID}/locations/$LOCATION/templates/${MA_TEMPLATE_ID}"
  }
  ]'
failOpen: true
EOF

gcloud beta service-extensions authz-extensions import ma-ext \
    --source=ma-content-authz-extension.yaml \
    --location=$LOCATION

Create the Model Armor Authorization Policy

Create a policy that associates the Model Armor extension with your Agent Gateway. Save this configuration as ma-content-authz-policy.yaml.

cat >ma-content-authz-policy.yaml <<EOF
name: ma-content-authz-policy
target:
  resources:
  -   "projects/$PROJECT_ID/locations/$LOCATION/gateways/mortgage-gateway"
policyProfile: CONTENT_AUTHZ
action: CUSTOM
customProvider:
  authzExtension:
    resources:
    -   "projects/$PROJECT_ID/locations/$LOCATION/authzExtensions/ma-ext"
EOF

gcloud network-security authz-policies import ma-content-authz-policy \
    --source=ma-content-authz-policy.yaml \
    --location=$LOCATION

7. Configure Gemini Enterprise

Enable observability

The mortgage agent is deployed with OpenTelemetry instrumentation and the following telemetry environment variables enabled by default:

  • GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY=true
  • OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true
  • OTEL_TRACES_SAMPLER=parentbased_traceidratio

For details on configuring traces and spans in Gemini Enterprise, see Manage observability settings.

Enable Model Armor in Gemini Enterprise

Apply Model Armor filtering to the Gemini Enterprise assistant to screen both user prompts and model responses. Global Gemini Enterprise apps require a Model Armor template in the us multi-region, so Terraform deploys a separate template for this purpose.

Retrieve the template name from Terraform output:

cd terraform
export GE_MA_TEMPLATE_NAME=$(terraform output -raw model_armor_gemini_enterprise_template_name)

Fetch the APP ID of your Gemini Enterprise Instance:

  1. In the Google Cloud console, go to the Gemini Enterprise page.
  2. From the navigation menu, click Apps.
  3. Copy the ID of the Gemini Enterprise Instance

Export the ID as an environment variable.

export APP_ID=<PASTE_APP_ID>

Patch the assistant to enable Model Armor:

curl -X PATCH \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
  -H "X-Goog-User-Project: ${PROJECT_ID}" \
  "https://global-discoveryengine.googleapis.com/v1/projects/${PROJECT_ID}/locations/global/collections/default_collection/engines/${APP_ID}/assistants/default_assistant?update_mask=customerPolicy" \
  -d '{
    "customerPolicy": {
      "modelArmorConfig": {
        "userPromptTemplate": "'"$GE_MA_TEMPLATE_NAME"'",
        "responseTemplate": "'"$GE_MA_TEMPLATE_NAME"'",
        "failureMode": "FAIL_OPEN"
      }
    }
  }'

Set failureMode to FAIL_OPEN to allow requests through when Model Armor is unavailable, or FAIL_CLOSED to block them.

8. Deploy Agent and Add Agent in Gemini Enterprise

Obtain authorization details

Follow these steps to obtain the authorization details.

  1. In the Google Cloud console, on the APIs & Services page, go to the Credentials page.
  2. Go to Credentials
  3. Click Create credentials and select OAuth client ID.
  4. In Application type, select Web application.
  5. In the Authorized redirect URIs section, add the following URIs:
  • https://vertexaisearch.cloud.google.com/oauth-redirect
  • https://vertexaisearch.cloud.google.com/static/oauth/oauth.html
  1. Click Create.
  2. Export Client ID and Client Secret for deployment.
export OAUTH_CLIENT_ID=<Client ID>
export OAUTH_CLIENT_SECRET=<Client Secret>

Deploy the mortgage agent

The src/mortgage-agent/deploy_agent.py script deploys the ADK agent to Agent Engine and optionally registers it in Gemini Enterprise. See Register and manage an A2A agent for background on the Gemini Enterprise registration flow.

Export variables from Terraform deployment.

export VPC_NAME=$(cd terraform && terraform output -raw vpc_name)
export PSC_ATTACHMENT=$(cd terraform && terraform output -raw psc_interface_network_attachment_id)
export DNS_PEERING_DOMAIN=$(cd terraform && terraform output -raw psc_interface_dns_peering_domain)

Install dependencies and deploy:

cd src/mortgage-agent
uv sync

Deploy Agent to Vertex Agent Engine and register to Gemini Enterprise:

uv run python deploy_agent.py \
    --project=${PROJECT_ID} \
    --dms-mcp-url=https://dms.${DNS_PEERING_DOMAIN%%.}/mcp \
    --income-verification-url=https://income-verification.${DNS_PEERING_DOMAIN%%.} \
    --email-mcp-url=https://email.${DNS_PEERING_DOMAIN%%.}/mcp \
    --network-attachment=projects/${PROJECT_ID}/regions/${REGION}/networkAttachments/${PSC_ATTACHMENT} \
    --dns-peering-domain=${DNS_PEERING_DOMAIN} \
    --dns-peering-target-project=${PROJECT_ID} \
    --dns-peering-target-network=${VPC_NAME} \
    --enable-agent-identity \
    --ge-deploy \
    --app-id=${APP_ID} \
    --oauth-client-id=${OAUTH_CLIENT_ID} \
    --agent-name=mortgage-agent

Flag reference

Flag

Default

Description

--project

$PROJECT_ID

GCP project ID

--dms-mcp-url

(required)

DMS MCP server URL

--income-verification-url

(required)

Income Verification API base URL; /mcp is appended automatically

--email-mcp-url

(required)

Email MCP server URL

--region

$REGION

GCP region

--staging-bucket

gs://PROJECT-staging

GCS bucket for staging

--display-name

Mortgage Assistant Agent

Display name for the deployed agent

--update

(optional)

Update an existing agent in-place (pass full resource name)

--network-attachment

(optional)

Network attachment for PSC Interface (full path or name)

--dns-peering-domain

(optional)

DNS domain for PSC-I DNS peering (must end with a dot)

--dns-peering-target-project

(optional)

Project hosting the target VPC network for DNS peering

--dns-peering-target-network

(optional)

VPC network name for DNS peering

--enable-agent-identity

false

Enable per-agent least-privilege credentials

--ge-deploy

false

Register agent in Gemini Enterprise after deploy

--app-id

(optional)

Gemini Enterprise engine ID (required with --ge-deploy)

--oauth-client-id

$OAUTH_CLIENT_ID

OAuth2 client ID (required with --ge-deploy)

--oauth-client-secret

$OAUTH_CLIENT_SECRET

OAuth2 client Secret (required with --ge-deploy)

--model

gemini-3.1-flash-lite-preview

Gemini model name for the agent

--agent-name

mortgage-agent

Gemini Enterprise authorization and agent name

--ge-deploy-only

(optional)

Register an existing reasoning engine in Gemini Enterprise without redeployment (pass full resource name)

Add permissioned users

To add permissioned users to an ADK agent using the Google Cloud console, see Add or modify users and their permissions.

9. Test your Agent

Now that you have deployed and configured the agent, GKE Gateway, and all the backend services, test the end-to-end flow to verify the security policies are working as expected. You will interact with the "mortgage-agent" within the Gemini Enterprise interface.

Accessing the Agent

  1. Navigate to the Gemini Enterprise page in the Google Cloud Console.
  2. Select the Gemini Enterprise App that you configured earlier, where the "mortgage-agent" is registered.
  3. On the Overview Tab, browse to the URL shown in the "Your Gemini Enterprise webapp is ready" section.
  4. Select the Agent Tab from the menu on the left called the Agent Gallery
  5. You should now be able to chat with the "Mortgage Assistant Agent".

Test Case 1: The "Happy Path" - Summarizing Data with PII Redaction

This test verifies that the agent can access backend systems through the Agent Gateway and that Data Loss Prevention (DLP) policies are enforced to redact sensitive information.

  1. Send the following prompt to the Mortgage Assistant Agent:
    I'm reviewing the Sterling family's current application. Can you summarize their 2024 and 2025 tax returns and verify if their total household income meets our 2026 debt-to-income requirements?
    
  2. What's happening behind the scenes:
    • The agent in Gemini Enterprise formulates requests to the necessary tools (e.g., Document Management System (DMS) for tax returns, Income Verification API).
    • Model Armor inspects the request and response payloads for threats.
    • The "Content-based Authorization Policy" you configured triggers the custom DLP extension (dlp-content-authz-ext). This extension inspects the data fetched from the backend systems.
    • The DLP service redacts any Personally Identifiable Information (PII), such as Social Security Numbers (SSNs), from the tax return data before it reaches the agent.
  3. Expected Outcome:The agent will return a summary of the tax returns and an income verification status. Critically, inspect the summary provided by the agent. You should observe that sensitive information, like Taxpayer IDs (SSNs), has been replaced with placeholders (e.g., [REDACTED]). This confirms the DLP policy execution via the gateway.

Observability and Auditing

Throughout these tests, the Agent Platform and associated services are collecting telemetry:

  • Cloud Logging: Detailed logs from the GKE Gateway, GKE workloads, and other services provide an audit trail of requests, policy evaluations, and outcomes.
  • Cloud Trace: The OpenTelemetry instrumentation configured in the agent and backend services allows you to visualize the entire call flow, from Gemini Enterprise through the GKE Gateway to the backend tools. This is invaluable for debugging and understanding the request lifecycle.

View Traces for your sessions:

  1. In the Google Cloud console, go to the Vertex AI Agent Engine page.
  2. Go to Agent Engine. Agent Engine instances that are part of the selected project appear in the list. You can use the Filter field to filter the list by your specified column.
  3. Click the name of your Agent Engine instance.
  4. Click the Traces tab.
  5. You can select Session view or Span view.
  6. Click a session or span to inspect trace details, including a directed acyclic graph (DAG) of its spans, inputs and outputs, and metadata attributes.

10. Optional: Transcoding REST APIs to MCP with Apigee

While our income verification service natively supports the Model Context Protocol, many enterprise legacy systems only provide RESTful APIs. In this optional step, you will use the Apigee MCP Discovery Proxy to transcode the REST endpoint of the income verification service into an MCP tool. This allows you to apply Apigee's advanced governance, rate limiting, and security policies to your legacy tools.

For more information, see the Apigee MCP overview.

Prerequisites

Before proceeding, ensure you have provisioned and configured Apigee API Hub. Follow the steps in the Provision API Hub documentation to set it up and attach your Apigee instance.

Step 1: Create a service attachment for Apigee

To allow Apigee to reach your internal services running on GKE, you must create a service attachment.

  1. Look up the internal GKE Gateway forwarding rule:
    export RULE_URI=$(gcloud compute forwarding-rules list \
      --filter="loadBalancingScheme=INTERNAL_MANAGED AND target~targetHttpsProxies" \
      --format="value(selfLink.segment(6), region.basename(), name)" | \
      awk '{print "projects/" $1 "/regions/" $2 "/forwardingRules/" $3}')
    
  2. Create the service attachment:
    gcloud compute service-attachments create internal-gke-gateway-apigee \
        --region=${REGION} \
        --target-service=$RULE_URI \
        --connection-preference=ACCEPT_AUTOMATIC \
        --nat-subnets=gateway-psc-subnet
    

Step 2: Enable Apigee with Terraform

Now, update your infrastructure configuration to provision the Apigee organization and instance.

  1. Navigate to the terraform directory:
    cd terraform
    
  2. Edit terraform.tfvars and set enable_apigee = true.
  3. Apply the changes:
    terraform apply
    

Step 3: Define the OpenAPI Specification

Apigee uses standard OpenAPI (OAS) definitions to discover and transcode tools. Create a file named income-verification-oas.yaml with the following content:

openapi: 3.0.0
info:
  title: Income Verification API
  description: Verify applicant income through third-party employer records.
  version: 1.0.0
servers:
  -   url: https://income-verification.internal.${DNS_PEERING_DOMAIN%%.}/api
paths:
  /income-verification/verify:
    post:
      summary: Verify applicant income
      operationId: verifyApplicant
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              properties:
                first_name:
                  type: string
                last_name:
                  type: string
      responses:
        '200':
          description: Successful verification
          content:
            application/json:
              schema:
                type: object

Step 4: Deploy the Apigee MCP Discovery Proxy

The MCP Discovery Proxy is a specialized Apigee proxy that acts as an MCP server.

  1. In the Apigee UI, navigate to Develop > API Proxies.
  2. Click Create New and select MCP Discovery Proxy.
  3. Name the proxy income-verification-discovery.
  4. In the OpenAPI Spec section, upload the income-verification-oas.yaml file you created.
  5. Set the Environment Group to the one where your internal gateway is accessible.
  6. Click Deploy.

(Optional) Add a security policy

Before deploying or sharing your proxy, you should secure it. You can add policies to enforce security requirements, such as OAuth tokens or API keys. For instructions on how to add a security policy, see the Apigee documentation on securing APIs.

Step 5: Verify Transcoded Tool Access

Once deployed, Apigee automatically transcodes the POST /verify endpoint into a verifyApplicant MCP tool.

  1. List the tools available through the Apigee MCP endpoint:
    curl -X POST https://api.internal.${DNS_PEERING_DOMAIN%%.}/income-verification-discovery/mcp \
      -H "Content-Type: application/json" \
      -d '{
        "jsonrpc": "2.0",
        "id": 1,
        "method": "tools/list",
        "params": {}
      }'
    
  2. You should see the verifyApplicant tool in the response, transcoded from your REST specification. You can now call this tool through Apigee, and Apigee will handle the translation to the underlying REST service while applying any security policies you have configured.

Step 6: Update the Mortgage Agent to use Apigee

Now that Apigee is successfully transcoding your REST API into an MCP-compliant tool, you must update the agent's deployment configuration. By pointing the agent to the Apigee endpoint, all income verification requests will now benefit from Apigee's enterprise-grade security, logging, and traffic management.

  1. Identify your Apigee MCP URL: Your endpoint should follow the pattern: https://api.internal.${DNS_PEERING_DOMAIN%%.}/income-verification-discovery/mcp.
  2. Re-run the deployment script: Use the --update flag along with the new --income-verification-url. This updates the existing agent in Agent Engine without requiring a full deletion.
    cd src/mortgage-agent
    
    # Update the agent to route income verification through Apigee
    uv run python deploy_agent.py \
        --project=${PROJECT_ID} \
        --update \
        --agent-name=mortgage-agent \
        --dms-mcp-url=https://dms.${DNS_PEERING_DOMAIN%%.}/mcp \
        --income-verification-url=https://api.internal.${DNS_PEERING_DOMAIN%%.}/income-verification-discovery/mcp \
        --email-mcp-url=https://email.${DNS_PEERING_DOMAIN%%.}/mcp \
        --network-attachment=projects/${PROJECT_ID}/regions/${REGION}/networkAttachments/${PSC_ATTACHMENT} \
        --dns-peering-domain=${DNS_PEERING_DOMAIN} \
        --dns-peering-target-project=${PROJECT_ID} \
        --dns-peering-target-network=${VPC_NAME} \
        --ge-deploy \
        --app-id=${APP_ID} \
        --oauth-client-id=${OAUTH_CLIENT_ID}
    
  1. Verify the change: Return to the Gemini Enterprise interface and ask the agent to verify an applicant's income.
    "Can you verify the joint income for the Sterlings using the verification service?"
    
    In the Apigee Debug tab, you should now see the incoming JSON-RPC request from the GKE Gateway being transformed into a standard REST POST request to your backend GKE service.

11. Clean up

To avoid incurring charges to your Google Cloud account for the resources created in this codelab, delete them when you are finished.

  1. Delete the service attachment if it was created manually:
    gcloud compute service-attachments delete internal-gke-gateway \
        --region=${REGION} --quiet
    
  2. Navigate to the terraform directory and destroy all resources:
    cd terraform
    terraform destroy
    
  3. Optionally, delete the Google Cloud project entirely:
    gcloud projects delete ${PROJECT_ID}
    

12. Congratulations

You completed this codelab and learned how to secure cross-cloud agentic enterprise deployments.

What you covered

  • Deployed an ADK mortgage assistant agent to Agent Engine with OpenTelemetry instrumentation
  • Deployed backend MCP servers to GKE behind an internal Gateway
  • Connected Agent Engine to a project VPC using a PSC Interface and DNS peering
  • Configured GKE Gateway for governed agent egress with DLP redaction and MCP authorization
  • Applied AI guardrails with Model Armor for prompt and response screening
  • Optionally transcoded REST APIs to MCP using the Apigee MCP Discovery Proxy

Next steps