Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. Developed by Google and released as open-source, Kubernetes is now maintained by a diverse community and shepherded by the Cloud Native Computing Foundation.

The easiest way to run a Kubernetes cluster is using Google Kubernetes Engine, a managed version of Kubernetes hosted on Google Cloud Platform. Kubernetes Engine, also known as GKE, is a managed environment for deploying containerized applications. It brings our latest innovations in developer productivity, resource efficiency, automated operations, and open source flexibility to accelerate your time to market.

This codelab shows you some of the advanced features of Google Kubernetes Engine, and will show you how to run a service which makes the most of Google Cloud Platform's features. It assumes you have basic familiarity with Docker containers and Kubernetes concepts.

Self-paced environment setup

If you don't already have a Google Account (Gmail or Google Apps), you must create one. Sign-in to Google Cloud Platform console (console.cloud.google.com) and create a new project:

Screenshot from 2016-02-10 12:45:26.png

Remember the project ID, a unique name across all Google Cloud projects (the name above has already been taken and will not work for you, sorry!). It will be referred to later in this codelab as PROJECT_ID.

Next, you'll need to enable billing in the Cloud Console in order to use Google Cloud resources.

Running through this codelab shouldn't cost you more than a few dollars, but it could be more if you decide to use more resources or if you leave them running (see "cleanup" section at the end of this document).

New users of Google Cloud Platform are eligible for a $300 free trial.

Google Cloud Shell

While Google Kubernetes Engine can be operated remotely from your laptop, in this codelab we will be using Google Cloud Shell, a command line environment running in the Cloud.

This Debian-based virtual machine is loaded with all the development tools you'll need. It offers a persistent 5GB home directory, and runs on the Google Cloud, greatly enhancing network performance and authentication. This means that all you will need for this codelab is a browser (yes, it works on a Chromebook).

To activate Google Cloud Shell, from the developer console simply click the button on the top right-hand side (it should only take a few moments to provision and connect to the environment):

activateCloudShell.png

Then accept the terms of service and click the "Start Cloud Shell" link:

x.png

Screen Shot 2017-06-14 at 10.13.43 PM.png

Once connected to the cloud shell, you should see that you are already authenticated and that the project is already set to your PROJECT_ID :

gcloud auth list

Command output

Credentialed accounts:
 - <myaccount>@<mydomain>.com (active)
gcloud config list project

Command output

[core]
project = <PROJECT_ID>

If for some reason the project is not set, simply issue the following command :

gcloud config set project <PROJECT_ID>

Looking for your PROJECT_ID? Check out what ID you used in the setup steps or look it up in the console dashboard:

Project_ID.png

IMPORTANT: Finally, set the default zone and project configuration:

gcloud config set compute/zone us-central1-f

You can choose a variety of different zones. Learn more in the Regions & Zones documentation.

Enable APIs

In order to use the necessary APIs for this workshop we need to enable a few Google Cloud APIs. Navigate to the APIs & Services page in the Google Cloud Console. Click on the button labeled Enable APIs and Services.

Search for the Kubernetes Engine API. Click on the search result and click the Enable button.

Next, search for the Stackdriver API. Click on the search result and click the Enable button.

Next, search for the Stackdriver Monitoring API. Click on the search result and click the Enable button.

Next, search for the Stackdriver Logging API. Click on the search result and click the Enable button.

Create a Stackdriver Monitoring Account

Select Monitoring from the menu in the Google Cloud Console. This will open Stackdriver Monitoring in a new tab. Here you will be prompted to create an account. Stackdriver Monitoring allows monitoring multiple projects from a single account. For the purposes of this workshop we will only select a single project.

You will be presented with a page like this:

From here just click on the Create Account button.

  1. On the next page click Continue.
  2. On the next page click Skip AWS Setup.
  3. On the next page click Continue.
  4. On the next page select No Reports and click Continue.
  5. After a moment, a Launch Monitoring button should appear. Click on the Launch Monitoring button.
  6. Click on the Continue the free trial button.
  7. Once the Stackdriver Monitoring home page has loaded you can close the tab.

Lets create a Kubernetes cluster. You will use this cluster for all the upcoming exercises.

First, we will set our zone to be used throughout the workshop.

$ gcloud config set compute/zone europe-west1-d

Updated property [compute/zone].

When creating a cluster we will need to specify the software version for our cluster. Kubernetes Engine version numbers look like X.Y.Z.gke.N. The X.Y.Z part indicates the Kubernetes version. The gke.N part specifies a patch version for Kubernetes Engine, which include security updates or bug fixes on top of the open-source upstream Kubernetes.

With Kubernetes Engine you can specify an exact version (e.g. 1.9.0.gke.0), or you can specify a version alias. Here we will specify the 1.9 version alias, which will give us the latest patch version in the 1.9 version series.

$ gcloud container clusters create gke-workshop \
       --enable-network-policy \
       --cluster-version=1.9

Creating cluster gke-workshop...done.                                                                                                                                                                     
Created [https://container.googleapis.com/v1/projects/codelab/zones/europe-west1-d/clusters/gke-workshop].
kubeconfig entry generated for gke-workshop.
NAME            LOCATION       MASTER_VERSION  MASTER_IP       MACHINE_TYPE   NODE_VERSION  NUM_NODES  STATUS
gke-workshop    europe-west1-d 1.9.6-gke.0     35.193.171.226  n1-standard-1  1.9.6-gke.0   3          RUNNING

When you create a cluster, gcloud adds a context to your kubectl configuration file (~/.kube/config). It then sets it as the current context, to let you operate on this cluster immediately.

$ kubectl config current-context

gke_codelab_europe-west1-d_gke-workshop

To test it, try a kubectl command line:

$ kubectl get nodes

NAME                                            STATUS    ROLES     AGE       VERSION
gke-gke-workshop-default-pool-1acc373c-1txb   Ready     <none>    6m        v1.9.6-gke.0
gke-gke-workshop-default-pool-1acc373c-cklc   Ready     <none>    6m        v1.9.6-gke.0
gke-gke-workshop-default-pool-1acc373c-kzjv   Ready     <none>    6m        v1.9.6-gke.0

If you navigate to "Kubernetes Engine" in the Google Cloud Platform console, you will see the cluster listed:

Run a deployment

Let's run a sample deployment to verify that our cluster is working. The kubectl run command is shorthand for creating a Kubernetes deployment without the need for a YAML or JSON spec file.

$ kubectl run hello-web --image=gcr.io/google-samples/hello-app:1.0 \
       --port=8080 --replicas=3

deployment "hello-web" created

$ kubectl get pods -o wide

NAME                         READY     STATUS    RESTARTS   AGE       IP          NODE
hello-web-5d9cdb689c-5qh2h   1/1       Running   0          2s        10.60.2.6   gke-gke-workshop-default-pool-1acc373c-1txb
hello-web-5d9cdb689c-sp2bj   1/1       Running   0          2s        10.60.1.9   gke-gke-workshop-default-pool-1acc373c-cklc
hello-web-5d9cdb689c-xfcm5   1/1       Running   0          2s        10.60.1.8   gke-gke-workshop-default-pool-1acc373c-cklc

Look at the Google Cloud dashboard

Workloads that are deployed to your Kubernetes Engine cluster are displayed in the Google Cloud Console. Navigate to Kubernetes Engine, and then Workloads.

You can see the hello-web Deployment we just created. Feel free to click on it, and explore the user interface.

Congrats on deploying your application to GKE!

What we've covered

A Kubernetes Engine cluster consists of a master and nodes. Kubernetes doesn't handle provisioning of nodes, so Google Kubernetes Engine handles this for you with a concept called node pools.

A node pool is a subset of node instances within a cluster that all have the same configuration. They map to instance templates in Google Compute Engine, which provides the VMs used by the cluster. By default a Kubernetes Engine cluster has a single node pool, but you can add or remove them as you wish to change the shape of your cluster.

In the previous example, you created a Kubernetes Engine cluster. This gave us three nodes (three n1-standard-1 VMs, 100 GB of disk each) in a single node pool (called default-pool). Let's inspect the node pool:

$ gcloud container node-pools list --cluster gke-workshop
NAME          MACHINE_TYPE   DISK_SIZE_GB  NODE_VERSION
default-pool  n1-standard-1  100           1.9.6-gke.0

If you want to add more nodes of this type, you can grow this node pool. If you want to add more nodes of a different type, you can add other node pools.

A common method of moving a cluster to larger nodes is to add a new node pool, move the work from the old nodes to the new, and delete the old node pool.

Let's add a second node pool, and migrate our workload over to it. This time we will use the larger n1-standard-2 machine type, but only create one instance.

$ gcloud container node-pools create new-pool --cluster gke-workshop \
    --machine-type n1-standard-2 --num-nodes 3
Creating node pool new-pool...done.                                                                                                                                   
Created [https://container.googleapis.com/v1/projects/codelab/zones/europe-west1-d/clusters/gke-workshop/nodePools/new-pool].
NAME         MACHINE_TYPE   DISK_SIZE_GB  NODE_VERSION
new-pool     n1-standard-1  100           1.9.6-gke.0

$ gcloud container node-pools list --cluster gke-workshop
NAME          MACHINE_TYPE   DISK_SIZE_GB  NODE_VERSION
default-pool  n1-standard-1  100           1.9.6-gke.0
new-pool      n1-standard-2  100           1.9.6-gke.0

$ kubectl get nodes
NAME                                            STATUS    ROLES     AGE       VERSION
gke-gke-workshop-default-pool-1acc373c-1txb   Ready     <none>    56m       v1.9.6-gke.0
gke-gke-workshop-default-pool-1acc373c-cklc   Ready     <none>    56m       v1.9.6-gke.0
gke-gke-workshop-default-pool-1acc373c-kzjv   Ready     <none>    57m       v1.9.6-gke.0
gke-gke-workshop-new-pool-97a76573-l57x       Ready     <none>    1m        v1.9.6-gke.0

Kubernetes does not reschedule Pods as long as they are running and available, so your workload remains running on the nodes in the default pool.

Look at one of your nodes using kubectl describe. Just like you can attach labels to pods, nodes are automatically labelled with useful information which lets the scheduler make decisions and the administrator perform action on groups of nodes.

Replace "[NODE NAME]" with the name of one of your nodes from the previous step.

$ kubectl describe node [NODE NAME] | head -n 20
Name:               gke-gke-workshop-default-pool-1acc373c-1txb
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/fluentd-ds-ready=true
                    beta.kubernetes.io/instance-type=n1-standard-1
                    beta.kubernetes.io/masq-agent-ds-ready=true
                    beta.kubernetes.io/os=linux
                    cloud.google.com/gke-nodepool=default-pool
                    failure-domain.beta.kubernetes.io/region=europe-west1
                    failure-domain.beta.kubernetes.io/zone=europe-west1-d
                    kubernetes.io/hostname=gke-gke-workshop-default-pool-1acc373c-1txb
                    projectcalico.org/ds-ready=true
Annotations:        node.alpha.kubernetes.io/ttl=0
                    volumes.kubernetes.io/controller-managed-attach-detach=true
<...>

You can also select nodes by node pool using the cloud.google.com/gke-nodepool label. We'll use this powerful construct shortly.

$ kubectl get nodes -l cloud.google.com/gke-nodepool=default-pool
NAME                                          STATUS    ROLES     AGE       VERSION
gke-gke-workshop-default-pool-1acc373c-1txb   Ready     <none>    54m       v1.9.6-gke.0
gke-gke-workshop-default-pool-1acc373c-cklc   Ready     <none>    54m       v1.9.6-gke.0
gke-gke-workshop-default-pool-1acc373c-kzjv   Ready     <none>    54m       v1.9.6-gke.0

Migrating pods to the new Node Pool

To migrate your pods to the new node pool, we will perform the following steps:

  1. Cordon the existing node pool: This operation marks the nodes in the existing node pool (default-pool) as unschedulable. Kubernetes stops scheduling new Pods to these nodes once you mark them as unschedulable.
  2. Drain the existing node pool: This operation evicts the workloads running on the nodes of the existing node pool (default-pool) gracefully.

You could cordon an individual node using the kubectl cordon command, but running this command on each node individually would be tedious. To speed up the process, we can embed the command in a loop. Be sure you copy the whole line - it will have scrolled off the screen to the right!

$ for node in $(kubectl get nodes -l cloud.google.com/gke-nodepool=default-pool -o=name); do kubectl cordon "$node"; done
node "gke-gke-workshop-default-pool-1acc373c-1txb" cordoned
node "gke-gke-workshop-default-pool-1acc373c-cklc" cordoned
node "gke-gke-workshop-default-pool-1acc373c-kzjv" cordoned

This loop utilizes the command kubectl get nodes to select all nodes in the default pool (using the cloud.google.com/gke-nodepool=default-pool label), and then it iterates through and runs kubectl cordon on each one.

After running the loop, you should see that the default-pool nodes have SchedulingDisabled status in the node list:

$ kubectl get nodes
NAME                                            STATUS                     ROLES     AGE       VERSION
gke-gke-workshop-default-pool-1acc373c-1txb   Ready,SchedulingDisabled   <none>    1h        v1.9.6-gke.0
gke-gke-workshop-default-pool-1acc373c-cklc   Ready,SchedulingDisabled   <none>    1h        v1.9.6-gke.0
gke-gke-workshop-default-pool-1acc373c-kzjv   Ready,SchedulingDisabled   <none>    1h        v1.9.6-gke.0
gke-gke-workshop-new-pool-97a76573-l57x       Ready                      <none>    10m       v1.9.6-gke.0

Next, we want to evict the Pods already scheduled on each node. To do this, we will construct another loop, this time using the kubectl drain command:

$ for node in $(kubectl get nodes -l cloud.google.com/gke-nodepool=default-pool -o=name); do kubectl drain --force --ignore-daemonsets --delete-local-data "$node"; done

<...>
pod "hello-web-59d96f7bd6-scknr" evicted
pod "hello-web-59d96f7bd6-v5dbq" evicted
pod "hello-web-59d96f7bd6-p92b6" evicted
node "gke-gke-workshop-default-pool-cf484031-cl8m" drained
<...>

As each node is drained, the pods running on it are evicted. Eviction makes sure to follow rules to provide the least disruption to the applications as possible. Users in production may want to look at more advanced features like Pod Disruption Budgets.

Because the default node pool is unschedulable, the pods are now running on the single machine in the new node pool:

$ kubectl get pods -o wide
NAME                         READY     STATUS    RESTARTS   AGE       IP          NODE
hello-web-5d9cdb689c-54vnv   1/1       Running   0          2m        10.60.6.5   gke-gke-workshop-new-pool-97a76573-l57x
hello-web-5d9cdb689c-pn25c   1/1       Running   0          2m        10.60.6.8   gke-gke-workshop-new-pool-97a76573-l57x
hello-web-5d9cdb689c-s6g6r   1/1       Running   0          2m        10.60.6.6   gke-gke-workshop-new-pool-97a76573-l57x

You can now delete the original node pool:

$ gcloud container node-pools delete default-pool --cluster gke-workshop
The following node pool will be deleted.
[default-pool] in cluster [gke-workshop] in [europe-west1-d]

Do you want to continue (Y/n)?  y

Deleting node pool default-pool...done.                                                                                                                                  
Deleted [https://container.googleapis.com/v1/projects/codelab/zones/europe-west1-d/clusters/gke-workshop/nodePools/default-pool].

Creating a secondary node pool with GPUs

In addition to migrating from one node pool to another there may be situations where you would like to have a subset of your nodes to have a different configuration. For instance, perhaps some of your applications require the use of GPU hardware. However, it would be unnecessarily expensive if you were to attach GPUs to all nodes in your cluster.

In that case you can create a separate pool of nodes that have GPUs attached and schedule pods to use those GPU enabled nodes. Google Compute Engine allows you to attach up to 8 GPUs per node, assuming you have quota and the GPU device type is available in the zone you are using.

Let's first create a node pool where each node has one GPU attached:

$ gcloud beta container node-pools create nvidia-tesla-k80-pool --cluster gke-workshop --machine-type n1-standard-1 --num-nodes 1 --accelerator type=nvidia-tesla-k80,count=1

Machines with GPUs have certain limitations which may affect your workflow.
Learn more at https://cloud.google.com/kubernetes-engine/docs/concepts/gpus

Creating node pool nvidia-tesla-k80-pool...done.                                                       
Created [https://container.googleapis.com/v1beta1/projects/codelab/zones/europe-west1-d/clusters/gke-workshop/nodePools/nvidia-tesla-k80-pool].
NAME                    MACHINE_TYPE   DISK_SIZE_GB  NODE_VERSION
nvidia-tesla-k80-pool  n1-standard-1  100           1.9.6-gke.0

Next, we must install the NVIDIA drivers.

$ kubectl create -f https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/k8s-1.9/nvidia-driver-installer/cos/daemonset-preloaded.yaml

daemonset "nvidia-driver-installer" created

Verify that the drivers are installed using the following command. Run the following commands to watch the status of the pod. When the pods are listed as 'Running", you will know that the drivers are finished installing. When you are done hit Ctrl-C to exit:

$ watch "kubectl get pods -n kube-system | grep nvidia-driver-installer"

nvidia-driver-installer-98ksb                                    1/1       Running   0          10m

You can verify that the new node has GPUs that are allocatable to pods with the following command:

$ kubectl get nodes -l cloud.google.com/gke-nodepool=nvidia-tesla-k80-pool -o yaml | grep allocatable -A4

    allocatable:
      cpu: 940m
      memory: 2708864Ki
      nvidia.com/gpu: "1"
      pods: "110"

Now let's create a pods that can consume GPUs:

$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: cuda-vector-add
spec:
  restartPolicy: OnFailure
  containers:
    - name: cuda-vector-add
      image: "k8s.gcr.io/cuda-vector-add:v0.1"
      resources:
        limits:
          nvidia.com/gpu: 1 # requesting 1 GPU
EOF

The pod should complete fairly quickly. You can verify that it was completed by watching the pod with the following command. Here we can see that the pod was run on one of the nodes with a GPU. When the pod enters 'Completed' status you can hit Ctrl-C to exit:

$ watch kubectl get pods --show-all -o wide

NAME              READY     STATUS      RESTARTS   AGE       IP          NODE
cuda-vector-add   0/1       Completed   0          10m       10.56.8.3   gke-gke-workshop-nvidia-tesla-k80-po-b310269d-gbs1

Verify the logs for the pod:

$ kubectl logs cuda-vector-add

[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done

Finally we will delete the GPU node pool:

$ gcloud container node-pools delete nvidia-tesla-k80-pool --cluster=gke-workshop

The following node pool will be deleted.
[nvidia-tesla-k80-pool] in cluster [gke-workshop] in [europe-west1-d]

Do you want to continue (Y/n)? Y

Deleting node pool nvidia-tesla-k80-pool...done.                                                                                                                                                                 
Deleted [https://container.googleapis.com/v1/projects/codelab/zones/europe-west1-d/clusters/gke-workshop/nodePools/nvidia-tesla-k80-pool].

Congratulations — you are now a master at node pools!

What we've covered

Google Kubernetes Engine comes with built-in integration with Stackdriver Logging & Monitoring. When you create a cluster these features are enabled by default.

When Stackdriver Logging support is enabled, all logs from containers in your cluster are exported to Stackdriver Logging. You can then view the logs using the Stackdriver Logging UI and perform powerful queries on your logging output. Kubernetes Engine also includes support for viewing logs emitted by the kubelet on each node in the cluster as well as audit logs from the API master.

Stackdriver Monitoring allows you to view information, such as the memory and CPU usage, of individual pods running in your cluster.

Examining Container Logs

First let's create a workload in the cluster to generate some logs. We will create an nginx Deployment and corresponding Service in order to emit some logs.

$ kubectl run nginx --image=nginx --expose --port=80 --service-overrides='{"spec":{"type":"LoadBalancer"}}'

service "nginx" created
deployment "nginx" created

Next check for the public IP of our nginx Service. When the IP address is returned you can hit Ctrl-C to exit.

$ watch kubectl get svc nginx -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

123.123.123.123

In your terminal run the following to generate load on the nginx Pod. Enter the IP address of your service instead of the IP address below.

$ while true; do curl -s http://123.123.123.123/ > /dev/null; done

This will run in a never ending loop sending requests to our nginx Pod. You should now be able to view the logs for the Pod. View the Stackdriver Logs by selecting Logging > Logs from the menu in the Google Cloud Platform console. You should see a page like the following:

From here select the first dropdown filter and select GKE Container > gke-workshop > default. This will show all logs from the 'default' Kubernetes namespace for our 'gke-workshop' cluster. Here we should see the logs generated by our nginx instance. If we had other Pods running in the cluster we would see those logs here as well. You can further limit which containers to show logs from by using the filter that says 'All logs'.

Viewing Metrics in Stackdriver Monitoring

Next, let's view the metrics for our pod in Stackdriver Monitoring. Navigate to Monitoring in the menu for the Google Cloud Platform Console. A new tab should be opened for Stackdriver Monitoring. Within Stackdriver Monitoring navigate to Resources > Kubernetes Engine. You should see a list of Pods. One of the pods should look like nginx-XXXXXXXXX-YYYYY. Click on that pod in the list.

This page will show you CPU, Memory, and Disk usage for the Pod. This data is aggregated for all containers in the Pod. If you have multiple containers in a Pod you can drill down on individual containers by clicking on them in the 'Containers' list.

You can also view these same metrics via the Workloads page. Navigate to Kubernetes Engine and then Workloads. Drill down on a workload by clicking on it in the cloud console.

When you are finished, stop sending requests to nginx by hitting Ctrl-C in the terminal to break the loop. Then you can delete the nginx Deployment.

$ kubectl delete deployment,service nginx

deployment "nginx" deleted
service "nginx" deleted

What we've covered

Kubernetes allows autoscaling with custom metrics. Kubernetes collects and exports information about the amount of CPU your application uses but sometimes this is not the right metric for autoscaling. Custom metrics allow you to autoscale applications any type of metric that your application can export. In this section we will export a metric to Stackdriver as a custom metric and use the custom metrics feature of HorizontalPodAutoscaler to autoscale our Pods.

First we will deploy the custom metric adapter for Stackdriver.

$ kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/adapter-beta.yaml
<...>

In order to use a Stackdriver custom metric as a metric for autoscaling, the metric must meet the following requirements.

Let's create our application. This application exports a static value to a custom metric. We will start out with a static value of 40.

$ cat <<EOF | kubectl apply -f -
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    run: custom-metric-sd
  name: custom-metric-sd
  namespace: default
spec:
  selector:
    matchLabels:
      run: custom-metric-sd
  template:
    metadata:
      labels:
        run: custom-metric-sd
    spec:
      containers:
      - command:
        - /bin/sh
        - -c
        - ./direct-to-sd --metric-name=foo --metric-value=40 --pod-id=\$(POD_ID)
        image: gcr.io/google-samples/sd-dummy-exporter:latest
        name: sd-dummy-exporter
        resources:
          requests:
            cpu: 100m
        env:
          - name: POD_ID
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.uid
EOF

deployment "custom-metric-sd" created

This application will start with a single Pod replica.

$ kubectl get pods | grep custom-metric-sd

custom-metric-sd-7b9b98f96d-xw2wp   1/1       Running   0          1m

Autoscale the Deployment

Next we will create a HorizontalPodAutoscaler to autoscale our Deployment. The HorizontalPodAutoscaler has a target average value of 20 per Pod.

$ cat <<EOF | kubectl apply -f -
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: custom-metric-sd
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1beta1
    kind: Deployment
    name: custom-metric-sd
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: Pods
    pods:
      metricName: foo
      targetAverageValue: 20

EOF

horizontalpodautoscaler "custom-metric-sd" created

Since the target average value of the autoscaler is 20 and our application is exporting a value of 40 we should quickly see that our application will begin scaling up from the initial single replica. Continue watching for a couple minutes to see the deployment scale up. HorizontalPodAutoscaler has a default upscale delay of 3 minutes. You can go on to the next step while continuing to watch the pods.

$ watch 'kubectl get pods | grep custom-metric-sd'

custom-metric-sd-7b9b98f96d-mvs8j   1/1       Running   0          1m
custom-metric-sd-7b9b98f96d-xw2wp   1/1       Running   0          6m

Viewing the Custom Metric in Stackdriver Monitoring

You can view the exported metric in Stackdriver. Navigate to the Monitoring menu item in the Cloud Console. From there navigate to Resources > Metrics Explorer. In the Metrics Explorer search for 'custom/foo'. You can then view the currently recorded values for the custom metric.

Viewing the results of Auto-Scaling

Coming back to our watch command, if it has been long enough we should see that the deployment has scaled up some more.

$ watch 'kubectl get pods | grep custom-metric-sd'

custom-metric-sd-7b9b98f96d-8c6dp   1/1       Running   0          3m 
custom-metric-sd-7b9b98f96d-mb7bh   1/1       Running   0          3m 
custom-metric-sd-7b9b98f96d-mvs8j   1/1       Running   0          6m
custom-metric-sd-7b9b98f96d-xw2wp   1/1       Running   0          11m

Let's now update our Deployment to change the statically exported value to 20.

$ cat <<EOF | kubectl apply -f -
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    run: custom-metric-sd
  name: custom-metric-sd
  namespace: default
spec:
  selector:
    matchLabels:
      run: custom-metric-sd
  template:
    metadata:
      labels:
        run: custom-metric-sd
    spec:
      containers:
      - command:
        - /bin/sh
        - -c
        - ./direct-to-sd --metric-name=foo --metric-value=20 --pod-id=\$(POD_ID)
        image: gcr.io/google-samples/sd-dummy-exporter:latest
        name: sd-dummy-exporter
        resources:
          requests:
            cpu: 100m
        env:
          - name: POD_ID
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.uid
EOF

deployment "custom-metric-sd" configured

The application will do a rolling update to update the pods to export the new value. If we then watch the pods for a few minutes we should see that the number of pods has leveled out and stopped scaling. Be sure to wait at least 3 minutes to verify that the Deployment has stopped auto-scaling.

$ watch 'kubectl get pods | grep custom-metric-sd'

custom-metric-sd-65b65f9b86-58f4t   1/1       Running   0          5m
custom-metric-sd-65b65f9b86-5nbhf   1/1       Running   0          5m
custom-metric-sd-65b65f9b86-j44k9   1/1       Running   0          5m
custom-metric-sd-65b65f9b86-m6bp5   1/1       Running   0          5m

If we view the custom metric in Stackdriver Monitoring again we should see the new value being reflected in the graph.

Finally let's clean up the Deployment and custom metrics adapter.

$ kubectl delete horizontalpodautoscaler custom-metric-sd

horizontalpodautoscaler "custom-metric-sd" deleted

$ kubectl delete deployment custom-metric-sd

deployment "custom-metric-sd" deleted

$ kubectl delete namespace custom-metrics

namespace "custom-metrics" deleted

What we've covered

Google Kubernetes Engine includes the ability to scale the cluster based on scheduled workloads. Kubernetes Engine's cluster autoscaler automatically resizes clusters based on the demands of the workloads you want to run.

Let us simulate some additional load to our web application by increasing the number of replicas:

$ kubectl scale deployment hello-web --replicas=20
deployment "hello-web" scaled
Allocatable:
 cpu:     940m
 memory:  2709028Ki
 pods:    110

Even with our larger machine, this is more work than we have space in our cluster to handle:

$ kubectl get pods
NAME                        READY     STATUS    RESTARTS   AGE
hello-web-967542450-0xvdf   1/1       Running   0          1m
hello-web-967542450-3p6xf   0/1       Pending   0          1m
hello-web-967542450-5wnkc   0/1       Pending   0          1m
hello-web-967542450-7cwl6   0/1       Pending   0          1m
hello-web-967542450-9jq4b   1/1       Running   0          16m
hello-web-967542450-gbglq   0/1       Pending   0          1m
hello-web-967542450-gvmcg   1/1       Running   0          1m
hello-web-967542450-hd07p   1/1       Running   0          1m
hello-web-967542450-hgr0x   1/1       Running   0          1m
hello-web-967542450-hxkv8   1/1       Running   0          1m
hello-web-967542450-jztp8   0/1       Pending   0          1m
hello-web-967542450-k1p01   0/1       Pending   0          1m
hello-web-967542450-kplff   1/1       Running   0          1m
hello-web-967542450-nbcsg   1/1       Running   0          1m
hello-web-967542450-pks0s   1/1       Running   0          1m
hello-web-967542450-pwh42   1/1       Running   0          17m
hello-web-967542450-v44dl   1/1       Running   0          17m
hello-web-967542450-w7293   1/1       Running   0          1m
hello-web-967542450-wx1m5   0/1       Pending   0          1m
hello-web-967542450-x9595   1/1       Running   0          1m

We can see that there are many pods that are stuck with status of "Pending". This means that Kubernetes has not yet been able to schedule that pod to a node.

Copy the name of one of the pods marked Pending, and look at its events with kubectl describe. You should see a message like the following:

$ kubectl describe pod hello-web-967542450-wx1m5
<...>
Events:
  Message
  -------
  FailedScheduling        No nodes are available that match all of the following predicates:: Insufficient cpu (3).

The scheduler was unable to assign this pod to a node because there is not sufficient CPU space left in the cluster. We can add nodes to the cluster in order to make have enough resources for all of the pods in our Deployment.

Cluster Autoscaler can be enabled when creating a cluster, or you can enable it by updating an existing node pool. We will enable cluster autoscaler on our new node pool.

$ gcloud container clusters update gke-workshop --enable-autoscaling \
      --min-nodes=0 --max-nodes=5 --node-pool=new-pool

Once autoscaling is enabled, Kubernetes Engine will automatically add new nodes to your cluster if you have created new Pods that don't have enough capacity to run; conversely, if a node in your cluster is underutilized, Kubernetes Engine can scale down the cluster by deleting the node.

After the command above completes, we can see that the autoscaler has noticed that there are pods in Pending, and creates new nodes to give them somewhere to go. After a few minutes, you will see a new node has been created, and all the pods are now Running. Hit Ctrl-C when you are done:

$ watch kubectl get nodes,pods
NAME                                           STATUS    AGE       VERSION
gke-gke-workshop-new-pool-97a76573-l57x      Ready     1h        v1.9.6-gke.0
gke-gke-workshop-new-pool-97a76573-t2v0      Ready     1m        v1.9.6-gke.0

NAME                        READY     STATUS    RESTARTS   AGE
hello-web-967542450-0xvdf   1/1       Running   0          17m
hello-web-967542450-3p6xf   1/1       Running   0          17m
hello-web-967542450-5wnkc   1/1       Running   0          17m
hello-web-967542450-7cwl6   1/1       Running   0          17m
hello-web-967542450-9jq4b   1/1       Running   0          32m
hello-web-967542450-gbglq   1/1       Running   0          17m
hello-web-967542450-gvmcg   1/1       Running   0          17m
hello-web-967542450-hd07p   1/1       Running   0          17m
hello-web-967542450-hgr0x   1/1       Running   0          17m
hello-web-967542450-hxkv8   1/1       Running   0          17m
hello-web-967542450-jztp8   1/1       Running   0          17m
hello-web-967542450-k1p01   1/1       Running   0          17m
hello-web-967542450-kplff   1/1       Running   0          17m
hello-web-967542450-nbcsg   1/1       Running   0          17m
hello-web-967542450-pks0s   1/1       Running   0          17m
hello-web-967542450-pwh42   1/1       Running   0          33m
hello-web-967542450-v44dl   1/1       Running   0          33m
hello-web-967542450-w7293   1/1       Running   0          17m
hello-web-967542450-wx1m5   1/1       Running   0          17m
hello-web-967542450-x9595   1/1       Running   0          17m

Cluster Autoscaler will scale down as well as up. When you enabled the autoscaler, you set a minimum of one node. If you were to resize to one Pod, or delete the Deployment and wait about 10 minutes, you would see that all but one of your nodes are considered unnecessary, and removed.

What we've covered

Preemptible VMs are Google Compute Engine VM instances that last a maximum of 24 hours and provide no availability guarantees. Preemptible VMs are priced substantially lower than standard Compute Engine VMs and offer the same machine types and options.

If your workload can handle nodes disappearing, using Preemptible VMs with the Cluster Autoscaler lets you run work at a lower cost. To specify that you want to use Preemptible VMs you simply use the --preemptible flag when you create the node pool. But if you're using Preemptible VMs to cut costs, then you don't need them sitting around idle. So let's create a node pool of Preemptible VMs that starts with zero nodes, and autoscales as needed.

Hold on though: before we create it, how do we schedule work on the Preemptible VMs? These would be a special set of nodes for a special set of work - probably low priority or batch work. For that we'll use a combination of a NodeSelector and taints/tolerations. Preemptible nodes are currently in beta so we will use the beta version of the Kubernetes Engine API and the beta commands in gcloud. The full command we'll run is:

$ gcloud config set container/use_v1_api false

Updated property [container/use_v1_api].

$ gcloud beta container node-pools create preemptible-pool \
    --cluster gke-workshop --preemptible --num-nodes 0 \
    --enable-autoscaling --min-nodes 0 --max-nodes 5 \
    --node-taints=pod=preemptible:PreferNoSchedule

Creating node pool preemptible-pool...done.
Created [https://container.googleapis.com/v1/projects/codelab/zones/europe-west1-d/clusters/gke-workshop/nodePools/preemptible-pool].
NAME               MACHINE_TYPE   DISK_SIZE_GB  NODE_VERSION
preemptible-pool   n1-standard-1  100           1.9.6-gke.0

$ kubectl get nodes 

NAME                                                STATUS    ROLES     AGE       VERSION
gke-gke-workshop-new-pool-97a76573-l57x           Ready     <none>    1h        v1.9.6-gke.0
gke-gke-workshop-new-pool-97a76573-t2v0           Ready     <none>    1h        v1.9.6-gke.0

We now have two node pools, but the new "preemptible" pool is autoscaled and is sized to zero initially so we only see the three nodes from the autoscaled node pool that we created in the previous section.

Usually as far as Kubernetes is concerned, all nodes are valid places to schedule pods. We may prefer to reserve the preemptible pool for workloads that are explicitly marked as suiting preemption — workloads which can be replaced if they die, versus those that generally expect their nodes to be long-lived.

To direct the scheduler to schedule pods onto the nodes in the preemptible pool we must first mark the new nodes with a special label called a taint. This makes the scheduler avoid using it for certain Pods.


We can then mark pods that we want to run on the preemptible nodes with a matching toleration, which says they are OK to be assigned to nodes with that taint.

Let's create a new workload that's designed to run on preemptible nodes and nowhere else.

$ cat <<EOF | kubectl apply -f -
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    run: hello-web
  name: hello-preempt
spec:
  replicas: 20
  selector:
    matchLabels:
      run: hello-web
  template:
    metadata:
      labels:
        run: hello-web
    spec:
      containers:
      - image: gcr.io/google-samples/hello-app:1.0
        name: hello-web
        ports:
        - containerPort: 8080
          protocol: TCP
        resources:
          requests:
            cpu: "50m"
      tolerations:
      - key: pod
        operator: Equal
        value: preemptible
        effect: PreferNoSchedule
      nodeSelector:
        cloud.google.com/gke-preemptible: "true"
EOF

deployment "hello-preempt" created

$ watch kubectl get nodes,pods -o wide

NAME                                              STATUS    ROLES     AGE <...>      
gke-gke-workshop-new-pool-97a76573-l57x           Ready     <none>    1h <...>
gke-gke-workshop-new-pool-97a76573-t2v0           Ready     <none>    1h <...>
gke-gke-workshop-preemptible-pool-8f5d3454-2wmb   Ready     <none>    16m <...>
gke-gke-workshop-preemptible-pool-8f5d3454-9bcw   Ready     <none>    16m <...>
gke-gke-workshop-preemptible-pool-8f5d3454-rk9c   Ready     <none>    14m <...>

NAME                            READY     STATUS    RESTARTS   AGE       IP            NODE
hello-preempt-79cbc76b4-45msl   1/1       Running   0          6s        10.60.7.18    gke-gke-workshop-preemptible-pool-8f5d3454-vmpf
hello-preempt-79cbc76b4-78z7s   1/1       Running   0          6s        10.60.9.18    gke-gke-workshop-preemptible-pool-8f5d3454-2wmb
hello-preempt-79cbc76b4-8cjlr   1/1       Running   0          6s        10.60.8.21    gke-gke-workshop-preemptible-pool-8f5d3454-s5zv
hello-preempt-79cbc76b4-fdtmx   1/1       Running   0          6s        10.60.11.12   gke-gke-workshop-preemptible-pool-8f5d3454-rk9c
hello-preempt-79cbc76b4-fw6cg   1/1       Running   0          6s        10.60.7.21    gke-gke-workshop-preemptible-pool-8f5d3454-vmpf
hello-preempt-79cbc76b4-gqnvp   1/1       Running   0          6s        10.60.10.20   gke-gke-workshop-preemptible-pool-8f5d3454-9bcw
hello-preempt-79cbc76b4-hb72t   1/1       Running   0          6s        10.60.11.13   gke-gke-workshop-preemptible-pool-8f5d3454-rk9c
hello-preempt-79cbc76b4-kps8r   1/1       Running   0          6s        10.60.7.20    gke-gke-workshop-preemptible-pool-8f5d3454-vmpf
hello-preempt-79cbc76b4-mnfvv   1/1       Running   0          6s        10.60.10.22   gke-gke-workshop-preemptible-pool-8f5d3454-9bcw
hello-preempt-79cbc76b4-plxsj   1/1       Running   0          6s        10.60.8.22    gke-gke-workshop-preemptible-pool-8f5d3454-s5zv
hello-preempt-79cbc76b4-pxw2w   1/1       Running   0          6s        10.60.10.23   gke-gke-workshop-preemptible-pool-8f5d3454-9bcw
hello-preempt-79cbc76b4-sqcst   1/1       Running   0          6s        10.60.11.14   gke-gke-workshop-preemptible-pool-8f5d3454-rk9c
hello-preempt-79cbc76b4-tnmdt   1/1       Running   0          6s        10.60.7.19    gke-gke-workshop-preemptible-pool-8f5d3454-vmpf
hello-preempt-79cbc76b4-v4wjw   1/1       Running   0          6s        10.60.9.20    gke-gke-workshop-preemptible-pool-8f5d3454-2wmb
hello-preempt-79cbc76b4-vg976   1/1       Running   0          6s        10.60.11.11   gke-gke-workshop-preemptible-pool-8f5d3454-rk9c
hello-preempt-79cbc76b4-vkjv7   1/1       Running   0          6s        10.60.9.19    gke-gke-workshop-preemptible-pool-8f5d3454-2wmb
hello-preempt-79cbc76b4-w6jvc   1/1       Running   0          6s        10.60.8.23    gke-gke-workshop-preemptible-pool-8f5d3454-s5zv
hello-preempt-79cbc76b4-x5hcs   1/1       Running   0          6s        10.60.10.21   gke-gke-workshop-preemptible-pool-8f5d3454-9bcw
hello-preempt-79cbc76b4-x6v5t   1/1       Running   0          6s        10.60.8.20    gke-gke-workshop-preemptible-pool-8f5d3454-s5zv
hello-preempt-79cbc76b4-z5sxm   1/1       Running   0          6s        10.60.9.21    gke-gke-workshop-preemptible-pool-8f5d3454-2wmb

Due to the NodeSelector, initially there were no nodes on which we could schedule the work. The scheduler works in tandem with the Cluster Autoscaler to provision new nodes in the pool with the node labels that match the NodeSelector. We haven't demonstrated it here, but the taint would mean prefer to prevent workloads with pods that don't tolerate the taint from being scheduled on these nodes.

As we do the cleanup for this section, let's delete the preemptible node pool and see what happens to the pods that we just created. This isn't something you would want to do in production!

$ gcloud container node-pools delete preemptible-pool --cluster \
    gke-workshop

The following node pool will be deleted.
[preemptible-pool] in cluster [gke-workshop] in [europe-west1-d]
Do you want to continue (Y/n)?

Deleting node pool preemptible-pool...|

$ watch kubectl get pods 

NAME                            READY     STATUS    RESTARTS   AGE
hello-preempt-79cbc76b4-2292l   0/1       Pending   0          3m
hello-preempt-79cbc76b4-29njq   0/1       Pending   0          3m
hello-preempt-79cbc76b4-42vv6   0/1       Pending   0          3m
hello-preempt-79cbc76b4-47fgt   0/1       Pending   0          23m
hello-preempt-79cbc76b4-4d9wn   0/1       Pending   0          23m
hello-preempt-79cbc76b4-4kqsr   0/1       Pending   0          23m
hello-preempt-79cbc76b4-5hh6f   0/1       Pending   0          3m
...

As you can see, because of the NodeSelector, none of the pods are running. Now, delete the deployment.

$ kubectl delete deployment hello-preempt

deployment "hello-preempt" deleted

What we've covered

Node auto-repair

Kubernetes Engine's node auto-repair feature helps you keep the nodes in your cluster in a healthy, running state. When enabled, Kubernetes Engine makes periodic checks on the health state of each node in your cluster. If a node fails consecutive health checks over an extended time period (approximately 10 minutes), Kubernetes Engine initiates a repair process for that node.

$ gcloud container node-pools update new-pool --cluster gke-workshop --enable-autorepair

This will enable the autorepair feature for nodes. Please see
https://cloud.google.com/kubernetes-engine/docs/node-auto-repair for more
information on node autorepairs.

Updating node pool new-pool...done.
Updated [https://container.googleapis.com/v1beta1/projects/codelab/zones/europe-west1/clusters/gke-workshop/nodePools/new-pool].

Now, for some fun: let's break a VM!

This gcloud command will find the VM in your regional node pool which is in the default zone, and SSH into it. If you are asked to generate an SSH key just answer 'Y' at the prompt and hit enter to not set a passphrase.

$ gcloud compute ssh $(gcloud compute instances list | \
                       grep -m 1 gke-workshop-new | \
                       grep $(gcloud config get-value compute/zone) | \
                       awk '{ print $1 }')

You can simulate a node failure by removing the kubelet binary, which is responsible for running Pods on every Kubernetes node:

$ sudo rm /home/kubernetes/bin/kubelet && sudo systemctl restart kubelet
$ logout

Now when we check the node status we see the node is NotReady.

$ watch kubectl get nodes

NAME                                          STATUS     ROLES     AGE       VERSION
gke-gke-workshop-new-pool-42b33f8c-9grf   Ready      <none>    6m        v1.9.6-gke.1
gke-gke-workshop-new-pool-847e18a1-f1bp   Ready      <none>    6m        v1.9.6-gke.1
gke-gke-workshop-new-pool-8c4b26e9-fq8p   NotReady   <none>    6m        v1.9.6-gke.1

The Kubernetes Engine node repair agent will wait a few minutes in case the problem is intermittent. We'll come back to this in a minute.

Define a maintenance window

You can configure a maintenance window to have more control over when automatic upgrades are applied to Kubernetes on your cluster.

Creating a maintenance window instructs Kubernetes Engine to automatically trigger any automated tasks in your clusters, such as master upgrades, node pool upgrades, and maintenance of internal components, during a specific timeframe.

The times are specified in UTC, so select an appropriate time and set up a maintenance window for your cluster.

Open the cloud console and navigate to Kubernetes Engine. Click on the gke-workshop cluster and click the edit button at the top. Find the Maintenance Window option and select 3AM. Finally, click Save to update the cluster.

Enable node auto-upgrades

Whenever a new version of Kubernetes is released, Google upgrades your master to that version. You can then choose to upgrade your nodes to that version, bringing functionality and security updates to both the OS and the Kubernetes components.

Node Auto-Upgrades use the same update mechanism as manual node upgrades, but does the scheduled upgrades during your maintenance window.

Auto-upgrades are enabled per node pool.

$ gcloud container node-pools update new-pool --cluster gke-workshop --enable-autoupgrade

This will enable the autoupgrade feature for nodes. Please see
https://cloud.google.com/kubernetes-engine/docs/node-management for more
information on node autoupgrades.

Updating node pool new-pool...done.
Updated [https://container.googleapis.com/v1beta1/projects/codelab/zones/europe-west1/clusters/gke-workshop/nodePools/new-pool].

Check your node repair

How is that node repair coming?

After a few minutes, you will see that the master drained the node, and then removed it.

$ watch kubectl get nodes
NAME                                          STATUS                        ROLES     AGE       VERSION
gke-gke-workshop-new-pool-42b33f8c-9grf   Ready                         <none>    12m       v1.9.6-gke.1
gke-gke-workshop-new-pool-847e18a1-f1bp   Ready                         <none>    12m       v1.9.6-gke.1
gke-gke-workshop-new-pool-8c4b26e9-fq8p   NotReady,SchedulingDisabled   <none>    12m       v1.9.6-gke.1

A few minutes after that, a new node was turned on in its place:

NAME                                          STATUS    ROLES     AGE       VERSION
gke-gke-workshop-new-pool-42b33f8c-9grf   Ready     <none>    15m       v1.9.6-gke.1
gke-gke-workshop-new-pool-847e18a1-f1bp   Ready     <none>    15m       v1.9.6-gke.1
gke-gke-workshop-new-pool-8c4b26e9-fq8p   Ready     <none>    2m        v1.9.6-gke.1

Because our Deployments were managed by Kubernetes controllers, we were able to survive the downtime with no problems.

$ kubectl get pods

NAME                                 READY     STATUS    RESTARTS   AGE
hello-web-98975b8cd-4cdnn   1/1       Running   0          8s
hello-web-98975b8cd-8sls4   1/1       Running   0          8s
hello-web-98975b8cd-9bcrq   1/1       Running   0          7s
hello-web-98975b8cd-bdwpc   1/1       Running   0          8s
hello-web-98975b8cd-d2fjj   1/1       Running   0          8s
hello-web-98975b8cd-ssbgw   1/1       Running   0          8s
hello-web-98975b8cd-tdrhz   1/1       Running   0          7s
hello-web-98975b8cd-v47mj   1/1       Running   0          8s
hello-web-98975b8cd-vzlgj   1/1       Running   0          8s

Let's clean up the Deployment:

$ kubectl delete deployment hello-web

deployment "hello-web" deleted

What we've covered

Along with defining the applications you run in your Kubernetes environment, you can define policies about which pods are able to talk to each other. By default, there is no network policy in place, and all pods can communicate with all other pods in the same cluster.

Deploy a sample application

Start some pods to act as tiers of a hypothetical application:

$ kubectl run frontend \
  --image=gcr.io/google-samples/hello-app:1.0 \
  --labels=app=frontend \
  --port=8080

deployment "frontend" created

$ kubectl expose deployment frontend

service "frontend" exposed

$ kubectl run backend \
  --image=gcr.io/google-samples/hello-app:1.0 \
  --labels=app=backend \
  --port=8080

deployment "backend" created

$ kubectl expose deployment backend

service "backend" exposed

$ kubectl run untrusted \
  --image=gcr.io/google-samples/hello-app:1.0 \
  --labels=app=untrusted \
  --port=8080

deployment "untrusted" created

$ kubectl expose deployment untrusted

service "untrusted" exposed

In this example, pods with the label app=frontend should be able to connect to pods with the label app=backend. We also run some untrusted code, and that pod should not be able to connect to frontend or backend pods. (This simulates a situation where a malicious actor gets access to a container on your network through a vulnerability in its code.)

Verify connectivity

Using kubectl exec we can run a command on our untrusted pod:

$ UNTRUSTED_POD=$(kubectl get pods -l app=untrusted -o jsonpath='{.items[0].metadata.name}')
$ kubectl exec -it $UNTRUSTED_POD -- wget -qO- backend:8080

Hello, world!
Version: 1.0.0
Hostname: backend-b58bc5ff7-qdkz7

Create a network policy

We can now create a NetworkPolicy which defines that only traffic from "frontend" is allowed to access "backend".

$ cat <<EOF | kubectl create -f -
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: frontend-to-backend
spec:
  podSelector:
    matchLabels:
      app: backend
  ingress:
  - from:
      - podSelector:
          matchLabels:
            app: frontend
EOF

networkpolicy "frontend-to-backend" created

Verify your network policy

Now, test again:

$ kubectl exec -it $UNTRUSTED_POD -- wget -qO- frontend:8080

Hello, world!
Version: 1.0.0
Hostname: frontend-6b57bff885-lb4gn
$ kubectl exec -it $UNTRUSTED_POD -- wget -qO- --timeout=2 backend:8080
wget: download timed out
command terminated with exit code 1

You can connect to the frontend service — but not the backend service, as our policy prohibits it.

Can we connect from the frontend to the backend?

$ FRONTEND_POD=$(kubectl get pods -l app=frontend -o jsonpath='{.items[0].metadata.name}')

$ kubectl exec -it $FRONTEND_POD -- wget -qO- backend:8080

Hello, world!
Version: 1.0.0
Hostname: backend-b58bc5ff7-kr86j

You see our policy does exactly what was asked: the frontend pod can connect to the backend pod, but nothing else can.

If you delete the policy, connection with the untrusted pod is restored:

$ kubectl delete networkpolicy frontend-to-backend

networkpolicy "frontend-to-backend" deleted

$ kubectl exec -it $UNTRUSTED_POD -- wget -qO- backend:8080
Hello, world!
Version: 1.0.0
Hostname: backend-b58bc5ff7-kr86j

To clean up delete the deployments and services:

$ kubectl delete deployment,service frontend backend untrusted

deployment "frontend" deleted
deployment "backend" deleted
deployment "untrusted" deleted
service "frontend" deleted
service "backend" deleted
service "untrusted" deleted

What we've covered

To connect to services in Google Cloud Platform, you need to provide an identity. While you might use a user's identity if operating interactively, services running on Compute Engine instances normally use an IAM service account.

Applications running on Compute Engine can access and use the service account associated with the instance, and as Kubernetes Engine nodes are Compute Engine instances, containers can access the identities provided by the node.

However, granting that identity access to a service could grant unnecessary privileges because any container that runs on the node will have access to it.

A better practice is to create a service account for your own application, and provide that to your application using Kubernetes secrets.

We will use a sample application which reads messages posted to a Google Cloud Pub/Sub topic. Cloud Pub/Sub is a simple, reliable, scalable foundation for stream analytics and event-driven computing systems. It's a global service, which makes it great for exchanging data between clusters in different regions.

Create a Pub/Sub topic

The Pub/Sub subscriber application you will deploy uses a subscription named echo-read on a Pub/Sub topic called echo. Create these resources before deploying the application:

$ gcloud pubsub topics create echo
$ gcloud pubsub subscriptions create echo-read --topic=echo

Deploy an application

Our sample application reads messages that are published to a Pub/Sub topic. This application is written in Python using Google Cloud Pub/Sub client libraries and you can find the source code on GitHub.

$ kubectl create -f https://raw.githubusercontent.com/GoogleCloudPlatform/kubernetes-engine-samples/master/cloud-pubsub/deployment/pubsub.yaml

deployment "pubsub" created

Look at the pod:

$ kubectl get pods -l app=pubsub

NAME                     READY     STATUS    RESTARTS   AGE
pubsub-8dc9647d6-lzvvb   0/1       Pending   0          39s

You can see that the container is failing to start and went into a CrashLoopBackOff state. Inspect the logs from the Pod by running:

$ kubectl logs -l app=pubsub
...
google.gax.errors.RetryError: GaxError(Exception occurred in retry method
that was not classified as transient, caused by <_Rendezvous of RPC that
terminated with (StatusCode.PERMISSION_DENIED, Request had insufficient
authentication scopes.)>)

The stack trace and the error message indicates that the application does not have permissions to query the Cloud Pub/Sub service. This is because the "Compute Engine default service account" is not assigned any roles giving it permission to Cloud Pub/Sub.

$ PROJECT=$(gcloud config get-value project)
<...>
$ gcloud iam service-accounts create pubsub-sa --display-name "Pub/Sub demo service account"
<...>
$ gcloud projects add-iam-policy-binding $PROJECT \
    --member serviceAccount:pubsub-sa@$PROJECT.iam.gserviceaccount.com \
    --role roles/pubsub.subscriber
<...>
$ gcloud iam service-accounts keys create key.json \
    --iam-account pubsub-sa@$PROJECT.iam.gserviceaccount.com

Import secret into Kubernetes

Load in the Secret:

$ kubectl create secret generic pubsub-key --from-file=key.json=key.json

We now have a secret called pubsub-key which contains a file called key.json. You can update your Deployment to mount this Secret, and you can override the application to use this service account instead of the default identity of the node.

Configure the application with the Secret

This manifest file defines the following to make the credentials available to the application:

       volumeMounts:
        - name: google-cloud-key
          mountPath: /var/secrets/google
        env:
        - name: GOOGLE_APPLICATION_CREDENTIALS
          value: /var/secrets/google/key.json

Apply this configuration:

$ kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/kubernetes-engine-samples/master/cloud-pubsub/deployment/pubsub-with-secret.yaml

Your pod will now start:

$ kubectl get pods -l app=pubsub
NAME                     READY     STATUS    RESTARTS   AGE
pubsub-94476cd97-kzd9j   1/1       Running   0          1m

Test receiving Pub/Sub messages

Validate that your application is now able to read from Google Cloud Pub/Sub:

$ gcloud pubsub topics publish echo --message "Hello world"
NAME                     READY     STATUS    RESTARTS   AGE
pubsub-94476cd97-kzd9j   1/1       Running   0          1m
$ kubectl logs -l app=pubsub
Pulling messages from Pub/Sub subscription...
[2017-11-28 21:44:10.537970] ID=177003642109951 Data=b'Hello, world'

You have successfully configured an application on Kubernetes Engine to authenticate to Pub/Sub API using service account credentials!

Delete your subscriptions and your pod:

$ gcloud pubsub subscriptions delete echo-read
$ gcloud pubsub topics delete echo
$ kubectl delete pods -l app=pubsub

What we've covered

To publish a service to the world via HTTP, Kubernetes has an object called an Ingress. The Ingress object and its associated controller programs a load balancer, like the Google Cloud Platform HTTP Load Balancer.

Deploy an application


Let's redeploy our faithful hello-web app:

$ kubectl run hello-web --image=gcr.io/google-samples/hello-app:1.0 --port=8080 --replicas=3

deployment "hello-web" created

Reserve a static IP address

When you create an Ingress, you will be allocated a random IP address. It's good practice to instead reserve an IP address, and then allocate that to your Ingress, so that you don't need to change DNS if you ever need to delete and recreate your load balancer.

Google Cloud Platform has two types of IP addresses - regional (for Network Load Balancers, as used by Services) and global (for HTTP Load Balancers).

Reserve a static external IP address named nginx-static-ip by running:

$ gcloud compute addresses create hello-web-static-ip --global

$ gcloud compute addresses list --filter="name=hello-web-static-ip"
NAME                 REGION  ADDRESS       STATUS
hello-web-static-ip          35.227.247.6  RESERVED

This is the value you would program into DNS.

Create a Service

We now need to tell Kubernetes that these pods comprise a Service, which can be used as a target for our traffic.

In many other examples you might have used --type=LoadBalancer. That creates a Network Load Balancer, which exists in one region. By using a Service of type NodePort, we are instead exposing the service to the VM IPs, and we can then direct an Ingress object, which corresponds to a global HTTP(s) Load Balancer, to the NodePort service on those VMs.

$ kubectl expose deployment hello-web --target-port=8080 --type=NodePort

$ kubectl get service hello-web
NAME        TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
hello-web   NodePort   10.63.253.73   <none>        8080:32231/TCP   2m

In this example, you can see that any traffic from the Internet to port 32231 on any node will be routed to a healthy container on port 8080. There is, however, no "EXTERNAL-IP" allocated which would have been the case if a Service of type LoadBalancer would have been used.

Create an Ingress

In order to reach the service on a single IP, we will create an Ingress object wiring together the static IP and the service:

$ cat <<EOF | kubectl create -f -
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: hello-web-ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: hello-web-static-ip
spec:
  backend:
    serviceName: hello-web
    servicePort: 8080
EOF

After a couple of minutes, the HTTP Load Balancer will be created, using the IP you reserved previously, and mapped to the ports exposed by the NodePort service.

$ watch kubectl get ingress
NAME                HOSTS     ADDRESS        PORTS     AGE
hello-web-ingress   *         35.227.247.6   80        2m

You can now hit (for example) http://35.227.247.6/:

If you no longer need them, delete the Ingress and release the static IP:

$ kubectl delete ingress hello-web-ingress
$ gcloud compute addresses delete hello-web-static-ip --global

What we've covered

Kubernetes Engine clusters are zonal by default. That means that the Kubernetes master is located in a single zone. Masters located in this zone are fully managed and will be restarted when failures arise but are susceptible to zone wide failures. Zonal clusters can allow you to create smaller clusters in a single zone and control the exact number of nodes in the cluster at a granular level. Multi-zone clusters allow you to spread your cluster across multiple zones but the master is still resident is one zone.

Kubernetes Engine also provides Regional Clusters. In regional clusters the Kubernetes master is replicated across all zones in the region. In the event of a zone wide failure, other replicas can still respond to Kubernetes API traffic. Regional clusters also spread the cluster nodes across all zones in the region, not just those specified, to provide for better availability of applications running on the cluster as well.

Let's create a regional cluster:

$ gcloud beta container clusters create gke-workshop-regional \
       --enable-network-policy \
       --cluster-version=1.9 \
       --num-nodes 1 \
       --machine-type n1-standard-2 \
       --region europe-west1

Creating cluster gke-workshop-regional...done.
Created [https://container.googleapis.com/v1beta1/projects/codelab/zones/europe-west1/clusters/gke-workshop-regional].
To inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/europe-west1/gke-workshop-regional?project=codelab
kubeconfig entry generated for gke-workshop-regional.
NAME                   LOCATION      MASTER_VERSION  MASTER_IP       MACHINE_TYPE   NODE_VERSION  NUM_NODES  STATUS
gke-workshop-regional  europe-west1  1.9.6-gke.1     35.205.125.117  n1-standard-2  1.9.6-gke.1   3          RUNNING

We can access the new cluster in the same way as zonal clusters.

$ kubectl get nodes

NAME                                                  STATUS    ROLES     AGE       VERSION
gke-gke-workshop-regiona-default-pool-28c69f2a-7dxm   Ready     <none>    1m        v1.9.6-gke.1
gke-gke-workshop-regiona-default-pool-9130a11a-r6zn   Ready     <none>    1m        v1.9.6-gke.1
gke-gke-workshop-regiona-default-pool-f7c489c5-x8vt   Ready     <none>    1m        v1.9.6-gke.1

We can also verify the each node's zone:

$ gcloud compute instances list | grep gke-workshop-reg

gke-gke-workshop-regiona-default-pool-9130a11a-r6zn  europe-west1-b  n1-standard-2               10.132.0.7   35.195.234.17  RUNNING
gke-gke-workshop-regiona-default-pool-28c69f2a-7dxm  europe-west1-c  n1-standard-2               10.132.0.6   35.195.69.111  RUNNING
gke-gke-workshop-regiona-default-pool-f7c489c5-x8vt  europe-west1-d  n1-standard-2               10.132.0.5   35.195.211.28  RUNNING
\

Scheduling a distributed workload

In much the same way as multi-zone clusters. You can create workloads that are distributed among each zone for higher tolerance to failure.

$ cat <<EOF | kubectl create -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-web-regional
  labels:
    app: hello-web
spec:
  replicas: 9
  selector:
    matchLabels:
      app: hello-web
  template:
    metadata:
      labels:
        app: hello-web
    spec:
      containers:
      - name: hello-web
        image: gcr.io/google-samples/hello-app:1.0
        ports:
        - containerPort: 8080
        resources:
          requests:
            cpu: "50m"
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - hello-web
                topologyKey: failure-domain.beta.kubernetes.io/zone
EOF

deployment "hello-web-regional" created

$ kubectl get pods -o wide

NAME                                 READY     STATUS    RESTARTS   AGE       IP           NODE
hello-web-regional-98975b8cd-5r2nq   1/1       Running   0          13s       10.48.1.8    gke-gke-workshop-regiona-default-pool-28c69f2a-7dxm
hello-web-regional-98975b8cd-5wfhg   1/1       Running   0          13s       10.48.2.5    gke-gke-workshop-regiona-default-pool-9130a11a-r6zn
hello-web-regional-98975b8cd-9g4b5   1/1       Running   0          13s       10.48.0.11   gke-gke-workshop-regiona-default-pool-f7c489c5-x8vt
hello-web-regional-98975b8cd-cfvls   1/1       Running   0          13s       10.48.2.6    gke-gke-workshop-regiona-default-pool-9130a11a-r6zn
hello-web-regional-98975b8cd-mn48f   1/1       Running   0          13s       10.48.0.9    gke-gke-workshop-regiona-default-pool-f7c489c5-x8vt
hello-web-regional-98975b8cd-nm2mr   1/1       Running   0          13s       10.48.1.10   gke-gke-workshop-regiona-default-pool-28c69f2a-7dxm
hello-web-regional-98975b8cd-pdxvd   1/1       Running   0          13s       10.48.0.10   gke-gke-workshop-regiona-default-pool-f7c489c5-x8vt
hello-web-regional-98975b8cd-s5m28   1/1       Running   0          13s       10.48.2.7    gke-gke-workshop-regiona-default-pool-9130a11a-r6zn
hello-web-regional-98975b8cd-xz8zm   1/1       Running   0          13s       10.48.1.9    gke-gke-workshop-regiona-default-pool-28c69f2a-7dxm

Finally let's delete our regional cluster to clean up:

$ gcloud beta container clusters delete gke-workshop-regional --region europe-west1

The following clusters will be deleted.
 - [gke-workshop-regional] in [europe-west1]

Do you want to continue (Y/n)?  Y

Deleting cluster gke-workshop-regional...done.
Deleted [https://container.googleapis.com/v1beta1/projects/codelab/zones/europe-west1/clusters/gke-workshop-regional].

What we've covered

By default, each node in a Kubernetes Engine cluster has a public IP address and the master is accessible from the internet. Google Kubernetes Engine master API servers are protected by the same security authentication mechanisms as all Google Cloud Platform APIs and require authentication for operations. Public IP addresses for nodes also allow easy access to public internet services. However, some users may want to limit access to their master API server and nodes.

Google Kubernetes Engine allows you to configure authorized networks, to restrict access to the API server, and Private Clusters to isolate nodes from the internet with private IPs. Let's take a look at how to use them.

Configure Master Authorized Networks

Master authorized networks are network addresses or ranges that you can set to limit access to the master API server. They effectively serve as a firewall for your cluster's API server.

First let's enable master authorized networks for our cluster. This will enable master authorized networks without setting any authorized networks, completely removing access to our master from outside of Google Cloud IP addresses.

$ gcloud container clusters update gke-workshop --enable-master-authorized-networks

Updating gke-workshop...done.                                                                                                                                                                                     
Updated [https://container.googleapis.com/v1/projects/codelab/zones/europe-west1-d/clusters/gke-workshop].
To inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/europe-west1-d/gke-workshop?project=codelab

This does not affect your ability to communicate with your master via Cloud Shell but it will affect access from your laptop. Run the following to get your API master's IP address.

$ gcloud container clusters describe gke-workshop --format='value(endpoint)'

35.123.123.123

Then try the following from your laptop. Replace the IP address with the IP address of your master. You should see that the connection times out because the connection is firewalled.

$ curl -k --connect-timeout 5 https://[IP ADDRESS]/

curl: (28) Connection timed out after 5000 milliseconds

Create a Private Cluster

Public IP addresses for each node allow traffic to be routed to the internet from pods running on each node. This makes it easy to access public services on the internet. However, security minded users would like to lock down internet access to and from the cluster.

In order to give users more flexibility into how nodes in the cluster can be accessed, and what pods running on the cluster can access, Kubernetes Engine allows you to create Private Clusters. The nodes in private clusters do not have a public IP address and thus cannot be accessed from the internet at all. They also cannot send data to the internet without an internet gateway.

Let's create a private cluster. Regional clusters are currently in beta so we will use the beta version of the Kubernetes Engine API and the beta commands in gcloud:

$ gcloud beta container clusters create gke-private --private-cluster --master-ipv4-cidr 172.16.0.16/28 --enable-ip-alias --create-subnetwork "" --num-nodes 2 --enable-network-policy --cluster-version 1.9

Creating cluster gke-private...done.                                                                                                       
Created [https://container.googleapis.com/v1beta1/projects/codelab/zones/europe-west1-d/clusters/gke-private].
To inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/europe-west1-d/gke-private?project=codelab
kubeconfig entry generated for gke-private.
NAME         LOCATION       MASTER_VERSION  MASTER_IP      MACHINE_TYPE   NODE_VERSION  NUM_NODES  STATUS
gke-private  europe-west1-d  1.9.6-gke.0     35.224.230.57  n1-standard-1  1.9.6-gke.0   2          RUNNING

The private cluster is now created. gcloud has set up kubectl to authenticate with the private cluster but the cluster's API server will only accept connections from the primary range of your subnetwork, and the secondary range of your subnetwork that is used for pods.

Connecting to a Private Cluster

Let's try to connect to the cluster:

$ kubectl get pods

Unable to connect to the server: dial tcp 35.456.456.456:443: i/o timeout

This fails because private clusters firewall traffic to the master by default. In order to connect to the cluster you need to make use of the master authorized networks feature. Here we will enable master authorized networks and whitelist the IP address for our Cloud Shell instance, to allow access to the master:

$ gcloud container clusters update gke-private --enable-master-authorized-networks --master-authorized-networks $(curl ipinfo.io/ip)/32

Updating gke-private...done.                                                                                                                                                                                     
Updated [https://container.googleapis.com/v1/projects/codelab/zones/europe-west1-d/clusters/gke-private].
To inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/europe-west1-d/gke-private?project=codelab

Now we can access the API server using kubectl:

$ kubectl run hello-web --image=gcr.io/google-samples/hello-app:1.0 \
       --port=8080 --replicas=1

deployment "hello-web" created

$ kubectl get pods

NAME                         READY     STATUS    RESTARTS   AGE
hello-web-59d96f7bd6-ds769   1/1       Running   0          10s

$ kubectl delete deployment hello-web

deployment "hello-web" deleted

Testing Outbound Traffic

Most outbound traffic is not routable in private clusters so access to the internet is limited. This isolates pods that are running sensitive workloads.

$ cat <<EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
  name: wget
spec:
  containers:
  - name: wget
    image: alpine
    command: ['wget', '-T', '5', 'http://www.example.com/']
  restartPolicy: Never
EOF

pod "wget" created

$ kubectl logs wget

Connecting to www.example.com (93.184.216.34:80)
wget: download timed out

We can see here that we were not able to connect to IP addresses on the internet.

Finally let's delete our private cluster to clean up:

$ gcloud container clusters delete gke-private

The following clusters will be deleted.
 - [gke-private] in [europe-west1-d]
Do you want to continue (Y/n)?  Y
Deleting cluster gke-private...done.                                                                                                                                                                                            
Deleted [https://container.googleapis.com/v1/projects/codelab/zones/europe-west1-d/clusters/gke-private].

What we've covered

We will now delete the resources you have created (to stop the charging and to be a good Cloud citizen):

$ gcloud container clusters delete gke-workshop

The following clusters will be deleted.
 - [gke-workshop] in [europe-west1-d]
Do you want to continue (Y/n)?  Y
Deleting cluster gke-workshop...done.                                                                                                                                                                                            
Deleted [https://container.googleapis.com/v1/projects/codelab/zones/europe-west1-d/clusters/gke-workshop].

You could instead delete the entire Google Cloud Platform project but you would also lose any billing setup you have done (disabling project billing first is required before deleting a project). Additionally, deleting a project will only stop all billing after the current billing cycle ends.

Enjoy your new-found GKE skills!