Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. Developed by Google and released as open-source, Kubernetes is now maintained by a diverse community and shepherded by the Cloud Native Computing Foundation.

The easiest way to run a Kubernetes cluster is using Google Kubernetes Engine, a managed version of Kubernetes hosted on Google Cloud Platform. Kubernetes Engine, also known as GKE, is a managed environment for deploying containerized applications. It brings our latest innovations in developer productivity, resource efficiency, automated operations, and open source flexibility to accelerate your time to market.

This codelab shows you some of the advanced features of Google Kubernetes Engine, and will show you how to run a service which makes the most of Google Cloud Platform's features. It assumes you have basic familiarity with Docker containers and Kubernetes concepts.

Self-paced environment setup

If you don't already have a Google Account (Gmail or G Suite), you must create one. Sign-in to Google Cloud Platform console (console.cloud.google.com) and create a new project from the Manage resources page :

Remember the project ID, a unique name across all Google Cloud projects (the name above has already been taken and will not work for you, sorry!). It will be referred to later in this codelab as PROJECT_ID.

Next, you'll need to enable billing in the Developers Console in order to use Google Cloud resources.

Running through this codelab shouldn't cost you more than a few dollars, but it could be more if you decide to use more resources or if you leave them running (see "cleanup" section at the end of this document). The Google Cloud Platform pricing calculator is available here.

New users of Google Cloud Platform are eligible for a $300 free trial.

Google Cloud Shell

While Google Kubernetes Engine can be operated remotely from your laptop, in this codelab we will be using Google Cloud Shell, a command line environment running in the Cloud.

This Debian-based virtual machine is loaded with all the development tools you'll need. It offers a persistent 5GB home directory, and runs on the Google Cloud, greatly enhancing network performance and authentication. This means that all you will need for this codelab is a browser (yes, it works on a Chromebook).

To activate Google Cloud Shell, from the developer console simply click the button on the top right-hand side (it should only take a few moments to provision and connect to the environment):

Then accept the terms of service and click the "Start Cloud Shell" link:

Once connected to the cloud shell, you should see that you are already authenticated and that the project is already set to your PROJECT_ID :

gcloud auth list

Command output

Credentialed accounts:
 - <myaccount>@<mydomain>.com (active)
gcloud config list project

Command output

[core]
project = <PROJECT_ID>

If for some reason the project is not set, simply issue the following command :

gcloud config set project <PROJECT_ID>

Looking for your PROJECT_ID? Check out what ID you used in the setup steps or look it up in the console dashboard:

IMPORTANT: Finally, set the default zone and project configuration:

gcloud config set compute/zone us-central1-f

You can choose a variety of different zones. Learn more in the Regions & Zones documentation.

First, lets create a Kubernetes cluster. You will use this cluster for all the upcoming exercises.

$ gcloud container clusters create gke-highlights \
       --enable-network-policy \
       --cluster-version=1.8.3-gke.0

Creating cluster gke-highlights...done.                                                                                                                                                                     
Created [https://container.googleapis.com/v1/projects/codelab/zones/us-central1-f/clusters/gke-highlights].
kubeconfig entry generated for gke-highlights.
NAME            ZONE           MASTER_VERSION  MASTER_IP       MACHINE_TYPE   NODE_VERSION  NUM_NODES  STATUS
gke-highlights  us-central1-f  1.8.3-gke.0     35.193.171.226  n1-standard-1  1.8.3-gke.0   3          RUNNING

When you create a cluster, gcloud adds a context to your kubectl configuration file (~/.kube/config). It then sets it as the current context, to let you operate on this cluster immediately.

$ kubectl config current-context

gke_codelab_us-central1-f_gke-highlights

To test it, try a kubectl command line:

$ kubectl get nodes

NAME                                            STATUS    ROLES     AGE       VERSION
gke-gke-highlights-default-pool-1acc373c-1txb   Ready     <none>    6m        v1.8.3-gke.0
gke-gke-highlights-default-pool-1acc373c-cklc   Ready     <none>    6m        v1.8.3-gke.0
gke-gke-highlights-default-pool-1acc373c-kzjv   Ready     <none>    6m        v1.8.3-gke.0

If you navigate to "Kubernetes Engine" in the Google Cloud Platform console, you will see the cluster listed:

Run a deployment

Most Kubernetes tutorials start by having you run a handful of containers, and this one is no different.

$ kubectl run hello-web --image=gcr.io/google-samples/hello-app:1.0 \
       --port=8080 --replicas=3

deployment "hello-web" created

$ kubectl get pods -o wide

NAME                         READY     STATUS    RESTARTS   AGE       IP          NODE
hello-web-5d9cdb689c-5qh2h   1/1       Running   0          2s        10.60.2.6   gke-gke-highlights-default-pool-1acc373c-1txb
hello-web-5d9cdb689c-sp2bj   1/1       Running   0          2s        10.60.1.9   gke-gke-highlights-default-pool-1acc373c-cklc
hello-web-5d9cdb689c-xfcm5   1/1       Running   0          2s        10.60.1.8   gke-gke-highlights-default-pool-1acc373c-cklc

In this example, there are three replicas, with two running on the same node. You may have slightly different results.

Look at the Google Cloud dashboard

Workloads that are deployed to your Kubernetes Engine cluster are displayed in the Google Cloud Console. Navigate to Kubernetes Engine and then Workloads.

You can see the hello-web deployment we just created. Click on it, and explore the user interface.

What we've covered

A Kubernetes Engine cluster consists of a master and nodes. A node pool is a subset of node instances within a cluster that all have the same configuration. By default a Kubernetes Engine cluster has a single node pool, but you can add or remove them as you wish to change the shape of your cluster.

In the previous example, you created a cluster with autoscaling enabled, this gave us three nodes (three n1-standard-1 VMs, 100 GB of disk each). This created a single node pool (called default-pool), which we can inspect:

$ gcloud container node-pools list --cluster gke-highlights
NAME          MACHINE_TYPE   DISK_SIZE_GB  NODE_VERSION
default-pool  n1-standard-1  100           1.8.3-gke.0

If you want to add more nodes of this type, you can grow this node pool. If you want to add more nodes of a different type, you can add other node pools.

A common method of moving a cluster to larger nodes is to add a new node pool, move the work from the old nodes to the new, and delete the old node pool.

Let's add a second node pool, and migrate our workload over to it.

$ gcloud container node-pools create new-pool --cluster gke-highlights \
    --machine-type n1-standard-2 --num-nodes 1
Creating node pool new-pool...done.                                                                                                                                   
Created [https://container.googleapis.com/v1/projects/codelab/zones/europe-west1-c/clusters/gke-highlights/nodePools/new-pool].
NAME         MACHINE_TYPE   DISK_SIZE_GB  NODE_VERSION
new-pool     n1-standard-1  100           1.8.3-gke.0

$ gcloud container node-pools list --cluster gke-highlights
NAME          MACHINE_TYPE   DISK_SIZE_GB  NODE_VERSION
default-pool  n1-standard-1  100           1.8.3-gke.0
new-pool      n1-standard-2  100           1.8.3-gke.0

$ kubectl get nodes
NAME                                            STATUS    ROLES     AGE       VERSION
gke-gke-highlights-default-pool-1acc373c-1txb   Ready     <none>    56m       v1.8.3-gke.0
gke-gke-highlights-default-pool-1acc373c-cklc   Ready     <none>    56m       v1.8.3-gke.0
gke-gke-highlights-default-pool-1acc373c-kzjv   Ready     <none>    57m       v1.8.3-gke.0
gke-gke-highlights-new-pool-97a76573-l57x       Ready     <none>    1m        v1.8.3-gke.0

Kubernetes does not reschedule Pods as long as they are running and available, so your workload remains running on the nodes in the default pool.

Look at one of your nodes using kubectl describe. Just like you can attach labels to pods, nodes are automatically labelled with useful information which lets the scheduler make decisions and the administrator perform action on groups of nodes.

Replace "gke-gke-highlights-default-pool-1acc373c-1txb" with the name of one of your nodes from the previous step.

$ kubectl describe node gke-gke-highlights-default-pool-1acc373c-1txb
Name:               gke-gke-highlights-default-pool-1acc373c-1txb
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/fluentd-ds-ready=true
                    beta.kubernetes.io/instance-type=n1-standard-1
                    beta.kubernetes.io/masq-agent-ds-ready=true
                    beta.kubernetes.io/os=linux
                    cloud.google.com/gke-nodepool=default-pool
                    failure-domain.beta.kubernetes.io/region=us-central1
                    failure-domain.beta.kubernetes.io/zone=us-central1-f
                    kubernetes.io/hostname=gke-gke-highlights-default-pool-1acc373c-1txb
                    projectcalico.org/ds-ready=true
Annotations:        node.alpha.kubernetes.io/ttl=0
                    volumes.kubernetes.io/controller-managed-attach-detach=true

You can also select nodes by node pool using the cloud.google.com/gke-nodepool label. We'll use this powerful construct shortly.

$ kubectl get nodes -l cloud.google.com/gke-nodepool=default-pool
NAME                                            STATUS    ROLES     AGE       VERSION
gke-gke-highlights-default-pool-1acc373c-1txb   Ready     <none>    54m       v1.8.3-gke.0
gke-gke-highlights-default-pool-1acc373c-cklc   Ready     <none>    54m       v1.8.3-gke.0
gke-gke-highlights-default-pool-1acc373c-kzjv   Ready     <none>    54m       v1.8.3-gke.0

Migrating pods to the new Node Pool

To migrate your pods to the new node pool, you must perform the following steps:

  1. Cordon the existing node pool: This operation marks the nodes in the existing node pool (default-pool) as unscheduleable. Kubernetes stops scheduling new Pods to these nodes once you mark them as unscheduleable.
  2. Drain the existing node pool: This operation evicts the workloads running on the nodes of the existing node pool (default-pool) gracefully.

You could cordon an individual node using the kubectl cordon command, but running this command on each node individually would be tedious. To speed up the process, we can embed the command in a loop. Be sure you copy the whole line - it will have scrolled off the screen to the right!

$ for node in $(kubectl get nodes -l cloud.google.com/gke-nodepool=default-pool -o=name); do kubectl cordon "$node"; done
node "gke-gke-highlights-default-pool-1acc373c-1txb" cordoned
node "gke-gke-highlights-default-pool-1acc373c-cklc" cordoned
node "gke-gke-highlights-default-pool-1acc373c-kzjv" cordoned

This loop utilizes the command kubectl get nodes to select all nodes in the default pool (using the cloud.google.com/gke-nodepool=default-pool label), and then it iterates through and runs kubectl drain on each one

After running the loop, you should see that the default-pool nodes have SchedulingDisabled status in the node list:

$ kubectl get nodes
NAME                                            STATUS                     ROLES     AGE       VERSION
gke-gke-highlights-default-pool-1acc373c-1txb   Ready,SchedulingDisabled   <none>    1h        v1.8.3-gke.0
gke-gke-highlights-default-pool-1acc373c-cklc   Ready,SchedulingDisabled   <none>    1h        v1.8.3-gke.0
gke-gke-highlights-default-pool-1acc373c-kzjv   Ready,SchedulingDisabled   <none>    1h        v1.8.3-gke.0
gke-gke-highlights-new-pool-97a76573-l57x       Ready                      <none>    10m       v1.8.3-gke.0

Next, we want to evict the Pods already scheduled on each node. To do this, we will construct another loop, this time using the kubectl drain command:

$ for node in $(kubectl get nodes -l cloud.google.com/gke-nodepool=default-pool -o=name); do kubectl drain --force --ignore-daemonsets "$node"; done

As each node is drained, the pods running on it are evicted. Because the default node pool is unschedulable, the pods are now running on the new node pool:

$ kubectl get pods -o wide
NAME                         READY     STATUS    RESTARTS   AGE       IP          NODE
hello-web-5d9cdb689c-54vnv   1/1       Running   0          2m        10.60.6.5   gke-gke-highlights-new-pool-97a76573-l57x
hello-web-5d9cdb689c-pn25c   1/1       Running   0          2m        10.60.6.8   gke-gke-highlights-new-pool-97a76573-l57x
hello-web-5d9cdb689c-s6g6r   1/1       Running   0          2m        10.60.6.6   gke-gke-highlights-new-pool-97a76573-l57x

You can now delete the original node pool:

$ gcloud container node-pools delete default-pool --cluster gke-highlights
The following node pool will be deleted.
[default-pool] in cluster [gke-highlights] in [us-central1-f]

Do you want to continue (Y/n)?  y

Deleting node pool default-pool...done.                                                                                                                                  
Deleted [https://container.googleapis.com/v1/projects/codelab/zones/us-central1-f/clusters/gke-highlights/nodePools/default-pool].

What we've covered

Google Kubernetes Engine includes the ability to scale the cluster based on scheduled workloads.Kubernetes Engine's cluster autoscaler automatically resizes clusters based on the demands of the workloads you want to run.

They say you can't buy friends, but you can simulate popularity. Let us simulate some additional load to our web application by increasing the number of replicas:

$ kubectl scale deployment hello-web --replicas=20
deployment "hello-web" scaled
Allocatable:
 cpu:     940m
 memory:  2709028Ki
 pods:    110

Even with our larger machine, this is more work than we have space in our cluster to handle:

$ kubectl get pods
NAME                        READY     STATUS    RESTARTS   AGE
hello-web-967542450-0xvdf   1/1       Running   0          1m
hello-web-967542450-3p6xf   0/1       Pending   0          1m
hello-web-967542450-5wnkc   0/1       Pending   0          1m
hello-web-967542450-7cwl6   0/1       Pending   0          1m
hello-web-967542450-9jq4b   1/1       Running   0          16m
hello-web-967542450-gbglq   0/1       Pending   0          1m
hello-web-967542450-gvmcg   1/1       Running   0          1m
hello-web-967542450-hd07p   1/1       Running   0          1m
hello-web-967542450-hgr0x   1/1       Running   0          1m
hello-web-967542450-hxkv8   1/1       Running   0          1m
hello-web-967542450-jztp8   0/1       Pending   0          1m
hello-web-967542450-k1p01   0/1       Pending   0          1m
hello-web-967542450-kplff   1/1       Running   0          1m
hello-web-967542450-nbcsg   1/1       Running   0          1m
hello-web-967542450-pks0s   1/1       Running   0          1m
hello-web-967542450-pwh42   1/1       Running   0          17m
hello-web-967542450-v44dl   1/1       Running   0          17m
hello-web-967542450-w7293   1/1       Running   0          1m
hello-web-967542450-wx1m5   0/1       Pending   0          1m
hello-web-967542450-x9595   1/1       Running   0          1m

We can see that there are many pods that are stuck with status of "Pending". This means that Kubernetes has not yet been able to schedule that pod to a node.

Copy the name of one of the pods marked Pending, and look at its events with kubectl describe. You should see a message like the following:

$ kubectl describe pod hello-web-967542450-wx1m5
...
Events:
  Message
  -------
  FailedScheduling        No nodes are available that match all of the following predicates:: Insufficient cpu (3).

The scheduler was unable to assign this pod to a node because there is not sufficient CPU space left in the cluster. We can add nodes to the cluster in order to make have enough resources for all of the pods in our Deployment.

Cluster Autoscaler can be enabled when creating a cluster, or you can enable it by updating an existing node pool. We will enable cluster autoscaler on our new node pool.

$ gcloud container clusters update gke-highlights --enable-autoscaling \
      --min-nodes=0 --max-nodes=5 --node-pool=new-pool

Once autoscaling is enabled, Kubernetes Engine will automatically add new nodes to your cluster if you have created new Pods that don't have enough capacity to run; conversely, if a node in your cluster is underutilized, Kubernetes Engine can scale down the cluster by deleting the node.

After the command above completes, we can see that the autoscaler has noticed that there are pods in Pending, and creates new nodes to give them somewhere to go. After a few minutes, you will see a new node has been created, and all the pods are now Running:

$ kubectl get nodes
NAME                                           STATUS    AGE       VERSION
gke-gke-highlights-new-pool-97a76573-l57x      Ready     1h        v1.8.3-gke.0
gke-gke-highlights-new-pool-97a76573-t2v0      Ready     1m        v1.8.3-gke.0

$ kubectl get pods
NAME                        READY     STATUS    RESTARTS   AGE
hello-web-967542450-0xvdf   1/1       Running   0          17m
hello-web-967542450-3p6xf   1/1       Running   0          17m
hello-web-967542450-5wnkc   1/1       Running   0          17m
hello-web-967542450-7cwl6   1/1       Running   0          17m
hello-web-967542450-9jq4b   1/1       Running   0          32m
hello-web-967542450-gbglq   1/1       Running   0          17m
hello-web-967542450-gvmcg   1/1       Running   0          17m
hello-web-967542450-hd07p   1/1       Running   0          17m
hello-web-967542450-hgr0x   1/1       Running   0          17m
hello-web-967542450-hxkv8   1/1       Running   0          17m
hello-web-967542450-jztp8   1/1       Running   0          17m
hello-web-967542450-k1p01   1/1       Running   0          17m
hello-web-967542450-kplff   1/1       Running   0          17m
hello-web-967542450-nbcsg   1/1       Running   0          17m
hello-web-967542450-pks0s   1/1       Running   0          17m
hello-web-967542450-pwh42   1/1       Running   0          33m
hello-web-967542450-v44dl   1/1       Running   0          33m
hello-web-967542450-w7293   1/1       Running   0          17m
hello-web-967542450-wx1m5   1/1       Running   0          17m
hello-web-967542450-x9595   1/1       Running   0          17m

Cluster Autoscaler will scale down as well as up. When you enabled the autoscaler, you set a minimum of one node. If you were to resize to one Pod, or delete the Deployment and wait about 10 minutes, you would see that all but one of your nodes are considered unnecessary, and removed.

Remove your Deployment:

$ kubectl delete deployment hello-web

What we've covered

Preemptible VMs are Google Compute Engine VM instances that last a maximum of 24 hours and provide no availability guarantees. Preemptible VMs are priced substantially lower than standard Compute Engine VMs and offer the same machine types and options.

If your workload can handle nodes disappearing, using Preemptible VMs with the Cluster Autoscaler lets you run work at a lower cost. To specify that you want to use Preemptible VMs you simply use the --preemptible flag when you create the node pool. But if you're using Preemptible VMs to cut costs, then you don't need them sitting around idle. So let's create a node pool of Preemptible VMs that starts with zero nodes, and autoscales as needed.

Hold on though: before we create it, how do we schedule work on the preemptible VMs? These would be a special set of nodes for a special set of work - probably low priority or batch work. For that we'll use a combination of a NodeSelector and taints/tolerations. The full command we'll run is:

$ gcloud config set container/use_v1_api_client false

$ gcloud beta container node-pools create preemptible-pool \
    --cluster gke-highlights --preemptible --num-nodes 0 \
    --enable-autoscaling --min-nodes 0 --max-nodes 5 \
    --node-taints=pod=preemptible:PreferNoSchedule

Creating node pool preemptible-pool...done.
Created [https://container.googleapis.com/v1/projects/codelab/zones/us-central1-f/clusters/gke-highlights/nodePools/preemptible-pool].
NAME               MACHINE_TYPE   DISK_SIZE_GB  NODE_VERSION
preemptible-pool   n1-standard-1  100           1.8.3-gke.0

$ kubectl get nodes 

NAME                                                STATUS    ROLES     AGE       VERSION
gke-gke-highlights-new-pool-97a76573-l57x           Ready     <none>    1h        v1.8.3-gke.0
gke-gke-highlights-new-pool-97a76573-t2v0           Ready     <none>    1h        v1.8.3-gke.0

We now have two node pools, but the new "preemptible" pool is autoscaled and is sized to zero initially so we only see the three nodes from the autoscaled node pool that we created in the previous section.

Usually as far as Kubernetes is concerned, all nodes are valid places to schedule pods. We may prefer to reserve the preemptible pool for workloads that are explicitly marked as suiting preemption — workloads which can be replaced if they die, versus those that generally expect their nodes to be long-lived.

To direct the scheduler to schedule pods onto the nodes in the preemptible pool we first label the new nodes

We can mark the preemptible nodes with a taint, which makes the scheduler avoid using it for certain Pods.


We can then mark pods that we want to run on the preemptible nodes with a matching toleration, which says they are OK to be assigned to nodes with that taint.

Let's create a new workload that's designed to run on preemptible nodes and nowhere else.

$ cat <<EOF | kubectl create -f -
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    run: hello-web
  name: hello-preempt
spec:
  replicas: 20
  selector:
    matchLabels:
      run: hello-web
  template:
    metadata:
      labels:
        run: hello-web
    spec:
      containers:
      - image: gcr.io/google-samples/hello-app:1.0
        name: hello-web
        ports:
        - containerPort: 8080
          protocol: TCP
        resources:
          requests:
            cpu: "50m"
      tolerations:
      - key: pod
        operator: Equal
        value: preemptible
        effect: PreferNoSchedule
      nodeSelector:
        cloud.google.com/gke-preemptible: "true"
EOF

deployment "hello-preempt" created

$ kubectl get nodes

gke-gke-highlights-new-pool-97a76573-l57x           Ready     <none>    1h        v1.8.3-gke.0
gke-gke-highlights-new-pool-97a76573-t2v0           Ready     <none>    1h        v1.8.3-gke.0
gke-gke-highlights-preemptible-pool-8f5d3454-2wmb   Ready     <none>    16m       v1.8.3-gke.0
gke-gke-highlights-preemptible-pool-8f5d3454-9bcw   Ready     <none>    16m       v1.8.3-gke.0
gke-gke-highlights-preemptible-pool-8f5d3454-rk9c   Ready     <none>    14m       v1.8.3-gke.0

$ kubectl get pods -o wide
NAME                            READY     STATUS    RESTARTS   AGE       IP            NODE
hello-preempt-79cbc76b4-45msl   1/1       Running   0          6s        10.60.7.18    gke-gke-highlights-preemptible-pool-8f5d3454-vmpf
hello-preempt-79cbc76b4-78z7s   1/1       Running   0          6s        10.60.9.18    gke-gke-highlights-preemptible-pool-8f5d3454-2wmb
hello-preempt-79cbc76b4-8cjlr   1/1       Running   0          6s        10.60.8.21    gke-gke-highlights-preemptible-pool-8f5d3454-s5zv
hello-preempt-79cbc76b4-fdtmx   1/1       Running   0          6s        10.60.11.12   gke-gke-highlights-preemptible-pool-8f5d3454-rk9c
hello-preempt-79cbc76b4-fw6cg   1/1       Running   0          6s        10.60.7.21    gke-gke-highlights-preemptible-pool-8f5d3454-vmpf
hello-preempt-79cbc76b4-gqnvp   1/1       Running   0          6s        10.60.10.20   gke-gke-highlights-preemptible-pool-8f5d3454-9bcw
hello-preempt-79cbc76b4-hb72t   1/1       Running   0          6s        10.60.11.13   gke-gke-highlights-preemptible-pool-8f5d3454-rk9c
hello-preempt-79cbc76b4-kps8r   1/1       Running   0          6s        10.60.7.20    gke-gke-highlights-preemptible-pool-8f5d3454-vmpf
hello-preempt-79cbc76b4-mnfvv   1/1       Running   0          6s        10.60.10.22   gke-gke-highlights-preemptible-pool-8f5d3454-9bcw
hello-preempt-79cbc76b4-plxsj   1/1       Running   0          6s        10.60.8.22    gke-gke-highlights-preemptible-pool-8f5d3454-s5zv
hello-preempt-79cbc76b4-pxw2w   1/1       Running   0          6s        10.60.10.23   gke-gke-highlights-preemptible-pool-8f5d3454-9bcw
hello-preempt-79cbc76b4-sqcst   1/1       Running   0          6s        10.60.11.14   gke-gke-highlights-preemptible-pool-8f5d3454-rk9c
hello-preempt-79cbc76b4-tnmdt   1/1       Running   0          6s        10.60.7.19    gke-gke-highlights-preemptible-pool-8f5d3454-vmpf
hello-preempt-79cbc76b4-v4wjw   1/1       Running   0          6s        10.60.9.20    gke-gke-highlights-preemptible-pool-8f5d3454-2wmb
hello-preempt-79cbc76b4-vg976   1/1       Running   0          6s        10.60.11.11   gke-gke-highlights-preemptible-pool-8f5d3454-rk9c
hello-preempt-79cbc76b4-vkjv7   1/1       Running   0          6s        10.60.9.19    gke-gke-highlights-preemptible-pool-8f5d3454-2wmb
hello-preempt-79cbc76b4-w6jvc   1/1       Running   0          6s        10.60.8.23    gke-gke-highlights-preemptible-pool-8f5d3454-s5zv
hello-preempt-79cbc76b4-x5hcs   1/1       Running   0          6s        10.60.10.21   gke-gke-highlights-preemptible-pool-8f5d3454-9bcw
hello-preempt-79cbc76b4-x6v5t   1/1       Running   0          6s        10.60.8.20    gke-gke-highlights-preemptible-pool-8f5d3454-s5zv
hello-preempt-79cbc76b4-z5sxm   1/1       Running   0          6s        10.60.9.21    gke-gke-highlights-preemptible-pool-8f5d3454-2wmb

Because of the NodeSelector initially there were no nodes on which we could schedule the work. The scheduler works in tandem with the Cluster Autoscaler to provision new nodes in the pool with the node labels that match the NodeSelector. We haven't demonstrated it here but the taint would mean prefer to prevent workloads with pods that don't tolerate the taint from being scheduled on these nodes.

As we do the cleanup for this section, let's delete the preemptible node pool and see what happens to the pods that we just created. This isn't something you would want to in production!

$ gcloud container node-pools delete preemptible-pool --cluster \
    gke-highlights

The following node pool will be deleted.
[preemptible-pool] in cluster [gke-highlights] in [us-central1-f]
Do you want to continue (Y/n)?

Deleting node pool preemptible-pool...|

$ kubectl get pods 

NAME                            READY     STATUS    RESTARTS   AGE
hello-preempt-79cbc76b4-2292l   0/1       Pending   0          3m
hello-preempt-79cbc76b4-29njq   0/1       Pending   0          3m
hello-preempt-79cbc76b4-42vv6   0/1       Pending   0          3m
hello-preempt-79cbc76b4-47fgt   0/1       Pending   0          23m
hello-preempt-79cbc76b4-4d9wn   0/1       Pending   0          23m
hello-preempt-79cbc76b4-4kqsr   0/1       Pending   0          23m
hello-preempt-79cbc76b4-5hh6f   0/1       Pending   0          3m
...

As you can see, because of the NodeSelector, none of the pods are running. Now delete the deployment.

$ kubectl delete deployment hello-preempt

deployment "hello-preempt" deleted

What we've covered

A best practice for running applications in Google Cloud Platform is to distribute your resources across multiple zones, to tolerate outages. Google designs zones to be independent from each other: a zone usually has power, cooling, networking, and control planes that are isolated from other zones, and most single failure events will affect only a single zone.

A cluster can be created or reconfigured to have additional zones, separate from the primary zone (where the master runs). When you enable additional zones, all a cluster's node pools will be expanded to run across those zones.

$ gcloud container clusters delete gke-highlights

The following clusters will be deleted.
 - [gke-highlights] in [us-central1-f]
Do you want to continue (Y/n)?
Deleting cluster gke-highlights...\   
               
$ gcloud beta container clusters create gke-highlights \
    --node-locations us-central1-f,us-central1-a,us-central1-b \
    --num-nodes 1 \
    --enable-network-policy \
    --cluster-version=1.8.3-gke.0 \
    --machine-type n1-standard-2


Creating cluster gke-highlights...|
Created [https://container.googleapis.com/v1beta1/projects/codelab/zones/us-central1-f/clusters/gke-highlights].
kubeconfig entry generated for gke-highlights.
NAME            ZONE           MASTER_VERSION  MASTER_IP     MACHINE_TYPE   NODE_VERSION  NUM_NODES  STATUS
gke-highlights  us-central1-f  1.8.3-gke.0     35.226.64.30  n1-standard-1  1.8.3-gke.0   3          RUNNING

When you create a multi-zone cluster, Kubernetes Engine makes the resource footprint the same in all zones:

$ kubectl get nodes
NAME                                             STATUS    AGE       VERSION
gke-gke-highlights-default-pool-da6def67-xs79   Ready     2m        v1.8.3-gke.0
gke-gke-highlights-default-pool-af10e4de-nxh9   Ready     2m        v1.8.3-gke.0
gke-gke-highlights-default-pool-97a76573-7zp9   Ready     31s       v1.8.3-gke.0

We created a cluster with only one node but asked for it to be spread across three zones in a single region. In this case, you will see we got a total of 3 nodes, with one node in each zone.

Resources are spread evenly across zones to ensure that Pods can be scheduled evenly across zones. Doing so improves availability and failure recovery. The nodes have labels applied in Kubernetes that indicate their failure domain, so that they can be taken into account by the Kubernetes scheduler.

Verify that the VMs are indeed in different zones using gcloud compute:

$ gcloud compute instances list | grep gke-highlights
gke-gke-highlights-default-pool-da6def67-xs79  us-central1-a   n1-standard-2               10.128.0.18  35.193.194.216   RUNNING
gke-gke-highlights-default-pool-af10e4de-nxh9  us-central1-b   n1-standard-2               10.128.0.15  104.154.197.216  RUNNING
gke-gke-highlights-default-pool-97a76573-7zp9  us-central1-f   n1-standard-2               10.128.0.8   104.198.218.200  RUNNING

Scheduling a distributed workload

You may recall when we first scheduled a workload, two pods ended up on the same node. Now our nodes are in different regions, we can use the "failure-domain" labels that are attached to them as an indication to the Kubernetes scheduler that pods should be spread evenly among our nodes.

$ cat <<EOF | kubectl create -f -
apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: hello-web-regional
  labels:
    app: hello-web
spec:
  replicas: 9
  selector:
    matchLabels:
      app: hello-web
  template:
    metadata:
      labels:
        app: hello-web
    spec:
      containers:
      - name: hello-web
        image: gcr.io/google-samples/hello-app:1.0
        ports:
        - containerPort: 8080
        resources:
          requests:
            cpu: "50m"
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
             - topologyKey: failure-domain.beta.kubernetes.io/zone
EOF
deployment "hello-web-regional" created

$ kubectl get pods -owide
NAME                                 READY     STATUS    RESTARTS   AGE       IP            NODE
hello-web-regional-bf7c94b77-ggwlv   1/1       Running   0          2m        10.60.11.19   gke-gke-highlights-new-pool-da6def67-xs79
hello-web-regional-bf7c94b77-kr5vv   1/1       Running   0          2m        10.60.11.20   gke-gke-highlights-new-pool-da6def67-xs79
hello-web-regional-bf7c94b77-m29sq   1/1       Running   0          2m        10.60.7.16    gke-gke-highlights-new-pool-af10e4de-nxh9
hello-web-regional-bf7c94b77-nh997   1/1       Running   0          2m        10.60.11.18   gke-gke-highlights-new-pool-da6def67-xs79
hello-web-regional-bf7c94b77-pfgx6   1/1       Running   0          2m        10.60.7.14    gke-gke-highlights-new-pool-af10e4de-nxh9
hello-web-regional-bf7c94b77-sfh6q   1/1       Running   0          2m        10.60.4.15    gke-gke-highlights-new-pool-97a76573-7zp9
hello-web-regional-bf7c94b77-tvz76   1/1       Running   0          2m        10.60.4.14    gke-gke-highlights-new-pool-97a76573-7zp9
hello-web-regional-bf7c94b77-wpwvh   1/1       Running   0          2m        10.60.4.16    gke-gke-highlights-new-pool-97a76573-7zp9
hello-web-regional-bf7c94b77-xr4gr   1/1       Running   0          2m        10.60.7.15    gke-gke-highlights-new-pool-af10e4de-nxh9

$ kubectl delete deployment hello-web-regional

What we've covered

Node auto-repair

Kubernetes Engine's node auto-repair feature helps you keep the nodes in your cluster in a healthy, running state. When enabled, Kubernetes Engine makes periodic checks on the health state of each node in your cluster. If a node fails consecutive health checks over an extended time period (approximately 10 minutes), Kubernetes Engine initiates a repair process for that node.

$ gcloud beta container node-pools update default-pool --cluster gke-highlights --enable-autorepair

Now, for some fun: let's break a VM!

This gcloud command will find the VM in your regional node pool which is in the default zone, and SSH into it.

$ gcloud compute ssh $(gcloud compute instances list | \
                       grep gke-highlights-default | \
                       grep $(gcloud config get-value compute/zone) | \
                       awk '{ print $1 }')

You can simulate a node failure by removing the kubelet binary, which is responsible for running Pods on every Kubernetes node:

$ sudo rm /home/kubernetes/bin/kubelet && sudo systemctl restart kubelet
$ logout

Now when we check the node status we see the node is NotReady.

$ kubectl get nodes
NAME                                                STATUS     AGE       VERSION
gke-gke-highlights-regional-pool-62b7bad4-26v0      NotReady   4d        v1.8.3-gke.0
gke-gke-highlights-regional-pool-884432cb-qw4n      Ready      4d        v1.8.3-gke.0
gke-gke-highlights-regional-pool-ca7f6c0d-dbsr      Ready      4d        v1.8.3-gke.0

The Kubernetes Engine node repair agent will wait a few minutes in case the problem is intermittent. We'll come back to this in a minute.

Define a maintenance window

You can configure a maintenance window to have more control over when automatic upgrades are applied to Kubernetes on your cluster.

Creating a maintenance window instructs Kubernetes Engine to automatically trigger any automated tasks in your clusters, such as master upgrades, node pool upgrades, and maintenance of internal components, during a specific timeframe.

The times are specified in UTC, so select an appropriate time and set up a maintenance window for your cluster.

$ gcloud container clusters update gke-highlights --maintenance-window=08:00

Enable node auto-upgrades

Whenever a new version of Kubernetes is released, Google upgrades your master to that version. You can then choose to upgrade your nodes to that version, bringing functionality and security updates to both the OS and the Kubernetes components.

Node Auto-Upgrades use the same update mechanism as manual node upgrades, but does the scheduled upgrades during your maintenance window.

Auto-upgrades are enabled per node pool.

$ gcloud container node-pools update default-pool --cluster gke-highlights --enable-autoupgrade

Check your node repair

How is that node repair coming?

After a few minutes, you will see that the master drained the node, and then removed it.

gke-gke-highlights-default-pool-62b7bad4-26v0      NotReady,SchedulingDisabled   4d        v1.8.3-gke.0

A few minutes after that, a new node was turned on in its place:

$ kubectl get nodes
NAME                                                STATUS    AGE       VERSION
gke-gke-highlights-regional-pool-62b7bad4-26v0      Ready     1m        v1.8.3-gke.0
gke-gke-highlights-regional-pool-884432cb-qw4n      Ready     4d        v1.8.3-gke.0
gke-gke-highlights-regional-pool-ca7f6c0d-dbsr      Ready     4d        v1.8.3-gke.0

Because our Deployments were managed by Kubernetes controllers, we were able to survive the downtime with no problems.

What we've covered

Along with defining the applications you run in your Kubernetes environment, you can define policies about which pods are able to talk to which others.

Deploy a sample application

Start some pods to act as tiers of a hypothetical application:

$ kubectl run frontend \
  --image=gcr.io/google-samples/hello-app:1.0 \
  --labels=app=frontend \
  --port=8080
$ kubectl expose deployment frontend
$ kubectl run backend \
  --image=gcr.io/google-samples/hello-app:1.0 \
  --labels=app=backend \
  --port=8080 
$ kubectl expose deployment backend
$ kubectl run untrusted \
  --image=gcr.io/google-samples/hello-app:1.0 \
  --labels=app=untrusted \
  --port=8080 
$ kubectl expose deployment untrusted

In this example, pods with the label app=frontend should be able to connect to pods with the label app=backend. We also run some untrusted code, and that pod should not be able to connect to frontend or backend pods. (This simulates a situation where a malicious actor gets access to a container on your network through a vulnerability in its code.)

Verify connectivity

Using kubectl exec we can run a command on our untrusted pod:

$ UNTRUSTED_POD=$(kubectl get pods -l app=untrusted -o jsonpath='{.items[0].metadata.name}')

$ kubectl exec -it $UNTRUSTED_POD -- wget -qO- frontend:8080

Hello, world!
Version: 1.0.0
Hostname: backend-b58bc5ff7-qdkz7

Create a network policy

We can now create a NetworkPolicy which defines that no traffic from "frontend" is allowed to access "backend"

$ cat <<EOF | kubectl create -f -
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: frontend-to-backend
spec:
  podSelector:
    matchLabels:
      app: backend
  ingress:
  - from:
      - podSelector:
          matchLabels:
            app: frontend
EOF

Verify your network policy

Now, test again:

$ kubectl exec -it $UNTRUSTED_POD -- wget -qO- frontend:8080

Hello, world!
Version: 1.0.0
Hostname: frontend-6b57bff885-lb4gn
$ kubectl exec -it $UNTRUSTED_POD -- wget -qO- --timeout=2 backend:8080
wget: download timed out
command terminated with exit code 1

You can connect to the frontend service — but not the backend service, as our policy prohibits it.

Can we connect from the frontend to the backend?

$ FRONTEND_POD=$(kubectl get pods -l app=frontend -o jsonpath='{.items[0].metadata.name}')

$ kubectl exec -it $FRONTEND_POD -- wget -qO- backend:8080

Hello, world!
Version: 1.0.0
Hostname: backend-b58bc5ff7-kr86j

You see our policy does exactly what was asked: the frontend pod can connect to the backend pod, but nothing else can.

Delete the policy:

$ kubectl delete networkpolicy frontend-to-backend

$ kubectl exec -it $UNTRUSTED_POD -- wget -qO- backend:8080
Hello, world!
Version: 1.0.0
Hostname: backend-b58bc5ff7-kr86j

What we've covered

To connect to services in Google Cloud Platform, you need to provide an identity. While you might use a user's identity if operating interactively, services running on Compute Engine instances normally use a service account.

Applications running on Compute Engine can access and use the service account associated with the instance, and as Kubernetes Engine nodes are Compute Engine instances, containers can access the identities provided by the node.

However, if you grant that identity access to a service, then any container that runs on the node will have access to it.

A better practice is to create a service account for your own application, and provide that to your application using Kubernetes secrets.

We will use a sample application which reads messages posted to a Google Cloud Pub/Sub queue.

Create a Pub/Sub topic

The Pub/Sub subscriber application you will deploy uses a subscription named echo-read on a Pub/Sub topic called echo. Create these resources before deploying the application:

$ gcloud beta pubsub topics create echo
$ gcloud beta pubsub subscriptions create echo-read --topic=echo

Deploy an application

Our sample application reads messages that are published to a Pub/Sub topic. This application is written in Python using Google Cloud Pub/Sub client libraries and you can find the source code on GitHub.

$ kubectl create -f https://raw.githubusercontent.com/GoogleCloudPlatform/kubernetes-engine-samples/master/cloud-pubsub/deployment/pubsub.yaml

Look at the pod:

$ kubectl get pods -l app=pubsub

You can see that the container is failing to start and went into a CrashLoopBackOff state. Inspect the logs from the Pod by running:

$ kubectl logs -l app=pubsub
...
google.gax.errors.RetryError: GaxError(Exception occurred in retry method
that was not classified as transient, caused by <_Rendezvous of RPC that
terminated with (StatusCode.PERMISSION_DENIED, Request had insufficient
authentication scopes.)>)

The stack trace and the error message indicates that the application does not have permissions to query the Cloud Pub/Sub service. This is because the "Compute Engine default service account" is not assigned any roles giving it permission to Cloud Pub/Sub.

$ PROJECT=$(gcloud config get-value project)
$ gcloud iam service-accounts create pubsub-sa --display-name "Pub/Sub demo service account"
$ gcloud projects add-iam-policy-binding $PROJECT \
    --member serviceAccount:pubsub-sa@$PROJECT.iam.gserviceaccount.com \
    --role roles/pubsub.subscriber
$ gcloud iam service-accounts keys create key.json \
    --iam-account pubsub-sa@$PROJECT.iam.gserviceaccount.com

Import secret into Kubernetes

Load in the Secret:

$ kubectl create secret generic pubsub-key --from-file=key.json=key.json

We now have a secret called pubsub-key which contains a file called key.json. You can update your Deployment to mount this Secret, you can override the application to use this service account instead of the default identity of the node.

Configure the application with the Secret

This manifest file defines the following to make the credentials available to the application:

       volumeMounts:
        - name: google-cloud-key
          mountPath: /var/secrets/google
        env:
        - name: GOOGLE_APPLICATION_CREDENTIALS
          value: /var/secrets/google/key.json

Apply this configuration:

$ kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/kubernetes-engine-samples/master/cloud-pubsub/deployment/pubsub-with-secret.yaml

Your pod will now start:

$ kubectl get pods -l app=pubsub
NAME                     READY     STATUS    RESTARTS   AGE
pubsub-94476cd97-kzd9j   1/1       Running   0          1m

Test receiving Pub/Sub messages

Validate that your application is now able to read from Google Cloud Pub/Sub:

$ gcloud beta pubsub topics publish echo "Hello world"
NAME                     READY     STATUS    RESTARTS   AGE
pubsub-94476cd97-kzd9j   1/1       Running   0          1m
$ kubectl logs -l app=pubsub
Pulling messages from Pub/Sub subscription...
[2017-11-28 21:44:10.537970] ID=177003642109951 Data=b'Hello, world'

You have successfully configured an application on Kubernetes Engine to authenticate to Pub/Sub API using service account credentials!

Delete your subscriptions and your pod:

$ gcloud beta pubsub subscriptions delete echo-read
$ gcloud beta pubsub topics delete echo
$ kubectl delete pods -l app=pubsub

What we've covered

To publish a service to the world via HTTP, Kubernetes has an object called an Ingress. The Ingress object and its associated controller program a load balancer, like the Google Cloud Platform HTTP Load Balancer.

Deploy an application


Let's redeploy our faithful hello-web app:

$ kubectl run hello-web --image=gcr.io/google-samples/hello-app:1.0 --port=8080 --replicas=3

deployment "hello-web" created

Reserve a static IP address

When you create an Ingress, you will be allocated an random IP address. It's sensible to instead reserve an IP address, and then allocate that to your Ingress, so that you don't need to change DNS if you ever need to delete and recreate your load balancer.

Google Cloud Platform has two types of IP addresses - regional (for Network Load Balancers, as used by Services) and global (for HTTP Load Balancers).

Reserve a static external IP address named nginx-static-ip by running:

$ gcloud compute addresses create hello-web-static-ip --global

$ gcloud compute addresses list --filter="name=hello-web-static-ip"
NAME                 REGION  ADDRESS       STATUS
hello-web-static-ip          35.227.247.6  IN_USE

This is the value you would program into DNS.

Create a Service

We now need to tell Kubernetes that these pods comprise a Service, which can be used as a target for our traffic.

In many other examples you might have used --type=LoadBalancer. That creates a Network Load Balancer, which exists in one region. By using NodePorts, we are instead just exposing the service to the VM IPs, and we can then have our Ingress address those VMs.

$ kubectl expose deployment hello-web --target-port=8080 --type=NodePort

$ kubectl get service hello-web
NAME        TYPE       CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
hello-web   NodePort   10.63.253.73   <none>        8080:32231/TCP   2m

In this example, you can see that any traffic from the Internet to port 32231 on any node will be routed to a healthy container on port 8080. You can also see that there is currently no way to contact it from outside the cluster.

Create an Ingress

To fix that, we will create an Ingress object using our static IP and our service:

$ cat <<EOF | kubectl create -f -
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: hello-web-ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: hello-web-static-ip
spec:
  backend:
    serviceName: hello-web
    servicePort: 8080
EOF

You should see:

After a couple of minutes, the load balancer will be created, and mapped to your IP address:

$ kubectl get ingress
NAME                HOSTS     ADDRESS        PORTS     AGE
hello-web-ingress   *         35.227.247.6   80        2m

You can now hit http://35.227.247.6/:

If you no longer need them, delete the Ingress and release the static IP:

$ kubectl delete ingress hello-web-ingress
$ gcloud compute addresses delete hello-web-static-ip --global

What we've covered

You can delete the resources you have created (to save on cost and to be a good cloud citizen):

$ gcloud container clusters delete gke-highlights

The following clusters will be deleted.
 - [gke-highlights] in [us-central1-f]
Do you want to continue (Y/n)?  Y
Deleting cluster gke-highlights...done.                                                                                                                                                                                            
Deleted [https://container.googleapis.com/v1/projects/codelab/zones/us-central1-f/clusters/gke-highlights].

Of course, you can also delete the entire project but you would lose any billing setup you have done (disabling project billing first is required). Additionally, deleting a project will only stop all billing after the current billing cycle ends.

Congratulations, you're now a certified #GKEnius. Enjoy the rest of KubeCon!