This lab uses a fake web application called the Sock Shop to demonstrate the features of Google Stackdriver. In order to have something interesting to troubleshoot, there are two existing problems that you will look for and analyze - but don't worry, you won't need to fix them! During the lab you'll use the following Google Cloud Platform applications to monitor your web app:

Self-paced environment setup

If you don't already have a Google Account (Gmail or Google Apps), you must create one. Sign-in to Google Cloud Platform console (console.cloud.google.com) and create a new project:

Remember the project ID, a unique name across all Google Cloud projects (the name above has already been taken and will not work for you, sorry!). It will be referred to later in this codelab as PROJECT_ID.

Next, you'll need to enable billing in the Developers Console in order to use Google Cloud resources.

Running through this codelab shouldn't cost you more than a few dollars, but it could be more if you decide to use more resources or if you leave them running (see "cleanup" section at the end of this document). Google Kubernetes Engine pricing is documented here.

New users of Google Cloud Platform are eligible for a $300 free trial.

Enable the Google Kubernetes Engine API

Take the following steps to enable the Google Kubernetes Engine API:

  1. Visit the Kubernetes Engine page in the Google Cloud Platform Console.
  2. Select the project you created above, if it isn't already selected.
  3. Wait for the API and related services to be enabled. This can take several minutes.

Activate Google Cloud Shell

From the GCP Console, click the Cloud Shell icon on the top right toolbar:

Then click "Start Cloud Shell".

In Cloud Shell you'll create a Kubernetes cluster in which to deploy the webstore app.

Clone the web store demo to your GCE project:

git clone https://github.com/mtwo/microservices-demo

Create the container cluster, container for the app, and enable Google Cloud services for your project:

gcloud container clusters create "sockshop" --scopes  "https://www.googleapis.com/auth/cloud-platform" --num-nodes "3" --network "default"  --enable-cloud-logging --enable-cloud-monitoring --subnetwork "default" --enable-legacy-authorization --zone us-central1-a

This will take about 5 minutes to build.

kubectl create namespace sock-shop

Go into the microservices-demo directory:

cd microservices-demo

Deploy the demo on Kubernetes:

kubectl apply -f deploy/kubernetes/complete-demo.yaml
kubectl apply -f deploy/kubernetes/manifests-zipkin/zipkin-ns.yaml -f deploy/kubernetes/manifests-zipkin

To see the set of microservices that make up your webstore application, in the left menu of the Console, go to Kubernetes Engine > Discovery & load balancing.

You should see something like the following:

These microservices also map to Kubernetes workloads. See those by clicking on Workloads. When all workloads show OK status, you're ready to use the web store.

This lab puts you in charge of a fake webstore, called Sock Shop, which runs on Google Kubernetes Engine. Before exploring what it takes to troubleshoot an issue with the webstore, let's explore the application.

To view the application, you need to find the endpoint.

In the left menu of the Console, go to Kubernetes Engine > Discovery & load balancing. Find the service called "front-end". Under the Endpoints column, click on the IP address.

You should see the homepage of your webstore in a new browser window.

Next you'll buy a few pairs of socks just like a customer would to create some interesting customer data to work with.

Click on the Catalogue button, then add the "Crossed" set of socks to your cart.

Then click on the button to view your cart.

Change the quantity in the cart to 0. The arrows won't let you go below 0, but you can select the 1, they type a 0. Then click on Update basket.

Notice that the website accepts this but does not remove the socks from your cart!

Go back to the catalog and add the "Colourful" socks to your cart. Notice that it seems to take longer than usual.

Now log in as "user" with password "password".

Go back into your cart and click Proceed to Checkout to try to complete the transaction. You won't be able to until the 0 is updated. That's fine - the logs are getting generated.

Place a few more orders by adding an item to your cart and then checking out. 3 or 4 orders in total should give you enough data to investigate.

Now, let's go look for the issues with the webstore. Leave the window for the webstore open - you'll come back to it as you troubleshoot the application.

It is very important to make sure that your web application performance is up to the requirements that you have set. Stackdriver Trace is a key tool that helps you understand latency in your applications by showing how long actual customer requests take through every step of their journey through a distributed system.

Traces are immensely powerful for debugging and management, and require an instrumentation library to be linked to an application for them to be captured. Stackdriver Trace supplies its own set of language-specific libraries for this, as do our competitors in this space. This app, Sock Shop, was instrumented with Zipkin, an open source tracing solution, prior to arriving on GCP. Conveniently, the Stackdriver Zipkin adapter is offered for situations like these, which allows customers to switch over to Stackdriver Trace with their existing Zipkin instrumentation, and without modifying their application at all.

There is a traffic generator included in the project, sending traffic to your application. This is sufficient for Traces to provide more information about the latency.

In the Console, in the Stackdriver section, click on Trace and then Trace List to see the recent traces and their latencies.

While the website generates some fake traffic to keep the trace list populated, filtering to "http:/orders" (include the quotes) will show highly detailed traces of a completed order. You should have a few traces if you completed a few of orders in the previous step.

You can see a few Traces for the /orders endpoint. Click on one in the scatterplot points to see the complete trace for that request. You will now be able to visualize the components of the Trace, called Spans. Each Span represents one of the steps that was completed during the processing of this request. If a Span is longer than you expected, it indicates that something is taking longer during that particular step.

Stackdriver Error Reporting automatically processes your logs to identify and group errors. You'll use it now to get some insight into the health of the Sock Shop application.

Click on Error Reporting in the Stackdriver section of the Console.

You should see a new error in your gke_instances, "Error: Tried to set quantity < 0".

Click on the error for "Tried to set quantity<0" to view the error details. From here you can see:

Click on the "View Logs" link at the bottom right of the Recent samples to drill into the actual logs to see what was happening around this time.

Since you entered Stackdriver Logging through Error Reporting, the search is already scoped to:

There are quite a few log entries displayed. Stackdriver Logging organizes all the logs from the project and allows you to search them quickly to better understand the problem.

Filter the logs to just look for errors by clicking on the Any log level dropdown and selecting Error.

Now you should see just the logs that have the Error status.

In the next section you'll continue to work with this error.

Filter Based on Existing Logs

In Error Reporting, you saw that the app's error was "Error: Tried to set quantity < 0", and you've filtered your log to only show errors. Now you'll see the detailed information available from the error that can be used to help you analyze the issue.

Click the arrow next to the "Tried to set quantity < 0" log entry error to expand and see details.

Then expand the labels field by clicking on the down arrow. You can see this error is coming from one of the front end pods.

Let's look at what else was happening at this pod. Click on the pod name and select Show Matching Entries.

Showing the matching entries automatically switched the mode to advanced mode, where you are able to specify complex conditions for searching and exploring your logs. This is an easy way to find the correct value for the advanced filters you want to use. (You can also use Hide Matching Entries to remove messages that are distracting you from the problem you are trying to fix).

Remove the restriction that the severity must be Error or higher to see more context around the logs at this time by deleting the highlighted row, then click Submit Filter (or press Command+Enter to submit the query).

From here you can see the log entries surrounding the error in the front-end microservice, which may help you understand what else was going on at the same time. This can be extremely useful when debugging.

Saving and Sharing

Once you've figured out the correct filters, you may want to save them or share them with others on your team.

Click the triangle in the filter box, then click Get link to filter to get a link to the filter you just created. You can share the link directly with teammates, save it in documentation, or save it in your favorites bar.

Searching Logs with the Advanced Filter

Let's discuss quickly the Advanced Filter now that you're using it. You don't need to run any of the examples in this section to complete your lab. The Advanced Filter has a special syntax editor to let you build your own filter.

To manually switch between basic and advanced modes, click on the triangle at the right edge of the filter box, then select the mode you'd like to use. If you're in basic mode, you'll be able to click on Convert to advanced filter; if you're in advanced mode, you'll be able to Clear filters and return to basic mode.

For example, this filter limits the view to just log messages from GKE:

resource.type="container"

To add more clauses to your filter, you can add more lines. You can also use operators like AND, OR, NOT, >, <, etc. Here's a filter that shows only ERROR and CRITICAL messages.

severity=("ERROR" OR "CRITICAL")

Stackdriver maps severity to numeric values, so you can also use mathematical comparisons on it like this:

resource.type="container"
severity >= ERROR

The user interface will auto-complete what it can, including field names and possible values for fields. More options for the advanced filter syntax include boolean expressions, global restrictions, and ip_in_net function which determines if an IP address in a log entry is contained in a subnet.

Now that you have some context about what's wrong, it's time to dive into the problem more. A snapshot captures the local variables and call stack at a specific line location in your source code. Snapshots are useful if you want to debug a particular piece of code or inspect the variables while your application is still running.

Let's add a debug snapshot to the function that adds items to the shopping cart so that you can look for the issue you know exists.

Return to Error Reporting in the Stackdriver section of the Console.

Click on the error name to see its details. You should see a Stack trace sample for the error in question. Make a note of the location of the stack trace "/usr/src/app/api/cart/index.js:165"

In the next section you'll dive even deeper into the error.

Add source code

Stackdriver Debug can be used without a source view, but that's no fun!

Our first step is to tell Stackdriver where to find the source code for our front-end microservice. front-end is a Node.js application available on the public GitHub repo: https://github.com/mtwo/front-end. Using your GitHub account is recommended, but it's not required.

In the Stackdriver section of the Console, click on Debug.

If you have a GitHub account:

  1. Click on GitHub. You may have to authorize GitHub to grant access to Google Stackdriver.
  2. Specify that you want to use the repository mtwo/front-end, with the master branch.

If you do not have a Github account:

  1. To to https://github.com/mtwo/front-end
  2. Click on the green "Clone or download" button on the right side and then click "Download ZIP"
  3. Extract the ZIP file
  4. In the Local files section, click Select Source and upload the extracted .zip file that you just downloaded

Once you've added the file, you'll see them in the leftmost pane. Open the folders for api, then cart then click on index.js.

Take the debug snapshot

Click in the margin on line 92, the Snapshot is automatically taken.

The application will continue running, waiting for the snapshot to hit when something is added to the cart.

Now go back to the Sock Shop app and add the Colourful socks to the cart to trigger a snapshot.

Return to the tab with Debug in the Console and look at the snapshot.

When the snapshot was activated you see the Variables and Call Stack sections get populated.

Expand the item variable to see that it now has the records corresponding to the item you added to your cart. Note the final character of the item id as well: "3395a43e-2d88-40de-b95f-e00e1502085b".

As applications are developed, developers put log statements into the code with the hope that the logs include enough information to figure out issues. But that is not always enough. It's usually only when debugging an issue that you realize that a few more log statements here and there are needed. The usual workflow is to modify the code, put in the additional log statements, re-deploy the code, and monitor the logs.

Stackdriver lets you add log statements (called logpoints) to your application while it is running. You don't need to go through the process of stopping the application, changing the code, and redeploying. Instead, you can manage your list of logpoints from outside the application using the logpoints support.

Earlier in this lab you saw a new error recorded on line 165 when a quantity of less than one was entered in the app. Looking at the code in the Error Logs you can see that the developer forgot to finish up and delete the item from the cart. Let's log the values input so that you can keep track of every incorrect interaction until the bug is fixed.

Click on the Logpoint tab, next to the Snapshot tab, then click on line 166.

Debug will automatically insert a logpoint template. To add a logpoint your script needs to do the following things:

In the template, copy and paste the text in bold to make your logpoint:

if (true) logpoint("illegal quantity: id={req.body.id}, quantity={req.body.quantity}")

Click Add.

Click on the Logpoint History tab at the bottom of the screen to see the logpoint below the code.

To see this in action, go back to your cart, change the quantity of one set of socks to be -1 and click Update Basket.

Return to Stackdriver Logging to view your logs. Filter your search to text:LOGPOINT.

In the details you will see the logpoint getting fired and the log message.

This lab touches upon just a few of the capabilities that the Stackdriver platform brings to the table. There is a lot more to explore, including how Stackdriver Monitoring works together with Stackdriver Logging, Trace and Debug to provide observability for your cloud application.

For example, Stackdriver Monitoring allows you to:

You have successfully:

Next Steps / Learn More

Learn more about Stackdriver in our Stackdriver Monitoring codelab.

Here are also links to the full Stackdriver documentation, with details for each product:

Additionally, you can learn about the origin of logpoints in this blog post: Add Application Logs to an application with no restarts.

Google Cloud Training & Certification

...helps you make the most of Google Cloud technologies. Our classes include technical skills and best practices to help you get up to speed quickly and continue your learning journey. We offer fundamental to advanced level training, with on-demand, live, and virtual options to suit your busy schedule. Certifications help you validate and prove your skill and expertise in Google Cloud technologies.

Manual Last Updated February 22, 2018

Lab Last Tested February 22, 2018