Setup Basic OpenTelemetry Plugin in gRPC C++

1. Introduction

In this codelab, you'll use gRPC to create a client and server that form the foundation of a route-mapping application written in C++.

By the end of the tutorial, you will have a simple gRPC HelloWorld application instrumented with the gRPC OpenTelemetry plugin and be able to see the exported observability metrics in Prometheus.

What you'll learn

• How to setup OpenTelemetry Plugin for existing gRPC C++ application
• Running a local Prometheus instance
• Exporting metrics to Prometheus
• View metrics from Prometheus dashboard

2. Before you begin

What you'll need

• git
• curl
• build-essential
• clang
• bazel to build examples in this codelab

Install the prerequisites:

sudo apt-get update -y
sudo apt-get upgrade -y
sudo apt-get install -y git curl build-essential clang

bazel can be installed via bazelisk. The latest version can be found here.

A simple way to set it up is to install it as the bazel binary in your PATH as follows:

sudo cp bazelisk-linux-amd64 /usr/local/bin/bazel
sudo chmod a+x /usr/local/bin/bazel

Alternatively, you can also use CMake. Instructions for using CMake can be found here.

Get the code

To streamline your learning, this codelab offers a pre-built source code scaffold to help you get started. The following steps will guide you through instrumenting the gRPC OpenTelemetry Plugin in an application.

grpc-codelabs

The scaffold source code for this codelab is available in this github directory. If you prefer not to implement the code yourself, the completed source code is available in the completed directory.

First, clone the grpc codelab repo and cd into grpc-cpp-opentelemetry folder:

git clone https://github.com/grpc-ecosystem/grpc-codelabs.git
cd grpc-codelabs/codelabs/grpc-cpp-opentelemetry/

Alternatively, you can download the .zip file containing only the codelab directory and manually unzip it.

Download Zip

Build the gRPC library using bazel:

bazel build start_here/...

Note: Building gRPC might take a few minutes. We can move on to the next step to instrument our gRPC example with the gRPC OpenTelemetry plugin in the meantime.

3. Register the OpenTelemetry Plugin

We need a gRPC application to add the gRPC OpenTelemetry plugin. In this codelab, we will use a simple gRPC HelloWorld client and server that we will instrument with the gRPC OpenTelemetry plugin.

Your first step is to register the OpenTelemetry Plugin configured with a Prometheus exporter in the client. Open codelabs/grpc-cpp-opentelemetry/start_here/greeter_callback_client.cc with your favorite editor, and transform main() to look like this -

int main(int argc, char **argv) {
  absl::ParseCommandLine(argc, argv);

  // Codelab Solution: Register a global gRPC OpenTelemetry plugin configured
  // with a prometheus exporter.
  opentelemetry::exporter::metrics::PrometheusExporterOptions opts;
  opts.url = absl::GetFlag(FLAGS_prometheus_endpoint);
  auto prometheus_exporter =
      opentelemetry::exporter::metrics::PrometheusExporterFactory::Create(opts);
  auto meter_provider =
      std::make_shared<opentelemetry::sdk::metrics::MeterProvider>();

  // The default histogram boundaries are not granular enough for RPCs. Override
  // the "grpc.client.attempt.duration" view as recommended by
  // https://github.com/grpc/proposal/blob/master/A66-otel-stats.md.
  AddLatencyView(meter_provider.get(), "grpc.client.attempt.duration", "s");
  meter_provider->AddMetricReader(std::move(prometheus_exporter));
  auto status = grpc::OpenTelemetryPluginBuilder()
                    .SetMeterProvider(std::move(meter_provider))
                    .BuildAndRegisterGlobal();
  if (!status.ok()) {
    std::cerr << "Failed to register gRPC OpenTelemetry Plugin: "
              << status.ToString() << std::endl;
    return static_cast<int>(status.code());
  }

  // Continuously send RPCs.
  RunClient(absl::GetFlag(FLAGS_target));

  return 0;
}

Note: A Prometheus Exporter is being set up on the OpenTelemetry Meter Provider. (There are other ways to export the metrics as well. This codelab chooses the prometheus exporter.) This MeterProvider is provided to gRPC's OpenTelemetry plugin as configuration. Once the OpenTelemetry plugin is registered globally all gRPC clients and servers will be instrumented with OpenTelemetry.

Next step is to add the OpenTelemetry plugin to the server. Open codelabs/grpc-cpp-opentelemetry/start_here/greeter_callback_server.cc and transform main to look like this:

int main(int argc, char** argv) {
  absl::ParseCommandLine(argc, argv);

  // Register a global gRPC OpenTelemetry plugin configured with a prometheus
  // exporter.
  opentelemetry::exporter::metrics::PrometheusExporterOptions opts;
  opts.url = absl::GetFlag(FLAGS_prometheus_endpoint);
  auto prometheus_exporter =
      opentelemetry::exporter::metrics::PrometheusExporterFactory::Create(opts);
  auto meter_provider =
      std::make_shared<opentelemetry::sdk::metrics::MeterProvider>();

  // The default histogram boundaries are not granular enough for RPCs. Override
  // the "grpc.server.call.duration" view as recommended by
  // https://github.com/grpc/proposal/blob/master/A66-otel-stats.md.
  AddLatencyView(meter_provider.get(), "grpc.server.call.duration", "s");
  meter_provider->AddMetricReader(std::move(prometheus_exporter));
  auto status = grpc::OpenTelemetryPluginBuilder()
                    .SetMeterProvider(std::move(meter_provider))
                    .BuildAndRegisterGlobal();
  if (!status.ok()) {
    std::cerr << "Failed to register gRPC OpenTelemetry Plugin: "
              << status.ToString() << std::endl;
    return static_cast<int>(status.code());
  }
  RunServer(absl::GetFlag(FLAGS_port));
  return 0;
}

The required header files and build dependencies have already been added for convenience.

#include "opentelemetry/exporters/prometheus/exporter_factory.h"
#include "opentelemetry/exporters/prometheus/exporter_options.h"
#include "opentelemetry/sdk/metrics/meter_provider.h"

#include <grpcpp/ext/otel_plugin.h>

Build dependencies are also already added in BUILD file -

cc_binary(
    name = "greeter_callback_client",
    srcs = ["greeter_callback_client.cc"],
    defines = ["BAZEL_BUILD"],
    deps = [
        "//util:util",
        "@com_github_grpc_grpc//:grpc++",
        "@com_github_grpc_grpc//:grpcpp_otel_plugin",
        "@com_google_absl//absl/flags:flag",
        "@com_google_absl//absl/flags:parse",
        "@io_opentelemetry_cpp//exporters/prometheus:prometheus_exporter",
        "@io_opentelemetry_cpp//sdk/src/metrics",
    ],
)

4. Running the example and viewing metrics

To run the server, run -

bazel run start_here:greeter_callback_server

With a successful setup, you will see the following output for the server -

Server listening on 0.0.0.0:50051

While, the server is running, on another terminal, run the client -

bazel run start_here:greeter_callback_client

A successful run will look like -

Greeter received: Hello world
Greeter received: Hello world
Greeter received: Hello world
Greeter received: Hello world
Greeter received: Hello world
Greeter received: Hello world
Greeter received: Hello world
Greeter received: Hello world
Greeter received: Hello world
Greeter received: Hello world
Greeter received: Hello world

Since we have set-up the gRPC OpenTelemetry plugin to export metrics using Prometheus. Those metrics will be available on localhost:9464 for server and localhost:9465 for client.

To see client metrics -

curl localhost:9465/metrics

The result would be of the form -

# HELP exposer_transferred_bytes_total Transferred bytes to metrics services
# TYPE exposer_transferred_bytes_total counter
exposer_transferred_bytes_total 0
# HELP exposer_scrapes_total Number of times metrics were scraped
# TYPE exposer_scrapes_total counter
exposer_scrapes_total 0
# HELP exposer_request_latencies Latencies of serving scrape requests, in microseconds
# TYPE exposer_request_latencies summary
exposer_request_latencies_count 0
exposer_request_latencies_sum 0
exposer_request_latencies{quantile="0.5"} Nan
exposer_request_latencies{quantile="0.9"} Nan
exposer_request_latencies{quantile="0.99"} Nan
# HELP target Target metadata
# TYPE target gauge
target_info{otel_scope_name="grpc-c++",otel_scope_version="1.67.0-dev",service_name="unknown_service",telemetry_sdk_version="1.13.0",telemetry_sdk_name="opentelemetry",telemetry_sdk_language="cpp"} 1 1721958543107
# HELP grpc_client_attempt_rcvd_total_compressed_message_size_bytes Compressed message bytes received per call attempt
# TYPE grpc_client_attempt_rcvd_total_compressed_message_size_bytes histogram
grpc_client_attempt_rcvd_total_compressed_message_size_bytes_count{grpc_method="helloworld.Greeter/SayHello",grpc_status="OK",grpc_target="dns:///localhost:50051",otel_scope_name="grpc-c++",otel_scope_version="1.67.0-dev"} 96 1721958543107
grpc_client_attempt_rcvd_total_compressed_message_size_bytes_sum{grpc_method="helloworld.Greeter/SayHello",grpc_status="OK",grpc_target="dns:///localhost:50051",otel_scope_name="grpc-c++",otel_scope_version="1.67.0-dev"} 1248 1721958543107
grpc_client_attempt_rcvd_total_compressed_message_size_bytes_bucket{grpc_method="helloworld.Greeter/SayHello",grpc_status="OK",grpc_target="dns:///localhost:50051",otel_scope_name="grpc-c++",otel_scope_version="1.67.0-dev",le="0"} 0 1721958543107
grpc_client_attempt_rcvd_total_compressed_message_size_bytes_bucket{grpc_method="helloworld.Greeter/SayHello",grpc_status="OK",grpc_target="dns:///localhost:50051",otel_scope_name="grpc-c++",otel_scope_version="1.67.0-dev",le="5"} 0 1721958543107
grpc_client_attempt_rcvd_total_compressed_message_size_bytes_bucket{grpc_method="helloworld.Greeter/SayHello",grpc_status="OK",grpc_target="dns:///localhost:50051",otel_scope_name="grpc-c++",otel_scope_version="1.67.0-dev",le="10"} 0 1721958543107
grpc_client_attempt_rcvd_total_compressed_message_size_bytes_bucket{grpc_method="helloworld.Greeter/SayHello",grpc_status="OK",grpc_target="dns:///localhost:50051",otel_scope_name="grpc-c++",otel_scope_version="1.67.0-dev",le="25"} 96 1721958543107
grpc_client_attempt_rcvd_total_compressed_message_size_bytes_bucket{grpc_method="helloworld.Greeter/SayHello",grpc_status="OK",grpc_target="dns:///localhost:50051",otel_scope_name="grpc-c++",otel_scope_version="1.67.0-dev",le="50"} 96 1721958543107
grpc_client_attempt_rcvd_total_compressed_message_size_bytes_bucket{grpc_method="helloworld.Greeter/SayHello",grpc_status="OK",grpc_target="dns:///localhost:50051",otel_scope_name="grpc-c++",otel_scope_version="1.67.0-dev",le="75"} 96 1721958543107

Similarly, for the server side metrics -

curl localhost:9464/metrics

5. Viewing metrics on Prometheus

Here, we will setup a prometheus instance that will scrape our gRPC example client and server that are exporting metrics using prometheus.

Download the latest release of Prometheus for your platform, then extract and run it:

tar xvfz prometheus-*.tar.gz
cd prometheus-*

Create a prometheus configuration file with the following -

cat > grpc_otel_cpp_prometheus.yml <<EOF
scrape_configs:
  - job_name: "prometheus"
    scrape_interval: 5s
    static_configs:
      - targets: ["localhost:9090"]
  - job_name: "grpc-otel-cpp"
    scrape_interval: 5s
    static_configs:
      - targets: ["localhost:9464", "localhost:9465"]
EOF

Start prometheus with the new configuration -

./prometheus --config.file=grpc_otel_cpp_prometheus.yml

This will configure the metrics from the client and server codelab processes to be scraped every 5 seconds.

Go to http://localhost:9090/graph to view the metrics. For example, the query -

histogram_quantile(0.5, rate(grpc_client_attempt_duration_seconds_bucket[1m]))

will show a graph with the median attempt latency using 1minute as the time window for the quantile calculation.

Rate of queries -

increase(grpc_client_attempt_duration_seconds_bucket[1m])

6. (Optional) Exercise for User

In the prometheus dashboards, you'll notice that the QPS is low. See if you can identify some suspicious code in the example that is limiting the QPS.

For the enthusiastic, the client code limits itself to only have a single pending RPC at a given moment. This can be modified so that the client sends more RPCs without waiting for the previous ones to complete. (The solution for this has not been provided.)