Using the Video Intelligence API with C#

Google Cloud Video Intelligence API allows developers to use Google video analysis technology as part of their applications.

It can be used to do:

The REST API enables users to annotate videos stored locally or in Google Cloud Storage with contextual information at the level of the entire video, per segment, per shot, and per frame.

In this codelab, you will focus on using the Video Intelligence API with C#. You will learn how to analyze videos for labels, shot changes and explicit content detection.

What you'll learn

  • How to use the Cloud Shell
  • How to enable the Video Intelligence API
  • How to Authenticate API requests
  • How to install the Google Cloud client library for C#
  • How to analyze videos for labels
  • How to analyze videos for shot changes
  • How to analyze videos for explicit content detection

What you'll need

  • A Google Cloud Platform Project
  • A Browser, such Chrome or Firefox
  • Familiarity using C#

Survey

How will you use this tutorial?

Read it through only Read it and complete the exercises

How would you rate your experience with C#?

Novice Intermediate Proficient

How would you rate your experience with using Google Cloud Platform services?

Novice Intermediate Proficient

Self-paced environment setup

  1. Sign in to Cloud Console and create a new project or reuse an existing one. (If you don't already have a Gmail or G Suite account, you must create one.)

dMbN6g9RawQj_VXCSYpdYncY-DbaRzr2GbnwoV7jFf1u3avxJtmGPmKpMYgiaMH-qu80a_NJ9p2IIXFppYk8x3wyymZXavjglNLJJhuXieCem56H30hwXtd8PvXGpXJO9gEUDu3cZw

ci9Oe6PgnbNuSYlMyvbXF1JdQyiHoEgnhl4PlV_MFagm2ppzhueRkqX4eLjJllZco_2zCp0V0bpTupUSKji9KkQyWqj11pqit1K1faS1V6aFxLGQdkuzGp4rsQTan7F01iePL5DtqQ

8-tA_Lheyo8SscAVKrGii2coplQp2_D1Iosb2ViABY0UUO1A8cimXUu6Wf1R9zJIRExL5OB2j946aIiFtyKTzxDcNnuznmR45vZ2HMoK3o67jxuoUJCAnqvEX6NgPGFjCVNgASc-lg

Remember the project ID, a unique name across all Google Cloud projects (the name above has already been taken and will not work for you, sorry!). It will be referred to later in this codelab as PROJECT_ID.

  1. Next, you'll need to enable billing in Cloud Console in order to use Google Cloud resources.

Running through this codelab shouldn't cost much, if anything at all. Be sure to to follow any instructions in the "Cleaning up" section which advises you how to shut down resources so you don't incur billing beyond this tutorial. New users of Google Cloud are eligible for the $300USD Free Trial program.

Start Cloud Shell

While Google Cloud can be operated remotely from your laptop, in this codelab you will be using Google Cloud Shell, a command line environment running in the Cloud.

Activate Cloud Shell

  1. From the Cloud Console, click Activate Cloud Shell H7JlbhKGHITmsxhQIcLwoe5HXZMhDlYue4K-SPszMxUxDjIeWfOHBfxDHYpmLQTzUmQ7Xx8o6OJUlANnQF0iBuUyfp1RzVad_4nCa0Zz5LtwBlUZFXFCWFrmrWZLqg1MkZz2LdgUDQ.

zlNW0HehB_AFW1qZ4AyebSQUdWm95n7TbnOr7UVm3j9dFcg6oWApJRlC0jnU1Mvb-IQp-trP1Px8xKNwt6o3pP6fyih947sEhOFI4IRF0W7WZk6hFqZDUGXQQXrw21GuMm2ecHrbzQ

If you've never started Cloud Shell before, you'll be presented with an intermediate screen (below the fold) describing what it is. If that's the case, click Continue (and you won't ever see it again). Here's what that one-time screen looks like:

kEPbNAo_w5C_pi9QvhFwWwky1cX8hr_xEMGWySNIoMCdi-Djx9AQRqWn-__DmEpC7vKgUtl-feTcv-wBxJ8NwzzAp7mY65-fi2LJo4twUoewT1SUjd6Y3h81RG3rKIkqhoVlFR-G7w

It should only take a few moments to provision and connect to Cloud Shell.

pTv5mEKzWMWp5VBrg2eGcuRPv9dLInPToS-mohlrqDASyYGWnZ_SwE-MzOWHe76ZdCSmw0kgWogSJv27lrQE8pvA5OD6P1I47nz8vrAdK7yR1NseZKJvcxAZrPb8wRxoqyTpD-gbhA

This virtual machine is loaded with all the development tools you'll need. It offers a persistent 5GB home directory and runs in Google Cloud, greatly enhancing network performance and authentication. Much, if not all, of your work in this codelab can be done with simply a browser or your Chromebook.

Once connected to Cloud Shell, you should see that you are already authenticated and that the project is already set to your project ID.

  1. Run the following command in Cloud Shell to confirm that you are authenticated:
gcloud auth list

Command output

 Credentialed Accounts
ACTIVE  ACCOUNT
*       <my_account>@<my_domain.com>

To set the active account, run:
    $ gcloud config set account `ACCOUNT`
gcloud config list project

Command output

[core]
project = <PROJECT_ID>

If it is not, you can set it with this command:

gcloud config set project <PROJECT_ID>

Command output

Updated property [core/project].

Before you can begin using the Video Intelligence API, you must enable the API. You can enable the API by using the following command in the Cloud Shell:

gcloud services enable videointelligence.googleapis.com

In order to make requests to the Video Intelligence API, you need to use a Service Account. A Service Account belongs to your project and it is used by the Google Client C# library to make Video Intelligence API requests. Like any other user account, a service account is represented by an email address. In this section, you will use the Cloud SDK to create a service account and then create credentials you will need to authenticate as the service account.

First, set an environment variable with your PROJECT_ID which you will use throughout this codelab:

export GOOGLE_CLOUD_PROJECT=$(gcloud config get-value core/project)

Next, create a new service account to access the Video Intelligence API by using:

gcloud iam service-accounts create my-video-int-sa \
  --display-name "my video intelligence codelab service account"

Next, create credentials that your C# code will use to login as your new service account. Create these credentials and save it as a JSON file "~/key.json" by using the following command:

gcloud iam service-accounts keys create ~/key.json \
  --iam-account  my-video-int-sa@${GOOGLE_CLOUD_PROJECT}.iam.gserviceaccount.com

Finally, set the GOOGLE_APPLICATION_CREDENTIALS environment variable, which is used by the Video Intelligence API C# library, covered in the next step, to find your credentials. The environment variable should be set to the full path of the credentials JSON file you created, by using:

export GOOGLE_APPLICATION_CREDENTIALS="/home/${USER}/key.json"

You can read more about authenticating the Video Intelligence API.

First, create a simple C# console application that you will use to run Video Intelligence API samples:

dotnet new console -n VideoIntApiDemo

You should see the application created and dependencies resolved:

The template "Console Application" was created successfully.
Processing post-creation actions...
...
Restore succeeded.

Next, navigate to VideoIntApiDemo folder:

cd VideoIntApiDemo/

And add Google.Cloud.VideoIntelligence.V1 NuGet package to the project:

dotnet add package Google.Cloud.VideoIntelligence.V1
info : Adding PackageReference for package 'Google.Cloud.VideoIntelligence.V1' into project '/home/atameldev/VideoIntApiDemo/VideoIntApiDemo.csproj'.
log  : Restoring packages for /home/atameldev/VideoIntApiDemo/VideoIntApiDemo.csproj...
...
info : PackageReference for package 'Google.Cloud.VideoIntelligence.V1' version '1.0.0' added to file '/home/atameldev/VideoIntApiDemo/VideoIntApiDemo.csproj'.

Now, you're ready to use Video Intelligence API!

Label analysis detects labels in a video either stored locally or in Google Cloud Storage. In this section, you will analyze a video for labels stored in Google Cloud Storage.

First, open the code editor from the top right side of the Cloud Shell:

9b8f365ab5ec7f71.png

Navigate to the Program.cs file inside the VideoIntApiDemo folder and replace the code with the following:

using System;
using System.Collections.Generic;
using Google.Cloud.VideoIntelligence.V1;

namespace VideoIntApiDemo
{
    class Program
    {
        static void Main(string[] args)
        {
            var client = VideoIntelligenceServiceClient.Create();
            var request = new AnnotateVideoRequest
            {
                InputUri = "gs://cloudmleap/video/next/gbikes_dinosaur.mp4",
                Features = { Feature.LabelDetection }
            };
            var op = client.AnnotateVideo(request).PollUntilCompleted();
            foreach (var result in op.Result.AnnotationResults)
            {
                PrintLabels("Video", result.SegmentLabelAnnotations);
                PrintLabels("Shot", result.ShotLabelAnnotations);
                PrintLabels("Frame", result.FrameLabelAnnotations);
            }
        }

        static void PrintLabels(string labelName,
            IEnumerable<LabelAnnotation> labelAnnotations)
        {
            foreach (var annotation in labelAnnotations)
            {
                Console.WriteLine($"{labelName} label: {annotation.Entity.Description}");
                foreach (var entity in annotation.CategoryEntities)
                {
                    Console.WriteLine($"{labelName} label category: {entity.Description}");
                }
                foreach (var segment in annotation.Segments)
                {
                    Console.Write("Segment location: ");
                    Console.Write(segment.Segment.StartTimeOffset);
                    Console.Write(":");
                    Console.WriteLine(segment.Segment.EndTimeOffset);
                    Console.WriteLine($"Confidence: {segment.Confidence}");
                }
            }
        }
    }
}

Take a minute or two to study the code and see how the video is being labelled*.*

Back in Cloud Shell, run the app:

dotnet run

It takes several seconds for Video Intelligence API to extract labels but eventually, you should see the following output:

Video label: bicycle
Video label category: vehicle
Segment location: "0s":"42.766666s"
Confidence: 0.475821
Video label: tyrannosaurus
Video label category: dinosaur
Segment location: "0s":"42.766666s"
Confidence: 0.4222222
Video label: tree
Video label category: plant
Segment location: "0s":"42.766666s"
Confidence: 0.4231415
...

Summary

In this step, you were able to list all the labels in a video using the Video Intelligence API. You can read more on Label detection page.

You can use the Video Intelligence API to detect shot changes in a video stored locally or in Google Cloud Storage. In this section, you will perform video analysis for shot changes on a file located in Google Cloud Storage.

To detect shot changes, navigate to the Program.cs file inside the VideoIntApiDemo folder and replace the code with the following:

using System;
using Google.Cloud.VideoIntelligence.V1;

namespace VideoIntApiDemo
{
    class Program
    {
        static void Main(string[] args)
        {
            var client = VideoIntelligenceServiceClient.Create();
            var request = new AnnotateVideoRequest
            {
                InputUri = "gs://cloudmleap/video/next/gbikes_dinosaur.mp4",
                Features = { Feature.ShotChangeDetection }
            };
            var op = client.AnnotateVideo(request).PollUntilCompleted();
            foreach (var result in op.Result.AnnotationResults)
            {
                foreach (var annotation in result.ShotAnnotations)
                {
                    Console.Out.WriteLine("Start Time Offset: {0}\tEnd Time Offset: {1}",
                        annotation.StartTimeOffset, annotation.EndTimeOffset);
                }
            }
        }
    }
}

Take a minute or two to study the code and see how the shot detection is performed.

Back in Cloud Shell, run the app. You should see the following output:

dotnet run

You should see the following output:

Start Time Offset: "0s" End Time Offset: "5.166666s"
Start Time Offset: "5.233333s"  End Time Offset: "10.066666s"
Start Time Offset: "10.100s"    End Time Offset: "28.133333s"
Start Time Offset: "28.166666s" End Time Offset: "42.766666s"

Summary

In this step, you were able to use the Video Intelligence API to detect shot changes in a file stored in Google Cloud Storage. Read more about Shot changes.

Explicit Content Detection detects adult content within a video. Adult content is content generally appropriate for 18 years of age and older, including but not limited to nudity, sexual activities, and pornography (including cartoons or anime). The response includes a bucketized likelihood value, from VERY_UNLIKELY to VERY_LIKELY.

When Explicit Content Detection evaluates a video, it does so on a per-frame basis and considers visual content only. The audio component of the video is not used to evaluate the explicit content level.

To detect explicit content, navigate to the Program.cs file inside the VideoIntApiDemo folder and replace the code with the following:

using System;
using Google.Cloud.VideoIntelligence.V1;

namespace VideoIntApiDemo
{
    class Program
    {
        static void Main(string[] args)
        {
            var client = VideoIntelligenceServiceClient.Create();
            var request = new AnnotateVideoRequest
            {
                InputUri = "gs://cloudmleap/video/next/gbikes_dinosaur.mp4",
                Features = { Feature.ExplicitContentDetection }
            };
            var op = client.AnnotateVideo(request).PollUntilCompleted();
            foreach (var result in op.Result.AnnotationResults)
            {
                foreach (var frame in result.ExplicitAnnotation.Frames)
                {
                    Console.WriteLine("Time Offset: {0}", frame.TimeOffset);
                    Console.WriteLine("Pornography Likelihood: {0}", frame.PornographyLikelihood);
                    Console.WriteLine();
                }
            }
        }
    }
}

Take a minute or two to study the code and see how the explicit content detection was performed*.*

Back in Cloud Shell, run the app:

dotnet run

It might take several seconds but eventually, you should see the following output:

dotnet run

Time Offset: "0.056149s"
Pornography Likelihood: VeryUnlikely

Time Offset: "1.166841s"
Pornography Likelihood: VeryUnlikely
...
Time Offset: "41.678209s"
Pornography Likelihood: VeryUnlikely

Time Offset: "42.596413s"
Pornography Likelihood: VeryUnlikely

Summary

In this step, you were able to perform explicit content detection on a video using the Video Intelligence API. Read more about Explicit content detection.

You learned how to use the Video Intelligence API using C#!

Clean up

To avoid incurring charges to your Google Cloud Platform account for the resources used in this quickstart:

  • Go to the Cloud Platform Console.
  • Select the project you want to shut down, then click ‘Delete' at the top: this schedules the project for deletion.

Learn More

License

This work is licensed under a Creative Commons Attribution 2.0 Generic License.