Integrating Dialogflow with Hangouts Chat

In this codelab, you'll learn how Dialogflow connects with G Suite APIs to create a fully functioning Appointment Scheduler with Google Calendar with dynamic responses in Hangouts Chat.

If you do not still have the set up from the previous codelabs or feel confident with Dialogflow and fulfillment, you may download the starter code and agent config from here

https://github.com/googlecodelabs/appointment-scheduler-hangouts

Prerequisites

Before proceeding, you need to complete the following codelabs:

  1. Build an Appointment Scheduler with Dialogflow
  2. Integrate Dialogflow with Actions on Google
  3. Understand entities in Dialogflow
  4. Understand fulfillment by integrating Dialogflow with Calendar

You also need to understand the basic concepts and constructs of Dialogflow, which you can glean from the following videos found in the Build a chatbot with Dialogflow pathway.

What you'll learn

  • How to deploy and configure a Hangouts Chat bot
  • How to build display cards in Hangouts Chat
  • How to build custom payloads in Dialogflow fulfillment

What you'll build

  • An integration between Dialogflow, Hangouts Chat and Calendar
  • (Optional) Integrate Google Maps for richer cards

5b4ca104638f9e8d.png

What you'll need

  • A web browser and an email address to log into the Dialogflow console
  • Calendar and Hangouts enabled in your G Suite domain

We will start with the Dialogflow Agent you have created in the previous codelabs.

  1. In the Dialogflow console, click d7d792687e597dd5.png.
  2. In the General tab, scroll to Project ID, then click Google Cloud f2bffd4fcdb84fa9.png.

34be16fcd4c5aeff.png

  1. In the Google Cloud Console, click Navigation menu ☰ > APIs & Services > Library.
  2. Search for "Hangouts Chat API," then click Enable to use the API on your Google Cloud project.
  3. Now we will configure your chatbot, navigate to the Configuration page for the API. Please note each GCP project can have at most one chatbot. 85794eaaaedc7eb5.png
  4. You should see the Dialogflow fields populating the configuration options.
  5. Change the Avatar URL field to this car image: **https://fonts.gstatic.com/s/i/googlematerialicons/directions_car/v5/black-48dp/1x/gm_directions_car_black_48dp.png**
  6. Enable your bot for both direct messages and rooms 9d439f492c8b71bb.png

Click Save and exit the Cloud Console.

Additionally Dialogflow Integrations. Hangouts Chat is enabled by default but if you would like a bot to serve multiple platforms, you may navigate to the Integrations page in the Dialogflow Console to enable them.

63296523b678ff8d.png

Now that your chatbot is configured, let's add it to a chat room and test it out. Open Hangouts Chat and create a test chat room. In the upper right corner of the room click the drop down and select Add people & bots.

f0dd1f5e205ef8e2.png

Search for AppointmentScheduler and add the bot to the room.

606dcb861f7f3144.png

You can now interact with the bot you already have built in Hangouts by simply typing @AppointmentScheduler in the room.

b7c73c64cf4c11.png

Test it out on this platform with the same input from the previous codelabs:

  1. User: "@AppointmentScheduler Set an appointment for vehicle registration at 2 PM tomorrow."
  2. Chatbot: "Ok, let me see if we can fit you in. April 24, 2 PM is fine!"

Next we will use the customizations in Hangouts to add richer responses.

With Hangouts Chat, you can have your bot return the user a basic Text Response or a Card Response which allows you a richer interface built by different widgets including images, buttons, etc. Now that we have connected your Dialogflow Agent to a Hangouts Chat bot, you simply need to return the JSON in the correct format to display in Hangouts in your fulfillment code. Let's look at some JSON examples.

The basic text response looks like:

 {
  "text": "Your pizza delivery is here!"
}

7e1cd3efb4fe9d1f.png

A sample card response with widgets looks like:

{
  "cards": [
    {
      "sections": [
        {
          "widgets": [
            {
              "image": { "imageUrl": "https://..." }
            },
            {
              "buttons": [
                {
                  "textButton": {
                    "text": "OPEN IN GOOGLE MAPS",
                    "onClick": {
                      "openLink": {
                        "url": "https://..."
                      }
                    }
                  }
                }
              ]
            }
          ]
        }
      ]
    }
  ]
}

72ac1b2903ed60b6.png

To learn more about cards please see the Message Format documentation. In the next section we will add cards to our chatbot.

Custom payloads in Dialogflow allow for platform-specific rich response messages. This is where we will add our Hangout Chat JSON cards where they will be served back to the user by the agent.

Let's start by adding a basic card for the welcome intent. In the Dialogflow Console, navigate to the Default Welcome Intent and scroll down to the responses section.

9624208f0d266afd.png

Click on Hangouts and unselect Use responses from the DEFAULT tab as the first responses, then ADD RESPONSES > Custom Payload.

You will see a JSON skeleton.

bb064f7ec1237fd3.png

Copy and paste the following code below. We have set up a card with a TextParagraph widget.

{
 "hangouts": {
   "header": {
     "title": "Appointment Scheduler"
   },
   "sections": [
     {
       "widgets": [
         {
           "textParagraph": {
             "text": "Welcome, I am the scheduler agent. Please request an appointment type and time."
           }
         }
       ]
     }
   ]
 }
}

Hit Save then let's go to your test chat room to see this card is rendered. In the chat room, type ‘@AppointmentScheduler hello'

91af6321f3a26e19.png

Next we will add a custom payload in the fulfillment code so we can dynamically generate the content with our code.

Now we will create a card with several widgets to show a scheduled appointment. Let's add a function called getHangoutsCard where we will render the inputs: appointmentType, date, and time.

Add the function below to your code in index.js at the bottom.

function getHangoutsCard(appointmentType, date, time) {
   const cardHeader = {
     title: 'Appointment Confirmation',
     subtitle: appointmentType,
     imageUrl: ICON_IMAGE_URL,
     imageStyle: 'IMAGE',
   };
    const dateWidget = {
     keyValue: {
       content: 'Date',
       bottomLabel: date,
     },
   };
    const timeWidget = {
     keyValue: {
       content: 'Time',
       bottomLabel: time,
     },
   };

    const buttonWidget = {
     buttons: [
       {
         textButton: {
           text: 'View Appointment',
           onClick: {
             openLink: {
               url: CALENDAR_URL,
             },
           },
         },
       },
     ],
   };
    const infoSection = {widgets: [dateWidget, timeWidget,
     buttonWidget]};
    return {
     'hangouts': {
       'name': 'Confirmation Card',
       'header': cardHeader,
       'sections': [infoSection],
     },
   };
 }
 

This card contains a key value pair and button widget. The widgets are then ordered in a section and the card contains a list of sections to render.

Now let's call this function when a calendar event is created. In your createCalendarEvent function, replace the contents with the code below and deploy the function.

return createCalendarEvent(dateTimeStart, dateTimeEnd, appointmentType).then(() => {
   agent.add(`Ok, let me see if we can fit you in. ${appointmentTimeString} is fine!.`);

   const dateString = dateTimeStart.toLocaleString(
     'en-US',
       {month: 'long', day: 'numeric'},
   );
   const dateParts = appointmentTimeString.split(',');
   const json = getHangoutsCard(appointmentType, dateParts[0], dateParts[1]);
   const payload = new Payload(
       'hangouts',
       json,
       {rawPayload: true, sendAsMessage: true},
   );
   agent.add(payload);

 }).catch(() => {
   agent.add(`I'm sorry, there are no slots available for ${appointmentTimeString}.`);
 });
}

When the agent is responding in hangouts, the agent will know to return the card constructed above. Test it out with a scheduling request.

If you would like to make your response card more visual, you can add a static map of the appointment location generated by the Maps Static API. The API allows you to construct a Maps image by URL parameters. We can then use the image in an image widget in our card.

5b4ca104638f9e8d.png

  1. Enable the Maps Static API in the Google Cloud Console, the same way you enabled the Calendar and Hangouts APIs.
  2. Create and save an API key for your project in the APIs & Services page in the console. For more information on retrieval and best practices on API keys with Maps, please see here. c0238236d3c709c5.png
  3. Copy the API key and the following constants to the top of your file in index.js
const API_KEY = 'YOUR_API_KEY';
const MAP_IMAGE_URL = 'https://maps.googleapis.com/maps/api/staticmap?center=Googleplex&zoom=14&size=200x200&key=' + API_KEY;
const ICON_IMAGE_URL = 'https://fonts.gstatic.com/s/i/googlematerialicons/calendar_today/v5/black-48dp/1x/gm_calendar_today_black_48dp.png';
const CALENDAR_URL = 'YOUR_CALENDAR_URL';
  1. Next add an image widget in your getHangoutsCard function
  const mapImageWidget = {
    'image': {
      'imageUrl': MAP_IMAGE_URL,
      'onClick': {
        'openLink': {
          'url': MAP_IMAGE_URL,
        },
      },
    },
  };
  1. Add mapImageWidget into the list of widgets in your infoSection variable, save and deploy.
  2. Test out your bot in your chat room and you should a rich response based on the code you added today!

If you're planning to complete other Dialogflow codelabs, then skip this section for now and return to it later.

Delete the Dialogflow agent

  1. Click dc4ac6f9c0ae94e9.png next to your existing agent.

520c1c6bb9f46ea6.png

  1. In the General tab, scroll to the bottom and click Delete This Agent.
  2. Type Delete into the dialog and click Delete.

You created a chatbot in Dialogflow and integrated it with Calendar and now Hangouts, you're a G Suite Pro!

Learn more

To learn more, check out the code samples on the Dialogflow Github page.