Spotted a bug? Have a great idea? Help us make google.dev great!

In this codelab you will learn how to integrate Dialogflow with Vision API to provide rich and dynamic ML based responses to user provided image inputs. You will create a chatbot application that will take an image as input, process it in Vision API and and return the identified landmark to the user.

  • You will create a Dialogflow agent
  • Implement a django front-end to upload a file
  • Implement Dialogflow fulfillment to invoke vision API against the uploaded image.

What you'll learn

  • Create a Dialogflow Agent
  • Update Dialogflow Agent to accommodate the file uploads.
  • How to set up Vision API connection with Dialogflow fulfillment.
  • How to setup and run Django front-end app for Dialogflow
  • How to deploy the Django front-end app to Google Cloud on App Engine
  • How to test Dialogflow application from a custom front-end

Prerequisites

  • Basic concepts and constructs of Dialogflow. For introductory Dialogflow tutorial videos that covers basic conversational design check out the following videos:
  • Build an Appointment Scheduler Chatbot using Dialogflow.
  • Understanding Entities (Video to be added here)

We are creating a new conversation experience with a custom Django front-end and extending in further to integrate with Vision API. We will build the front-end with Django-framework, run and test it locally and then deploy it on Google App Engine. The front-end will look like this:

The request flow will be:

  1. The user will send a request via this front-end.
  2. This will trigger a call to the Dialogflow DetectIntent API to map the user's utterance to the right intent.
  3. Once the "Explore Landmark" intent is detected, Dialogflow fulfillment will then send a request to the Vision API, receive a response back and send it back to the user.

Overall architecture:

Google cloud Vision API is a pre trained Machine Learning model that helps derive insights from images. You can get insights including image labeling, face and landmark detection, optical character recognition (OCR), and tagging of explicit content. Here is the link to learn more specifically about the Vision API.

  1. Go to the Dialogflow Console.
  2. Sign-in, if you are a first time user, then use your email to sign up
  3. Accept the terms and conditions and you will be in the console
  4. Create an Agent. To create, click on the dropdown menu in the left pane in order to see "Create new agent"

  1. Call this "VisionAPI"
  2. Dialogflow creates a Google Cloudproject for you to access logs, cloud functions etc. You can select an existing project as well.
  3. When you're ready, click Create.
  4. Dialogflow creates two default intents as a part of the agent.
  1. Default Welcome intent helps greet your users
  2. Default Fallback intent helps catch all the questions that your bot does not understand.
  1. At this point, we have a functional bot that greets the users. But we need to update it slightly to let the user know that they can upload an image to explore landmarks.

Update Default Welcome Intent to notify the user to upload image

  1. Click on "Default Welcome Intent"
  2. Update the "Response" to "Hi! You can upload a picture to explore landmarks."

Create Entity

  1. Click on "Entities"

  1. Create a new entity, name it "filename" and "save".

Create new Intent

  1. Click on "Intents"
  2. Name it "Explore uploaded image"
  3. Click on "Training phrases" and add "file is demo.jpg" and "file is taj.jpeg" with @filename as the entity.

  1. Click on "Responses" and add "Assessing file" as the Text Response
  2. Click on "Fulfillment" and toggle "Enable webhook call for this intent"

  1. Navigate to Dialogflow Agent "VisionAPI" and click on "Fulfillment"
  2. Enable inline code editor by toggling the switch.

  1. Update the index.js file with the following code and update YOUR-BUCKET-NAME with the name of your storage bucket.
'use strict';

const functions = require('firebase-functions');
const {google} = require('googleapis');
const {WebhookClient} = require('dialogflow-fulfillment');
const vision = require('@google-cloud/vision');
  /**
   * TODO(developer): Uncomment the following lines before running the sample.
   */
const bucketName = 'YOUR-BUCKET-NAME';
const timeZone = 'America/Los_Angeles';
const timeZoneOffset = '-07:00';

exports.dialogflowFirebaseFulfillment = functions.https.onRequest((request, response) => {
  const agent = new WebhookClient({ request, response });
  console.log("Parameters", agent.parameters);

  function applyML(agent){
    const filename = agent.parameters.filename;
    console.log("filename is: ", filename);

    // call vision API to detect text
    return callVisionApi(agent, bucketName, filename).then(result => {
                      console.log(`result is ${result}`);
                      agent.add(`file is being processed, here are the results:  ${result}`);
            //agent.add(`file is being processed ${result}`);
        }).catch((error)=> {
            agent.add(`error occurred at apply ml function`  + error);
        });
  }

  let intentMap = new Map();
  intentMap.set('Explore uploaded image', applyML);
  agent.handleRequest(intentMap);
});


async function callVisionApi(agent, bucketName, fileName){
    // [START vision_text_detection_gcs]
  // Imports the Google Cloud client libraries
  // Creates a client
  
  const client = new vision.ImageAnnotatorClient();
    try {
        // Performs text detection on the gcs file
        const [result] = await client.landmarkDetection(`gs://${bucketName}/${fileName}`);
        const detections = result.landmarkAnnotations;
        var detected = [];
        detections.forEach(text => {
            console.log(text.description);
            detected.push(text.description);
        });
        return detected;
    }
    catch(error) {
        console.log('fetch failed', error);
        return [];
    }
}
  1. Click package.json and paste the following to replace its contents.
{
  "name": "dialogflowFirebaseFulfillment",
  "description": "Dialogflow fulfillment for the bike shop sample",
  "version": "0.0.1",
  "private": true,
  "license": "Apache Version 2.0",
  "author": "Google Inc.",
  "engines": {
    "node": "6"
  },
  "scripts": {
    "lint": "semistandard --fix \"**/*.js\"",
    "start": "firebase deploy --only functions",
    "deploy": "firebase deploy --only functions"
  },
  "dependencies": {
    "firebase-functions": "2.0.2",
    "firebase-admin": "^5.13.1",
    "actions-on-google": "2.2.0", 
    "googleapis": "^27.0.0",
    "dialogflow-fulfillment": "^0.6.1",
    "@google-cloud/bigquery": "^1.3.0",
    "@google-cloud/storage": "^2.0.0",
    "@google-cloud/vision": "^0.25.0"
  }
}
  1. Click Deploy at the bottom of the page.
  1. Clone this repository to your local machine:
https://github.com/priyankavergadia/visionapi-dialogflow.git
  1. Change to the directory that contains the code/. Alternatively, you can download the sample as a zip and extract it.
cd visionapi-dialogflow

When deployed, your app uses the Cloud SQL Proxy that is built in to the App Engine environment to communicate with your Cloud SQL instance. However, to test your app locally, you must install and use a local copy of the Cloud SQL Proxy in your development environment.

Learn more about the Cloud SQL Proxy here.

To perform basic admin tasks on your Cloud SQL instance, you can use the MySQL client.

Install the Cloud SQL Proxy

Download and install the Cloud SQL Proxy. The Cloud SQL Proxy is used to connect to your Cloud SQL instance when running locally.

Download the proxy:

curl -o cloud_sql_proxy https://dl.google.com/cloudsql/cloud_sql_proxy.darwin.amd64

Make the proxy executable:

chmod +x cloud_sql_proxy

Create a Cloud SQL instance

  1. Create a Cloud SQL for MySQL Second Generation instance. Name the instance "polls-instance" or similar. It can take a few minutes for the instance to be ready. After the instance is ready, it should be visible in the instance list. Be sure to create a Second Generation instance.
  2. Now use the Cloud SDK to run the following command where [YOUR_INSTANCE_NAME] represents the name of your Cloud SQL instance. Make a note of the value shown for connectionName for the next step. The connectionName value is in the format [PROJECT_NAME]:[REGION_NAME]:[INSTANCE_NAME]
gcloud sql instances describe [YOUR_INSTANCE_NAME]

Alternatively, you can get the connectionName from the console by clicking on the instance

Initialize your Cloud SQL instance

Start the Cloud SQL Proxy using the connectionName from the previous step.

./cloud_sql_proxy -instances="[YOUR_INSTANCE_CONNECTION_NAME]"=tcp:3306

Replace [YOUR_INSTANCE_CONNECTION_NAME] with the value of connectionName that you recorded in the previous step.

This step establishes a connection from your local computer to your Cloud SQL instance for local testing purposes. Keep the Cloud SQL Proxy running the entire time you test your app locally.

Next you create a new Cloud SQL user and database.

  1. Create a new database using the Google CloudConsole for your Cloud SQL instance polls-instance. For example, you can use the name polls.
  2. Create a new user using the Google CloudConsole for your Cloud SQL instance polls-instance.

Configure the database settings

  1. Open mysite/settings-changeme.py for editing.
  2. Rename the file to setting.py
  3. In two places, replace [YOUR-USERNAME] and [YOUR-PASSWORD] with the database username and password you created previously in the step "Create a Cloud SQL instance". This helps set up the connection to the database for both App Engine deployment and local testing.
  4. In line ‘HOST': ‘cloudsql/ [PROJECT_NAME]:[REGION_NAME]:[INSTANCE_NAME]' replace [PROJECT_NAME]:[REGION_NAME]:[INSTANCE_NAME] with your instance name acquired in previous step.
  5. Run the following command. Copy the outputted connectionName value for the next step.
gcloud sql instances describe [YOUR_INSTANCE_NAME]
  1. Replace [YOUR-CONNECTION-NAME] with connectionName from the previous step
  2. Replace [YOUR-DATABASE] with the name you choose during the "Initialize your Cloud SQL instance" step
# [START db_setup]
if os.getenv('GAE_APPLICATION', None):
    # Running on production App Engine, so connect to Google Cloud SQL using
    # the unix socket at /cloudsql/<your-cloudsql-connection string>
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.mysql',
            'HOST': '/cloudsql/[PROJECT_NAME]:[REGION_NAME]:[INSTANCE_NAME]',
            'USER': '[YOUR-USERNAME]',
            'PASSWORD': '[YOUR-PASSWORD]',
            'NAME': '[YOUR-DATABASE]',
        }
    }
else:
    # Running locally so connect to either a local MySQL instance or connect to
    # Cloud SQL via the proxy. To start the proxy via command line:
    #     $ cloud_sql_proxy -instances=[INSTANCE_CONNECTION_NAME]=tcp:3306
    # See https://cloud.google.com/sql/docs/mysql-connect-proxy
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.mysql',
            'HOST': '127.0.0.1',
            'PORT': '3306',
            'NAME': '[YOUR-DATABASE]',
            'USER': '[YOUR-USERNAME]',
            'PASSWORD': '[YOUR-PASSWORD]'
        }
    }
# [END db_setup]
  1. Close and save settings.py.
  1. In Dialogflow's console, go to settings ⚙ and under the general tab, you'll see the Google Project section. Click on the service account. This will open the Google Cloud Console.
  2. In the Google Cloudconsole, the presented webpage will show the Dialogflow service account. Click on the 3 dots in the "Actions" section to the far right and then click "Create Key"

  1. A JSON file will be downloaded to your computer that you will need in the setup sections below.
  1. Inside chat folder, replace the key-sample.json with your own credentials json file and call it key.json.
  2. In views.py in chat folder, Change the GOOGLE_PROJECT_ID = "<YOUR_PROJECT_ID>" to your project ID

Create GCS bucket for front-end static objects

  • Navigate to the Google CloudProject and click on Storage from the hamburger menu

  • Click on "Create New Bucket"
  • Provide a name - this has to be a globally unique name

  • Choose where to store your data - Pick "Regional" and select the location that best suits your needs.
  • Choose default storage class of your data as "Standard"

  • Choose how to control access to objects as "Set permissions uniformly at bucket-level" and then continue to create the bucket.

  • Once the bucket is created, click on the "Browser" and locate the bucket you just created.

  • Click on the three dots on the right hand side corresponding to the bucket and click "Edit bucket permissions"

  • In the presented side panel, click on "Add Members" and then add a new member "allUsers", then click on "Select a role" to add "Storage Object Viewer" role. We do this to provide view access to the static front end files to allUsers. This is not an ideal security setting for the files but for the purpose of this particular lab it will work.

Create GCS bucket for user uploaded images

Follow the same instructions and create a separate bucket to upload user images.

Set permissions to "allUsers" and role "Storage Object Creator" and "Storage Object Viewer"

Configure GCS bucket in settings.py

  • Open mysite/setting.py
  • Locate GCS_BUCKET variable and replace the "<YOUR-GCS-BUCKET-NAME>" with your GCS static bucket
  • Locate GS_MEDIA_BUCKET_NAME variable and replace the "<YOUR-GCS-BUCKET-NAME-MEDIA>" with your GCS bucket name for the images.
  • Locate GS_STATIC_BUCKET_NAME variable and replace the "<YOUR-GCS-BUCKET-NAME-STATIC>" with your GCS bucket name for the static files.
  • Save the file.
GCS_BUCKET = '<YOUR-GCS-BUCKET-NAME>'
GS_MEDIA_BUCKET_NAME = '<YOUR-GCS-BUCKET-NAME-MEDIA>'
GS_STATIC_BUCKET_NAME = '<YOUR-GCS-BUCKET-NAME-STATIC>'

Configure GCS bucket in home.html

  • Open chat -> templates -> home-changeme.html
  • Rename it to home.html
  • Look for <YOUR-GCS-BUCKET-NAME-MEDIA> and replace it with your bucket name for where you would like the user uploaded file to be saved. We do this so we don't store the user uploaded file in the front-end and keep the static assets all in GCS bucket. The Vision API calls the GCS bucket to pick up the file and do the prediction.

To run the Django app on your local computer, you'll need to set up a Python development environment, including Python, pip, and virtualenv. For instructions, refer to Setting Up a Python Development Environment for Google Cloud Platform.

  • Create an isolated Python environment, and install dependencies:
virtualenv env
source env/bin/activate
pip install -r requirements.txt
  • Run the Django migrations to set up your models:
python3 manage.py makemigrations
python3 manage.py makemigrations polls
python3 manage.py migrate
  • Start a local web server:
python3 manage.py runserver
  • In your web browser, enter this address http://localhost:8000/ You should see a simple webpage with the text: "Dialogflow" a textbox and submit button.

The sample app pages are delivered by the Django web server running on your computer. When you're ready to move forward, press Ctrl+C to stop the local web server.

Use the Django admin console

  • Create a superuser:
python3 manage.py createsuperuser
  • Start a local web server:
python3 manage.py runserver
  • Enter this address http://localhost:8000/admin/ in your web browser. To log on to the admin site, use the username and password you created when you ran createsuperuser.

Gather all the static content into one folder. This command moves all of the app's static files into the folder specified by STATIC_ROOT in settings.py:

python3 manage.py collectstatic

Upload the app by running the following command from within the directory of the application where the app.yaml file is located:

gcloud app deploy

Wait for the message that notifies you that the update has been completed.

In your web browser, enter this address:

https://<your_project_id>.appspot.com

This time, your request is served by a web server running in the App Engine standard environment.

This command deploys the application as described in app.yaml and sets the newly deployed version as the default version, causing it to serve all new traffic.

When you are ready to serve your content in production, make the following configuration change:

In mysite/settings.py, change the DEBUG variable to False.

Let's test our chatbot, you can test it in the simulator or use the web or google home integration we have learnt in previous codelabs.

  1. User: "hi"
  2. Chatbot response: "Hi! You can upload a picture to explore landmarks."
  3. User: uploads an image.
  4. Download this image, name it "demo.jpg" and use this.

  1. Chatbot response: "file is being processed, here are the results: Golden Gate Bridge,Golden Gate National Recreation Area,Golden Gate Bridge,Golden Gate Bridge,Golden Gate Bridge"
  2. Overall, it should look like this.

If you are planning on doing the other labs in this series, don't do the cleanup now, do it after you are done with all the labs in the series.

Delete the Dialogflow Agent

  • Click on the gear icon next to your existing agent

  • In the General tab scroll down to the bottom and click Delete this Agent.
  • Type DELETE into the window that appears and click Delete.

You created a chatbot in Dialogflow and integrated it with Google Calendar.. You're now a chatbot developer!

Check out these other resources: