How to use Gemini to explore and enhance an existing application

1. Overview

This Codelab will demonstrate how to use Gemini to build a client application on top of an existing API backend.

What you will learn

This lab is intended to illustrate using Gemini with an existing application. In this lab, you will learn how to do the following:

• Deploy the AI/ML Image Processing Jump Start Solution
• Use Gemini to explore the API application
• Use Gemini to make a client for the API application

Setup - Create a new project

Depending on your organization policies, you may not have the ability to correctly deploy this solution.

Recommendation: Use an account where you have full permissions (like a personal account) instead of a company or school account

Note: If you recently created a project where you have full permissions and you already have the Gemini API enabled on that project, then you can continue using that project and skip this step. For example, if you have recently completed another Gemini codelab, you can skip this step.

1. Go to the Google Cloud Console
2. Click the current project dropdown.
3. Click New Project
4. Add a project name and click CREATE
   • Recommendation: Use a unique project name so that the Project name and Project ID are the same

2. Deploy the Jump Start Solution

Jump Start Solutions are pre-built solution templates from Google Cloud. This lab uses a Jump Start Solution to demonstrate how Gemini can help enhance a pre-existing application.

1. Go to the AI/ML Image Processing Jump Start Solution details page
   • Click the hamburger menu
   • In the Products & solutions submenu, click Jump Start Solutions
   • Click AI/ML Image Processing on Cloud Functions
2. Click DEPLOY
   • If your project does not have billing enabled, enable billing.
   • Confirm the project name and click CONTINUE
   • Select us-central1 (Iowa) as the region.
   • If prompted for other input, accept the defaults.
   • Click DEPLOY.
   • This can take up to 15 minutes.
   • You do not need to make any changes, but feel free to explore the Jump Start Solution deployment.

Troubleshooting

1. Problem: Do you see an error that says Partial deployment.?
   • Solution: Unfortunately, this means you should likely delete the deployment and start over. There should be a button that says DELETE DEPLOYMENT. Click that button and then redeploy the Jump Start Solution.

3. Enable Gemini

Set up your code editing environment and enable Gemini

1. Go to the Cloud Shell Editor
   • Return to the Google Cloud Console
   • Type Cloud Shell Editor in the search bar and click Cloud Shell Editor from the dropdown.
   • Click Try the new Editor if you are using the Legacy Editor.
     • Note: If you do not see the Try the new Editor button, you are likely already using the new Editor. The Legacy Editor will work, but the views will look slightly different.
   • Click the Open in New Window icon
2. If prompted, log in or authorize your account.
   • Depending on your settings, you may be prompted to authorize your account several times during this lab, when asked, select authorize each time.
3. Click on the Cloud Code - Sign in button in the bottom status bar as shown. Authorize the plugin as instructed.
4. If prompted, click Select a Google Cloud project then select the project you are using
5. Enable Gemini
   • Click Cloud Code
     • Note: Depending on the size of your screen, it may take one step or two steps.
   • Click Change Settings in the HELP AND FEEDBACK section
   • Type Gemini: Enable
   • Select the checkbox (if it is not already selected)
   • Reload your IDE.
   • This enables Gemini in Cloud Code, and the Gemini status bar appears in your IDE.
6. Click on the Gemini button in the bottom right corner as shown and select the correct Google Cloud project.
   • If you see an error that Gemini has not been enabled for your selected project. Click Enable Gemini API
7. Confirm that you are able to see the correct Google Cloud project on the left (you will see your project name) and Gemini enabled on the right, in the status bar as shown below:

4. Download the Cloud Function code

Download the code and use Gemini to learn more about Jump Start Solutions and the code you deployed.

1. If you have navigated away, click Cloud Code
2. Click Cloud Functions
   • If you see a Select a project button, click it, and select the project you are working with.
   • Click the annotate-http function
   • Click the Download to new workspace icon
3. Use annotate-http-1 as the workspace name (should be the default) and click OK
4. Open the main.py file
   • Click the Explorer icon
   • Click the main.py file to open it
5. Click the Gemini Chat icon
   • Note: Depending on the size of your screen, it may take one step or two steps.
6. Try asking Gemini a question. Enter a prompt like:
   • What is a Jump Start Solution in Google Cloud?
   • Explain this code.

Resetting the Gemini Chat

Prompts impact future prompts. For example, if you make a typo in the project id or region, those will carry forward in future prompts and responses, which can negatively impact your experience. If you have made a typo or would like to clear the history for any other reason, you can use the trash can icon to reset the chat.

5. Calling the existing Cloud Function endpoint

Explore the code and call the endpoint you deployed in the Jump Start Solution.

1. Enter the prompt:
   • IMPORTANT Replace my-project-with-duet in the following prompt with your project id.
   • The region is us-central1 and the project id is my-project-with-duet. Write a curl post for the /annotate-http/annotate Cloud Function endpoint that performs label detection on the default eiffel tower image.
   • You should receive a response that includes a curl command like this:

     curl -X POST \
         -H "Authorization: Bearer $(gcloud auth print-access-token)" \
     ...
     

   • Copy this curl command for use in the next step
2. Test out the command in a new terminal
   • Click the hamburger menu
   • Click Terminal
   • Click New Terminal
   • Paste the curl command from the previous step into the terminal and press Enter to run the command
   • You should receive a response that starts like this:

     {
         "labelAnnotations": [
         {
         "mid": "/m/0csby",
         "description": "Cloud",
         "score": 0.97885925,
         "topicality": 0.97885925,
     ...
     

3. (Optional) Do this with another image from the public internet by prompting Duet with something like:
   • Do the same thing with the image https://storage.googleapis.com/cloud-samples-data/vision/eiffel_tower.jpg

Troubleshooting

Note about non-determinism: Gemini and other products powered by Large Language Models (LLMs) are non-deterministic: meaning given the same input, you may get different outputs. Even if you follow the steps perfectly, you may not get functioning commands. These troubleshooting steps should help work through those.

The curl command generated should look something like this:

curl -X POST \
    -H "Authorization: Bearer $(gcloud auth print-access-token)" \
    -H "Content-Type: application/json" \
    -d '{
  "image_uri": "gs://cloud-samples-data/vision/eiffel_tower.jpg",
  "features": "LABEL_DETECTION"
}' \
    "https://us-central1-my-project-with-duet.cloudfunctions.net/annotate-http/annotate"

Gemini is using the TEST_IMAGE in the main.py file, which is an image of the Eiffel Tower.

If the curl command isn't working for you, here are a few things to check:

1. Problem: Does the URL endpoint include .a.run.app?
   • Solution: Duet may think you are deploying to Cloud Run. Replace my-project-with-duet with your project id in https://us-central1-my-project-with-duet.cloudfunctions.net/annotate-http/annotate to get the correct Cloud Functions endpoint and use that for the curl command.
2. Problem: Does the URL endpoint include my-project-with-duet?
   • Solution: Replace my-project-with-duet with your project id.
3. Problem: Is the URL endpoint missing us-central1?
   • Solution: Make sure the URL starts with us-central1 like: https://us-central1-my-project-with-duet.cloudfunctions.net/annotate-http/annotate with your project id.
4. Problem: Depending on the prompt used, sometimes the endpoint generated will look like cloudfunctions.net/annotate-http or cloudfunctions.net/annotate. Is the URL endpoint missing /annotate-http/annotate at the end?
   • Solution: Make sure the URL endpoint ends with the full API application path cloudfunctions.net/annotate-http/annotate
5. Problem: The cURL request works, but it is not getting much information back. Does the request include features?
   • Solution: This is not a major problem. If you continue without changing anything the rest of the lab will still work. If you'd like to get more information back, you can ask Duet to add features to the cURL request.
6. Problem: The answers seem incorrect or seem to lack code-specific context. Are you using the Gemini Chat within Cloud Shell Editor?
   • Solution: Try opening the main.py file and highlighting the code related to your prompt while you use the Gemini Chat within Cloud Shell Editor. The Gemini assistant throughout Google Cloud has different context, which will lead to different answers.
7. Problem: The prompt response still isn't quite right.
   • Solution: Try a different prompt where you give Gemini even more direction around what you would like it to produce. Be sure to replace my-project-with-duet with your project id. Here is an example:

     Give me the curl command for:
     
     1- Cloud Function running in us-central1 and project my-project-with-duet
     2- HTTP POST to the following endpoint: /annotate-http/annotate endpoint
     3- JSON in the body: { "image_uri": "gs://cloud-samples-data/vision/eiffel_tower.jpg", "features": "LABEL_DETECTION" }
     

8. Problem: After all these troubleshooting steps, it still isn't working.
   • Solution: Because of the non-deterministic nature of LLMs, it's possible that Duet generated a response that won't work. If that happens, copy the example curl command above and try running that in the terminal. Be sure to replace my-project-with-duet with your project id.

6. Make a web application client for the API

Use Gemini to generate an index.html file that calls the Jump Start Solution Cloud Function endpoint. Then deploy the index.html file to Firebase Hosting to see the code in action.

1. Generate the client-side code
   • Back in the Gemini chat box, enter the prompt:
     • Write an index.html file that takes two inputs: a text input for the image_uri and a dropdown for the features. It should make a post request and display the raw json result to the DOM. The fetch call should use the full url beginning with https. The default image_uri should be gs://cloud-samples-data/vision/eiffel_tower.jpg.
   • Copy the index.html file generated in the response.
     • NOTE: You will need this code later, so save it somewhere if you are worried you might lose it.
   • Close main.py.
2. Create a new frontend directory
   • You can ask Gemini something like:
     • What are the console commands to create and navigate to a new frontend directory?
   • You should see results like mkdir frontend and cd frontend.
   • Use the Cloud Shell Terminal to confirm you are still in the annotate-http-1 directory and run those commands:
     • cd ~/annotate-http-1
     • mkdir frontend
     • cd frontend
3. Confirm you are logged in to the correct project in the Cloud Shell terminal
   • You can ask Gemini something like:
     • What is the gcloud command to set my project?
   • You should see a result like gcloud config set project my-project-with-duet
     • Confirm the project id is correct, then run the command in the Cloud Shell terminal.
4. Enable Firebase
   • In a new tab (leave your current Cloud Shell Editor open), visit https://console.firebase.google.com/
   • Click Add Project
   • Type your project id and wait for it to appear in the dropdown
   • Click your project id
   • Click Continue
   • Click Confirm plan
   • Click Continue
   • Deselect Enable Google Analytics for this project
     • Firebase does recommend using Google Analytics, but they will not be used in this application.
   • Click Continue
   • Close this tab and return to Cloud Shell Editor
   • Note: These steps are only needed for users who have not previously used Firebase, but this allows everyone to follow the same path. If you do this again in the future, you can skip this step.
5. Log in to Firebase in Cloud Shell
   • In the Cloud Shell Editor terminal, run the command firebase login --no-localhost
   • Visit the URL provided
     • Click Yes, I just ran this command
     • Click Yes, this is my session ID
     • Click Copy
   • Return to the Cloud Shell Editor terminal
   • Paste the copied value in the terminal where it says: Enter authorization code:
   • Press Enter
6. Deploy a new website to Firebase Hosting
   • Initialize a new firebase project
     • You can ask Gemini something like:
       • What are the commands to initialize and deploy a new firebase application?
     • You should see results like firebase init and firebase deploy with instructions.
     • Since there are many possible ways to do this that Gemini might recommend depending on your preferences, here is a specific recommendation for you to follow:
       • firebase init
         • Arrow down to Hosting: Configure files for Firebase Hosting and (optionally) set up GitHub Action deploys
           • Press the Space bar
           • Press Enter
         • Arrow down to Use an existing project
           • Press Enter
         • If you see a list of projects, arrow down to the project you are using today
           • Press Enter
         • If you are asked to Please input the ID of the Google Cloud Project you would like to add Firebase:
           • Type the project ID for your project.
           • Press Enter
         • For all remaining prompts, press Enter
       • firebase deploy
7. Re-deploy your application with a call to the Cloud Functions API endpoint
   • The previous commands should create a public directory with an index.html file inside of the frontend directory you created earlier. The ~/annotate-http-1/frontend/public/index.html file is where you should make the next few changes.
   • Open the index.html file.
     • Click the Explorer icon
     • Click the frontend folder
     • Click the public folder
     • Click the index.html file
   • Empty out the existing index.html file.
   • Paste the code you copied earlier into the index.html file
   • Run firebase deploy to deploy the new application
   • Visit the firebase Hosting URL to see if it works!
     • Note: You will likely need to do a hard refresh of the website to see your updates.

Troubleshooting

The index.html file should look something like this:

<!DOCTYPE html>
<html>
<head>
<title>Vision API Annotate Image</title>
</head>
<body>
<h1>Vision API Annotate Image</h1>
<form id="annotate-form" action="https://us-central1-my-project-with-duet.cloudfunctions.net/annotate-http/annotate" method="POST">
    <label for="image-uri">Image URI:</label>
    <input type="text" id="image-uri" name="image_uri" value="gs://cloud-samples-data/vision/eiffel_tower.jpg">
    <br>
    <label for="features">Features:</label>
    <select id="features" name="features">
    <option value="LABEL_DETECTION">LABEL_DETECTION</option>
    <option value="TEXT_DETECTION">TEXT_DETECTION</option>
    <option value="IMAGE_PROPERTIES">IMAGE_PROPERTIES</option>
    </select>
    <br>
    <input type="submit" value="Annotate">
</form>

<div id="result"></div>

<script>
    const form = document.getElementById('annotate-form');

    form.addEventListener('submit', (event) => {
    event.preventDefault();

    const imageUri = document.getElementById('image-uri').value;
    const features = document.getElementById('features').value;

    const data = {
        image_uri: imageUri,
        features: features,
    };

    fetch('https://us-central1-my-project-with-duet.cloudfunctions.net/annotate-http/annotate', {
        method: 'POST',
        headers: {
        'Content-Type': 'application/json',
        },
        body: JSON.stringify(data),
    })
        .then((response) => response.json())
        .then((result) => {
        const resultElement = document.getElementById('result');
        resultElement.innerHTML = JSON.stringify(result, null, 2);
        });
    });
</script>
</body>
</html>

If the website isn't working for you, here are a few things to check:

1. Problem: 404 (Not Found) Does the URL used in the fetch method include my-project-with-duet?
   • Solution: Replace my-project-with-duet with your project id.
2. Problem: 404 (Not Found) Is the URL used in the fetch method missing the full URL?
   • Solution: Be sure to include the full URL in the fetch method. It should look something like https://us-central1-my-project-with-duet.cloudfunctions.net/annotate-http/annotate
3. Problem: The prompt response still isn't quite right.
   • Solution: Try a different prompt where you give Gemini even more direction around what you would like it to produce. Be sure to replace my-project-with-duet with your project id. Here is an example:
     • Write an index.html file that takes two inputs: a text input for the image_uri and a dropdown for the features. It should make a post request and display the raw json result to the DOM. The fetch call should use https://us-central1-my-project-with-duet.cloudfunctions.net/annotate-http/annotate. The default image_uri should be gs://cloud-samples-data/vision/eiffel_tower.jpg.
4. Problem: Is your site deployed, but it isn't working?
   • Solution: Because of the non-deterministic nature of LLMs, it's possible that Duet generated a response that won't work. If that happens, copy the example index.html above and try redeploying that with firebase deploy. Be sure to replace my-project-with-duet with your project id.

7. (Optional) Open Exploration

1. Use Gemini and the in-editor assistant to add CSS styles to your application and deploy the application again when you're done! If you're looking for inspiration, here is an example prompt you could try:
   • Make this index.html file use material design.
2. Other things to try:
   • The generated website should work with any public http image. Work with Duet to display the image on the page.

8. (Optional) Delete the project

If you would like to avoid being charged for the resources you created today, you can ask Duet how to do that.

1. Enter the prompt:
   • How can I delete my project?
   • You should see a result like gcloud projects delete my-project-with-duet.
   • IMPORTANT Replace my-project-with-duet in the previous command with your project id.
2. Run the gcloud projects delete command to remove all the resources you created today.