Codelab - Build a contextual Yoga Poses recommender app with Firestore, Vector Search, Langchain and Gemini (Python version)

Codelab - Build a contextual Yoga Poses recommender app with Firestore, Vector Search, Langchain and Gemini (Python version)

About this codelab

subjectLast updated Feb 12, 2025
account_circleWritten by Romin Irani and Alvin Prayuda Juniarta Dwiyantoro

1. Introduction

In this codelab, you will build out an application that uses vector search to recommend Yoga poses.

Through the codelab, you will employ a step by step approach as follows:

  1. Utilize an existing Hugging Face Dataset of Yoga poses (JSON format).
  2. Enhance the dataset with an additional field description that uses Gemini to generate descriptions for each of the poses.
  3. Use Langchain to create a Document, use Firestore Langchain integration to create the collection and embeddings in Firestore.
  4. Create a composite index in Firestore to allow for Vector search.
  5. Utilize the Vector Search in a Flask Application that brings everything together as shown below:

84e1cbf29cbaeedc.png

What you'll do

  • Design, Build and Deploy a web application that employs Vector Search to recommend Yoga poses.

What you'll learn

  • How to use Gemini to generate text content and within the context of this codelab, generate descriptions for yoga poses
  • How to use Langchain Document Loader for Firestore to load records from an enhanced dataset from Hugging Face into Firestore along with Vector Embeddings
  • How to use Langchain Vector Store for Firestore to search for data based on a natural language query
  • How to use Google Cloud Text to Speech API to generate Audio content

What you'll need

  • Chrome web browser
  • A Gmail account
  • A Cloud Project with billing enabled

This codelab, designed for developers of all levels (including beginners), uses Python in its sample application. However, Python knowledge isn't required for understanding the concepts presented.

2. Before you begin

Create a project

  1. In the Google Cloud Console, on the project selector page, select or create a Google Cloud project.
  2. Make sure that billing is enabled for your Cloud project. Learn how to check if billing is enabled on a project .
  3. You'll use Cloud Shell, a command-line environment running in Google Cloud that comes preloaded with bq. Click Activate Cloud Shell at the top of the Google Cloud console.

Activate Cloud Shell button image

  1. Once connected to Cloud Shell, you check that you're already authenticated and that the project is set to your project ID using the following command:
gcloud auth list
  1. Run the following command in Cloud Shell to confirm that the gcloud command knows about your project.
gcloud config list project
  1. If your project is not set, use the following command to set it:
gcloud config set project <YOUR_PROJECT_ID>
  1. Enable the required APIs via the command shown below. This could take a few minutes, so please be patient.
gcloud services enable firestore.googleapis.com \
                       
compute.googleapis.com \
                       
cloudresourcemanager.googleapis.com \
                       
servicenetworking.googleapis.com \
                       
run.googleapis.com \
                       
cloudbuild.googleapis.com \
                       
cloudfunctions.googleapis.com \
                       
aiplatform.googleapis.com \
                       
texttospeech.googleapis.com

On successful execution of the command, you should see a message similar to the one shown below:

Operation "operations/..." finished successfully.

The alternative to the gcloud command is through the console by searching for each product or using this link.

If any API is missed, you can always enable it during the course of the implementation.

Refer documentation for gcloud commands and usage.

Clone repository and setup environment settings

The next step is to clone the sample repository that we will be referencing in the rest of the codelab. Assuming that you are in Cloud Shell, give the following command from your home directory:

git clone https://github.com/rominirani/yoga-poses-recommender-python

To launch the editor, click Open Editor on the toolbar of the Cloud Shell window. Click on the menu bar in the top left corner and select File → Open Folder as shown below:

66221fd0d0e5202f.png

Select the yoga-poses-recommender-python folder and you should see the folder open with the following files as shown below:

44699efc7fb1b911.png

We need to now set up the environment variables that we shall be using. Click on the config.template.yaml file and you should see the contents as shown below:

project_id: your-project-id
location: us-central1
gemini_model_name: gemini-1.5-flash-002
embedding_model_name: text-embedding-004
image_generation_model_name: imagen-3.0-fast-generate-002
database: (default)
collection: poses
test_collection: test-poses
top_k: "3"

Please update the values for project_id and location as per what you have selected while creating the Google Cloud Project and Firestore Database region. Ideally, we would like the values of the location to be the same for the Google Cloud Project and the Firestore Database, for e.g. us-central1.

For the purpose of this codelab, we are going to go with the pre-configured values (except of course for project_id and location, which you need to set as per your configuration.

Please save this file as config.yaml in the same folder as the config.template.yaml file.

The final step now is to create a Python environment that we shall use locally with all the Python dependencies set up for us. Take a look at the pyproject.toml file that contains the details for the same, the contents of which are shown below:

dependencies = [
   
"datasets>=3.2.0",
   
"flask>=3.1.0",
   
"google-cloud-aiplatform>=1.78.0",
   
"google-cloud-texttospeech>=2.24.0",
   
"langchain-community>=0.3.15",
   
"langchain-core>=0.3.31",
   
"langchain-google-community>=2.0.4",
   
"langchain-google-firestore>=0.5.0",
   
"langchain-google-vertexai>=2.0.7",
   
"pydantic-settings>=2.7.1",
   
"pyyaml>=6.0.2",
   
"tenacity>=9.0.0",
]

These dependencies are already version locked in the requirements.txt. To summarize, we need to create a virtual Python environment with the Python package dependencies in requirements.txt to be installed in the virtual environment. To do that, go to the Command Palette (Ctrl+Shift+P) in Cloud Shell IDE and type in Python: Create Environment. Follow the next few steps to select a Virtual Environment(venv), Python 3.x interpreter and the requirements.txt file.

Once the environment is created, we will need to activate the created environment with the following command

source .venv/bin/activate

You should see (.venv) in your console. E.g. -> (.venv) yourusername@cloudshell:

Great ! We are now all set to move on to the task of setting up the Firestore database.

3. Setup Firestore

Cloud Firestore is a fully-managed serverless document database that we will use as a backend for our application data. Data in Cloud Firestore is structured in collections of documents.

Firestore Database initialization

Visit the Firestore page in the Cloud console.

If you have not initialized a Firestore database before in the project, do create the default database by clicking on Create Database. During creation of the database, go with the following values:

  • Firestore mode: Native.
  • Location: Go with the default location settings.
  • For the Security Rules, go with Test rules.
  • Create the Database.

504cabdb99a222a5.png

In the next section, we will set the groundwork for creating a collection named poses in our default Firestore database. This collection will hold sample data (documents) or Yoga poses information, that we will then use in our application.

This completes the section for setting up of the Firestore database.

4. Prepare the Yoga poses dataset

Our first task is to prepare the Yoga Poses dataset that we shall be using for the application. We will start with an existing Hugging Face dataset and then enhance it with additional information.

Check out the Hugging Face Dataset for Yoga Poses. Note that while this codelab uses one of the datasets, you can in fact use any other dataset and follow the same techniques demonstrated to enhance the dataset.

298cfae7f23e4bef.png

If we go to the Files and versions section, we can get the JSON data file for all the poses.

3fe6e55abdc032ec.png

We have downloaded the yoga_poses.json and provided that file to you. This file is named as yoga_poses_alldata.json and it's there in the /data folder.

Go to the data/yoga_poses.json file in the Cloud Shell Editor and take a look at the list of JSON objects, where each JSON object represents a Yoga pose. We have a total of 3 records and a sample record is shown below:

{
   
"name": "Big Toe Pose",
   
"sanskrit_name": "Padangusthasana",
   
"photo_url": "https://pocketyoga.com/assets/images/full/ForwardBendBigToe.png",
   
"expertise_level": "Beginner",
   
"pose_type": ["Standing", "Forward Bend"]
 
}

Now is a great opportunity for us to introduce Gemini and how we can use the default model itself to generate a description field for it.

In the Cloud Shell Editor, go to the generate-descriptions.py file. The contents of this file are shown below:

import json
import time
import logging
import vertexai
from langchain_google_vertexai import VertexAI
from tenacity import retry, stop_after_attempt, wait_exponential
from settings import get_settings

settings = get_settings()
logging.basicConfig(
   
level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s"
)
# Initialize Vertex AI SDK
vertexai.init(project=settings.project_id, location=settings.location)
logging.info("Done Initializing Vertex AI SDK")


@retry(
   
stop=stop_after_attempt(5),
   
wait=wait_exponential(multiplier=1, min=4, max=10),
)
def generate_description(pose_name, sanskrit_name, expertise_level, pose_types):
    """Generates a description for a yoga pose using the Gemini API."""

   
prompt = f"""
    Generate a concise description (max 50 words) for the yoga pose: {pose_name}
    Also known as: {sanskrit_name}
    Expertise Level: {expertise_level}
    Pose Type: {", ".join(pose_types)}

    Include key benefits and any important alignment cues.
    """
   
try:
       
model = VertexAI(model_name=settings.gemini_model_name, verbose=True)
       
response = model.invoke(prompt)
       
return response
   
except Exception as e:
       
logging.info(f"Error generating description for {pose_name}: {e}")
       
return ""


def add_descriptions_to_json(input_file, output_file):
    """Loads JSON data, adds descriptions, and saves the updated data."""

   
with open(input_file, "r") as f:
       
yoga_poses = json.load(f)

   
total_poses = len(yoga_poses)
   
processed_count = 0

   
for pose in yoga_poses:
       
if pose["name"] != " Pose":
           
start_time = time.time()  # Record start time
           
pose["description"] = generate_description(
               
pose["name"],
               
pose["sanskrit_name"],
               
pose["expertise_level"],
               
pose["pose_type"],
           
)
           
end_time = time.time()  # Record end time

           
processed_count += 1
           
end_time = time.time()  # Record end time
           
time_taken = end_time - start_time
           
logging.info(
               
f"Processed: {processed_count}/{total_poses} - {pose['name']} ({time_taken:.2f} seconds)"
           
)

       
else:
           
pose["description"] = ""
           
processed_count += 1
           
logging.info(
               
f"Processed: {processed_count}/{total_poses} - {pose['name']} ({time_taken:.2f} seconds)"
           
)
       
# Adding a delay to avoid rate limit
       
time.sleep(30)

   
with open(output_file, "w") as f:
       
json.dump(yoga_poses, f, indent=2)


def main():
   
# File paths
   
input_file = "./data/yoga_poses.json"
   
output_file = "./data/yoga_poses_with_descriptions.json"

   
# Add descriptions and save the updated JSON
   
add_descriptions_to_json(input_file, output_file)


if __name__ == "__main__":
   
main()

This application will add a new description field to each Yoga pose JSON record. It will obtain the description via a call to the Gemini model, where we will provide it with the necessary prompt. The field is added to the JSON file and the new file is written to data/yoga_poses_with_descriptions.json file.

Let's go through the main steps:

  1. In the main() function, you will find that it invokes the add_descriptions_to_json function and provides the input file and the output file expected.
  2. The add_descriptions_to_json function does the following for each JSON record i.e. Yoga post information:
  3. It extracts out the pose_name, sanskrit_name, expertise_level and pose_types.
  4. It invokes the generate_description function that constructs a prompt and then invokes the Langchain VertexAI model class to get the response text.
  5. This response text is then added to the JSON object.
  6. The updated JSON list of objects is then written to the destination file.

Let us run this application. Launch a new terminal window (Ctrl+Shift+C) and give the following command:

python generate-descriptions.py

If you are asked for any authorization, please go ahead and provide that.

You will find that the application starts executing. We have added a delay of 30 seconds between records to avoid any rate limit quotas that could be there on new Google Cloud accounts, so please be patient.

A sample run in progress is shown below:

8e830d9ea9b6c60.png

Once all the 3 records have been enhanced with the Gemini call, a file data/yoga_poses_with_description.json will be generated. You can take a look at that.

We are now ready with our data file and the next step is to understand how to populate a Firestore Database with it, along with embeddings generation.

5. Import Data into Firestore and generate Vector Embeddings

We have the data/yoga_poses_with_description.json file and now need to populate the Firestore Database with it and importantly, generate the Vector Embeddings for each of the records. The Vector Embeddings will be useful later on when we have to do a similarity search on them with the user query that has been provided in natural language.

We will be using the Langchain Firestore components to implement the above process.

The steps to do that will be as follows:

  1. We will convert the list of JSON objects into a list of Langchain Document objects. Each document will have two attributes: page_content and metadata. The metadata object will contain the entire JSON object that has attributes like name, description, sanskrit_name, etc. The page_content will be a string text that will be a concatenation of a few fields.
  2. Once we have a list of Document objects, we will be using the FirestoreVectorStore Langchain class and specifically the from_documents method with this list of documents, a collection name (we are using the TEST_COLLECTION variable that points to test-poses), a Vertex AI Embedding class and the Firestore connection details (PROJECT_ID and DATABASE name). This will create the collection and will also generate an embedding field for each of the attributes.

The code for import-data.py is given below (parts of the code have been truncated for brevity):

... 

def create_langchain_documents(poses):
   """Creates a list of Langchain Documents from a list of poses."""
   
documents = []
   
for pose in poses:
       
# Convert the pose to a string representation for page_content
       
page_content = (
           
f"name: {pose.get('name', '')}\n"
           
f"description: {pose.get('description', '')}\n"
           
f"sanskrit_name: {pose.get('sanskrit_name', '')}\n"
           
f"expertise_level: {pose.get('expertise_level', 'N/A')}\n"
           
f"pose_type: {pose.get('pose_type', 'N/A')}\n"
       
).strip()
       
# The metadata will be the whole pose
       
metadata = pose

       
document = Document(page_content=page_content, metadata=metadata)
       
documents.append(document)
   
logging.info(f"Created {len(documents)} Langchain documents.")
   
return documents

def main():
   
all_poses = load_yoga_poses_data_from_local_file(
       
"./data/yoga_poses_with_descriptions.json"
   
)
   
documents = create_langchain_documents(all_poses)
   
logging.info(
       
f"Successfully created langchain documents. Total documents: {len(documents)}"
   
)

   
embedding = VertexAIEmbeddings(
       
model_name=settings.embedding_model_name,
       
project=settings.project_id,
       
location=settings.location,
   
)

   
client = firestore.Client(project=settings.project_id, database=settings.database)

   
vector_store = FirestoreVectorStore.from_documents(
       
client=client,
       
collection=settings.test_collection,
       
documents=documents,
       
embedding=embedding,
   
)
   
logging.info("Added documents to the vector store.")


if __name__ == "__main__":
   
main()

Let us run this application. Launch a new terminal window (Ctrl+Shift+C) and give the following command:

python import-data.py

If all goes well, you should see a message similar to the one below:

2025-01-21 14:50:06,479 - INFO - Added documents to the vector store.

To check if the records have been inserted successfully and the embeddings have been generated, visit the Firestore page in the Cloud console.

504cabdb99a222a5.png

Click on the (default) database, this should show the test-poses collection and multiple documents under that collection. Each document is one Yoga pose.

d0708499e403aebc.png

Click on any of the documents to investigate the fields. In addition to the fields that we imported, you will also find the embedding field, which is a Vector field that has been generated automatically for you via the Langchain VertexAIEmbeddings class that we used, in which we provided the text-embedding-004 Vertex AI Embedding model.

d67113e2dc63cd6b.png

Now that we have the records uploaded into the Firestore Database with the embeddings in place, we can move to the next step and see how to do Vector Similarity Search in Firestore.

6. Import full Yoga poses into Firestore Database collection

We will now create the poses collection, which is a full list of 160 Yoga poses, for which we have generated a database import file that you can directly import. This is done to save time in the lab. The process to generate the database that contains the description and embeddings, is the same that we saw in the previous section.

Import the database by following the steps given below:

  1. Create a bucket in your project with the gsutil command given below. Replace the <PROJECT_ID> variable in the command below with your Google Cloud Project Id.
gsutil mb -l us-central1 gs://<PROJECT_ID>-my-bucket
  1. Now that the bucket is created, we need to copy the database export that we have prepared into this bucket, before we can import it into the Firebase database. Use the command given below:
gsutil cp -r gs://yoga-database-firestore-export-bucket/2025-01-27T05:11:02_62615  gs://<PROJECT_ID>-my-bucket

Now that we have the data to import, we can move to the final step of importing the data into the Firebase database (default) that we've created.

  1. Use the gcloud command given below:
gcloud firestore import gs://<PROJECT_ID>-my-bucket/2025-01-27T05:11:02_62615

The import will take a few seconds and once it's ready, you can validate your Firestore database and the collection by visiting https://console.cloud.google.com/firestore/databases, select the default database and the poses collection as shown below:

a8f5a6ba69bec69b.png

This completes the creation of the Firestore collection that we will be using in our application.

7. Perform Vector Similarity Search in Firestore

To perform Vector Similarity search, we will take in the query from the user. An example of this query can be "Suggest me some exercises to relieve back pain".

Take a look at the search-data.py file. The key function to look at is the search function, which is shown below. At a high level, it creates an embedding class that shall be used to generate the embedding for the user query. It then uses the FirestoreVectorStore class to invoke its similarity_search function.

def search(query: str):
    """Executes Firestore Vector Similarity Search"""
   
embedding = VertexAIEmbeddings(
       
model_name=settings.embedding_model_name,
       
project=settings.project_id,
       
location=settings.location,
   
)

   
client = firestore.Client(project=settings.project_id, database=settings.database)

   
vector_store = FirestoreVectorStore(
       
client=client, collection=settings.collection, embedding_service=embedding
   
)

   
logging.info(f"Now executing query: {query}")
   
results: list[Document] = vector_store.similarity_search(
       
query=query, k=int(settings.top_k), include_metadata=True
   
)
   
for result in results:
       
print(result.page_content)

Before you run this with a few query examples, you must first generate a Firestore composite index, which is needed for your search queries to succeed. If you run the application without creating the index, an error indicating that you need to create the index first will be displayed with the command to create the index first.

The gcloud command to create the composite index is shown below:

gcloud firestore indexes composite create --project=<YOUR_PROJECT_ID> --collection-group=poses --query-scope=COLLECTION --field-config=vector-config='{"dimension":"768","flat": "{}"}',field-path=embedding

The index will take a few minutes to complete since there are 150+ records that are present in the database. Once it is complete, you can view the index via the command shown below:

gcloud firestore indexes composite list

You should see the index that you just created in the list.

Try out the following command now:

python search-data.py --prompt "Recommend me some exercises for back pain relief"

You should have a few recommendations provided to you. A sample run is shown below:

2025-01-21 15:48:51,282 - INFO - Now executing query: Recommend me some exercises for back pain relief
name: Supine Spinal Twist Pose
description: A gentle supine twist (Supta Matsyendrasana), great for beginners.  Releases spinal tension, improves digestion, and calms the nervous system.  Keep shoulders flat on the floor and lengthen the spine.

sanskrit_name: Supta Matsyendrasana
expertise_level: Beginner
pose_type: ['Supine', 'Twist']
name: Cow Pose
description: Cow Pose (Bitilasana) is a gentle backbend, stretching the chest, shoulders, and abdomen.  Maintain a neutral spine, lengthen the tailbone, and avoid hyperextension.  Benefits include improved posture and stress relief.

sanskrit_name: Bitilasana
expertise_level: Beginner
pose_type: ['Arm Leg Support', 'Back Bend']
name: Locust I Pose
description: Locust Pose I (Shalabhasana A) strengthens the back, glutes, and shoulders.  Lie prone, lift chest and legs simultaneously, engaging back muscles.  Keep hips grounded and gaze slightly forward.

sanskrit_name: Shalabhasana A
expertise_level: Intermediate
pose_type: ['Prone', 'Back Bend']

Once you have this working, we have now understood how to work the Firestore Vector Database to upload records, generate embeddings and do a Vector Similarity Search. We can now create a web application which will integrate the vector search into a web front-end.

8. The Web Application

The Python Flask web application is available in main.py file and the front-end HTML file is present in the templates/index.html.

It is recommended that you take a look at both the files. Start first with the main.py file that contains the /search handler, which takes the prompt that has been passed from the front-end HTML index.html file. This then invokes the search method, which does the Vector Similarity search that we looked at in the previous section.

The response is then sent back to the index.html with the list of recommendations. The index.html then displays the recommendations as different cards.

Run the application locally

Launch a new terminal window (Ctrl+Shift+C) or any existing terminal window and give the following command:

python main.py

A sample execution is shown below:

 * Serving Flask app 'main'
 
* Debug mode: on
2025-01-21 16:02:37,473 - INFO - WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 
* Running on all addresses (0.0.0.0)
 
* Running on http://127.0.0.1:8080
 
* Running on http://10.88.0.4:8080
2025-01-21 16:02:37,473 - INFO - Press CTRL+C to quit
2025-01-21 16:02:37,474 - INFO -  * Restarting with stat
2025-01-21 16:02:41,462 - WARNING -  * Debugger is active!
2025-01-21 16:02:41,484 - INFO -  * Debugger PIN: 440-653-555

Once up and running, visit the home URL of the application, by clicking the Web Preview button shown below:

de297d4cee10e0bf.png

It should show you the index.html file served as shown below:

20240a0e885ac17b.png

Provide a sample query (Example : Provide me some exercises for back pain relief) and click on the Search button. This should retrieve some recommendations from the database. You will also see a Play Audio button, which will generate an audio stream based on the description, which you can hear directly.

789b4277dc40e2be.png

9. (Optional) Deploying to Google Cloud Run

Our final step will be to deploy this application to Google Cloud Run. The deployment command is shown below, ensure that before you deploy it, you replace the values for the variable (<<YOUR_PROJECT_ID>>) with the ones that are specific to your project. These are values that you will be able to retrieve from the config.yaml file.

gcloud run deploy yogaposes --source . \
 
--port=8080 \
 
--allow-unauthenticated \
 
--region=us-central1 \
 
--platform=managed  \
 
--project=<<YOUR_PROJECT_ID>> \
 
--env-vars-file=config.yaml

Execute the above command from the root folder of the application. You may also be asked to enable Google Cloud APIs, give your acknowledgement for various permissions, please do so.

The deployment process will take about 5-7 minutes to complete, so please be patient.

3a6d86fd32e4a5e.png

Once successfully deployed, the deployment output will provide the Cloud Run service URL. It will be of the form:

Service URL: https://yogaposes-<<UNIQUEID>.us-central1.run.app

Visit that public URL and you should see the same web application deployed and running successfully.

84e1cbf29cbaeedc.png

You can also visit Cloud Run from the Google Cloud console and you will see the list of services in Cloud Run. The yogaposes service should be one of the services (if not the only one) listed there.

f2b34a8c9011be4c.png

You can view the details of the service like URL, configurations, logs and more by clicking on the specific service name (yogaposes in our case).

faaa5e0c02fe0423.png

This completes the development and deployment of our Yoga poses recommender web application on Cloud Run.

10. Congratulations

Congratulations, you've successfully built an application that uploads a dataset to Firestore, generates the embeddings and does a Vector Similarity Search based on the users query.

Reference docs