Codelab - Build a contextual Yoga Poses recommender app with Firestore, Vector Search, Langchain and Gemini (Node.js version)
About this codelab
1. Introduction
In this codelab, you will build out an application that uses vector search to recommend Yoga poses.
Through the codelab, you will employ a step by step approach as follows:
- Utilize an existing Hugging Face Dataset of Yoga poses (JSON format).
- Enhance the dataset with an additional field description that uses Gemini to generate descriptions for each of the poses.
- Load the Yoga poses data as a collection of Documents in Firestore collection with generated embeddings.
- Create a composite index in Firestore to allow for Vector search.
- Utilize the Vector Search in a Node.js Application that brings everything together as shown below:
What you'll do
- Design, Build and Deploy a web application that employs Vector Search to recommend Yoga poses.
What you'll learn
- How to use Gemini to generate text content and within the context of this codelab, generate descriptions for yoga poses
- How to load records from an enhanced dataset from Hugging Face into Firestore along with Vector Embeddings
- How to use Firestore Vector Search to search for data based on a natural language query
- How to use Google Cloud Text to Speech API to generate Audio content
What you'll need
- Chrome web browser
- A Gmail account
- A Cloud Project with billing enabled
This codelab, designed for developers of all levels (including beginners), uses JavaScript and Node.js in its sample application. However, JavaScript and Node.js knowledge isn't required for understanding the concepts presented.
2. Before you begin
Create a project
- In the Google Cloud Console, on the project selector page, select or create a Google Cloud project.
- Make sure that billing is enabled for your Cloud project. Learn how to check if billing is enabled on a project .
- You'll use Cloud Shell, a command-line environment running in Google Cloud that comes preloaded with bq. Click Activate Cloud Shell at the top of the Google Cloud console.
- Once connected to Cloud Shell, you check that you're already authenticated and that the project is set to your project ID using the following command:
gcloud auth list
- Run the following command in Cloud Shell to confirm that the gcloud command knows about your project.
gcloud config list project
- If your project is not set, use the following command to set it:
gcloud config set project <YOUR_PROJECT_ID>
- Enable the required APIs via the command shown below. This could take a few minutes, so please be patient.
gcloud services enable firestore.googleapis.com \
compute.googleapis.com \
cloudresourcemanager.googleapis.com \
servicenetworking.googleapis.com \
run.googleapis.com \
cloudbuild.googleapis.com \
cloudfunctions.googleapis.com \
aiplatform.googleapis.com \
texttospeech.googleapis.com
On successful execution of the command, you should see a message similar to the one shown below:
Operation "operations/..." finished successfully.
The alternative to the gcloud command is through the console by searching for each product or using this link.
If any API is missed, you can always enable it during the course of the implementation.
Refer documentation for gcloud commands and usage.
Clone repository and setup environment settings
The next step is to clone the sample repository that we will be referencing in the rest of the codelab. Assuming that you are in Cloud Shell, give the following command from your home directory:
git clone https://github.com/rominirani/yoga-poses-recommender-nodejs
To launch the editor, click Open Editor on the toolbar of the Cloud Shell window. Click on the menu bar in the top left corner and select File → Open Folder as shown below:
Select the yoga-poses-recommender-nodejs
folder and you should see the folder open with the following files as shown below:
We need to now set up the environment variables that we shall be using. Click on the env-template
file and you should see the contents as shown below:
PROJECT_ID=<YOUR_GOOGLE_CLOUD_PROJECT_ID>
LOCATION=us-<GOOGLE_CLOUD_REGION_NAME>
GEMINI_MODEL_NAME=<GEMINI_MODEL_NAME>
EMBEDDING_MODEL_NAME=<GEMINI_EMBEDDING_MODEL_NAME>
IMAGE_GENERATION_MODEL_NAME=<IMAGEN_MODEL_NAME>
DATABASE=<FIRESTORE_DATABASE_NAME>
COLLECTION=<FIRESTORE_COLLECTION_NAME>
TEST_COLLECTION=test-poses
TOP_K=3
Please update the values for PROJECT_ID
and LOCATION
as per what you have selected while creating the Google Cloud Project and Firestore Database region. Ideally, we would like the values of the LOCATION
to be the same for the Google Cloud Project and the Firestore Database, for e.g. us-central1
.
For the purpose of this codelab, we are going to go with the following values (except of course for PROJECT_ID
and LOCATION
, which you need to set as per your configuration.
PROJECT_ID=<YOUR_GOOGLE_CLOUD_PROJECT_ID>
LOCATION=us-<GOOGLE_CLOUD_REGION_NAME>
GEMINI_MODEL_NAME=gemini-1.5-flash-002
EMBEDDING_MODEL_NAME=text-embedding-004
IMAGE_GENERATION_MODEL_NAME=imagen-3.0-fast-generate-001
DATABASE=(default)
COLLECTION=poses
TEST_COLLECTION=test-poses
TOP_K=3
Please save this file as .env
in the same folder as the env-template
file.
Go to the main menu on the top left in Cloud Shell IDE and then Terminal → New Terminal
.
Navigate to the root folder of the repository that you cloned via the following command:
cd yoga-poses-recommender-nodejs
Install the Node.js dependencies via the command:
npm install
Great ! We are now all set to move on to the task of setting up the Firestore database.
3. Setup Firestore
Cloud Firestore is a fully-managed serverless document database that we will use as a backend for our application data. Data in Cloud Firestore is structured in collections of documents.
Firestore Database initialization
Visit the Firestore page in the Cloud console.
If you have not initialized a Firestore database before in the project, do create the default
database by clicking on Create Database
. During creation of the database, go with the following values:
- Firestore mode:
Native.
- Location: Go with the default location settings.
- For the Security Rules, go with
Test rules
. - Create the Database.
In the next section, we will set the groundwork for creating a collection named poses
in our default Firestore database. This collection will hold sample data (documents) or Yoga poses information, that we will then use in our application.
This completes the section for setting up of the Firestore database.
4. Prepare the Yoga poses dataset
Our first task is to prepare the Yoga Poses dataset that we shall be using for the application. We will start with an existing Hugging Face dataset and then enhance it with additional information.
Check out the Hugging Face Dataset for Yoga Poses. Note that while this codelab uses one of the datasets, you can in fact use any other dataset and follow the same techniques demonstrated to enhance the dataset.
If we go to the Files and versions
section, we can get the JSON data file for all the poses.
We have downloaded the yoga_poses.json
and provided that file to you. This file is named as yoga_poses_alldata.json
and it's there in the /data
folder.
Go to the data/yoga_poses.json
file in the Cloud Shell Editor and take a look at the list of JSON objects, where each JSON object represents a Yoga pose. We have a total of 3 records and a sample record is shown below:
{
"name": "Big Toe Pose",
"sanskrit_name": "Padangusthasana",
"photo_url": "https://pocketyoga.com/assets/images/full/ForwardBendBigToe.png",
"expertise_level": "Beginner",
"pose_type": ["Standing", "Forward Bend"]
}
Now is a great opportunity for us to introduce Gemini and how we can use the default model itself to generate a description
field for it.
In the Cloud Shell Editor, go to the generate-descriptions.js
file. The contents of this file are shown below:
import { VertexAI } from "@langchain/google-vertexai";
import fs from 'fs/promises'; // Use fs/promises for async file operations
import dotenv from 'dotenv';
import pRetry from 'p-retry';
import { promisify } from 'util';
const sleep = promisify(setTimeout);
// Load environment variables
dotenv.config();
async function callGemini(poseName, sanskritName, expertiseLevel, poseTypes) {
const prompt = `
Generate a concise description (max 50 words) for the yoga pose: ${poseName}
Also known as: ${sanskritName}
Expertise Level: ${expertiseLevel}
Pose Type: ${poseTypes.join(', ')}
Include key benefits and any important alignment cues.
`;
try {
// Initialize Vertex AI Gemini model
const model = new VertexAI({
model: process.env.GEMINI_MODEL_NAME,
location: process.env.LOCATION,
project: process.env.PROJECT_ID,
});
// Invoke the model
const response = await model.invoke(prompt);
// Return the response
return response;
} catch (error) {
console.error("Error calling Gemini:", error);
throw error; // Re-throw the error for handling in the calling function
}
}
// Configure logging (you can use a library like 'winston' for more advanced logging)
const logger = {
info: (message) => console.log(`INFO - ${new Date().toISOString()} - ${message}`),
error: (message) => console.error(`ERROR - ${new Date().toISOString()} - ${message}`),
};
async function generateDescription(poseName, sanskritName, expertiseLevel, poseTypes) {
const prompt = `
Generate a concise description (max 50 words) for the yoga pose: ${poseName}
Also known as: ${sanskritName}
Expertise Level: ${expertiseLevel}
Pose Type: ${poseTypes.join(', ')}
Include key benefits and any important alignment cues.
`;
const req = {
contents: [{ role: 'user', parts: [{ text: prompt }] }],
};
const runWithRetry = async () => {
const resp = await generativeModel.generateContent(req);
const response = await resp.response;
const text = response.candidates[0].content.parts[0].text;
return text;
};
try {
const text = await pRetry(runWithRetry, {
retries: 5,
onFailedAttempt: (error) => {
logger.info(
`Attempt ${error.attemptNumber} failed. There are ${error.retriesLeft} retries left. Waiting ${error.retryDelay}ms...`
);
},
minTimeout: 4000, // 4 seconds (exponential backoff will adjust this)
factor: 2, // Exponential factor
});
return text;
} catch (error) {
logger.error(`Error generating description for ${poseName}: ${error}`);
return '';
}
}
async function addDescriptionsToJSON(inputFile, outputFile) {
try {
const data = await fs.readFile(inputFile, 'utf-8');
const yogaPoses = JSON.parse(data);
const totalPoses = yogaPoses.length;
let processedCount = 0;
for (const pose of yogaPoses) {
if (pose.name !== ' Pose') {
const startTime = Date.now();
pose.description = await callGemini(
pose.name,
pose.sanskrit_name,
pose.expertise_level,
pose.pose_type
);
const endTime = Date.now();
const timeTaken = (endTime - startTime) / 1000;
processedCount++;
logger.info(`Processed: ${processedCount}/${totalPoses} - ${pose.name} (${timeTaken.toFixed(2)} seconds)`);
} else {
pose.description = '';
processedCount++;
logger.info(`Processed: ${processedCount}/${totalPoses} - ${pose.name} (${timeTaken.toFixed(2)} seconds)`);
}
// Add a delay to avoid rate limit
await sleep(30000); // 30 seconds
}
await fs.writeFile(outputFile, JSON.stringify(yogaPoses, null, 2));
logger.info(`Descriptions added and saved to ${outputFile}`);
} catch (error) {
logger.error(`Error processing JSON file: ${error}`);
}
}
async function main() {
const inputFile = './data/yoga_poses.json';
const outputFile = './data/yoga_poses_with_descriptions.json';
await addDescriptionsToJSON(inputFile, outputFile);
}
main();
This application will add a new description
field to each Yoga pose JSON record. It will obtain the description via a call to the Gemini model, where we will provide it with the necessary prompt. The field is added to the JSON file and the new file is written to data/yoga_poses_with_descriptions.json
file.
Let's go through the main steps:
- In the
main()
function, you will find that it invokes theadd_descriptions_to_json
function and provides the input file and the output file expected. - The
add_descriptions_to_json
function does the following for each JSON record i.e. Yoga post information: - It extracts out the
pose_name
,sanskrit_name
,expertise_level
andpose_types
. - It invokes the
callGemini
function that constructs a prompt and then invokes the LangchainVertexAI model class to get the response text. - This response text is then added to the JSON object.
- The updated JSON list of objects is then written to the destination file.
Let us run this application. Launch a new terminal window (Ctrl+Shift+C) and give the following command:
npm run generate-descriptions
If you are asked for any authorization, please go ahead and provide that.
You will find that the application starts executing. We have added a delay of 30 seconds between records to avoid any rate limit quotas that could be there on new Google Cloud accounts, so please be patient.
A sample run in progress is shown below:
Once all the 3 records have been enhanced with the Gemini call, a file data/yoga_poses_with_description.json
will be generated. You can take a look at that.
We are now ready with our data file and the next step is to understand how to populate a Firestore Database with it, along with embeddings generation.
5. Import Data into Firestore and generate Vector Embeddings
We have the data/yoga_poses_with_description.json
file and now need to populate the Firestore Database with it and importantly, generate the Vector Embeddings for each of the records. The Vector Embeddings will be useful later on when we have to do a similarity search on them with the user query that has been provided in natural language.
The steps to do that will be as follows:
- We will convert the list of JSON objects into a list of objects. Each document will have two attributes:
content
andmetadata
. The metadata object will contain the entire JSON object that has attributes likename
,description
,sanskrit_name
, etc. Thecontent
will be a string text that will be a concatenation of a few fields. - Once we have a list of documents, we will be using the Vertex AI Embeddings class to generate the embedding for the content field. This embedding will be added to each document record and then we will use Firestore API to save this list of document objects in the collection (we are using the
TEST_COLLECTION
variable that points totest-poses
).
The code for import-data.js
is given below (parts of the code have been truncated for brevity):
import { Firestore,
FieldValue,
} from '@google-cloud/firestore';
import { VertexAIEmbeddings } from "@langchain/google-vertexai";
import * as dotenv from 'dotenv';
import fs from 'fs/promises';
// Load environment variables
dotenv.config();
// Configure logging
const logger = {
info: (message) => console.log(`INFO - ${new Date().toISOString()} - ${message}`),
error: (message) => console.error(`ERROR - ${new Date().toISOString()} - ${message}`),
};
async function loadYogaPosesDataFromLocalFile(filename) {
try {
const data = await fs.readFile(filename, 'utf-8');
const poses = JSON.parse(data);
logger.info(`Loaded ${poses.length} poses.`);
return poses;
} catch (error) {
logger.error(`Error loading dataset: ${error}`);
return null;
}
}
function createFirestoreDocuments(poses) {
const documents = [];
for (const pose of poses) {
// Convert the pose to a string representation for pageContent
const pageContent = `
name: ${pose.name || ''}
description: ${pose.description || ''}
sanskrit_name: ${pose.sanskrit_name || ''}
expertise_level: ${pose.expertise_level || 'N/A'}
pose_type: ${pose.pose_type || 'N/A'}
`.trim();
// The metadata will be the whole pose
const metadata = pose;
documents.push({ pageContent, metadata });
}
logger.info(`Created ${documents.length} Langchain documents.`);
return documents;
}
async function main() {
const allPoses = await loadYogaPosesDataFromLocalFile('./data/yoga_poses_with_descriptions.json');
const documents = createFirestoreDocuments(allPoses);
logger.info(`Successfully created Firestore documents. Total documents: ${documents.length}`);
const embeddings = new VertexAIEmbeddings({
model: process.env.EMBEDDING_MODEL_NAME,
});
// Initialize Firestore
const firestore = new Firestore({
projectId: process.env.PROJECT_ID,
databaseId: process.env.DATABASE,
});
const collectionName = process.env.TEST_COLLECTION;
for (const doc of documents) {
try {
// 1. Generate Embeddings
const singleVector = await embeddings.embedQuery(doc.pageContent);
// 2. Store in Firestore with Embeddings
const firestoreDoc = {
content: doc.pageContent,
metadata: doc.metadata, // Store the original data as metadata
embedding: FieldValue.vector(singleVector), // Add the embedding vector
};
const docRef = firestore.collection(collectionName).doc();
await docRef.set(firestoreDoc);
logger.info(`Document ${docRef.id} added to Firestore with embedding.`);
} catch (error) {
logger.error(`Error processing document: ${error}`);
}
}
logger.info('Finished adding documents to Firestore.');
}
main();
Let us run this application. Launch a new terminal window (Ctrl+Shift+C) and give the following command:
npm run import-data
If all goes well, you should see a message similar to the one below:
INFO - 2025-01-28T07:01:14.463Z - Loaded 3 poses.
INFO - 2025-01-28T07:01:14.464Z - Created 3 Langchain documents.
INFO - 2025-01-28T07:01:14.464Z - Successfully created Firestore documents. Total documents: 3
INFO - 2025-01-28T07:01:17.623Z - Document P46d5F92z9FsIhVVYgkd added to Firestore with embedding.
INFO - 2025-01-28T07:01:18.265Z - Document bjXXISctkXl2ZRSjUYVR added to Firestore with embedding.
INFO - 2025-01-28T07:01:19.285Z - Document GwzZMZyPfTLtiX6qBFFz added to Firestore with embedding.
INFO - 2025-01-28T07:01:19.286Z - Finished adding documents to Firestore.
To check if the records have been inserted successfully and the embeddings have been generated, visit the Firestore page in the Cloud console.
Click on the (default) database, this should show the test-poses
collection and multiple documents under that collection. Each document is one Yoga pose.
Click on any of the documents to investigate the fields. In addition to the fields that we imported, you will also find the embedding
field, which is a Vector field, whose value we generated via the text-embedding-004
Vertex AI Embedding model.
Now that we have the records uploaded into the Firestore Database with the embeddings in place, we can move to the next step and see how to do Vector Similarity Search in Firestore.
6. Import full Yoga poses into Firestore Database collection
We will now create the poses
collection, which is a full list of 160 Yoga poses, for which we have generated a database import file that you can directly import. This is done to save time in the lab. The process to generate the database that contains the description and embeddings, is the same that we saw in the previous section.
Import the database by following the steps given below:
- Create a bucket in your project with the
gsutil
command given below. Replace the<PROJECT_ID>
variable in the command below with your Google Cloud Project Id.
gsutil mb -l us-central1 gs://<PROJECT_ID>-my-bucket
- Now that the bucket is created, we need to copy the database export that we have prepared into this bucket, before we can import it into the Firebase database. Use the command given below:
gsutil cp -r gs://yoga-database-firestore-export-bucket/2025-01-27T05:11:02_62615 gs://<PROJECT_ID>-my-bucket
Now that we have the data to import, we can move to the final step of importing the data into the Firebase database (default
) that we've created.
- Use the gcloud command given below:
gcloud firestore import gs://<PROJECT_ID>-my-bucket/2025-01-27T05:11:02_62615
The import will take a few seconds and once it's ready, you can validate your Firestore database and the collection by visiting https://console.cloud.google.com/firestore/databases, select the default
database and the poses
collection as shown below:
This completes the creation of the Firestore collection that we will be using in our application.
7. Perform Vector Similarity Search in Firestore
To perform Vector Similarity search, we will take in the query from the user. An example of this query can be "Suggest me some exercises to relieve back pain"
.
Take a look at the search-data.js
file. The key function to look at is the search
function, which is shown below. At a high level, it creates an embedding class that shall be used to generate the embedding for the user query. It then establishes a connection to the Firestore database and collection. Then on the collection it invokes the findNearest method, that does a Vector Similarity Search.
async function search(query) {
try {
const embeddings = new VertexAIEmbeddings({
model: process.env.EMBEDDING_MODEL_NAME,
});
// Initialize Firestore
const firestore = new Firestore({
projectId: process.env.PROJECT_ID,
databaseId: process.env.DATABASE,
});
log.info(`Now executing query: ${query}`);
const singleVector = await embeddings.embedQuery(query);
const collectionRef = firestore.collection(process.env.COLLECTION);
let vectorQuery = collectionRef.findNearest(
"embedding",
FieldValue.vector(singleVector), // a vector with 768 dimensions
{
limit: process.env.TOP_K,
distanceMeasure: "COSINE",
}
);
const vectorQuerySnapshot = await vectorQuery.get();
for (const result of vectorQuerySnapshot.docs) {
console.log(result.data().content);
}
} catch (error) {
log.error(`Error during search: ${error.message}`);
}
}
Before you run this with a few query examples, you must first generate a Firestore composite index, which is needed for your search queries to succeed. If you run the application without creating the index, an error indicating that you need to create the index first will be displayed with the command to create the index first.
The gcloud
command to create the composite index is shown below:
gcloud firestore indexes composite create --project=<YOUR_PROJECT_ID> --collection-group=poses --query-scope=COLLECTION --field-config=vector-config='{"dimension":"768","flat": "{}"}',field-path=embedding
The index will take a few minutes to complete since there are 150+ records that are present in the database. Once it is complete, you can view the index via the command shown below:
gcloud firestore indexes composite list
You should see the index that you just created in the list.
Try out the following command now:
node search-data.js --prompt "Recommend me some exercises for back pain relief"
You should have a few recommendations provided to you. A sample run is shown below:
2025-01-28T07:09:05.250Z - INFO - Now executing query: Recommend me some exercises for back pain relief
name: Sphinx Pose
description: A gentle backbend, Sphinx Pose (Salamba Bhujangasana) strengthens the spine and opens the chest. Keep shoulders relaxed, lengthen the tailbone, and engage the core for optimal alignment. Beginner-friendly.
sanskrit_name: Salamba Bhujangasana
expertise_level: Beginner
pose_type: ['Prone']
name: Supine Spinal Twist Pose
description: A gentle supine twist (Supta Matsyendrasana), great for beginners. Releases spinal tension, improves digestion, and calms the nervous system. Keep shoulders flat on the floor and lengthen your spine throughout the twist.
sanskrit_name: Supta Matsyendrasana
expertise_level: Beginner
pose_type: ['Supine', 'Twist']
name: Reverse Corpse Pose
description: Reverse Corpse Pose (Advasana) is a beginner prone pose. Lie on your belly, arms at your sides, relaxing completely. Benefits include stress release and spinal decompression. Ensure your forehead rests comfortably on the mat.
sanskrit_name: Advasana
expertise_level: Beginner
pose_type: ['Prone']
Once you have this working, we have now understood how to work the Firestore Vector Database to upload records, generate embeddings and do a Vector Similarity Search. We can now create a web application which will integrate the vector search into a web front-end.
8. The Web Application
The Python Flask web application is available in app.js
file and the front-end HTML file is present in the views/index.html.
It is recommended that you take a look at both the files. Start first with the app.js
file that contains the /search
handler, which takes the prompt that has been passed from the front-end HTML index.html
file. This then invokes the search method, which does the Vector Similarity search that we looked at in the previous section.
The response is then sent back to the index.html
with the list of recommendations. The index.html
then displays the recommendations as different cards.
Run the application locally
Launch a new terminal window (Ctrl+Shift+C) or any existing terminal window and give the following command:
npm run start
A sample execution is shown below:
...
Server listening on port 8080
Once up and running, visit the home URL of the application, by clicking the Web Preview button shown below:
It should show you the index.html
file served as shown below:
Provide a sample query (Example : Provide me some exercises for back pain relief
) and click on the Search
button. This should retrieve some recommendations from the database. You will also see a Play Audio
button, which will generate an audio stream based on the description, which you can hear directly.
9. (Optional) Deploying to Google Cloud Run
Our final step will be to deploy this application to Google Cloud Run. The deployment command is shown below, ensure that before you deploy it, you replace the values that are shown in bold below. These are values that you will be able to retrieve from the .env
file.
gcloud run deploy yogaposes --source . \
--port=8080 \
--allow-unauthenticated \
--region=<<YOUR_LOCATION>> \
--platform=managed \
--project=<<YOUR_PROJECT_ID>> \
--set-env-vars=PROJECT_ID="<<YOUR_PROJECT_ID>>",LOCATION="<<YOUR_LOCATION>>",EMBEDDING_MODEL_NAME="<<EMBEDDING_MODEL_NAME>>",DATABASE="<<FIRESTORE_DATABASE_NAME>>",COLLECTION="<<FIRESTORE_COLLECTION_NAME>>",TOP_K=<<YOUR_TOP_K_VALUE>>
Execute the above command from the root folder of the application. You may also be asked to enable Google Cloud APIs, give your acknowledgement for various permissions, please do so.
The deployment process will take about 5-7 minutes to complete, so please be patient.
Once successfully deployed, the deployment output will provide the Cloud Run service URL. It will be of the form:
Service URL: https://yogaposes-<UNIQUEID>.us-central1.run.app
Visit that public URL and you should see the same web application deployed and running successfully.
You can also visit Cloud Run from the Google Cloud console and you will see the list of services in Cloud Run. The yogaposes
service should be one of the services (if not the only one) listed there.
You can view the details of the service like URL, configurations, logs and more by clicking on the specific service name (yogaposes
in our case).
This completes the development and deployment of our Yoga poses recommender web application on Cloud Run.
10. Congratulations
Congratulations, you've successfully built an application that uploads a dataset to Firestore, generates the embeddings and does a Vector Similarity Search based on the users query.