# Implementing Semantic Search with Gemini API ## Introduction In today's data-driven world, the sheer volume of information available can be overwhelming. Traditional keyword-based search methods often fall short when it comes to understanding the context or intent behind a query. This is where [Semantic Search](https://www.elastic.co/what-is/semantic-search) comes into play. Semantic Search goes beyond simple keyword matching by understanding the meaning behind the words. It leverages advanced algorithms to interpret the context, making it possible to deliver more accurate and relevant search results. With Cosmocloud's services, integrating Semantic Search into your applications is simple. This tutorial will show you how to use the Gemini API and MongoDB Atlas Vector Search to create vector embeddings and enabling intelligent search that understands user intent and delivers relevant results. ## Our Agenda In this tutorial, we will cover: * Implementing Semantic Search using [Vector Search Index](https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-overview/) from Cosmocloud and [Vector Embeddings generated from Gemini](https://ai.google.dev/gemini-api/docs/embeddings) to implement a Semantic Search. ## Prerequisites * Our tutorial assumes that you have already [created a project](https://app.gitbook.com/s/ZCdm9aJ8vkvDIIbg04AL/getting-started) on Cosmocloud, picking the cloud of your choice. * [Connecting your database to your Cosmocloud Project](https://app.gitbook.com/s/ZCdm9aJ8vkvDIIbg04AL/getting-started/3.-connect-your-database). ## Creating Semantic Search To implement semantic search, you will need to store vector embeddings of your data and query these embeddings using a Vector Search Index. We will use the following sample data generated by the Gemini chatbot for hotel information - {% code title="Hotel Sample Data" %} ```json [ { "hotelName": "Desert Oasis", "city": "Phoenix", "rating": 4.4, "ammenities": [ "Spa", "Golf course", "Free breakfast" ], "price": 210.5, "details": "Desert Oasis lived up to its name. The spa treatments were incredibly relaxing and professionally done. The golf course was well-maintained and enjoyable to play on. The free breakfast had a good variety of choices. The hotel grounds were beautiful and well-kept. We had a very relaxing and enjoyable stay.", }, // ... ] ``` {% endcode %} The complete dataset can be found here: [Hotel's Data](https://api.npoint.io/848c8b755df98cd1b4e7) We will create the following two APIs: 1. `POST /hotels`: This endpoint will generate vector embeddings for each record inserted into the database, based on the `details` field. The vector embeddings will be generated using the Gemini API. 2. `GET /hotels/_search`: This endpoint will allow you to search for hotels based on a query describing the hotel. The search will use the [Vector Search Index](https://app.gitbook.com/s/ZCdm9aJ8vkvDIIbg04AL/resources/vector-search/create-a-vector-search-index) created in Cosmocloud Platform. ### Creating Hotel DB Model The first step in Cosmocloud is to build your database models for the entities defined in your system. We'll start by creating our `Hotels` model. {% hint style="info" %} The name of the model should match the collection name (table name) in your database. Keeping it Hotels would create a collection named Hotels in your MongoDB database. {% endhint %} * Navigate to **Application Layer** -> **DB Models** to create new database models in Cosmocloud. * Click on **New Model** button. * Enter `Hotels` as the model name, provide a description, and click `Create`. * Switch to the **Schema** tab at the top to define the schema for the `Hotels` model. **Defining the Schema of our Hotels Model** As Cosmocloud is connected to your MongoDB database, we can define a Document Model based schema for our `Hotels` model. * Click on **JSON to Schema** button to quickly generate schema from a sample `Hotel` record. * Copy the snippet below as the sample record in our `Hotels` model. {% hint style="info" %} We will not add `_id` as it is going to be auto-generated by our database (MongoDB) for us The `detailsEmbeddings` field will store the vector embeddings of the `details` field which will be generated from Gemini API. We'll see this later below. {% endhint %} ```json { "hotelName": "Desert Oasis", "city": "Phoenix", "rating": 4.4, "ammenities": [ "Spa", "Golf course", "Free breakfast" ], "price": 210.5, "details": "Desert Oasis lived up to its name. The spa treatments were incredibly relaxing and professionally done. The golf course was well-maintained and enjoyable to play on. The free breakfast had a good variety of choices. The hotel grounds were beautiful and well-kept. We had a very relaxing and enjoyable stay.", "detailsEmbeddings": [1.23, 1.23] } ``` * Click on Save button to instantly generate the schema of your `Hotels` model as well as save it in the system. If the schema doesn't look as the image given below, do necessary changes manually.

Hotels Schema

### Store Secret of Gemini API After creating your Gemini API key from [Google AI Studio](https://aistudio.google.com/app/apikey), you'll need to store it as a [Custom Secret](https://app.gitbook.com/s/ZCdm9aJ8vkvDIIbg04AL/resources/secrets) in Cosmocloud.
### Creating Subflow to Generate Embeddings Next, we'll create a [Subflow](https://app.gitbook.com/s/ZCdm9aJ8vkvDIIbg04AL/examples-how-to/reusable-flows-subflows) to generate embeddings from a text using the Gemini API. We are creating a Subflow as it will allow us to quickly reuse this flow in multiple APIs such as storing data in the database and when querying it. 1. Head over to **Application Layer** -> **Subflows**. 2. Name the subflow as `create_embeddings`. 3. Set the argument key as `text`, type as `String`, and mark it as required by setting the first option to `true`. 4. Click on `Create` button to create the new subflow.

create_embeddings Subflow

Here comes the main part, we'll now edit the main flow of generating the embeddings of a text from Gemini. Head over to the `Flow` tab. #### Build JSON Object * Add a [Build JSON Object](https://app.gitbook.com/s/ZCdm9aJ8vkvDIIbg04AL/flow-builder/node-types/variable-nodes/json/build-json-object) node, and name it `jsonPayload`. * This node will create the request body for the Gemini API. We are going to use `$.variables.text` which is coming as an argument to our Subflow. The JSON content should look like this: ```json { "model": "models/text-embedding-004", "content": { "parts": [ { "text": "$.variables.text" } ] } } ```

jsonPayload JSON Object

#### Concat Strings * Add a [Concat Strings](https://app.gitbook.com/s/ZCdm9aJ8vkvDIIbg04AL/flow-builder/node-types/variable-nodes/strings/concat-strings) node and name it as `gemini_url`. * This node will concatenate the base URL `https://generativelanguage.googleapis.com/v1beta/models/text-embedding-004:embedContent?key=` with the `gemini_api_key` stored as a secret, forming the full API endpoint. Refer this [doc](https://ai.google.dev/gemini-api/docs/embeddings) to learn more about the Base URL.

gemini_url - Conact Strings Node

#### API Call Add an [API Call](https://app.gitbook.com/s/ZCdm9aJ8vkvDIIbg04AL/flow-builder/node-types/external-nodes/api-call) node to make a POST request to the above `gemini_url` URL generated, using the `jsonPayload` as the request body. This call will generate the text embeddings.

Gemini API Call Node

#### Subflow Response Add a `Subflow Response` node at the end to return the embeddings generated by the previous `API Call` node.

Subflow Response Node

In the end, the flow will look like this:

create_embeddings Final Subflow

Pheww, a lot of work right! Take a moment to relax, you've made great progress! There's still more to come, but the upcoming sections will get even more interesting as you continue to build out your Semantic Search functionality. ### Creating POST API In this section, we'll use the `create_embeddings` Subflow to generate embeddings for each record added to the database. These embeddings will be stored alongside the Hotels data for future querying. Start by using the [CRUD API's Template](https://app.gitbook.com/s/ZCdm9aJ8vkvDIIbg04AL/templates/crud-apis) to streamline the creation of your API. We'll specifically choose the `/hotels POST` API and the `Create Hotels RB` Request Body. This will provide a boilerplate to simplify the creation of the POST API's request body.

Hotels CRUD API Template

Once the template is applied, head over to the Flow Builder for the generated `POST /hotels` API and start customising the flow. #### Execute Subflow * Add an [Execute Subflow](https://app.gitbook.com/s/ZCdm9aJ8vkvDIIbg04AL/flow-builder/node-types/external-nodes/execute-subflow) node and select `create_embeddings` under **Select Subflow to execute** * Pass the `details` field from the request body to the subflow to generate the embeddings: ```json { "text": "$.request.body.details" } ```

Execute Subflow Node

#### Update JSON Object * Add an [Update JSON Object](https://app.gitbook.com/s/ZCdm9aJ8vkvDIIbg04AL/flow-builder/node-types/variable-nodes/json/update-json-object) node to update the request body JSON with generated embeddings. * Use `$.request.body` as the **Existing Object Name** and update the `detailsEmbeddings` field in the request body with the embeddings generated by the subflow: ```json { "detailsEmbeddings": "$..result" } ```

Update JSON Object Node

The remaining nodes - `Insert One`, `Build JSON Object` and `HTTP Response` can be left as they are. These nodes will handle inserting the record into the database, building the final JSON response, and returning the HTTP response, respectively. The final flow should somewhat look like this:

Hotels POST API Final Flow

This setup ensures that each hotel record inserted into the database will have its `detailsEmbeddings` field populated with vector embeddings, making it ready for efficient semantic search. ### Vector Search Index To enable semantic search capabilities, we'll leverage Cosmocloud's [Vector Search Index](https://app.gitbook.com/s/ZCdm9aJ8vkvDIIbg04AL/resources/vector-search). This powerful feature will allow you to query data based on the vector embeddings we've generated for each hotel record. Start by creating a new vector search index named `hotels_vector_search`. Follow the detailed instructions in this tutorial: [Create a Vector Search Index](https://app.gitbook.com/s/ZCdm9aJ8vkvDIIbg04AL/resources/vector-search/create-a-vector-search-index). {% hint style="info" %} Gemini AI `models/text-embedding-004` creates an embedding of size 768, hence we will use this value as Number of Dimensions while creating our Vector Search below. You can check [Gemini docs](https://ai.google.dev/gemini-api/docs/embeddings#embeddings-models) for more info. {% endhint %} You’ll need to configure the fields according to the image below to guide your setup:

Hotels Vector Search Index

By properly configuring your vector search index, you'll be set to perform advanced semantic searches on your hotel data. This index will allow you to query records not just by keyword but by the meaning and context of the hotel descriptions, resulting in more accurate and relevant search results. ### Defining Search Query Parameters First, we’ll create a [Query Params](https://app.gitbook.com/s/ZCdm9aJ8vkvDIIbg04AL/resources/models) model for the search query parameters, which will be used in the API to handle incoming queries. 1. Navigate to **Application Layer** -> **Request Models**. 2. Click on **Create Model** and set the **Model Name** as `Search Hotels QP` and the **Model Type** as `Query Params`. 3. Once created, switch to `Schema` tab and paste the following JSON inside the widget - `JSON to Schema`: ```json { "query": "text", "limit": 10, "offset": 0 } ``` {% hint style="warning" %} Don't forget to mark all query, limit and offset as **required**. (Red dot against the field name) {% endhint %}

Search Hotels Query Parms Model

### Creating the Search Query API Next, we’ll create the `GET /hotels/_search` API to handle the search functionality. 1. Navigate to **Application Layer** -> **APIs**. 2. Click on **Create API** and choose **Start from Scratch**. 3. Name the API `Search Hotels`, set the **Request method** to `GET`, and set the endpoint to `/hotels/_search`. Select the **Query Params** as `Search Hotels QP`, which we created in the previous step. 4. Click `Create` to generate the API. Once the template is applied, head over to the flow builder for the generated `GET /hotels/_search` API and start customising the flow. #### Execute Subflow * Add an [Execute Subflow](https://docs.cosmocloud.io/flow-builder/node-types/external-nodes/execute-subflow) node and select `create_embeddings` Subflow under **Select Subflow to execute** * Pass the `query` parameter from the request query params to generate the embeddings: ```json { "text": "$.request.queryParams.query" } ```

Execute Subflow Node

#### Run Aggregation Pipeline * Below the Execute Subflow node, add a [Run Aggregation Pipeline](https://app.gitbook.com/s/ZCdm9aJ8vkvDIIbg04AL/flow-builder/node-types/database-nodes/run-aggregation-pipeline) node and select the **Database Collection** as `Hotels`. Select the **Response Type** as `Array`. * Edit the **Aggregate Query** as follows: ```json [ { "$vectorSearch": { "index": "hotels_vector_search", "path": "detailsEmbeddings", "queryVector": "$.node_9.result", "numCandidates": 15, "limit": "$.request.queryParams.limit" } }, { "$facet": { "data": [ { "$project": { "hotelName": 1, "details": 1, "score": { "$meta": "vectorSearchScore" } } }, {"$skip": "$.request.queryParams.offset"}, {"$limit": "$.request.queryParams.limit"} ], "count": [ {"$count": "totalCount"} ] } } ] ```

Run Aggregation Pipeline Node

#### HTTP Response * Add an `HTTP Response` node at the end of the flow. * Set the **Response value** to `$..result` and the **Status Code** to `200`.

HTTP Response Node

The final flow for the `GET /hotels/_search` API should resemble the structure below, enabling the API to perform sophisticated semantic searches:

Search Query API Final Flow

## Testing APIs ### Adding Data To add hotel records to the database, use the `POST /hotels` API. We'll be using the [Hotel's Data](https://api.npoint.io/848c8b755df98cd1b4e7) dataset, and each record should be added individually. {% hint style="warning" %} Follow the instructions in the [Testing Free Tier APIs](https://app.gitbook.com/s/ZCdm9aJ8vkvDIIbg04AL/getting-started/6.-testing-free-tier-apis) document for detailed guidance on setting headers and URLs. {% endhint %}

Postman Add Hotels

Once the data is added, inspect your database. You'll notice that a `detailsEmbeddings` field with a `768`-dimensional vector added to each record. These embeddings are generated using the Gemini API and are essential for performing semantic searches.

MongoDB Hotels Collection

### Searching Data To search the data, use the `GET /hotels/_search` API. This API will allow you to perform semantic searches based on the vector embeddings stored in the database. For example, set the query parameter to `stylish city stay, rooftop bar with skyline views, well-equipped fitness center`, with limit as `10` and offset as `0`. The result will return records that are most relevant to the search query.

Postman Search Hotels

In the aggregation pipeline, we added a `score` field which represents the similarity between the query input and the records. This score helps in identifying the most relevant records. To understand more about the **Vector Search Score** refer this [documentation](https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-stage/#atlas-vector-search-score). ## Conclusion {% hint style="success" %} Congratulations on completing this tutorial :tada:! {% endhint %} You've successfully navigated through the process of integrating semantic search capabilities into your application using Cosmocloud and the Gemini API. From creating a Hotels database model, generating vector embeddings, setting up a vector search index, and building both POST and GET APIs, you now have a powerful system capable of understanding and interpreting user intent at a deeper level. Thank you for following along, and happy coding!