# RAG with Gemini and other Open AI Alternatives

In this module, we’ll cover how to build a RAG project using alternate APIs by Gemini, Replicate, and 300+ LLMs apart from OpenAI. 

Here you'll use Google Gemini Pro for chat completions/generation and Sentence Transformer for embeddings. 

If you've already built a pipeline using OpenAI, you can jump to [Step 4](#step-4-create-a-.env-file-and-put-your-gemini-api-key).

## Introduction

We’ll walk you through setting up a RAG project using Gemini Pro and Pathway.

### **Key Features:**

* Create an in-memory persistent vector store with real-time document indexing using Pathway that can easily work with documents in your Google Drive, Microsoft 365 SharePoint, Databases, Local directory, etc.
* Connect an LLM model of choice (<https://docs.litellm.ai/docs/providers>) to your knowledge base.
* Get quality, accurate, and precise responses to your questions.
* Ask questions about folders, files, or all your documents easily, with the help of filtering options.
* Get an executive outlook for a question on different files to easily access available knowledge in your documents.

## Prerequisites

Before we begin, ensure you have the following requirements we shared earlier:

* **Docker Desktop**: This tool allows you to run applications in isolated containers. Download Docker Desktop. (Note: Your antivirus software might block the installation, so temporarily disable it if needed.)
* **If you’re using VS Code,** consider installing the Docker extension to manage containers directly from the editor.

## Step-by-Step Process

## Step 1: Verify Docker Installation

First, let’s verify that Docker is properly installed and open in your system. Open your terminal (Command Prompt on Windows) and run:

{% code overflow="wrap" %}

```sh
docker --version
```

{% endcode %}

You should see the Docker version information if it's installed correctly.

## Step 2: Clone the LLM App Templates Repository

Next, clone the `llm-app` repository from GitHub. This repository contains all the files you’ll need.

{% code overflow="wrap" %}

```sh
git clone https://github.com/pathwaycom/llm-app.git
```

{% endcode %}

If you get an error because you have previously cloned an older version of the `llm-app` repository, **ensure you're in the correct repository** directory and update it using:

{% code overflow="wrap" %}

```sh
git pull
```

{% endcode %}

This will update your local repository with the latest changes from the remote repository.

## Step 3: Navigate to the Project Directory

Change to the directory where the relevant example of your current project is located.

{% code overflow="wrap" %}

```sh
cd examples/pipelines/demo-question-answering
```

{% endcode %}

## Step 4: Update your `.env` File with your Gemini API Key

If you've already built a pipeline with Open AI, this is where things get slightly different. Configure your key in a `.env` file by providing it as follows:

{% code overflow="wrap" %}

```sh
GEMINI_API_KEY=*******
```

{% endcode %}

Replace `*******` with your actual Gemini API key. Save the file as `.env` in the `demo-question-answering` folder.

## Step 5: Update `requirements.txt` File

Add the following dependencies to the `requirements.txt` file that enable us to use Pathway LiteLLM wrapper, Google's APIs, and Sentence Transformers (a.k.a. SBERT) for embeddings:

{% code overflow="wrap" %}

```shell
pathway[all]>=0.11.0
python-dotenv==1.0.1
mpmath==1.3.0
litellm>=1.35
Google-generativeai
Sentence-transformers
```

{% endcode %}

## Step 6: Update `app.py` Code

Replace the existing `app.py` code with the following:

{% code overflow="wrap" %}

```python
import logging
import sys
import os
import click
import pathway as pw
import yaml
from dotenv import load_dotenv

from pathway.udfs import DiskCache, ExponentialBackoffRetryStrategy
from pathway.xpacks.llm import embedders, llms, parsers, splitters
from pathway.xpacks.llm.question_answering import BaseRAGQuestionAnswerer
from pathway.xpacks.llm.vector_store import VectorStoreServer

# To use advanced features with Pathway Scale, get your free license key from
# https://pathway.com/features and paste it below.
# To use Pathway Community, comment out the line below.
pw.set_license_key("demo-license-key-with-telemetry")

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(name)s %(levelname)s %(message)s",
    datefmt="%Y-%m-%d %H:%M:%S",
)

load_dotenv()

def data_sources(source_configs) -> list[pw.Table]:
    sources = []
    for source_config in source_configs:
        if source_config["kind"] == "local":
            source = pw.io.fs.read(
                **source_config["config"],
                format="binary",
                with_metadata=True,
            )
            sources.append(source)
        elif source_config["kind"] == "gdrive":
            source = pw.io.gdrive.read(
                **source_config["config"],
                with_metadata=True,
            )
            sources.append(source)
        elif source_config["kind"] == "sharepoint":
            try:
                import pathway.xpacks.connectors.sharepoint as io_sp

                source = io_sp.read(**source_config["config"], with_metadata=True)
                sources.append(source)
            except ImportError:
                print(
                    "The Pathway Sharepoint connector is part of the commercial offering, "
                    "please contact us for a commercial license."
                )
                sys.exit(1)

    return sources

@click.command()
@click.option("--config_file", default="config.yaml", help="Config file to be used.")
def run(config_file: str = "config.yaml"):
    with open(config_file) as config_f:
        configuration = yaml.safe_load(config_f)

    LLM_MODEL = configuration["llm_config"]["model"]

    embedding_model = "avsolatorio/GIST-small-Embedding-v0"

    embedder = embedders.SentenceTransformerEmbedder(
        embedding_model,
        call_kwargs={"show_progress_bar": False}
    )

    chat = llms.LiteLLMChat(
        model=LLM_MODEL,
        retry_strategy=ExponentialBackoffRetryStrategy(max_retries=6),
        cache_strategy=DiskCache(),
    )

    host_config = configuration["host_config"]
    host, port = host_config["host"], host_config["port"]

    doc_store = VectorStoreServer(
        *data_sources(configuration["sources"]),
        embedder=embedder,
        splitter=splitters.TokenCountSplitter(max_tokens=400),
        parser=parsers.ParseUnstructured(),
    )

    rag_app = BaseRAGQuestionAnswerer(llm=chat, indexer=doc_store)

    rag_app.build_server(host=host, port=port)

    rag_app.run_server(with_cache=True, terminate_on_error=False)

if __name__ == "__main__":
    run()
```

{% endcode %}

### Key Changes:

* **Embedding Model Selection**: Chose `avsolatorio/GIST-small-Embedding-v0` for embedding chunked texts. This model is compact and performed well in tests. Other options include `mixedbread-ai/mxbai-embed-large-v1`and `avsolatorio/GIST-Embedding-v0`(For other possible choices, take a look at the [MTEB Leaderboard](https://huggingface.co/spaces/mteb/leaderboard) managed by HuggingFace)
* **LLM Initialization**: Integrated the Gemini Pro model by updating the configuration and initialization script to use `LLM_MODEL` instead of `GPT_MODEL`.

## Step 7: Update `config.yaml` File

Update the model specification in the `config.yaml` file to use Gemini Pro and keep rest of the items as is:

{% code overflow="wrap" %}

```yaml
llm_config:
  model: "gemini/gemini-pro"
host_config:
  host: "0.0.0.0"
  port: 8000
cache_options:
  with_cache: True
  cache_folder: "./Cache"
sources:
  - local_files:
    kind: local
    config:
      # Please refer to
      # https://pathway.com/developers/api-docs/pathway-io/fs#pathway.io.fs.read
      # for options definition
      path: "data/"
  # - google_drive_folder:
  #   kind: gdrive
  #   config:
  #     # Please refer to
  #     # https://pathway.com/developers/api-docs/pathway-io/gdrive#pathway.io.gdrive.read
  #     # for options definition
  #     # Please follow https://pathway.com/developers/user-guide/connectors/gdrive-connector/#setting-up-google-drive
  #     # for instructions on getting credentials
  #     object_id: "1cULDv2OaViJBmOfG5WB0oWcgayNrGtVs" # folder used in the managed demo
  #     service_user_credentials_file: SERVICE_CREDENTIALS
  #     refresh_interval: 5
  # - sharepoint_folder:
  #   kind: sharepoint
  #   config:
  #     # The sharepoint is part of our commercial offering, please contact us to use it
  #     # Please contact here: `contact@pathway.com`
  #     root_path: ROOT_PATH
  #     url: SHAREPOINT_URL
  #     tenant: SHAREPOINT_TENANT
  #     client_id: SHAREPOINT_CLIENT_ID
  #     cert_path: SHAREPOINT.pem
  #     thumbprint: SHAREPOINT_THUMBPRINT
  #     refresh_interval: 5
```

{% endcode %}

## Step 8: Changes in the `Dockerfile`:

Modify the `Dockerfile` with the following content:

```
FROM python:3.11
WORKDIR /app

RUN apt-get update
&& apt-get install -y python3-opencv
&& rm -rf /var/lib/apt/lists/* /var/cache/apt/archives/*

COPY requirements.txt . RUN pip install -U --no-cache-dir -r requirements.txt

COPY . .

EXPOSE 8000

CMD ["python", "app.py"]
```

## Step 9: Build the Docker Image

Build the Docker image. This step might take a few minutes depending on your machine. Ensure you have enough space (approximately 8 GB).

{% code overflow="wrap" %}

```sh
docker build -t raggem .
```

{% endcode %}

## Step 10: Run the Docker Container

Run the Docker container, mounting the data folder and exposing port 8000.&#x20;

### For Windows:

{% code overflow="wrap" %}

```sh
docker run -v "${PWD}/data:/app/data" -p 8000:8000 raggem
```

{% endcode %}

### For Linux/Mac:

{% code overflow="wrap" %}

```sh
docker run -v "$(pwd)/data:/app/data" -p 8000:8000 --env-file .env raggem
```

{% endcode %}

**Handling Port Conflicts**: If port 8000 is already in use, specify a different port. For example, to use port 8080:

### For Windows:

{% code overflow="wrap" %}

```sh
docker run -v "${PWD}/data:/app/data" -p 8080:8000 raggem
```

{% endcode %}

### For Linux/Mac:

{% code overflow="wrap" %}

```sh
docker run -v "$(pwd)/data:/app/data" -p 8080:8000 --env-file .env raggem
```

{% endcode %}

## Step 11: Check the List of Files

To see the list of files in the data folder, use the curl command. While doing this ensure that you'll need to update the port here as well if you changed it in the previous step while managing port conflicts:

{% code overflow="wrap" %}

```sh
curl -X 'POST' 'http://localhost:8000/v1/pw_list_documents' -H 'accept: */*' -H 'Content-Type: application/json'
```

{% endcode %}

This will return the list of files that are ingested as an external data source for your RAG project.

## Step 12: Last Step – Run the RAG Service

You can now run the RAG service. Start by asking a simple question. For example:

### **For Linux/Mac Users (`curl` command):**

You can use the following `curl` command to ask a simple question to the RAG service:

{% code overflow="wrap" %}

```python
curl -X 'POST' 'http://0.0.0.0:8000/v1/pw_ai_answer' -H 'accept: */*' -H 'Content-Type: application/json' -d '{
  "prompt": "What are the terms and conditions"
}'
```

{% endcode %}

*Note: If you change the port from `8000` to another value (e.g., `8080`), make sure to update the `curl` command accordingly. For example, replace `8000` with `8080` in the URL.*

It should return the following answer:

{% code overflow="wrap" %}

```
"The terms and conditions are: Rights Granted, Use of Titles, Warranties and Representations, Indemnification, Disclaimers, Limitation of Liability, Governing Law, Dispute Resolution, Term, Termination, Entire Agreement, Assignment, Waiver, Severability, Notices, Counterparts and Construction."
```

{% endcode %}

### **For Windows Users (PowerShell `Invoke-WebRequest`):**

If you're using PowerShell on Windows, use the `Invoke-WebRequest` command to ask the same question:

```python
Invoke-WebRequest -Uri 'http://0.0.0.0:8000/v1/pw_ai_answer' `
  -Method POST `
  -Headers @{ "accept" = "*/*"; "Content-Type" = "application/json" } `
  -Body '{"prompt": "What are the terms and conditions?}'
```

*Note: Just like with `curl`, if you change the port to a different value (e.g., `8080`), make sure to update the URL in the `Invoke-WebRequest` command.*

This will return the same response with the answer:

{% code overflow="wrap" %}

```
"The terms and conditions are: Rights Granted, Use of Titles, Warranties and Representations, Indemnification, Disclaimers, Limitation of Liability, Governing Law, Dispute Resolution, Term, Termination, Entire Agreement, Assignment, Waiver, Severability, Notices, Counterparts and Construction."
```

{% endcode %}

## Conclusion

This will help you set up a powerful RAG pipeline with Gemini Pro.

**If you get stuck,** you should explore the Pathway documentation here and try to find the issue yourself once. It will also help you understand the code better, and many of your queries can actually be figured out via LLMs as well. &#x20;

If still needed, you are very welcomed to ask it in the Discord channel for this bootcamp or also post your query on [LiteLLM's Discord](https://discord.com/invite/wuPM9dRgDw). It is generally a great practice to post your queries in the most relevant open source communities.  :smile: