Building with Open AI

Step 1: Verify Docker Installation

First, let’s verify that Docker is properly installed and open in your system. Open your terminal (Command Prompt on Windows) and run:

docker --version

You should see the Docker version information if it's installed correctly. If not, you might want to revist the prerequisites and the Docker Basics section.

Step 2: Clone the LLM App Templates Repository

Next, clone the llm-app repository from GitHub. This repository contains all the files you’ll need.

git clone https://github.com/pathwaycom/llm-app.git

If you get an error because you have previously cloned an older version of the llm-app repository, ensure you're in the correct repository directory and update it using:

git pull

This will update your local repository with the latest changes from the remote repository.

Step 3: Navigate to the relevant project directory

Change to the directory where the example is located.

cd examples/pipelines/demo-question-answering

Step 4: Changes in the requirements.txt, app.py and config.yaml files

In the Pathway LLM App repo (which you’ve already starred ⭐), you’ll see the demo question-answering template that uses YAML files to help you quickly set up the pipeline. While this makes Dockerized RAG applications 10x easier to productionize, in this bootcamp, we will take a different approach. You’ll skip the shortcuts and dive deeper into the application code, to build a solid foundation to create powerful customizations as you progress in your Gen AI journey.

Now that you’ve cloned the repo, let’s walk through some changes you’ll make to the files in the directory. This hands-on step will help you better understand how everything works under the hood.

Changes in the requirements.txt file:

To ensure you have all necessary dependencies, update the requirements.txt file with the following content:

pathway[all]>=0.11.0
python-dotenv==1.0.1
mpmath==1.3.0

Changes in the app.py file

After cloning the repository and navigating to the correct directory, modify the app.py file with the following content to ensure proper functionality:

import logging
import sys
import click
import pathway as pw
import yaml
from dotenv import load_dotenv
from pathway.udfs import DiskCache, ExponentialBackoffRetryStrategy
from pathway.xpacks.llm import embedders, llms, parsers, splitters
from pathway.xpacks.llm.question_answering import BaseRAGQuestionAnswerer
from pathway.xpacks.llm.vector_store import VectorStoreServer

# Set your Pathway license key here to use advanced features.
# You can obtain a free license key from: https://pathway.com/get-license
# If you're using the Community version, you can comment this out.
pw.set_license_key("demo-license-key-with-telemetry")

# Set up basic logging to capture key events and errors.
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s %(name)s %(levelname)s %(message)s",
    datefmt="%Y-%m-%d %H:%M:%S",
)

# Load environment variables (e.g., API keys) from the .env file.
load_dotenv()

# Function to handle data sources. The example uses a local folder first,
# but you can easily connect to Google Drive or SharePoint as alternative data sources.
def data_sources(source_configs) -> list[pw.Table]:
    sources = []
    for source_config in source_configs:
        if source_config["kind"] == "local":
            # Reading data from a local directory (default is the 'data' folder).
            # This is the first option used in this example, but it's flexible for other sources.
            source = pw.io.fs.read(
                **source_config["config"],
                format="binary",
                with_metadata=True,
            )
            sources.append(source)

        elif source_config["kind"] == "gdrive":
            # Reading data from a Google Drive folder.
            # Requires OAuth credentials specified in the config.yaml file.
            source = pw.io.gdrive.read(
                **source_config["config"],
                with_metadata=True,
            )
            sources.append(source)

        elif source_config["kind"] == "sharepoint":
            try:
                # Import the SharePoint connector for reading data from SharePoint.
                import pathway.xpacks.connectors.sharepoint as io_sp
                
                # Reading data from a SharePoint folder.
                # Note: The SharePoint connector is part of Pathway's commercial offering.
                source = io_sp.read(**source_config["config"], with_metadata=True)
                sources.append(source)
            except ImportError:
                # If SharePoint is configured but the connector isn't available, exit the program.
                print(
                    "The Pathway Sharepoint connector is part of the commercial offering. "
                    "Please contact us for a commercial license."
                )
                sys.exit(1)

    return sources

# Command-line interface (CLI) function to run the app with a specified config file.
@click.command()
@click.option("--config_file", default="config.yaml", help="Config file to be used.")
def run(config_file: str = "config.yaml"):
    # Load the configuration from the YAML file.
    with open(config_file) as config_f:
        configuration = yaml.safe_load(config_f)

    # Fetch the GPT model from the YAML configuration.
    GPT_MODEL = configuration["llm_config"]["model"]

    # Initialize the OpenAI Embedder to handle embeddings with caching enabled.
    embedder = embedders.OpenAIEmbedder(
        model="text-embedding-ada-002",
        cache_strategy=DiskCache(),
    )

    # Set up OpenAI's GPT model for answering questions with retry and caching strategies.
    chat = llms.OpenAIChat(
        model=GPT_MODEL,
        retry_strategy=ExponentialBackoffRetryStrategy(max_retries=6),
        cache_strategy=DiskCache(),
        temperature=0.05,  # Low temperature for less random responses.
    )

    # Host and port configuration for running the server.
    host_config = configuration["host_config"]
    host, port = host_config["host"], host_config["port"]

    # Initialize the vector store for storing document embeddings in memory.
    # This vector store updates the index dynamically whenever the data source changes
    # and can scale to handle over a million documents.
    doc_store = VectorStoreServer(
        *data_sources(configuration["sources"]),
        embedder=embedder,
        splitter=splitters.TokenCountSplitter(max_tokens=400),  # Split documents by token count.
        parser=parsers.ParseUnstructured(),  # Parse unstructured data for better handling.
    )

    # Create a RAG (Retrieve and Generate) question-answering application.
    rag_app = BaseRAGQuestionAnswerer(llm=chat, indexer=doc_store)

    # Build the server to handle requests at the specified host and port.
    rag_app.build_server(host=host, port=port)

    # Run the server with caching enabled, and handle errors without shutting down.
    rag_app.run_server(with_cache=True, terminate_on_error=False)

# Entry point to execute the app if the script is run directly.
if __name__ == "__main__":
    run()

Changes in the config.yaml file:

Rename the existing app.yaml file to config.yaml.update the content of the config.yaml file with the configuration below. It's easy to comprehend if you read the comments.:

llm_config:
  model: "gpt-3.5-turbo"  # Specify the GPT model to use for question answering.

host_config:
  host: "0.0.0.0"  # Host for running the app.
  port: 8000       # Port for the app.

cache_options:
  with_cache: True             # Enable caching for better performance.
  cache_folder: "./Cache"      # Directory to store cached data.

sources:
  - local_files:               # Data source is local files.
    kind: local
    config:
      path: "data/"            # Path to the local data directory.

  # Optionally, you can configure Google Drive or SharePoint as additional data sources.
  # For Google Drive:
  # - google_drive_folder:
  #   kind: gdrive
  #   config:
  #     object_id: "1cULDv2OaViJBmOfG5WB0oWcgayNrGtVs"
  #     service_user_credentials_file: SERVICE_CREDENTIALS
  #     refresh_interval: 5

  # For SharePoint (commercial offering):
  # - sharepoint_folder:
  #   kind: sharepoint
  #   config:
  #     root_path: ROOT_PATH
  #     url: SHAREPOINT_URL
  #     tenant: SHAREPOINT_TENANT
  #     client_id: SHAREPOINT_CLIENT_ID
  #     cert_path: SHAREPOINT.pem
  #     thumbprint: SHAREPOINT_THUMBPRINT
  #     refresh_interval: 5

Make sure to save the file after applying the changes.

Changes in the Dockerfile:

Modify the Dockerfile with the following content:

FROM python:3.11
WORKDIR /app

RUN apt-get update
&& apt-get install -y python3-opencv
&& rm -rf /var/lib/apt/lists/* /var/cache/apt/archives/*

COPY requirements.txt . RUN pip install -U --no-cache-dir -r requirements.txt

COPY . .

EXPOSE 8000

CMD ["python", "app.py"]

Step 5: Update your .env File with your OpenAI API Key

In the .env file already present in the project, replace the placeholder API key (sk-xxxxxx) with the OpenAI API key you retrieved from the website.

OPENAI_API_KEY=<your_openai_api_key>

Make sure to save the file after updating.

Step 6: Build the Docker Image

Now, let’s build the Docker image. This step might take a few minutes depending on your machine. Ensure you have enough space (approximately 8 GB).

docker build -t rag .

The -t rag part is tagging the Docker image with the name ‘rag’. Whereas the . at the end specifies the build context directory, which is the current directory. This tells Docker to look for the Dockerfile in the current directory and include any files/subdirectories in the build context.

Step 7: Run the Docker Container

Run the Docker container, mounting (described below) the data folder, and exposing port 8000.

For Windows:

docker run -v "%cd%/data:/app/data" -p 8000:8000 rag

For Linux/Mac:

docker run -v "$(pwd)/data:/app/data" -p 8000:8000 --env-file .env rag

Note: You will see the logs for parsing & embedding documents in the Docker image logs. Give it a few minutes to finish up on embeddings. You will see 0 entries (x minibatch(es)) have been... message. If there are no more updates, this means the app is ready for use!

Handling Port Conflicts: If port 8000 is already in use and you see an error related to it, you can specify a different port. For example, if you want to use port 8080 instead, modify the command as follows:

For Windows:

docker run -v "${PWD}/data:/app/data" -p 8000:8000 rag

For Linux/Mac:

docker run -v "$(pwd)/data:/app/data" -p 8080:8000 --env-file .env rag

This will map port 8080 on your local machine to port 8000 in the Docker container. Just remember to update the port in the next step as well.

Open up another terminal window and follow the next steps.

Step 8: Check the List of Files

You will see the logs for parsing & embedding documents in the Docker image logs. Give it a few minutes to finish up on embeddings, you will see 0 entries (x minibatch(es)) have been… message. If there are no more updates, this means the app is ready for use! Now, let's retrieve the list of files from which our LLMs will gather information. To check the available inputs and associated metadata, you can use either the curl command (for Linux/Mac users) or Invoke-WebRequest (for Windows PowerShell users) as described below:

For Linux/Mac Users (curl command):

Use the following curl command to query the app:

curl -X 'POST'   'http://localhost:8000/v1/pw_list_documents'   -H 'accept: */*'   -H 'Content-Type: application/json'

This will return the list of files e.g. if you start with the data folder provided in the demo, the answer will be as follows:

[{"created_at": null, "modified_at": 1718810417, "owner": "root", "path":"data/IdeanomicsInc_20160330_10-K_EX-10.26_9512211_EX-10.26_Content License Agreement.pdf", "seen_at": 1718902304}]

For Windows Users (PowerShell Invoke-WebRequest):

If you're using PowerShell on Windows, you can use the following Invoke-WebRequest command to perform the same query:

Invoke-WebRequest -Uri 'http://localhost:8000/v1/pw_list_documents' `
  -Method POST `
  -Headers @{ "accept" = "*/*"; "Content-Type" = "application/json" }

This will also return the list of files with the associated metadata, similar to the example above.

Key Differences:

• Use curl for Linux/Mac environments or for users who have installed curl on Windows.

• Use Invoke-WebRequest for users working within Windows PowerShell.

This ensures that no matter which environment you're using, you can retrieve the list of documents and associated metadata to confirm that the app is ready for queries.

Step 9: Last Step – Run the RAG Service

You can now run the RAG service. Start by asking a simple question. For example:

For Linux/Mac Users (curl command):

You can use the following curl command to ask a simple question to the RAG service:

curl -X 'POST' \
  'http://0.0.0.0:8000/v1/pw_ai_answer' \
  -H 'accept: */*' \
  -H 'Content-Type: application/json' \
  -d '{
  "prompt": "What is the start date of the contract?"
}'

Note: If you change the port from 8000 to another value (e.g., 8080), make sure to update the curl command accordingly. For example, replace 8000 with 8080 in the URL.

It should return the following answer:

December 21, 2015

For Windows Users (PowerShell Invoke-WebRequest):

If you're using PowerShell on Windows, use the Invoke-WebRequest command to ask the same question:

Invoke-WebRequest -Uri 'http://0.0.0.0:8000/v1/pw_ai_answer' `
  -Method POST `
  -Headers @{ "accept" = "*/*"; "Content-Type" = "application/json" } `
  -Body '{"prompt": "What is the start date of the contract?"}'

Note: Just like with curl, if you change the port to a different value (e.g., 8080), make sure to update the URL in the Invoke-WebRequest command.

This will return the same response with the answer:

December 21, 2015

---

But how do you tweak this for your use-case? Let's see that by understanding the contents of the repo which just used.