# MCP Vector Database Server — Hands-On Workshop

## Introduction

Welcome to the **MCP Vector Database Server Workshop**!

This workshop demonstrates how to build a **Model Context Protocol (MCP) server** that provides vector database capabilities through a standardized interface. MCP is an open protocol that enables seamless integration between AI applications and data sources, making it easier to build context-aware AI systems.

### What is MCP?

**Model Context Protocol (MCP)** is an open standard created by Anthropic that allows AI assistants to securely access external data sources and tools. Think of it as a universal adapter that lets AI applications connect to various services through a standardized interface.

**Key Benefits:**
- **Standardized Communication**: One protocol for all integrations
- **Security**: Controlled access to resources
- **Flexibility**: Easy to add new tools and data sources
- **Interoperability**: Works with any MCP-compatible client

### Project Overview

This project implements an MCP server that exposes two main capabilities:
1. **Vector Database Creation**: Generate FAISS indexes from knowledge bases
2. **Semantic Search**: Query the vector database using natural language

---

## Table of Contents

1. [Prerequisites](#prerequisites)
2. [Installation and Environment Setup](#installation-and-environment-setup)
3. [Project Structure](#project-structure)
4. [Step-by-Step Guide](#step-by-step-guide)
   - [File 1: Generate Vector Database](#file-1-generate-vector-database)
   - [File 2: Search Vector Database](#file-2-search-vector-database)
   - [File 3: MCP Vector Server](#file-3-mcp-vector-server)
   - [File 4: MCP Client](#file-4-mcp-client)
5. [Running the Complete System](#running-the-complete-system)
6. [Key Concepts](#key-concepts)
7. [Troubleshooting](#troubleshooting)
8. [Advanced Usage](#advanced-usage)

---

## Prerequisites

- **Operating System**: macOS, Linux, or Windows
- **Python**: Version 3.8 or higher
- **RAM**: At least 4GB available
- **Disk Space**: ~500MB for models and dependencies

---

## Installation and Environment Setup

### 1. Create Virtual Environment

```bash
# Navigate to the project directory
cd Terraform_Project/Hands-on/MCP2

# Create virtual environment
python3 -m venv venv

# Activate virtual environment
# On macOS/Linux:
source venv/bin/activate
# On Windows:
venv\Scripts\activate
```

### 2. Install Python Dependencies

```bash
# Install required packages
pip install -U sentence-transformers faiss-cpu numpy mcp fastmcp
```

### 3. Verify Installation

```bash
# Check if packages are installed
python3 -c "import faiss; import sentence_transformers; import mcp; print('✅ All packages installed successfully!')"
```

---

## Project Structure

```
MCP + Claude Desktop/
├── generate_vector_db.py          # Creates FAISS vector database from PDFs
├── search_vector_db.py             # Searches the vector database
├── mcp_vector_server.py            # MCP server implementation
├── client_vector.py                # MCP client for testing
├── data/                           # All data-related files
│   ├── PDF/                        # Place your PDF files here
│   │   └── *.pdf                   # Any PDF will be processed
│   ├── knowledge_base.json         # Generated - extracted text chunks
│   └── faiss_index/                # Generated - vector database
│       ├── faiss.index             # FAISS index (binary)
│       ├── embeddings.npy          # Stored embeddings (numpy)
│       └── texts.json              # Text chunks for retrieval
├── requirements.txt                # Python dependencies
├── Dockerfile                      # Docker configuration
├── docker-compose.yml              # Docker Compose setup
└── README.md                       # This file
```

### File Descriptions

| File | Purpose | Type |
|------|---------|------|
| `generate_vector_db.py` | Extracts text from PDFs and creates vector DB | Utility |
| `search_vector_db.py` | Standalone script to search vector DB | Utility |
| `mcp_vector_server.py` | MCP server exposing vector DB tools | Server |
| `client_vector.py` | MCP client for testing the server | Client |
| `data/PDF/` | Folder to place your PDF documents | Data Input |
| `data/knowledge_base.json` | Extracted text chunks (auto-generated) | Generated |
| `data/faiss_index/` | FAISS index and metadata files | Generated |

---

## Step-by-Step Guide

### File 1: Generate Vector Database

**File:** [`generate_vector_db.py`](./generate_vector_db.py)

#### Overview

This script creates a FAISS vector database by **automatically extracting text from all PDF files** in the `data/PDF/` folder. It demonstrates:
- PDF text extraction using PyPDF2
- Text chunking with overlap for better retrieval
- Generating embeddings using SentenceTransformers
- Creating a FAISS index with cosine similarity

#### Key Components

1. **PDF Extraction**: Uses PyPDF2 to extract text from all PDFs
2. **Text Chunking**: Splits text into ~1000 character chunks with 200 char overlap
3. **Embedding Model**: `all-MiniLM-L6-v2` (lightweight, 384 dimensions)
4. **FAISS Index**: `IndexFlatIP` for inner product (cosine similarity)
5. **Source Tracking**: Each chunk includes metadata about its source PDF

#### How It Works

```python
# 1. Find all PDFs in the knowledge base folder
pdf_files = list(pdf_dir.glob("*.pdf"))

# 2. Extract text from each PDF
for pdf_path in pdf_files:
    text = extract_text_from_pdf(pdf_path)
    chunks = split_text_into_chunks(text, chunk_size=1000, overlap=200)
    # Add source metadata
    chunks_with_source = [f"[Source: {pdf_path.name}]\n{chunk}" for chunk in chunks]
    all_chunks.extend(chunks_with_source)

# 3. Generate embeddings
model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = model.encode(all_chunks)
embeddings = normalize(embeddings)  # For cosine similarity

# 4. Create and save FAISS index
index = faiss.IndexFlatIP(dim)
index.add(embeddings)
faiss.write_index(index, "faiss_index/faiss.index")
```

#### How to Run

**Step 1**: Add your PDF files to the knowledge base folder:
```
data/PDF/
├── your-document-1.pdf
├── your-document-2.pdf
└── ...any other PDFs...
```

**Step 2**: Run the script:
```bash
python3 generate_vector_db.py
```

#### Expected Output

```
============================================================
🚀 FAISS Vector DB Generator from PDFs
============================================================
📂 Found 1 PDF file(s) in: /path/to/data/PDF
📄 Processing: CTW-2019-PCS-00043-book-of-delivery.pdf
   ✓ Extracted 27 chunks

💾 Knowledge base saved to: knowledge_base.json

📚 27 knowledge chunks to index.
⚙️ Generating embeddings with model: all-MiniLM-L6-v2
Batches: 100%|████████| 1/1 [00:01<00:00, 1.19s/it]

✅ FAISS vector DB created successfully!
📁 Location: faiss_index
📦 Entries: 27 | Dimensions: 384

============================================================
✨ Done! Vector database is ready for use.
============================================================
```

#### Generated Files

After running, you'll have:
- `faiss_index/faiss.index` - FAISS index file
- `faiss_index/embeddings.npy` - Numpy array of embeddings
- `faiss_index/texts.json` - Text chunks with source metadata
- `knowledge_base.json` - Backup of all extracted text

#### What You'll Learn

- How to extract text from PDFs programmatically
- Text chunking strategies for RAG systems
- How to create vector embeddings from text
- FAISS index creation and configuration

---

### File 2: Search Vector Database

**File:** [`search_vector_db.py`](./search_vector_db.py)

#### Overview

This script provides a search function for querying the FAISS vector database. It demonstrates:
- Loading a pre-built FAISS index
- Encoding query text to embeddings
- Performing similarity search
- Ranking and formatting results

#### Key Components

1. **Search Function**: `search_in_faiss(query_text, top_k, index_dir)`
2. **Similarity Threshold**: Minimum score of 0.35 for valid results
3. **Result Formatting**: Displays ranked results with cosine scores

#### How It Works

```python
# 1. Load FAISS index and texts
index = faiss.read_index("faiss_index/faiss.index")
with open("faiss_index/texts.json") as f:
    texts = json.load(f)

# 2. Encode query
model = SentenceTransformer("all-MiniLM-L6-v2")
query_embedding = model.encode([query])
query_embedding = normalize(query_embedding)

# 3. Search for similar vectors
similarities, indices = index.search(query_embedding, top_k)

# 4. Return ranked results
for rank, (idx, score) in enumerate(zip(indices[0], similarities[0])):
    print(f"Rank {rank+1}: {texts[idx]} (Score: {score:.4f})")
```

#### How to Run

**Prerequisites**: Run `generate_vector_db.py` first!

```bash
# Import and use in Python
python3 -c "
from search_vector_db import search_in_faiss
result = search_in_faiss('What are the quality principles?', top_k=2)
print(result)
"
```

#### Expected Output

```
🔎 Query: What are the quality principles?
Top 2 similar results (cosine)
============================================================
🏅 Rank 1 (Cosine: 0.8234)
Critical TechWorks Quality (Quality, Reliability, and Resilience):
We ensure delivery of high-quality, reliable, and resilient software...
------------------------------------------------------------
🏅 Rank 2 (Cosine: 0.5421)
Critical TechWorks Principles:
A principle is a rule, belief, or idea that guides action...
------------------------------------------------------------
```

#### What You'll Learn

- How to load and query FAISS indexes
- Understanding similarity scores
- Result ranking and filtering
- Handling edge cases (missing index, dimension mismatches)

---

### File 3: MCP Vector Server

**File:** [`mcp_vector_server.py`](./mcp_vector_server.py)

#### Overview

This is the **core MCP server** that exposes vector database functionality through the Model Context Protocol. It demonstrates:
- Creating an MCP server with FastMCP
- Defining MCP tools (functions callable by clients)
- Handling asynchronous operations
- Logging and error handling

#### Key Components

1. **FastMCP Server**: Simplified MCP server implementation
2. **Tool 1 - `create_vector_db`**: Creates a FAISS index from JSON
3. **Tool 2 - `search_vector_db`**: Searches the vector database
4. **Logging**: Outputs to stderr (doesn't interfere with MCP protocol)

#### MCP Tools Exposed

##### Tool 1: `create_vector_db`

**Purpose**: Create a FAISS vector database from a JSON knowledge base

**Parameters**:
- `json_path` (str): Path to the knowledge base JSON file
- `output_dir` (str): Directory to save the FAISS index

**Returns**: Success message with entry count

**Example**:
```python
{
  "json_path": "knowledge_base.json",
  "output_dir": "faiss_index"
}
```

##### Tool 2: `search_vector_db`

**Purpose**: Search the vector database using semantic similarity

**Parameters**:
- `query` (str): Search query in natural language
- `index_dir` (str): Directory containing the FAISS index
- `top_k` (int): Number of results to return (default: 5)

**Returns**: Formatted search results with scores

**Example**:
```python
{
  "query": "collaboration principles",
  "index_dir": "faiss_index",
  "top_k": 3
}
```

#### How to Run

```bash
# Start the MCP server
python3 mcp_vector_server.py
```

The server will start and wait for MCP client connections via stdio.

#### Server Architecture

```
┌─────────────────────────────────────┐
│       MCP Vector Server             │
│  (mcp_vector_server.py)             │
├─────────────────────────────────────┤
│                                     │
│  Tool 1: create_vector_db()         │
│  ├─ Load JSON knowledge base        │
│  ├─ Generate embeddings             │
│  ├─ Create FAISS index              │
│  └─ Save index files                │
│                                     │
│  Tool 2: search_vector_db()         │
│  ├─ Load FAISS index                │
│  ├─ Encode query                    │
│  ├─ Perform similarity search       │
│  └─ Return ranked results           │
│                                     │
└─────────────────────────────────────┘
         ↕ (MCP Protocol via stdio)
┌─────────────────────────────────────┐
│         MCP Client                  │
│    (client_vector.py or any         │
│     MCP-compatible client)          │
└─────────────────────────────────────┘
```

#### What You'll Learn

- How to create an MCP server with FastMCP
- Defining tools that clients can call
- MCP protocol communication (stdio)
- Error handling in server tools
- Logging without breaking MCP protocol

---

### File 4: MCP Client

**File:** [`client_vector.py`](./client_vector.py)

#### Overview

This is a **test client** for the MCP server. It demonstrates:
- Connecting to an MCP server via stdio
- Discovering available tools
- Calling server tools with parameters
- Handling responses

#### Key Components

1. **MCPClient Class**: Manages connection and communication
2. **Connection Method**: Establishes stdio connection to server
3. **Tool Calling**: Invokes server tools with arguments
4. **Response Handling**: Processes and displays results

#### How It Works

```python
# 1. Create client and connect to server
client = MCPClient()
await client.connect_to_server("mcp_vector_server.py")

# 2. List available tools
tools = await client.session.list_tools()
print("Available tools:", [t.name for t in tools.tools])

# 3. Call a tool
response = await client.call_tool(
    "search_vector_db",
    query="quality principles",
    index_dir="faiss_index",
    top_k=2
)

# 4. Process response
print(response)
```

#### How to Run

```bash
# Run the client (it will start the server automatically)
python3 client_vector.py mcp_vector_server.py
```

#### Expected Output

```
🔌 Starting MCP client for server: mcp_vector_server.py
[DEBUG] Initializing MCP session...
✅ MCP Handshake Complete

🛠 Tools detected: ['create_vector_db', 'search_vector_db']

⚙ Calling tool 'search_vector_db' with: {'query': 'critical techworks quality', 'index_dir': '/Users/...', 'top_k': 1}

📌 RAW RESPONSE FROM SERVER:
1) score=0.8234
Critical TechWorks Quality (Quality, Reliability, and Resilience):
We ensure delivery of high-quality, reliable, and resilient software...

🔚 Client closed.
```

#### What You'll Learn

- How to build an MCP client
- Connecting to MCP servers via stdio
- Tool discovery and invocation
- Handling MCP responses
- Async/await patterns in Python

---

## Running the Complete System

### Option 1: Standalone Scripts

**Step 1**: Generate the vector database
```bash
python3 generate_vector_db.py
```

**Step 2**: Test search functionality
```bash
python3 -c "
from search_vector_db import search_in_faiss
print(search_in_faiss('collaboration', top_k=2))
"
```

### Option 2: MCP Server + Client

**Step 1**: Ensure vector database exists
```bash
python3 generate_vector_db.py
```

**Step 2**: Run the MCP client (starts server automatically)
```bash
python3 client_vector.py mcp_vector_server.py
```

### Option 3: Use with MCP-Compatible Applications

The MCP server can be integrated with any MCP-compatible application (e.g., Claude Desktop, custom AI assistants).

**Example MCP Configuration** (for Claude Desktop):
```json
{
  "mcpServers": {
    "vector-db": {
      "command": "python3",
      "args": ["/path/to/mcp_vector_server.py"]
    }
  }
}
```

---

## Key Concepts

### 1. Model Context Protocol (MCP)

MCP is an open protocol that standardizes how AI applications communicate with external data sources and tools.

**Key Features**:
- **Tools**: Functions that can be called by AI assistants
- **Resources**: Data sources that can be read
- **Prompts**: Reusable prompt templates
- **Stdio Communication**: Uses standard input/output for communication

**Why MCP?**
- Eliminates the need for custom integrations
- Provides security through controlled access
- Enables interoperability between different AI systems
- Simplifies development of AI-powered applications

### 2. FastMCP

FastMCP is a Python library that simplifies MCP server creation.

**Benefits**:
- Decorator-based tool definition
- Automatic type validation
- Built-in error handling
- Easy to use and understand

**Example**:
```python
from mcp.server.fastmcp import FastMCP

mcp = FastMCP("MyServer")

@mcp.tool()
def my_tool(param: str) -> str:
    return f"Processed: {param}"

mcp.run()
```

### 3. Vector Databases

Vector databases store and retrieve data based on semantic similarity rather than exact matches.

**How They Work**:
1. **Embedding**: Convert text to numerical vectors
2. **Indexing**: Store vectors in an efficient data structure
3. **Querying**: Find vectors similar to a query vector
4. **Ranking**: Return results sorted by similarity

**Use Cases**:
- Semantic search
- Recommendation systems
- Document retrieval
- Question answering

### 4. FAISS (Facebook AI Similarity Search)

FAISS is a library for efficient similarity search and clustering of dense vectors.

**Key Features**:
- **Fast**: Optimized for large-scale search
- **Flexible**: Supports various distance metrics
- **Scalable**: Can handle billions of vectors
- **CPU & GPU**: Works on both architectures

**Index Types**:
- `IndexFlatIP`: Exact search with inner product (used in this project)
- `IndexFlatL2`: Exact search with L2 distance
- `IndexIVFFlat`: Approximate search with clustering
- And many more...

### 5. Embeddings

Embeddings are dense vector representations of text that capture semantic meaning.

**Properties**:
- Similar texts have similar embeddings
- Dimensionality typically ranges from 128 to 1536
- Generated by neural networks trained on large text corpora

**Model Used**: `all-MiniLM-L6-v2`
- **Dimensions**: 384
- **Size**: ~120MB
- **Speed**: Fast inference
- **Quality**: Good for general-purpose tasks

### 6. Cosine Similarity

Cosine similarity measures the cosine of the angle between two vectors.

**Formula**: `similarity = (A · B) / (||A|| × ||B||)`

**Range**: -1 to 1 (for normalized vectors: 0 to 1)
- 1 = Identical direction (very similar)
- 0 = Orthogonal (unrelated)
- -1 = Opposite direction (very dissimilar)

**Why Cosine?**
- Invariant to vector magnitude
- Works well for text embeddings
- Efficient to compute

---

## Troubleshooting

### Issue: Module not found

**Error**: `ModuleNotFoundError: No module named 'mcp'`

**Solution**:
```bash
pip install -U mcp fastmcp
```

### Issue: FAISS index not found

**Error**: `ERROR: faiss.index missing`

**Solution**:
```bash
# Generate the index first
python3 generate_vector_db.py
```

### Issue: Dimension mismatch

**Error**: `Dimension mismatch: query=384, index=768`

**Solution**: Ensure you're using the same embedding model (`all-MiniLM-L6-v2`) for both indexing and searching. Delete the old index and regenerate:
```bash
rm -rf faiss_index/
python3 generate_vector_db.py
```

### Issue: MCP server not responding

**Symptoms**: Client hangs or times out

**Solutions**:
1. Check if the server script path is correct
2. Ensure Python 3.8+ is being used
3. Check server logs in stderr
4. Verify all dependencies are installed

### Issue: Permission denied

**Error**: `PermissionError: [Errno 13] Permission denied`

**Solution**:
```bash
# Make scripts executable
chmod +x generate_vector_db.py
chmod +x mcp_vector_server.py
chmod +x client_vector.py
```

### Issue: Out of memory

**Error**: System runs out of RAM during embedding generation

**Solutions**:
- Close other applications
- Process texts in smaller batches
- Use a smaller embedding model
- Increase system swap space

---

## Advanced Usage

### Custom Knowledge Base

Add your own knowledge by placing PDF files in the `data/PDF/` folder:

```
data/PDF/
├── your-document-1.pdf
├── your-document-2.pdf
├── company-handbook.pdf
├── technical-docs.pdf
└── ...any PDF files...
```

Then regenerate the index:
```bash
python3 generate_vector_db.py
```

The script will:
1. Automatically find all `.pdf` files in the folder
2. Extract text from each PDF
3. Split text into chunks (~1000 chars with 200 char overlap)
4. Generate embeddings and create the FAISS index
5. Add source metadata to each chunk for traceability

> **Note**: If you need to add non-PDF content, you can manually edit `knowledge_base.json` after it's generated, then run the `create_vector_db` MCP tool to rebuild the index.

### Using Different Embedding Models

Modify the `EMBED_MODEL` constant in the scripts:

```python
# Smaller, faster model
EMBED_MODEL = "all-MiniLM-L6-v2"  # 384 dims, ~120MB

# Larger, more accurate model
EMBED_MODEL = "all-mpnet-base-v2"  # 768 dims, ~420MB

# Multilingual model
EMBED_MODEL = "paraphrase-multilingual-MiniLM-L12-v2"  # 384 dims
```

**Important**: Use the same model for both indexing and searching!

### Integrating with Claude Desktop

Add the MCP server to Claude Desktop's configuration:

**macOS**: `~/Library/Application Support/Claude/claude_desktop_config.json`

```json
{
  "mcpServers": {
    "vector-db": {
      "command": "python3",
      "args": ["/absolute/path/to/mcp_vector_server.py"],
      "env": {
        "PYTHONPATH": "/absolute/path/to/project"
      }
    }
  }
}
```

Restart Claude Desktop, and the tools will be available!

### Scaling to Larger Datasets

For larger knowledge bases (10,000+ documents):

1. **Use Approximate Search**:
```python
# Instead of IndexFlatIP
quantizer = faiss.IndexFlatIP(dim)
index = faiss.IndexIVFFlat(quantizer, dim, nlist=100)
index.train(embeddings)
index.add(embeddings)
```

2. **Batch Processing**:
```python
# Process embeddings in batches
batch_size = 1000
for i in range(0, len(texts), batch_size):
    batch = texts[i:i+batch_size]
    batch_embeddings = model.encode(batch)
    index.add(batch_embeddings)
```

3. **Use GPU** (if available):
```bash
pip install faiss-gpu
```

---

## Testing the System

### Test 1: Create Vector Database

```bash
python3 generate_vector_db.py
```

**Expected**: Index files created in `faiss_index/` directory

### Test 2: Search Standalone

```python
from search_vector_db import search_in_faiss

# Test different queries
queries = [
    "What are the collaboration principles?",
    "Tell me about quality",
    "How do we ensure visibility?"
]

for query in queries:
    print(f"\n{'='*60}")
    print(search_in_faiss(query, top_k=1))
```

### Test 3: MCP Server + Client

```bash
python3 client_vector.py mcp_vector_server.py
```

**Expected**: Client connects, discovers tools, executes search, displays results

---

## Next Steps

After completing this workshop, you can:

1. **Extend the MCP Server**: Add more tools (e.g., update index, delete entries)
2. **Build a Web Interface**: Create a Flask/FastAPI frontend for the vector DB
3. **Integrate with AI Assistants**: Connect to Claude, ChatGPT, or custom LLMs
4. **Scale Up**: Use cloud vector databases (Pinecone, Weaviate, Qdrant)
5. **Add Authentication**: Implement access control for the MCP server
6. **Monitor Performance**: Add metrics and logging for production use

---

## Additional Resources

- [MCP Documentation](https://modelcontextprotocol.io/)
- [FastMCP GitHub](https://github.com/jlowin/fastmcp)
- [FAISS Documentation](https://github.com/facebookresearch/faiss)
- [SentenceTransformers](https://www.sbert.net/)
- [Anthropic MCP Servers](https://github.com/anthropics/anthropic-quickstarts)

---

## Support

For questions or issues during the workshop, please contact the Terraform team or refer to the internal documentation.

---

**Happy Learning! 🚀**
