DDN INFINIA Multimodal Semantic Search Demo

An AI-powered semantic search application for images, videos, and documents using DDN INFINIA storage, CyborgDB encrypted vector search, and NVIDIA CLIP/BLIP models.

Overview

This demo showcases DDN INFINIA high-performance storage combined with CyborgDB encrypted vector search and NVIDIA's vision-language AI models for enterprise multimodal search applications. Upload content, and search across all media types using natural language queries.

Key Features

Feature	Description
Image Search	CLIP embeddings + BLIP captions for semantic image understanding
Video Search	Frame extraction and scene analysis with keyframe embeddings
Document Search	PDF/DOCX text extraction with AI summarization
Natural Language	Search using descriptive queries, not just keywords
Encrypted Vector Search	CyborgDB stores and queries CLIP embeddings with AES encryption

Quick Start

Prerequisites

• Python 3.9+
• Node.js 18+
• Docker (for CyborgDB)
• NVIDIA GPU (optional, but recommended for faster inference)

Option A: Docker Compose (Recommended)

This runs the backend and CyborgDB together in containers.

# 1. Clone and configure
cd backend
cp .env.example .env
# Edit backend/.env with your NVIDIA_API_KEY and CYBORGDB_API_KEY

# 2. Start backend + CyborgDB
docker compose up --build

# 3. Start frontend (separate terminal)
cd frontend
npm install && npm run dev
# App: http://localhost:5173

Option B: Local Backend (without Docker)

# 1. Install dependencies
./install.sh

# 2. Start backend (separate terminal)
cd backend
docker compose up cyborgdb  # Start CyborgDB
source venv/bin/activate
python main.py
# API: http://localhost:8001
# Docs: http://localhost:8001/docs

# 3. Start frontend (separate terminal)
cd frontend
npm run dev
# App: http://localhost:5173

Architecture

ddn-demo-rag-v3/
├── backend/                    # FastAPI Python backend
│   ├── app/
│   │   ├── api/routes.py       # REST API endpoints
│   │   ├── core/config.py      # Configuration management
│   │   ├── models/schemas.py   # Pydantic models
│   │   └── services/
│   │       ├── storage.py      # DDN INFINIA S3 handler
│   │       ├── ai_models.py    # CLIP, BLIP, document analysis
│   │       └── cyborgdb.py     # CyborgDB vector search client
│   ├── main.py
│   └── requirements.txt
│
├── frontend/                   # React + Vite + TypeScript
│   ├── src/
│   │   ├── components/         # Header, Sidebar
│   │   ├── pages/              # About, Config, Upload, Search, Browse
│   │   ├── services/api.ts     # API client
│   │   └── styles/             # Tailwind CSS + DDN colors
│   ├── public/                 # Production assets
│   └── package.json
│
├── docker-compose.yml          # Backend + CyborgDB containers
├── ARCHITECTURE.md             # Detailed architecture guide
└── install.sh                  # One-click installation (local dev)

Data Flow

Upload:
  Content → AI Analysis (CLIP/BLIP) → DDN INFINIA (files + metadata)
                                    → CyborgDB (encrypted CLIP embeddings)

Search:
  Query → CLIP text embedding → CyborgDB vector search → Ranked results
                               → Keyword fallback (if CyborgDB unavailable)

Features

Multimodal Content Support

• Images -- JPEG, PNG, WebP, GIF, BMP
• Videos -- MP4, AVI, MOV, WebM
• Documents -- PDF, DOCX, TXT

AI Analysis Pipeline

• CLIP -- Vision-language embeddings for semantic search
• BLIP -- Automatic image captioning
• BART -- Document summarization
• Object Detection -- Automatic tagging of detected objects

Storage & Search

• DDN INFINIA -- High-performance S3-compatible object storage
• CyborgDB -- Encrypted vector search for CLIP embeddings
• Metadata Enrichment -- AI-generated metadata stored with objects
• Presigned URLs -- Secure access to stored media

Environment Variables

Configure in backend/.env:

Variable	Required	Description
NVIDIA_API_KEY	Yes	NVIDIA NIM API key (get one free)
CYBORGDB_API_KEY	Yes	CyborgDB API key (get one free)
CYBORGDB_INDEX_NAME	No	Index name (default: media-embeddings)
CYBORGDB_INDEX_KEY	Yes	32-byte hex-encoded encryption key
HUGGINGFACE_TOKEN	No	For gated/private HuggingFace models

Storage credentials (DDN INFINIA) are configured via the app UI.

API Reference

Configuration

Method	Endpoint	Description
POST	/api/config/ddn	Configure DDN INFINIA
POST	/api/config/aws	Configure AWS S3
GET	/api/config/test/{provider}	Test connection

Upload

Method	Endpoint	Description
POST	/api/upload/image	Upload and analyze image
POST	/api/upload/video	Upload and analyze video
POST	/api/upload/document	Upload and analyze document

Search & Browse

Method	Endpoint	Description
POST	/api/search	Semantic search
POST	/api/browse	Browse all content
DELETE	/api/browse/{key}	Delete object

Health & Metrics

Method	Endpoint	Description
GET	/health	Health check
GET	/metrics	Storage metrics

Tech Stack

Backend

• FastAPI -- High-performance Python web framework
• PyTorch -- Deep learning framework
• Transformers -- Hugging Face model hub
• boto3 -- AWS SDK for S3-compatible storage
• CyborgDB -- Encrypted vector database SDK

Frontend

• React 18 + TypeScript
• Vite -- Build tool
• Tailwind CSS -- Styling
• TanStack Query -- Data fetching
• Framer Motion -- Animations

Infrastructure

• Docker Compose -- Container orchestration
• CyborgDB Service -- Vector search backend
• DDN INFINIA -- S3-compatible object storage

AI Models Used

Model	Purpose	Source
CLIP (ViT-B/32)	Image embeddings	OpenAI
BLIP	Image captioning	Salesforce
BART	Document summarization	Facebook

Troubleshooting

Issue	Solution
Backend won't start	Check Python 3.9+, reinstall deps
Models download slow	First run downloads ~2GB of models
Storage connection fails	Check endpoint URL, network access, and protocol (http vs https)
Frontend API fails	Check Vite proxy port matches backend (8080 for Docker, 8001 for local)
GPU not detected	Install CUDA toolkit, check PyTorch CUDA build
CyborgDB unhealthy	Check docker compose logs cyborgdb -- healthcheck uses Python, not curl
CyborgDB not used during search	Check CYBORGDB_API_KEY is set in .env and backend logs show "Connected to CyborgDB"

License

MIT

---

DDN INFINIA | CyborgDB | NVIDIA CLIP/BLIP | Multimodal Search