Private RAG Knowledge System
On-premise retrieval-augmented generation system for a professional services firm handling sensitive client documents.
The Challenge
A legal-adjacent services firm needed internal AI search across a large body of client documents but couldn't use cloud AI tools due to data sensitivity and compliance constraints.
The Approach
Assessed infrastructure constraints. Designed a private architecture using local models and vector storage. Chose Ollama for model serving and pgvector for embeddings storage.
The Solution
Deployed a private RAG stack: Ollama running Mistral 7B, LangChain for document chunking and retrieval, PostgreSQL with pgvector for embeddings, and a clean React UI for staff queries. All running on their existing server infrastructure.
Architecture
- Private DocumentsInternal client files
- Document ProcessingChunking and ingestion via LangChain
- Embedding & IndexingVector embeddings generated locally
- Vector StoragePostgreSQL + pgvector
- RetrievalRelevant passages selected for the query
- Local Model ServingOllama
- Grounded ResponseAnswer assembled from retrieved passages
- Staff InterfaceReact query UI
Designed for private infrastructure: the retrieval and generation path does not depend on an external model provider.
Related projects.
Ready to build something
that matters?
Bring me one business problem. I typically respond within 24 hours.