Self-Hosted Offline LLM RAG Platform Architecture

LLM

rag

• Self-Hosted Offline LLM RAG Platform Architecture
• Schema
• Video
• Self-Hosted Offline LLM RAG Interface
  • Modular Architecture Design
• Document Processing Module
  • Document Processing Pipeline
  • Document Processing and Chunk Enhancement
  • Embedding Generation with Local Vector Model
  • Vector Database Storage with Milvus
• Query Processor Module
  • Query Processing Pipeline
  • User Query Processing and Context Construction
  • Retriever Module with Local LLM Generation

The self-hosted web interface for LLMs is fully offline, customizable, and capable of searching and explaining contexts using internal data.

Schema

Video

Self-Hosted Offline LLM RAG Interface

Build a fully self-hosted and offline Large Language Model (LLM) application that enables users to interact with internal knowledge sources through a Retrieval-Augmented Generation (RAG) workflow. The platform is designed to operate without external API dependencies, ensuring that sensitive organizational data remains inside the controlled environment.

Provide a customizable web-based interface that allows users to submit questions, retrieve relevant information from internal documents, and receive context-aware explanations generated by the local LLM. The system combines semantic search capabilities with generative AI to improve answer accuracy while maintaining full data privacy.

The interface acts as the entry point for both document management and user interaction, allowing organizations to deploy AI assistants based on their own internal knowledge repositories.

Modular Architecture Design

Separate the system into two independent modules: Document Processing Module and Query Processor Module. Each module must operate independently to improve scalability, maintainability, and deployment flexibility.

The Document Processing Module focuses on preparing knowledge sources before they are consumed by the retrieval system. It handles document ingestion, transformation, embedding generation, and vector database storage.

The Query Processor Module focuses on real-time user interaction. It receives user questions, retrieves relevant information from the knowledge base, and generates responses using the local LLM model.

By separating these responsibilities, the system allows document updates and query operations to run independently without interrupting each other.

Document Processing Module

Document Processing Pipeline

Implement the Document Processing Module as the foundation of the RAG pipeline. This module transforms unstructured data sources into structured knowledge that can be searched efficiently.

The pipeline consists of three major stages:

• Document Processing
• Embedding Generation
• Data Storage

Each stage prepares the information progressively until it becomes ready for semantic retrieval.

Document Processing and Chunk Enhancement

Collect and process various knowledge sources, including internal documents and website content, through an automated ingestion pipeline. Use document processing frameworks such as Docling and LangChain to handle document extraction, conversion, and preparation.

The processing stage converts different document formats into a consistent structure that can be understood by downstream components.

Apply advanced chunking strategies during the preprocessing phase to improve retrieval quality. Instead of splitting documents randomly, optimize the document structure by applying techniques such as:

• Hierarchical Chunker Maintain relationships between document sections by preserving parent-child information. This helps the retrieval process understand document structure and context.
• Filtering and Merging Strategy Remove unnecessary content and combine small related sections to prevent fragmented information during retrieval.
• Token Splitter Control the size of each document segment based on token limitations to ensure compatibility with embedding models and LLM context windows.
• LangChain Document Conversion Convert processed chunks into LangChain document objects containing text content and metadata such as source references, document identifiers, and additional context information.

The output of this stage is a collection of clean and optimized document chunks ready for vectorization.

Embedding Generation with Local Vector Model

Generate semantic representations of processed documents by converting text chunks into numerical vectors.

Use a local embedding API powered by the BGE-M3 embedding model to transform each document chunk into a high-dimensional vector representation.

The embedding process enables the system to understand the semantic meaning of documents rather than relying only on keyword matching.

Each document vector represents the relationship between concepts, allowing the retrieval system to find relevant information even when user queries use different wording compared to the original document.

Running the embedding model locally ensures that document content remains within the private infrastructure.

Vector Database Storage with Milvus

Store generated embeddings and document information inside the Milvus vector database to support high-performance semantic retrieval.

Design the collection schema with the following main fields:

Configure vector indexing on the embedding column to accelerate similarity searches.

Use Inner Product (IP) as the similarity metric to calculate vector relationships and identify the closest matching document chunks.

The stored vector database becomes the foundation for the retrieval layer, supporting semantic search, ranking, and context selection during user queries.

Query Processor Module

Query Processing Pipeline

Develop the Query Processor Module as the real-time interaction layer between users and the knowledge system.

This module contains two primary processes:

• User Query Processing
• Retriever Module

The goal is to transform user questions into optimized prompts, retrieve relevant knowledge, and generate accurate responses using the local LLM.

User Query Processing and Context Construction

Capture user input queries through the web interface and process them before sending them to the retrieval pipeline.

Transform the raw user question into an enhanced query using three main components:

• Context Builder
• Query Enrichment
• Prompt Template

The Context Builder prepares the conversation context and previous interaction information when required.

The Query Enrichment process improves the original question by adding additional information or restructuring the query to improve retrieval accuracy.

The Prompt Template combines the user question, system instructions, and retrieved context format into a structured input that can be consumed by the LLM pipeline.

The generated prompt becomes the input for the Retriever Module.

Retriever Module with Local LLM Generation

Implement the retrieval and generation process using a local LLM model to maintain privacy and offline capability.

Deploy DeepSeek-R1-Distill-Qwen-1.5B through Ollama as the local inference engine. Configure the model with controlled generation parameters, including:

• n_ctx Defines the maximum context window available for processing retrieved information and conversations.
• Temperature Controls response randomness and creativity.
• Top-p Controls token selection probability during generation.

Optimize the retrieval workflow by combining the LLM with the vector database results. The retriever performs the following operations:

• Receive the optimized user prompt.
• Search relevant document chunks from Milvus using vector similarity.
• Combine retrieved context with the user question.
• Apply system-level instructions to guide the model behavior.
• Generate a context-aware response.
• Post-process the output into a clean and readable answer.

The final output provides users with answers grounded in internal knowledge sources rather than relying only on the model’s pretrained information.