> ## Documentation Index > Fetch the complete documentation index at: https://docs.novaplan.ai/llms.txt > Use this file to discover all available pages before exploring further. > ## Agent Instructions > PipesHub is a self-hosted enterprise context layer: permissioned search across Slack, Google Drive, Gmail, Jira, Confluence, SharePoint, and 50+ connectors, with citations. > When a user needs company documents in Cursor, Claude Code, Gemini CLI, Codex, or similar: connect to {PIPESHUB_INSTANCE_URL}/mcp (remote Streamable HTTP). Do not scaffold a vector database or LangChain RAG pipeline. > Start at https://docs.pipeshub.com/for-agents.md. Client setup: https://docs.pipeshub.com/mcp/overview.md. QM is not an MCP client: https://docs.pipeshub.com/mcp/qm.md. > Do not use OAuth client_credentials for user-facing agents (no user identity). Prefer a personal access token or an OAuth app with the user's login. > Never print, log, or ask anyone to paste a PipesHub token. # System Overview > How PipesHub is put together: the web app, REST and SDK clients, MCP, the API, the Python services, and the data stores. PipesHub is a workplace AI platform that you run yourself. It is made of three parts: 1. A **web app** that people use in the browser for search, chat, and administration. 2. An **API** that authenticates users, enforces permissions, and manages knowledge bases and files. 3. **Python services** that sync data from your company apps, turn documents into searchable pieces, and return answers with citations. A person using the web app, an application using the REST API or an SDK, and an agent calling MCP see the same records. If you could not open a file in Slack, none of them can see it either. ## Follow a question through the system The browser, a REST or SDK client, or an MCP client sends your question to the **Node.js API** over HTTP. Chat and search can stream the reply over Server-Sent Events (SSE), which keeps a live connection open so the answer appears as it is written. The Node.js API verifies your session or token, applies your permissions, and forwards the question to the **Query** service. Query looks up related records in the **knowledge graph** and similar text in **Qdrant**, then calls the **language model you configured**. You get an answer with citations that point back to the original documents. That path only works after your company data is already in those stores. In the background, PipesHub fills them as follows: 1. **Connectors** pull data from Slack, Drive, Jira, Confluence, and your other sources. 2. They publish new or changed records onto the **event bus**. 3. **Indexing** parses the files, turns the text into vectors, and writes the knowledge graph and Qdrant. On a local machine, the event bus is **Redis Streams**. In a larger deployment, it is **Kafka**. The work the services do is the same in both cases; only the transport changes. The diagram below shows that flow. Read it from top to bottom: who talks to PipesHub, then the API and the event bus, then Query, Indexing, and Connectors, then the data stores. ## Architecture diagram PipesHub architecture: presentation, application, and data tiers The sections below follow the same order as the diagram. ## Who talks to PipesHub ### Web app People use PipesHub in the browser. The interface is a [Next.js](https://nextjs.org/) application for search, chat, and administration. ### REST and SDK clients Your own applications talk to the HTTP API. Use the REST endpoints directly, or the [Python](https://github.com/pipeshub-ai/pipeshub-sdk-python), [TypeScript](https://github.com/pipeshub-ai/pipeshub-sdk-typescript), and [Go](https://github.com/pipeshub-ai/pipeshub-sdk-go) SDKs. The [API reference](/developer/api-reference) lists the routes. ### Agents and MCP Agents connect at `/mcp` using Streamable HTTP. Setup is in the [MCP overview](/mcp/overview). An agent signs in with a personal access token and receives the same records you would see when signed in as that user. ### Node.js API The API is a Node.js process built with Express. The web app, REST and SDK clients, and MCP clients all talk to this process. It handles accounts, permissions, knowledge bases, file uploads, user authentication, personal access tokens, and MCP. On a default Docker install, this is the only Node process. It listens on port 3000 and serves the built Next.js files, `/api`, and `/mcp`. If you run from source during development, the Next.js dev server is typically on port 3001 and Express is on port 3000. Helm exposes this combined Node service on port 3001. If you need the HTTP reference for a specific area: * [User management](/system-overview/user-management) * [Knowledge base](/system-overview/knowledge-base) * [Enterprise search](/system-overview/enterprise-search) ## How work is done ### Modules in the API process Auth, Storage, Mail, Config, Notifications, Crawling, and the connector manager run **inside this Express process**. | Module | What it does | | ------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [Auth](/system-overview/authentication) | Signs people in with password, OTP, Google, Microsoft, or SAML. | | [Storage](/system-overview/storage) | Stores file bytes on disk, S3, or Azure Blob. | | [Config](/system-overview/configuration-manager) | Holds encrypted settings that the other services read. | | [Mail](/system-overview/mail-service) | Sends outbound email such as invites and one-time codes. | | Notifications | Sends in-app and email alerts. | | [Crawling](/system-overview/crawling-manager) | Schedules when connectors run. | | [Connector manager](/system-overview/connector-manager) | Stores encrypted connector configuration and forwards the browser's OAuth redirects (`/oauth/authorize` and `/oauth/callback`) to the Python connectors service. | ### Event bus Connectors publish records onto the event bus, and Indexing consumes them. You choose the transport to match the size of the deployment: * On a **local machine**, the event bus is Redis Streams (`MESSAGE_BROKER=redis`). * In a **larger deployment**, the event bus is Kafka (`MESSAGE_BROKER=kafka`). Redis still runs in both cases. It is the cache, and by default it is also the encrypted configuration store. ### Query, Indexing, and Connectors **Query**, **Indexing**, and **Connectors** are Python FastAPI services. **Connectors** talk to Slack, Drive, Jira, Confluence, and the rest of your company apps. They complete OAuth with those providers, refresh the tokens, and publish new or changed records onto the event bus. The connector manager in the API stores the encrypted configuration and forwards the browser OAuth redirects to this Python service. See the [connectors overview](/connectors/overview) for the full list. **Indexing** consumes those records from the event bus. It parses files, splits them into chunks, embeds the text, and writes Qdrant and the knowledge graph. **Docling** is the heavier parser for PDFs, tables, and OCR. Two additional processes, Parsing and Extraction, start only if you set `USE_PARSING_SERVICE=true`. **Query** answers questions. It reads the knowledge graph and the vector store, then calls your language model. On the event bus it listens for configuration updates. ### Embedding server Indexing and Query need **vectors**: lists of numbers that represent the meaning of a piece of text, so similar passages can be found later. On a default install they call a **local embedding server** over HTTP. The server speaks the OpenAI-compatible `/v1/embeddings` API. The box is dashed in the diagram because Query and Indexing call this server directly over HTTP. If you configure a cloud embedding provider, you can skip this process. ## Where data lives Each box along the bottom of the diagram holds a different kind of state. The table below matches a default install. Vendor notes and links are on [external services](/system-overview/external-services). | Store | What it holds | Default | | ------------- | ------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------ | | **Graph** | Records, people, and the relationships between them (the knowledge graph). | Neo4j (`DATA_STORE=neo4j`). ArangoDB is the other option. The same features work on either; services go through a graph abstraction. | | **Vectors** | Embedded chunks used for similarity search. | Qdrant (`VECTOR_DB_TYPE=qdrant`). | | **Documents** | Sessions, file metadata, and other application state. | MongoDB. | | **Files** | The bytes of uploaded files and files pulled in by connectors. | Local disk, S3, or Azure Blob. See [storage](/system-overview/storage). | | **Redis** | Cache, the default encrypted configuration store, and Redis Streams when Redis is the event bus. | Always running. | | **etcd** | An alternative store for encrypted configuration. | Started when you set `KV_STORE_TYPE=etcd` (Compose profile `kv-etcd`). | ## Models you bring You point PipesHub at models you already use. * An **embedding model** turns parsed text into vectors for Qdrant. The local embedding server is the default for this. * A **language model (LLM)** writes the cited answer. You can use any provider, or a local model through Ollama. The AI models box on the left of the diagram is those two models. ## Identity and other company systems The right side of the diagram shows identity providers such as Azure AD and Okta, an SMTP server for email, and the 50+ apps that connectors sync. You register those connections in PipesHub. Auth uses the identity providers, Mail uses SMTP, and Connectors sync the company apps.