Skip to main content
This page describes the databases and brokers along the bottom of the system overview diagram. A default install starts these services for you. Use this page to learn what each service is for, and when you would switch to the other option. The sections follow the same order as the diagram: search data, application data, then how services pass work to each other and where configuration is stored.

​
Knowledge graph (Neo4j or ArangoDB)

The knowledge graph is how PipesHub stores that a Slack thread, a Drive file, and a person belong together. Permission checks and citations follow those links. You choose one graph database with the DATA_STORE environment variable. Application services talk to a graph abstraction, so the product behaves the same on either backend.
  • Neo4j is the default (DATA_STORE=neo4j). The default install, backend/env.template, and the Helm chart all use Neo4j.
  • ArangoDB is the other option (DATA_STORE=arangodb). If an existing install already has an ArangoDB data volume, the installer keeps ArangoDB.

​
Vector search (Qdrant)

After Indexing parses a document, an embedding model turns each chunk of text into a vector. Qdrant stores those vectors and finds similar text when Query answers a question. A default install uses Qdrant. The application also supports OpenSearch and Redis (VECTOR_DB_TYPE=opensearch or VECTOR_DB_TYPE=redis). Those values apply only when the app container receives VECTOR_DB_TYPE. The default Docker Compose file does not pass that variable, so Qdrant stays in use unless you add it to the service environment.

​
Application data (MongoDB)

MongoDB stores application data that is not part of the knowledge graph: sessions, file metadata, and similar state.

​
Files

The contents of files are not stored in MongoDB. They live on local disk, S3, or Azure Blob, through the storage module in the API.

​
How services hand work to each other

Connectors publish records, and Indexing consumes them. That path is the event bus in the architecture diagram.
  • On a local machine, the event bus is Redis Streams (MESSAGE_BROKER=redis). Kafka is not started.
  • In a larger deployment, the event bus is Apache Kafka (MESSAGE_BROKER=kafka).
The Python services stay the same. Only the transport between them changes.

​
Cache, configuration, and Redis

Redis always runs. It can do three jobs, depending on how you deploy:
  1. Cache — frequently used data and locks.
  2. Encrypted configuration — the default key-value store for settings (KV_STORE_TYPE=redis).
  3. Event bus — Redis Streams, when MESSAGE_BROKER=redis.
In a larger deployment, Kafka takes over the event bus. Redis continues to provide the cache, and it remains the configuration store unless you switch that store to etcd.

​
Optional: etcd

etcd is the other place to keep encrypted configuration. A default install uses Redis for this. To use etcd, set KV_STORE_TYPE=etcd and enable the Compose profile kv-etcd.