Skip to Navigation
Cluster Observer
-
The primary goal of the
Cluster Observeris to maintain the Replication Factor of each database in the cluster. -
This observer is always running on the Leader node.
Operation Flow
-
In order to maintain the Replication Factor, every newly elected Leader starts measuring the health of each node by creating dedicated maintenance TCP connections to all other nodes in the cluster.
-
Each node reports the current status of all its databases in intervals of 250 milliseconds (by default).
TheCluster Observerwill consume those reports every 500 milliseconds (by default). -
Upon a node failure, the Dynamic Database Distribution sequence
will take place in order to ensure that theReplication Factordoes not change.
Note
The Cluster Observer stores its information in memory, so when the Leader loses its leadership, the collected reports of the Cluster Observer and its decisions log are lost.
Interacting with the Cluster Observer
- You can interact with the
Cluster Observerusing the following REST API calls:
| URL | Method | Query Params. | Description |
|---|---|---|---|
/admin/cluster/observer/suspend |
POST | value=[bool] |
Setting false will suspend the Cluster Observer operation for the current Leader term. |
/admin/cluster/observer/decisions |
GET | Fetch the log of the recent decisions made by the cluster observer. | |
/admin/cluster/maintenance-stats |
GET | Fetch the latest reports of the Cluster Observer |
For Example
-
Let us assume a five node cluster, with servers A, B, C, D, E.
We create a database with a replication factor of 3 and define an ETL task. -
The newly created database will be distributed automatically to three of the cluster nodes.
Let's assume it is distributed to B, C and E (So the database group is [B,C,E]),
and the cluster decides that node C is the responsible for performing the ETL task. -
If node C goes offline or is not reachable, the Observer will notice it and relocate the database from node C to another available node. Meanwhile the ETL task will failover to be performed by another available node from the Database Group.