A worker communicates with the data subscription using a custom protocol on top of a long-lived TCP connection. Each successful batch processing consists of these stages:
The server sends documents a batch.
Worker sends acknowledgment message after it finishes processing the batch.
The server returns the client a notification that the acknowledgment persistence is done and it is ready to send the next batch.
When the responsible node handling the subscription is down, the subscription task can be manually reassigned to another node in the cluster.
With the Enterprise license the cluster will automatically reassign the work to another node.
The TCP connection is also used as the "state" of the worker process and as long as it's alive, the server will not allow other clients to consume the subscription.
The TCP connection is kept alive and monitored using "heartbeat" messages. If it's found nonfunctional, the current batch progress will be restarted.
See the sequence diagram below that summarizes the lifetime of a subscription connection.