Ongoing Tasks: Queue ETL Overview

  • Message brokers are high-throughput, distributed messaging services that host data they receive from producer applications and serve it to consumer clients via FIFO data queue/s.
  • RavenDB 5.4 and on can function as a Producer in this architecture.
    RavenDB 6.0 and on can also function as a Consumer.

    This overview and the other pages in the Queue ETL section explain only RavenDB's role as a Producer through the execution of ETL tasks.
    To learn about RavenDB's role as a Consumer please refer to the Queue Sink section.

  • RavenDB produces messages to broker queues via ETL tasks that -
    • Extract data from specified document collections.
    • Transform the data to new JSON objects.
    • Load the JSON objects to the message broker.
  • Prior to loading the JSON objects to the designated broker, RavenDB wraps them as CloudEvents messages using the CloudEvents Library.
  • Supported message brokers currently include Apache Kafka and RabbitMQ.

  • In this page:

Supported Message Brokers

Queue brokers that RavenDB can currently read data from include Apache Kafka and RabbitMQ.

Ongoing Tasks

Ongoing Tasks

  1. Ongoing Tasks
    Click to open the ongoing tasks view.
  2. Add a Database Task
    Click to create a new ongoing task.
  3. Info Hub
    Click for usage and licensing assistance.

    Info Hub

    Info Hub

Define ETL Task

Define ETL Task

  1. Kafka ETL
    Click to define a Kafka ETL task.
  2. RabbitMQ ETL
    Click to define a RabbitMQ ETL task.


After preparing a JSON object that needs to be sent to a message broker, the ETL task wraps it as a CloudEvents message using the CloudEvents Library.

To do that, the JSON object is provided with additional required attributes, added as headers to the message, including:

Attribute Type Description Default Value
Id string Event Identifier Document Change Vector
Type string Event Type "ravendb.etl.put"
Source string Event Context <ravendb-node-url>/<database-name>/<etl-task-name>

Optional Attributes

The optional partitionkey attribute can also be added. It is currently only implemented by Kafka.

Optional Attribute Type Description Default Value
partitionkey string events relationship/grouping definition Document ID

Task Statistics

Use Studio's ongoing tasks stats view to see various statistics related to data extraction, transformation, and loading to the target broker.

Queue Brokers Stats

Ongoing Tasks