Skip to Navigation
Ongoing Tasks: Queue ETL Overview
- Message brokers are high-throughput, distributed messaging services that host data they receive from producer applications and serve it to consumer clients via FIFO data queue/s.
- RavenDB 5.4 and on can function as a Producer in this architecture.
RavenDB 6.0 and on can also function as a Consumer.
This overview and the other pages in the Queue ETL section explain only RavenDB's role as a Producer through the execution of ETL tasks.
To learn about RavenDB's role as a Consumer please refer to the Queue Sink section. -
RavenDB produces messages to broker queues via ETL tasks that -
- Extract data from specified document collections.
- Transform the data to new JSON objects.
- Load the JSON objects to the message broker.
- Prior to loading the JSON objects to the designated broker, RavenDB wraps them as CloudEvents messages using the CloudEvents Library.
-
Supported message brokers currently include Apache Kafka and RabbitMQ.
-
In this page:
Supported Message Brokers
Queue brokers that RavenDB can currently read data from include Apache Kafka and RabbitMQ.
Ongoing Tasks
- Ongoing Tasks
Click to open the ongoing tasks view. - Add a Database Task
Click to create a new ongoing task. -
Info Hub
Click for usage and licensing assistance.
Info Hub
Define ETL Task
- Kafka ETL
Click to define a Kafka ETL task. - RabbitMQ ETL
Click to define a RabbitMQ ETL task.
CloudEvents
After preparing a JSON object that needs to be sent to a message broker, the ETL task wraps it as a CloudEvents message using the CloudEvents Library.
To do that, the JSON object is provided with additional required attributes, added as headers to the message, including:
| Attribute | Type | Description | Default Value |
|---|---|---|---|
| Id | string |
Event Identifier | Document Change Vector |
| Type | string |
Event Type | "ravendb.etl.put" |
| Source | string |
Event Context | <ravendb-node-url>/<database-name>/<etl-task-name> |
Optional Attributes
The optional partitionkey attribute can also be added. It is currently only implemented by Kafka.
| Optional Attribute | Type | Description | Default Value |
|---|---|---|---|
| partitionkey | string |
events relationship/grouping definition | Document ID |
Task Statistics
Use Studio's ongoing tasks stats view to see various statistics related to data extraction, transformation, and loading to the target broker.
Ongoing Tasks
Related Articles