Ongoing Tasks: Queue Sink Overview


  • Message brokers are high-throughput, distributed messaging services that host data they receive from producer applications and serve it to consumer clients via FIFO data queues.
  • RavenDB 5.4 and on can function as a Producer in this architecture.
    RavenDB 6.0 and on can also function as a Consumer.

    This overview and the other pages in the Queue Sink section explain only RavenDB's role as a Consumer through the implementation of a sink connector.
    To learn about RavenDB's role as a Producer please refer to the Queue ETL section.

  • RavenDB can run an ongoing Sink task that reads JSON formatted messages from broker queues, apply a user-defined script that can, among other things, construct documents from the retrieved messages, and potentially store manufactured documents in RavenDB's database.
  • Supported broker queues currently include Apache Kafka and RabbitMQ.

Using RavenDB as a message broker sink can benefit users who want to combine Kafka or RabbitMQ's immense capability to collect and stream data with RavenDB's ability to process this data, reveal and exploit its value.


Supported Message Brokers

Queue brokers currently supported by RavenDB include Apache Kafka and RabbitMQ.

Ongoing Tasks

Ongoing Tasks

  1. Ongoing Tasks
    Click to open the ongoing tasks view.
  2. Add a Database Task
    Click to create a new ongoing task.
  3. Info Hub
    Click for usage and licensing assistance.

    Info Hub

    Info Hub


Define Queue Sink Task

Define Queue Sink Task

  1. Kafka Sink
    Click to define a Kafka Queue Sink task.
  2. RabbitMQ Sink
    Click to define a RabbitMQ Queue Sink task.

Task Statistics

Use Studio's ongoing tasks stats view to see transfer statistics.

Queue Brokers Stats

Ongoing Tasks

  1. Kafka sink task statistics
    All statistics related to the sink task.
    Click the bars to expand or collide statistics.
    Hover over bar sections to expose statistics.
  2. RabbitMQ sink task statistics
  3. Sink statistics
    • Total duration
      The time it took to get a batch of documents (in MS)
    • Currently allocated
      Memory allocated for the task (in MB)
    • Number of processed messages
      The number of messages that were recognized and processed
    • Number of read messages
      The number of messages that were actually transferred to the database
    • Successfully processed
      Has this batch of messages been fully processed (yes/no)
  4. Queue readings
    The duration of reading from queues (in MS)
  5. Script processing
    The duration of script processing (in MS)

Licensing

Queue Sink is Available on an Enterprise license.

Learn more about licensing here.