See on GitHub

    Ongoing Tasks: Queue Sink Overview

    • Message brokers are high-throughput, distributed messaging services that host data they receive from producer applications and serve it to consumer clients via FIFO data queues.
    • RavenDB 5.4 and on can function as a Producer in this architecture.
      RavenDB 6.0 and on can also function as a Consumer.

      This overview and the other pages in the Queue Sink section explain only RavenDB's role as a Consumer through the implementation of a sink connector.
      To learn about RavenDB's role as a Producer please refer to the Queue ETL section.

    • RavenDB can run an ongoing Sink task that reads JSON formatted messages from broker queues, apply a user-defined script that can, among other things, construct documents from the retrieved messages, and potentially store manufactured documents in RavenDB's database.
    • Supported broker queues currently include Apache Kafka and RabbitMQ.

    Using RavenDB as a message broker sink can benefit users who want to combine Kafka or RabbitMQ's immense capability to collect and stream data with RavenDB's ability to process this data, reveal and exploit its value.

    Supported Message Brokers

    Queue brokers currently supported by RavenDB include Apache Kafka and RabbitMQ.

    Ongoing Tasks

    Ongoing Tasks

    1. Ongoing Tasks
      Click to open the ongoing tasks view.
    2. Add a Database Task
      Click to create a new ongoing task.

    3. Info Hub
      Click for usage and licensing assistance.

      Info Hub

      Info Hub

    Define Queue Sink Task

    Define Queue Sink Task

    1. Kafka Sink
      Click to define a Kafka Queue Sink task.
    2. RabbitMQ Sink
      Click to define a RabbitMQ Queue Sink task.

    Task Statistics

    Use Studio's ongoing tasks stats view to see transfer statistics.

    Queue Brokers Stats

    Ongoing Tasks

    1. Kafka sink task statistics
      All statistics related to the sink task.
      Click the bars to expand or collide statistics.
      Hover over bar sections to expose statistics.
    2. RabbitMQ sink task statistics
    3. Sink statistics
      • Total duration
        The time it took to get a batch of documents (in MS)
      • Currently allocated
        Memory allocated for the task (in MB)
      • Number of processed messages
        The number of messages that were recognized and processed
      • Number of read messages
        The number of messages that were actually transferred to the database
      • Successfully processed
        Has this batch of messages been fully processed (yes/no)
    4. Queue readings
      The duration of reading from queues (in MS)
    5. Script processing
      The duration of script processing (in MS)

    Licensing

    Queue Sink is Available on an Enterprise license.

    Learn more about licensing here.

    related articlesRelated Articles

    Server

    Studio

    Home / Docs / Server / Ongoing Tasks / Queue Sink / Overview