HDFS to Kafka Sync

Download, configure, and deploy

Ingest lines from Hadoop HDFS to Apache Kafka

The HDFS to Kafka Application Template continuously reads lines from configured Hadoop HDFS file(s) and writes each line as a message in configured Apache Kafka topic.

Features of the enterprise-grade application template include these advantages:

  • The application scales linearly with the number of record readers and number of Kafka brokers
  • The application is fault-tolerant and can withstand node and cluster outages without data loss
  • Highly performant, the application can process as fast as the Kafka broker can consume per topic
  • DataTorrent’s template drastically simplifies custom logic, providing you business value with top connectivity and operational details of Kafka consumer and HDFS reader
  • Configuration is also simple: users need only to provide source HDFS path and destination Kafka broker list and topic
  • Dramatic reduction in time-to-market and cost of operations

 

Download the application template and launch it to read your data from HDFS and write to Kafka. Follow the tutorial videos or walkthrough document below to launch the template and add custom logic to process the data during ingestion.

Get Started

Import, configure, and launch application template

Customize and Deploy

Add custom logic to the application template and go to production