Kafka to HDFS Sync

Download, configure, and deploy

Ingest data from Kafka to HDFS

The Kafka to HDFS Application Template continuously reads messages from configured Apache Kafka topic(s) and writes each message as a line in Hadoop HDFS file(s).

Features of the enterprise-grade application template include these advantages:

  • The application scales linearly with the number of Kafka brokers and Kafka topics
  • The application is fault-tolerant and can withstand node and cluster outages without data loss
  • Highly performant, the application can process faster than the Kafka broker can produce per topic
  • DataTorrent’s template drastically simplifies custom logic, providing you business value with top connectivity and operational details of Kafka consumer and HDFS writer
  • Configuration is also simple: users need only provide source Kafka broker list, topic list and destination HDFS path, and filename
  • Dramatic reduction in time-to-market and cost of operations

 

Download the application template and launch it to read your data from Kafka topics and write to HDFS. Follow the tutorial videos or walkthrough document below to launch the template and add custom logic to process the data during ingestion.

Get Started

Import, configure, and launch application template

Customize and Deploy

Add custom logic to the application template and go to production