Ingest data from Kafka to HDFS
The Kafka to HDFS Application Template continuously reads messages from configured Apache Kafka topic(s) and writes each message as a line in Hadoop HDFS file(s).
Features of the enterprise-grade application template include these advantages:
- The application scales linearly with the number of Kafka brokers and Kafka topics
- The application is fault-tolerant and can withstand node and cluster outages without data loss
- Highly performant, the application can process faster than the Kafka broker can produce per topic
- DataTorrent’s template drastically simplifies custom logic, providing you business value with top connectivity and operational details of Kafka consumer and HDFS writer
- Configuration is also simple: users need only provide source Kafka broker list, topic list and destination HDFS path, and filename
- Dramatic reduction in time-to-market and cost of operations
Download the application template and launch it to read your data from Kafka topics and write to HDFS. Follow the tutorial videos or walkthrough document below to launch the template and add custom logic to process the data during ingestion.
Import, configure, and launch application template
Customize and Deploy
Add custom logic to the application template and go to production
Have feedback or want to learn more?