Apache NiFi is a robust and secure framework for routing, transforming, and delivering data across a multitude of systems. NiFi can run in parallel with other applications, but it performs best when the entire system (or multiple systems in a cluster) are dedicated to it. It often uses SAN or RAID storage at the TB … Read more
For a great overview of Airflow and Apache NiFi checkout this reddit post: https://www.reddit.com/r/bigdata/comments/51mgk6/comparing_airbnb_airflow_and_apache_nifi/ For your specific use-case of ingesting Json files, enriching them and routing them to Kafka I believe NiFi is the right tool for the job. A couple of processors you could potentially use, as well as documentation for each, are below: … Read more
Suraj, Great question. My response is as a member of the open source Apache NiFi project management committee and as someone who is passionate about the dataflow management domain. I’ve been involved in the NiFi project since it was started in 2006. My knowledge of Streamsets is relatively limited so I’ll let them speak for … Read more