Required Apache NiFi freelancer for Apache Nifi aggregation Consultant job

1. Has good knowledge of Apache Nif (version 10) and has worked with this technology in the past.
2. The work involved is to use Nifi for log aggregation. Aggregation will involve clubbing logs of a similar type that occur during a defined period (say 1 minute/2 minute etc). This will amount to sending out a single log from Nifi with the count of the number of logs received of that type during the defined period. For example, let us say Nifi receives 10 logs of type "User login failure" within a period of 1 minute. Then Nifi should send out one log with the count field within the log incremented to 10.
3. Will need to conduct a POC using Nifi consisting of the following:
a. Build in Nifi the log aggregation logic where logs from devices like IPS, Firewalls, etc. are aggregated over a defined criterion
b. Build in capability to change the aggregation criteria externally through a UI - the aggregation criteria will mention the fields to be aggregated and the duration
c. Optimize Nifi to process aggregation of logs at speed of 20000 logs per second
d. Optimize Nifi to run in a cluster to scale up as required to support 20000 logs per second

Skills & Expertise Required

Apache NiFi