Nifi Tuning Tip - Using Backpressure and Max Retries

Apache Nifi supports buffering of queuing data. This is very useful when ur downstream processor is not working as expected (say due to a connection timeout) in which case Nifi queues up the flow files for processing.

However this can also turn out to be a negative if downstream system remains out for a longer than handled time. In such a case, the number of flow files can build up causing stress on the Nifi cluster and may even cause failure. Another scenario can be wherein a processor is translating an input message. A message in incorrect format might cause the processor to retry it an indefinite number of times causing queuing up of other messages and unnecessary stress on the cluster.

The best way to avoid this is to use a combination of Backpressure &Threshold, and impose a max limit on the number of retries. Backpressure implies limiting the amount or size of messages that can be queued up. Once the threshold value is reached, the connector doesn’t allow any further flow files or messages to get build up.  

Apache Nifi supports buffering of queuing data. This is very useful when ur downstream processor is not working as expected (say due to a connection timeout) in which case Nifi queues up the flow files for processing.

However this can also turn out to be a negative if downstream system remains out for a longer than handled time. In such a case, the number of flow files can build up causing stress on the Nifi cluster and may even cause failure. Another scenario can be wherein a processor is translating an input message. A message in incorrect format might cause the processor to retry it an indefinite number of times causing queuing up of other messages and unnecessary stress on the cluster.

The best way to avoid this is to use a combination of Backpressure &Threshold, and impose a max limit on the number of retries. Backpressure implies limiting the amount or size of messages that can be queued up. Once the threshold value is reached, the connector doesn’t allow any further flow files or messages to get build up.