Flow file replication

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Flow file replication

Provenzano Nicolas

Hi all,

 

Is there a way to replicate the repositories in order to be able to resume the processing of data when a nodes crashes ?

 

What is the recommendation to avoid losing data in case a node crashes ?

 

Thanks in advance

 

BR

 

Nicolas

 


Gfi Informatique Telecom
Nicolas Provenzano

BU Telecom / Customer Experience

[hidden email] 
— 
2 rue Andras Beck – 92360 Meudon La Forêt
Tél. : +33 (0)1 46 83 35 06
Mob. : +33 (0)6 82 07 19 21

www.gfi.world

signature-img

 

Reply | Threaded
Open this post in threaded view
|

Re: Flow file replication

Joe Witt
Hello Nicolas

So today if a NiFi node dies the data should still be recoverable unless the disks/storage itself is dead.  So losing a node should not cause data loss but it does impact availability.

Currently the way to achieve rapid availability in the presence of node failure would be through having the key repositories (content, flowfile, provenance) all mounted in such a way that you could remount them to another host.  This is a typical infrastructure/OS level approach that can be managed outside of NiFi itself.  It is admittedly non-trivial to get right.

However, there have been discussions on making this a first class feature of nifi itself.  https://cwiki.apache.org/confluence/display/NIFI/Data+Replication

Thanks
Joe



On Tue, Nov 22, 2016 at 12:30 PM, Provenzano Nicolas <[hidden email]> wrote:

Hi all,

 

Is there a way to replicate the repositories in order to be able to resume the processing of data when a nodes crashes ?

 

What is the recommendation to avoid losing data in case a node crashes ?

 

Thanks in advance

 

BR

 

Nicolas

 


Gfi Informatique Telecom
Nicolas Provenzano

BU Telecom / Customer Experience

[hidden email] 
— 
2 rue Andras Beck – 92360 Meudon La Forêt
Tél. : <a href="tel:%2B33%20%280%291%2046%2083%2035%2006" value="+33146833506" target="_blank">+33 (0)1 46 83 35 06
Mob. : <a href="tel:%2B33%20%280%296%C2%A082%2007%2019%2021" value="+33682071921" target="_blank">+33 (0)6 82 07 19 21

www.gfi.world

signature-img