Upgrade NiFi 1.11.4 Cluster External ZooKeeper 3.4.X

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Upgrade NiFi 1.11.4 Cluster External ZooKeeper 3.4.X

Sri Harsha Chavali
Hi All,

We have been running NiFi 1.9.2 (3 Node Cluster) with external ZooKeeper 3.4.X in both CDH 6.2.X and 5.15.X versions. We are trying to upgrade our NiFi cluster to version 1.11.4 but noticed that it has Zookeeper 3.5.5 dependency. I have a couple of questions after playing with NiFi 1.11.4 on my VM. 
  1. Is the ZK 3.5.5 a mandatory dependency or can we still live with ZK 3.4.x?
  2. If ZK 3.5.5 is a mandatory dependency. Can we reliably run embedded ZK on all cluster nodes while not using the existing ZK which comes with our cluster? I ask this question because if we run a 3 node NiFi cluster with embedded ZK, the ZK on that node goes down along with NiFi when there is an issue. This will cause Quorum/voting issues and will that break the NiFi cluster?
  3.  If piggybacking on the existing External ZK 3.4.X is an option what is the functionality we will lose? Is it all the processors that rely on state-management that get effected?
Thank you,
Harsha

Sent from Outlook

Reply | Threaded
Open this post in threaded view
|

Re: Upgrade NiFi 1.11.4 Cluster External ZooKeeper 3.4.X

Bryan Bende
Hello,

Starting with NiFi 1.10.0 [1], ZooKeeper 3.5.x is a requirement, it will not work with 3.4.x.

It is generally not recommended to use embedded ZK for production.

Thanks,

Bryan



On Thu, Jun 4, 2020 at 3:04 PM Sri Harsha Chavali <[hidden email]> wrote:
Hi All,

We have been running NiFi 1.9.2 (3 Node Cluster) with external ZooKeeper 3.4.X in both CDH 6.2.X and 5.15.X versions. We are trying to upgrade our NiFi cluster to version 1.11.4 but noticed that it has Zookeeper 3.5.5 dependency. I have a couple of questions after playing with NiFi 1.11.4 on my VM. 
  1. Is the ZK 3.5.5 a mandatory dependency or can we still live with ZK 3.4.x?
  2. If ZK 3.5.5 is a mandatory dependency. Can we reliably run embedded ZK on all cluster nodes while not using the existing ZK which comes with our cluster? I ask this question because if we run a 3 node NiFi cluster with embedded ZK, the ZK on that node goes down along with NiFi when there is an issue. This will cause Quorum/voting issues and will that break the NiFi cluster?
  3.  If piggybacking on the existing External ZK 3.4.X is an option what is the functionality we will lose? Is it all the processors that rely on state-management that get effected?
Thank you,
Harsha

Sent from Outlook

Reply | Threaded
Open this post in threaded view
|

Re: Upgrade NiFi 1.11.4 Cluster External ZooKeeper 3.4.X

Sri Harsha Chavali
Thank you for the inputs Bryan! Not much we can do at this time and need to figure out a way to incorporate secure 3.5.5 ZK running in parallel with ZK 3.4.x in our environment. 

Thanks,
Harsha

Sent from Outlook


From: Bryan Bende <[hidden email]>
Sent: Thursday, June 4, 2020 3:34 PM
To: [hidden email] <[hidden email]>
Subject: Re: Upgrade NiFi 1.11.4 Cluster External ZooKeeper 3.4.X
 
Hello,

Starting with NiFi 1.10.0 [1], ZooKeeper 3.5.x is a requirement, it will not work with 3.4.x.

It is generally not recommended to use embedded ZK for production.

Thanks,

Bryan



On Thu, Jun 4, 2020 at 3:04 PM Sri Harsha Chavali <[hidden email]> wrote:
Hi All,

We have been running NiFi 1.9.2 (3 Node Cluster) with external ZooKeeper 3.4.X in both CDH 6.2.X and 5.15.X versions. We are trying to upgrade our NiFi cluster to version 1.11.4 but noticed that it has Zookeeper 3.5.5 dependency. I have a couple of questions after playing with NiFi 1.11.4 on my VM. 
  1. Is the ZK 3.5.5 a mandatory dependency or can we still live with ZK 3.4.x?
  2. If ZK 3.5.5 is a mandatory dependency. Can we reliably run embedded ZK on all cluster nodes while not using the existing ZK which comes with our cluster? I ask this question because if we run a 3 node NiFi cluster with embedded ZK, the ZK on that node goes down along with NiFi when there is an issue. This will cause Quorum/voting issues and will that break the NiFi cluster?
  3.  If piggybacking on the existing External ZK 3.4.X is an option what is the functionality we will lose? Is it all the processors that rely on state-management that get effected?
Thank you,
Harsha

Sent from Outlook

Reply | Threaded
Open this post in threaded view
|

Re: Upgrade NiFi 1.11.4 Cluster External ZooKeeper 3.4.X

Boris Tyukin
wow this is a major pain as we have to stay on CDH 6.1/6.2 for a long time for some reasons. Do you know why Bryan they decided to require 3.5.5 now?




On Thu, Jun 4, 2020 at 8:42 PM Sri Harsha Chavali <[hidden email]> wrote:
Thank you for the inputs Bryan! Not much we can do at this time and need to figure out a way to incorporate secure 3.5.5 ZK running in parallel with ZK 3.4.x in our environment. 

Thanks,
Harsha

Sent from Outlook


From: Bryan Bende <[hidden email]>
Sent: Thursday, June 4, 2020 3:34 PM
To: [hidden email] <[hidden email]>
Subject: Re: Upgrade NiFi 1.11.4 Cluster External ZooKeeper 3.4.X
 
Hello,

Starting with NiFi 1.10.0 [1], ZooKeeper 3.5.x is a requirement, it will not work with 3.4.x.

It is generally not recommended to use embedded ZK for production.

Thanks,

Bryan



On Thu, Jun 4, 2020 at 3:04 PM Sri Harsha Chavali <[hidden email]> wrote:
Hi All,

We have been running NiFi 1.9.2 (3 Node Cluster) with external ZooKeeper 3.4.X in both CDH 6.2.X and 5.15.X versions. We are trying to upgrade our NiFi cluster to version 1.11.4 but noticed that it has Zookeeper 3.5.5 dependency. I have a couple of questions after playing with NiFi 1.11.4 on my VM. 
  1. Is the ZK 3.5.5 a mandatory dependency or can we still live with ZK 3.4.x?
  2. If ZK 3.5.5 is a mandatory dependency. Can we reliably run embedded ZK on all cluster nodes while not using the existing ZK which comes with our cluster? I ask this question because if we run a 3 node NiFi cluster with embedded ZK, the ZK on that node goes down along with NiFi when there is an issue. This will cause Quorum/voting issues and will that break the NiFi cluster?
  3.  If piggybacking on the existing External ZK 3.4.X is an option what is the functionality we will lose? Is it all the processors that rely on state-management that get effected?
Thank you,
Harsha

Sent from Outlook

Reply | Threaded
Open this post in threaded view
|

Re: Upgrade NiFi 1.11.4 Cluster External ZooKeeper 3.4.X

Bryan Bende
I believe the main reason was to finally get TLS support from ZK 3.5.x, which NiFi community has wanted for a long time, plus getting on a more recent version with other security related fixes.

On Fri, Jun 12, 2020 at 9:43 AM Boris Tyukin <[hidden email]> wrote:
wow this is a major pain as we have to stay on CDH 6.1/6.2 for a long time for some reasons. Do you know why Bryan they decided to require 3.5.5 now?




On Thu, Jun 4, 2020 at 8:42 PM Sri Harsha Chavali <[hidden email]> wrote:
Thank you for the inputs Bryan! Not much we can do at this time and need to figure out a way to incorporate secure 3.5.5 ZK running in parallel with ZK 3.4.x in our environment. 

Thanks,
Harsha

Sent from Outlook


From: Bryan Bende <[hidden email]>
Sent: Thursday, June 4, 2020 3:34 PM
To: [hidden email] <[hidden email]>
Subject: Re: Upgrade NiFi 1.11.4 Cluster External ZooKeeper 3.4.X
 
Hello,

Starting with NiFi 1.10.0 [1], ZooKeeper 3.5.x is a requirement, it will not work with 3.4.x.

It is generally not recommended to use embedded ZK for production.

Thanks,

Bryan



On Thu, Jun 4, 2020 at 3:04 PM Sri Harsha Chavali <[hidden email]> wrote:
Hi All,

We have been running NiFi 1.9.2 (3 Node Cluster) with external ZooKeeper 3.4.X in both CDH 6.2.X and 5.15.X versions. We are trying to upgrade our NiFi cluster to version 1.11.4 but noticed that it has Zookeeper 3.5.5 dependency. I have a couple of questions after playing with NiFi 1.11.4 on my VM. 
  1. Is the ZK 3.5.5 a mandatory dependency or can we still live with ZK 3.4.x?
  2. If ZK 3.5.5 is a mandatory dependency. Can we reliably run embedded ZK on all cluster nodes while not using the existing ZK which comes with our cluster? I ask this question because if we run a 3 node NiFi cluster with embedded ZK, the ZK on that node goes down along with NiFi when there is an issue. This will cause Quorum/voting issues and will that break the NiFi cluster?
  3.  If piggybacking on the existing External ZK 3.4.X is an option what is the functionality we will lose? Is it all the processors that rely on state-management that get effected?
Thank you,
Harsha

Sent from Outlook

Reply | Threaded
Open this post in threaded view
|

Re: Upgrade NiFi 1.11.4 Cluster External ZooKeeper 3.4.X

Joe Witt
Id add that there may well be a CDH compatible version of newer nifi releases for those distributions.  Contact the vendor.

But it was definitely important for the apache nifi community to benefit from the latest zk security features and fixes.

On Fri, Jun 12, 2020 at 6:54 AM Bryan Bende <[hidden email]> wrote:
I believe the main reason was to finally get TLS support from ZK 3.5.x, which NiFi community has wanted for a long time, plus getting on a more recent version with other security related fixes.

On Fri, Jun 12, 2020 at 9:43 AM Boris Tyukin <[hidden email]> wrote:
wow this is a major pain as we have to stay on CDH 6.1/6.2 for a long time for some reasons. Do you know why Bryan they decided to require 3.5.5 now?




On Thu, Jun 4, 2020 at 8:42 PM Sri Harsha Chavali <[hidden email]> wrote:
Thank you for the inputs Bryan! Not much we can do at this time and need to figure out a way to incorporate secure 3.5.5 ZK running in parallel with ZK 3.4.x in our environment. 

Thanks,
Harsha

Sent from Outlook


From: Bryan Bende <[hidden email]>
Sent: Thursday, June 4, 2020 3:34 PM
To: [hidden email] <[hidden email]>
Subject: Re: Upgrade NiFi 1.11.4 Cluster External ZooKeeper 3.4.X
 
Hello,

Starting with NiFi 1.10.0 [1], ZooKeeper 3.5.x is a requirement, it will not work with 3.4.x.

It is generally not recommended to use embedded ZK for production.

Thanks,

Bryan



On Thu, Jun 4, 2020 at 3:04 PM Sri Harsha Chavali <[hidden email]> wrote:
Hi All,

We have been running NiFi 1.9.2 (3 Node Cluster) with external ZooKeeper 3.4.X in both CDH 6.2.X and 5.15.X versions. We are trying to upgrade our NiFi cluster to version 1.11.4 but noticed that it has Zookeeper 3.5.5 dependency. I have a couple of questions after playing with NiFi 1.11.4 on my VM. 
  1. Is the ZK 3.5.5 a mandatory dependency or can we still live with ZK 3.4.x?
  2. If ZK 3.5.5 is a mandatory dependency. Can we reliably run embedded ZK on all cluster nodes while not using the existing ZK which comes with our cluster? I ask this question because if we run a 3 node NiFi cluster with embedded ZK, the ZK on that node goes down along with NiFi when there is an issue. This will cause Quorum/voting issues and will that break the NiFi cluster?
  3.  If piggybacking on the existing External ZK 3.4.X is an option what is the functionality we will lose? Is it all the processors that rely on state-management that get effected?
Thank you,
Harsha

Sent from Outlook

Reply | Threaded
Open this post in threaded view
|

Re: Upgrade NiFi 1.11.4 Cluster External ZooKeeper 3.4.X

Boris Tyukin
thanks Joe and Bryan, these are good reasons

On Fri, Jun 12, 2020 at 10:06 AM Joe Witt <[hidden email]> wrote:
Id add that there may well be a CDH compatible version of newer nifi releases for those distributions.  Contact the vendor.

But it was definitely important for the apache nifi community to benefit from the latest zk security features and fixes.

On Fri, Jun 12, 2020 at 6:54 AM Bryan Bende <[hidden email]> wrote:
I believe the main reason was to finally get TLS support from ZK 3.5.x, which NiFi community has wanted for a long time, plus getting on a more recent version with other security related fixes.

On Fri, Jun 12, 2020 at 9:43 AM Boris Tyukin <[hidden email]> wrote:
wow this is a major pain as we have to stay on CDH 6.1/6.2 for a long time for some reasons. Do you know why Bryan they decided to require 3.5.5 now?




On Thu, Jun 4, 2020 at 8:42 PM Sri Harsha Chavali <[hidden email]> wrote:
Thank you for the inputs Bryan! Not much we can do at this time and need to figure out a way to incorporate secure 3.5.5 ZK running in parallel with ZK 3.4.x in our environment. 

Thanks,
Harsha

Sent from Outlook


From: Bryan Bende <[hidden email]>
Sent: Thursday, June 4, 2020 3:34 PM
To: [hidden email] <[hidden email]>
Subject: Re: Upgrade NiFi 1.11.4 Cluster External ZooKeeper 3.4.X
 
Hello,

Starting with NiFi 1.10.0 [1], ZooKeeper 3.5.x is a requirement, it will not work with 3.4.x.

It is generally not recommended to use embedded ZK for production.

Thanks,

Bryan



On Thu, Jun 4, 2020 at 3:04 PM Sri Harsha Chavali <[hidden email]> wrote:
Hi All,

We have been running NiFi 1.9.2 (3 Node Cluster) with external ZooKeeper 3.4.X in both CDH 6.2.X and 5.15.X versions. We are trying to upgrade our NiFi cluster to version 1.11.4 but noticed that it has Zookeeper 3.5.5 dependency. I have a couple of questions after playing with NiFi 1.11.4 on my VM. 
  1. Is the ZK 3.5.5 a mandatory dependency or can we still live with ZK 3.4.x?
  2. If ZK 3.5.5 is a mandatory dependency. Can we reliably run embedded ZK on all cluster nodes while not using the existing ZK which comes with our cluster? I ask this question because if we run a 3 node NiFi cluster with embedded ZK, the ZK on that node goes down along with NiFi when there is an issue. This will cause Quorum/voting issues and will that break the NiFi cluster?
  3.  If piggybacking on the existing External ZK 3.4.X is an option what is the functionality we will lose? Is it all the processors that rely on state-management that get effected?
Thank you,
Harsha

Sent from Outlook