zookeeper SASL/Digest authentication....

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

zookeeper SASL/Digest authentication....

danoyoung
Hello everyone,

I'm trying to configure the zookeeper state provider in  NiFi to use the Access Policy of CreatorOnly vs Open using DIGEST vs Kerberos.   I believe I've setup zookeeper correctly for this, and partly Nifi, but when I startup nifi cluster, we seem to get stuck with the following:

2020-07-06 16:06:20,826 WARN [Clustering Tasks Thread-1] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:06:35,920 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:06:50,923 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:07:06,071 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected

I can see the znode in zookeeper, and it appears to at least have the correct permissions.  I created this znode in the CLI:

addauth digest nifi:<passwd>
create /nifi data digest:nifi<passwd digest>:cdrwa

The digest was generated via:

java -cp '/op/zookeeper/lib/zookeeper-3.5.8.jar:/opt/zookeeper/lib/slf4j-api-1.7.25.jar' org.apache.auth.AuthenticationProvider nifi:<passwd> 

[zk: nifi1-5:2181,nifi2-5:2181,nifi3-5:2181(CONNECTED) 4] getAcl /nifi
'digest,'nifi:the-passwd-digest'
: cdrwa


after starting up Nifi, doing and ls /nifi, the znode is empty.
[zk: nifi1-5:2181,nifi2-5:2181,nifi3-5:2181(CONNECTED) 4] ls /nifi
[]

Seems like we can't write the leaders or components value under the /nifi znode.

 
Looking at the nifi-app log

2020-07-06 16:05:46,554 INFO [main-SendThread(xx.xxx.x.xx:2181)] org.apache.zookeeper.Login Client successfully logged in.
2020-07-06 16:05:46,556 INFO [main-SendThread(xx.xxx.x.xx:2181)] o.a.zookeeper.client.ZooKeeperSaslClient Client will use DIGEST-MD5 as SASL mechanism.
2020-07-06 16:05:46,900 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2020-07-06 16:05:47,347 INFO [main-EventThread] o.a.c.framework.imps.EnsembleTracker New config event received: {server.1=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181, version=0, server.3=xx.xxx.x.xx:2888:3888:participant;0.0.0.0:2181, server.2=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181}
2020-07-06 16:05:47,354 INFO [main-EventThread] o.a.c.framework.imps.EnsembleTracker New config event received: {server.1=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181, version=0, server.3=xx.xxx.x.xx:2888:3888:participant;0.0.0.0:2181, server.2=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181}
2020-07-06 16:05:47,357 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
2020-07-06 16:05:47,364 DEBUG [main] org.apache.zookeeper.ZooKeeper Closing session: 0x3002a05b0c60006
2020-07-06 16:05:47,469 INFO [main/ org.apache.zookeeper.ZooKeeper Session: 0x3002a05b0c60006 closed



Any ideas on what configuration I could be missing or have wrong?  I have a jaas.conf file in the $NIFI_HOME/conf directory and have a java.arg.18--Djava.security.auth.login.config=<path to jaas.conf file>

One question I have, in the jaas.conf file, I put the passwd in there and not the digest I believe...I understand this would be passed around cleartext, but this is just for testing purposes currently....

Nifi 1.11.4
external zookeeper 3.5.8

Regards,

Dano

Reply | Threaded
Open this post in threaded view
|

Re: zookeeper SASL/Digest authentication....

danoyoung
Hello,

And a follow up on this, if I delete the znode in zookeeper, the leaders is written to the /nifi znode, but the ACL is open, 'world';'anyone....  I do have the Access COntrol set to CreatorOnly in the state-management.xml.  So one question, is the CreatorOnly only supported when we run in kerberos env?

Dano

On Mon, Jul 6, 2020 at 10:36 AM dan young <[hidden email]> wrote:
Hello everyone,

I'm trying to configure the zookeeper state provider in  NiFi to use the Access Policy of CreatorOnly vs Open using DIGEST vs Kerberos.   I believe I've setup zookeeper correctly for this, and partly Nifi, but when I startup nifi cluster, we seem to get stuck with the following:

2020-07-06 16:06:20,826 WARN [Clustering Tasks Thread-1] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:06:35,920 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:06:50,923 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:07:06,071 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected

I can see the znode in zookeeper, and it appears to at least have the correct permissions.  I created this znode in the CLI:

addauth digest nifi:<passwd>
create /nifi data digest:nifi<passwd digest>:cdrwa

The digest was generated via:

java -cp '/op/zookeeper/lib/zookeeper-3.5.8.jar:/opt/zookeeper/lib/slf4j-api-1.7.25.jar' org.apache.auth.AuthenticationProvider nifi:<passwd> 

[zk: nifi1-5:2181,nifi2-5:2181,nifi3-5:2181(CONNECTED) 4] getAcl /nifi
'digest,'nifi:the-passwd-digest'
: cdrwa


after starting up Nifi, doing and ls /nifi, the znode is empty.
[zk: nifi1-5:2181,nifi2-5:2181,nifi3-5:2181(CONNECTED) 4] ls /nifi
[]

Seems like we can't write the leaders or components value under the /nifi znode.

 
Looking at the nifi-app log

2020-07-06 16:05:46,554 INFO [main-SendThread(xx.xxx.x.xx:2181)] org.apache.zookeeper.Login Client successfully logged in.
2020-07-06 16:05:46,556 INFO [main-SendThread(xx.xxx.x.xx:2181)] o.a.zookeeper.client.ZooKeeperSaslClient Client will use DIGEST-MD5 as SASL mechanism.
2020-07-06 16:05:46,900 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2020-07-06 16:05:47,347 INFO [main-EventThread] o.a.c.framework.imps.EnsembleTracker New config event received: {server.1=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181, version=0, server.3=xx.xxx.x.xx:2888:3888:participant;0.0.0.0:2181, server.2=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181}
2020-07-06 16:05:47,354 INFO [main-EventThread] o.a.c.framework.imps.EnsembleTracker New config event received: {server.1=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181, version=0, server.3=xx.xxx.x.xx:2888:3888:participant;0.0.0.0:2181, server.2=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181}
2020-07-06 16:05:47,357 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
2020-07-06 16:05:47,364 DEBUG [main] org.apache.zookeeper.ZooKeeper Closing session: 0x3002a05b0c60006
2020-07-06 16:05:47,469 INFO [main/ org.apache.zookeeper.ZooKeeper Session: 0x3002a05b0c60006 closed



Any ideas on what configuration I could be missing or have wrong?  I have a jaas.conf file in the $NIFI_HOME/conf directory and have a java.arg.18--Djava.security.auth.login.config=<path to jaas.conf file>

One question I have, in the jaas.conf file, I put the passwd in there and not the digest I believe...I understand this would be passed around cleartext, but this is just for testing purposes currently....

Nifi 1.11.4
external zookeeper 3.5.8

Regards,

Dano

Reply | Threaded
Open this post in threaded view
|

Re: zookeeper SASL/Digest authentication....

Bryan Bende
Have you configured this in nifi.properties?
nifi.zookeeper.auth.type=sasl


On Mon, Jul 6, 2020 at 12:43 PM dan young <[hidden email]> wrote:
Hello,

And a follow up on this, if I delete the znode in zookeeper, the leaders is written to the /nifi znode, but the ACL is open, 'world';'anyone....  I do have the Access COntrol set to CreatorOnly in the state-management.xml.  So one question, is the CreatorOnly only supported when we run in kerberos env?

Dano

On Mon, Jul 6, 2020 at 10:36 AM dan young <[hidden email]> wrote:
Hello everyone,

I'm trying to configure the zookeeper state provider in  NiFi to use the Access Policy of CreatorOnly vs Open using DIGEST vs Kerberos.   I believe I've setup zookeeper correctly for this, and partly Nifi, but when I startup nifi cluster, we seem to get stuck with the following:

2020-07-06 16:06:20,826 WARN [Clustering Tasks Thread-1] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:06:35,920 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:06:50,923 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:07:06,071 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected

I can see the znode in zookeeper, and it appears to at least have the correct permissions.  I created this znode in the CLI:

addauth digest nifi:<passwd>
create /nifi data digest:nifi<passwd digest>:cdrwa

The digest was generated via:

java -cp '/op/zookeeper/lib/zookeeper-3.5.8.jar:/opt/zookeeper/lib/slf4j-api-1.7.25.jar' org.apache.auth.AuthenticationProvider nifi:<passwd> 

[zk: nifi1-5:2181,nifi2-5:2181,nifi3-5:2181(CONNECTED) 4] getAcl /nifi
'digest,'nifi:the-passwd-digest'
: cdrwa


after starting up Nifi, doing and ls /nifi, the znode is empty.
[zk: nifi1-5:2181,nifi2-5:2181,nifi3-5:2181(CONNECTED) 4] ls /nifi
[]

Seems like we can't write the leaders or components value under the /nifi znode.

 
Looking at the nifi-app log

2020-07-06 16:05:46,554 INFO [main-SendThread(xx.xxx.x.xx:2181)] org.apache.zookeeper.Login Client successfully logged in.
2020-07-06 16:05:46,556 INFO [main-SendThread(xx.xxx.x.xx:2181)] o.a.zookeeper.client.ZooKeeperSaslClient Client will use DIGEST-MD5 as SASL mechanism.
2020-07-06 16:05:46,900 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2020-07-06 16:05:47,347 INFO [main-EventThread] o.a.c.framework.imps.EnsembleTracker New config event received: {server.1=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181, version=0, server.3=xx.xxx.x.xx:2888:3888:participant;0.0.0.0:2181, server.2=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181}
2020-07-06 16:05:47,354 INFO [main-EventThread] o.a.c.framework.imps.EnsembleTracker New config event received: {server.1=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181, version=0, server.3=xx.xxx.x.xx:2888:3888:participant;0.0.0.0:2181, server.2=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181}
2020-07-06 16:05:47,357 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
2020-07-06 16:05:47,364 DEBUG [main] org.apache.zookeeper.ZooKeeper Closing session: 0x3002a05b0c60006
2020-07-06 16:05:47,469 INFO [main/ org.apache.zookeeper.ZooKeeper Session: 0x3002a05b0c60006 closed



Any ideas on what configuration I could be missing or have wrong?  I have a jaas.conf file in the $NIFI_HOME/conf directory and have a java.arg.18--Djava.security.auth.login.config=<path to jaas.conf file>

One question I have, in the jaas.conf file, I put the passwd in there and not the digest I believe...I understand this would be passed around cleartext, but this is just for testing purposes currently....

Nifi 1.11.4
external zookeeper 3.5.8

Regards,

Dano

Reply | Threaded
Open this post in threaded view
|

Re: zookeeper SASL/Digest authentication....

danoyoung
yes, and set the nifi.kerberos.service.principal to nifi, but when I did that I'm getting an  java.lang.StringIndexOutOfBoundsException: String index out of range: -1

Let me look at the code for setting the principal, maybe I have that borked....



On Mon, Jul 6, 2020 at 10:47 AM Bryan Bende <[hidden email]> wrote:
Have you configured this in nifi.properties?
nifi.zookeeper.auth.type=sasl


On Mon, Jul 6, 2020 at 12:43 PM dan young <[hidden email]> wrote:
Hello,

And a follow up on this, if I delete the znode in zookeeper, the leaders is written to the /nifi znode, but the ACL is open, 'world';'anyone....  I do have the Access COntrol set to CreatorOnly in the state-management.xml.  So one question, is the CreatorOnly only supported when we run in kerberos env?

Dano

On Mon, Jul 6, 2020 at 10:36 AM dan young <[hidden email]> wrote:
Hello everyone,

I'm trying to configure the zookeeper state provider in  NiFi to use the Access Policy of CreatorOnly vs Open using DIGEST vs Kerberos.   I believe I've setup zookeeper correctly for this, and partly Nifi, but when I startup nifi cluster, we seem to get stuck with the following:

2020-07-06 16:06:20,826 WARN [Clustering Tasks Thread-1] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:06:35,920 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:06:50,923 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:07:06,071 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected

I can see the znode in zookeeper, and it appears to at least have the correct permissions.  I created this znode in the CLI:

addauth digest nifi:<passwd>
create /nifi data digest:nifi<passwd digest>:cdrwa

The digest was generated via:

java -cp '/op/zookeeper/lib/zookeeper-3.5.8.jar:/opt/zookeeper/lib/slf4j-api-1.7.25.jar' org.apache.auth.AuthenticationProvider nifi:<passwd> 

[zk: nifi1-5:2181,nifi2-5:2181,nifi3-5:2181(CONNECTED) 4] getAcl /nifi
'digest,'nifi:the-passwd-digest'
: cdrwa


after starting up Nifi, doing and ls /nifi, the znode is empty.
[zk: nifi1-5:2181,nifi2-5:2181,nifi3-5:2181(CONNECTED) 4] ls /nifi
[]

Seems like we can't write the leaders or components value under the /nifi znode.

 
Looking at the nifi-app log

2020-07-06 16:05:46,554 INFO [main-SendThread(xx.xxx.x.xx:2181)] org.apache.zookeeper.Login Client successfully logged in.
2020-07-06 16:05:46,556 INFO [main-SendThread(xx.xxx.x.xx:2181)] o.a.zookeeper.client.ZooKeeperSaslClient Client will use DIGEST-MD5 as SASL mechanism.
2020-07-06 16:05:46,900 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2020-07-06 16:05:47,347 INFO [main-EventThread] o.a.c.framework.imps.EnsembleTracker New config event received: {server.1=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181, version=0, server.3=xx.xxx.x.xx:2888:3888:participant;0.0.0.0:2181, server.2=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181}
2020-07-06 16:05:47,354 INFO [main-EventThread] o.a.c.framework.imps.EnsembleTracker New config event received: {server.1=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181, version=0, server.3=xx.xxx.x.xx:2888:3888:participant;0.0.0.0:2181, server.2=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181}
2020-07-06 16:05:47,357 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
2020-07-06 16:05:47,364 DEBUG [main] org.apache.zookeeper.ZooKeeper Closing session: 0x3002a05b0c60006
2020-07-06 16:05:47,469 INFO [main/ org.apache.zookeeper.ZooKeeper Session: 0x3002a05b0c60006 closed



Any ideas on what configuration I could be missing or have wrong?  I have a jaas.conf file in the $NIFI_HOME/conf directory and have a java.arg.18--Djava.security.auth.login.config=<path to jaas.conf file>

One question I have, in the jaas.conf file, I put the passwd in there and not the digest I believe...I understand this would be passed around cleartext, but this is just for testing purposes currently....

Nifi 1.11.4
external zookeeper 3.5.8

Regards,

Dano

Reply | Threaded
Open this post in threaded view
|

Re: zookeeper SASL/Digest authentication....

danoyoung
In reply to this post by Bryan Bende
arrgh...sorry Bryan...seems like I had the principal borked...i've fixed that and the permissions look better now .  Going to test a processor that stores the state via zk...



On Mon, Jul 6, 2020 at 10:47 AM Bryan Bende <[hidden email]> wrote:
Have you configured this in nifi.properties?
nifi.zookeeper.auth.type=sasl


On Mon, Jul 6, 2020 at 12:43 PM dan young <[hidden email]> wrote:
Hello,

And a follow up on this, if I delete the znode in zookeeper, the leaders is written to the /nifi znode, but the ACL is open, 'world';'anyone....  I do have the Access COntrol set to CreatorOnly in the state-management.xml.  So one question, is the CreatorOnly only supported when we run in kerberos env?

Dano

On Mon, Jul 6, 2020 at 10:36 AM dan young <[hidden email]> wrote:
Hello everyone,

I'm trying to configure the zookeeper state provider in  NiFi to use the Access Policy of CreatorOnly vs Open using DIGEST vs Kerberos.   I believe I've setup zookeeper correctly for this, and partly Nifi, but when I startup nifi cluster, we seem to get stuck with the following:

2020-07-06 16:06:20,826 WARN [Clustering Tasks Thread-1] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:06:35,920 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:06:50,923 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:07:06,071 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected

I can see the znode in zookeeper, and it appears to at least have the correct permissions.  I created this znode in the CLI:

addauth digest nifi:<passwd>
create /nifi data digest:nifi<passwd digest>:cdrwa

The digest was generated via:

java -cp '/op/zookeeper/lib/zookeeper-3.5.8.jar:/opt/zookeeper/lib/slf4j-api-1.7.25.jar' org.apache.auth.AuthenticationProvider nifi:<passwd> 

[zk: nifi1-5:2181,nifi2-5:2181,nifi3-5:2181(CONNECTED) 4] getAcl /nifi
'digest,'nifi:the-passwd-digest'
: cdrwa


after starting up Nifi, doing and ls /nifi, the znode is empty.
[zk: nifi1-5:2181,nifi2-5:2181,nifi3-5:2181(CONNECTED) 4] ls /nifi
[]

Seems like we can't write the leaders or components value under the /nifi znode.

 
Looking at the nifi-app log

2020-07-06 16:05:46,554 INFO [main-SendThread(xx.xxx.x.xx:2181)] org.apache.zookeeper.Login Client successfully logged in.
2020-07-06 16:05:46,556 INFO [main-SendThread(xx.xxx.x.xx:2181)] o.a.zookeeper.client.ZooKeeperSaslClient Client will use DIGEST-MD5 as SASL mechanism.
2020-07-06 16:05:46,900 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2020-07-06 16:05:47,347 INFO [main-EventThread] o.a.c.framework.imps.EnsembleTracker New config event received: {server.1=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181, version=0, server.3=xx.xxx.x.xx:2888:3888:participant;0.0.0.0:2181, server.2=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181}
2020-07-06 16:05:47,354 INFO [main-EventThread] o.a.c.framework.imps.EnsembleTracker New config event received: {server.1=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181, version=0, server.3=xx.xxx.x.xx:2888:3888:participant;0.0.0.0:2181, server.2=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181}
2020-07-06 16:05:47,357 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
2020-07-06 16:05:47,364 DEBUG [main] org.apache.zookeeper.ZooKeeper Closing session: 0x3002a05b0c60006
2020-07-06 16:05:47,469 INFO [main/ org.apache.zookeeper.ZooKeeper Session: 0x3002a05b0c60006 closed



Any ideas on what configuration I could be missing or have wrong?  I have a jaas.conf file in the $NIFI_HOME/conf directory and have a java.arg.18--Djava.security.auth.login.config=<path to jaas.conf file>

One question I have, in the jaas.conf file, I put the passwd in there and not the digest I believe...I understand this would be passed around cleartext, but this is just for testing purposes currently....

Nifi 1.11.4
external zookeeper 3.5.8

Regards,

Dano

Reply | Threaded
Open this post in threaded view
|

Re: zookeeper SASL/Digest authentication....

danoyoung
In reply to this post by Bryan Bende
Hello Bryan,

Making some progress....any insight into this error with GenerateTableFectch processor?


On Mon, Jul 6, 2020 at 10:47 AM Bryan Bende <[hidden email]> wrote:
Have you configured this in nifi.properties?
nifi.zookeeper.auth.type=sasl


On Mon, Jul 6, 2020 at 12:43 PM dan young <[hidden email]> wrote:
Hello,

And a follow up on this, if I delete the znode in zookeeper, the leaders is written to the /nifi znode, but the ACL is open, 'world';'anyone....  I do have the Access COntrol set to CreatorOnly in the state-management.xml.  So one question, is the CreatorOnly only supported when we run in kerberos env?

Dano

On Mon, Jul 6, 2020 at 10:36 AM dan young <[hidden email]> wrote:
Hello everyone,

I'm trying to configure the zookeeper state provider in  NiFi to use the Access Policy of CreatorOnly vs Open using DIGEST vs Kerberos.   I believe I've setup zookeeper correctly for this, and partly Nifi, but when I startup nifi cluster, we seem to get stuck with the following:

2020-07-06 16:06:20,826 WARN [Clustering Tasks Thread-1] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:06:35,920 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:06:50,923 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:07:06,071 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected

I can see the znode in zookeeper, and it appears to at least have the correct permissions.  I created this znode in the CLI:

addauth digest nifi:<passwd>
create /nifi data digest:nifi<passwd digest>:cdrwa

The digest was generated via:

java -cp '/op/zookeeper/lib/zookeeper-3.5.8.jar:/opt/zookeeper/lib/slf4j-api-1.7.25.jar' org.apache.auth.AuthenticationProvider nifi:<passwd> 

[zk: nifi1-5:2181,nifi2-5:2181,nifi3-5:2181(CONNECTED) 4] getAcl /nifi
'digest,'nifi:the-passwd-digest'
: cdrwa


after starting up Nifi, doing and ls /nifi, the znode is empty.
[zk: nifi1-5:2181,nifi2-5:2181,nifi3-5:2181(CONNECTED) 4] ls /nifi
[]

Seems like we can't write the leaders or components value under the /nifi znode.

 
Looking at the nifi-app log

2020-07-06 16:05:46,554 INFO [main-SendThread(xx.xxx.x.xx:2181)] org.apache.zookeeper.Login Client successfully logged in.
2020-07-06 16:05:46,556 INFO [main-SendThread(xx.xxx.x.xx:2181)] o.a.zookeeper.client.ZooKeeperSaslClient Client will use DIGEST-MD5 as SASL mechanism.
2020-07-06 16:05:46,900 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2020-07-06 16:05:47,347 INFO [main-EventThread] o.a.c.framework.imps.EnsembleTracker New config event received: {server.1=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181, version=0, server.3=xx.xxx.x.xx:2888:3888:participant;0.0.0.0:2181, server.2=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181}
2020-07-06 16:05:47,354 INFO [main-EventThread] o.a.c.framework.imps.EnsembleTracker New config event received: {server.1=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181, version=0, server.3=xx.xxx.x.xx:2888:3888:participant;0.0.0.0:2181, server.2=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181}
2020-07-06 16:05:47,357 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
2020-07-06 16:05:47,364 DEBUG [main] org.apache.zookeeper.ZooKeeper Closing session: 0x3002a05b0c60006
2020-07-06 16:05:47,469 INFO [main/ org.apache.zookeeper.ZooKeeper Session: 0x3002a05b0c60006 closed



Any ideas on what configuration I could be missing or have wrong?  I have a jaas.conf file in the $NIFI_HOME/conf directory and have a java.arg.18--Djava.security.auth.login.config=<path to jaas.conf file>

One question I have, in the jaas.conf file, I put the passwd in there and not the digest I believe...I understand this would be passed around cleartext, but this is just for testing purposes currently....

Nifi 1.11.4
external zookeeper 3.5.8

Regards,

Dano

Reply | Threaded
Open this post in threaded view
|

Re: zookeeper SASL/Digest authentication....

danoyoung
Fat fingered... any insight into this error when the GenerateTableFectch?

Failed to set cluster-wide state in Zookeeper...
...
...

Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /nifi/components/2a056fd7-b63c-33b4-a5c4-bf767c1a2983
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:128)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
        at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1538)
        at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.createNode(ZooKeeperStateProvider.java:360)
        at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:321)
        ... 17 common frames omitted


On Mon, Jul 6, 2020 at 11:39 AM dan young <[hidden email]> wrote:
Hello Bryan,

Making some progress....any insight into this error with GenerateTableFectch processor?


On Mon, Jul 6, 2020 at 10:47 AM Bryan Bende <[hidden email]> wrote:
Have you configured this in nifi.properties?
nifi.zookeeper.auth.type=sasl


On Mon, Jul 6, 2020 at 12:43 PM dan young <[hidden email]> wrote:
Hello,

And a follow up on this, if I delete the znode in zookeeper, the leaders is written to the /nifi znode, but the ACL is open, 'world';'anyone....  I do have the Access COntrol set to CreatorOnly in the state-management.xml.  So one question, is the CreatorOnly only supported when we run in kerberos env?

Dano

On Mon, Jul 6, 2020 at 10:36 AM dan young <[hidden email]> wrote:
Hello everyone,

I'm trying to configure the zookeeper state provider in  NiFi to use the Access Policy of CreatorOnly vs Open using DIGEST vs Kerberos.   I believe I've setup zookeeper correctly for this, and partly Nifi, but when I startup nifi cluster, we seem to get stuck with the following:

2020-07-06 16:06:20,826 WARN [Clustering Tasks Thread-1] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:06:35,920 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:06:50,923 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:07:06,071 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected

I can see the znode in zookeeper, and it appears to at least have the correct permissions.  I created this znode in the CLI:

addauth digest nifi:<passwd>
create /nifi data digest:nifi<passwd digest>:cdrwa

The digest was generated via:

java -cp '/op/zookeeper/lib/zookeeper-3.5.8.jar:/opt/zookeeper/lib/slf4j-api-1.7.25.jar' org.apache.auth.AuthenticationProvider nifi:<passwd> 

[zk: nifi1-5:2181,nifi2-5:2181,nifi3-5:2181(CONNECTED) 4] getAcl /nifi
'digest,'nifi:the-passwd-digest'
: cdrwa


after starting up Nifi, doing and ls /nifi, the znode is empty.
[zk: nifi1-5:2181,nifi2-5:2181,nifi3-5:2181(CONNECTED) 4] ls /nifi
[]

Seems like we can't write the leaders or components value under the /nifi znode.

 
Looking at the nifi-app log

2020-07-06 16:05:46,554 INFO [main-SendThread(xx.xxx.x.xx:2181)] org.apache.zookeeper.Login Client successfully logged in.
2020-07-06 16:05:46,556 INFO [main-SendThread(xx.xxx.x.xx:2181)] o.a.zookeeper.client.ZooKeeperSaslClient Client will use DIGEST-MD5 as SASL mechanism.
2020-07-06 16:05:46,900 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2020-07-06 16:05:47,347 INFO [main-EventThread] o.a.c.framework.imps.EnsembleTracker New config event received: {server.1=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181, version=0, server.3=xx.xxx.x.xx:2888:3888:participant;0.0.0.0:2181, server.2=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181}
2020-07-06 16:05:47,354 INFO [main-EventThread] o.a.c.framework.imps.EnsembleTracker New config event received: {server.1=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181, version=0, server.3=xx.xxx.x.xx:2888:3888:participant;0.0.0.0:2181, server.2=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181}
2020-07-06 16:05:47,357 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
2020-07-06 16:05:47,364 DEBUG [main] org.apache.zookeeper.ZooKeeper Closing session: 0x3002a05b0c60006
2020-07-06 16:05:47,469 INFO [main/ org.apache.zookeeper.ZooKeeper Session: 0x3002a05b0c60006 closed



Any ideas on what configuration I could be missing or have wrong?  I have a jaas.conf file in the $NIFI_HOME/conf directory and have a java.arg.18--Djava.security.auth.login.config=<path to jaas.conf file>

One question I have, in the jaas.conf file, I put the passwd in there and not the digest I believe...I understand this would be passed around cleartext, but this is just for testing purposes currently....

Nifi 1.11.4
external zookeeper 3.5.8

Regards,

Dano

Reply | Threaded
Open this post in threaded view
|

Re: zookeeper SASL/Digest authentication....

Bryan Bende
You set <property name="Access Control">CreatorOnly</property> in the ZK state manager ?

On Mon, Jul 6, 2020 at 1:40 PM dan young <[hidden email]> wrote:
Fat fingered... any insight into this error when the GenerateTableFectch?

Failed to set cluster-wide state in Zookeeper...
...
...

Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /nifi/components/2a056fd7-b63c-33b4-a5c4-bf767c1a2983
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:128)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
        at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1538)
        at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.createNode(ZooKeeperStateProvider.java:360)
        at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:321)
        ... 17 common frames omitted


On Mon, Jul 6, 2020 at 11:39 AM dan young <[hidden email]> wrote:
Hello Bryan,

Making some progress....any insight into this error with GenerateTableFectch processor?


On Mon, Jul 6, 2020 at 10:47 AM Bryan Bende <[hidden email]> wrote:
Have you configured this in nifi.properties?
nifi.zookeeper.auth.type=sasl


On Mon, Jul 6, 2020 at 12:43 PM dan young <[hidden email]> wrote:
Hello,

And a follow up on this, if I delete the znode in zookeeper, the leaders is written to the /nifi znode, but the ACL is open, 'world';'anyone....  I do have the Access COntrol set to CreatorOnly in the state-management.xml.  So one question, is the CreatorOnly only supported when we run in kerberos env?

Dano

On Mon, Jul 6, 2020 at 10:36 AM dan young <[hidden email]> wrote:
Hello everyone,

I'm trying to configure the zookeeper state provider in  NiFi to use the Access Policy of CreatorOnly vs Open using DIGEST vs Kerberos.   I believe I've setup zookeeper correctly for this, and partly Nifi, but when I startup nifi cluster, we seem to get stuck with the following:

2020-07-06 16:06:20,826 WARN [Clustering Tasks Thread-1] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:06:35,920 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:06:50,923 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:07:06,071 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected

I can see the znode in zookeeper, and it appears to at least have the correct permissions.  I created this znode in the CLI:

addauth digest nifi:<passwd>
create /nifi data digest:nifi<passwd digest>:cdrwa

The digest was generated via:

java -cp '/op/zookeeper/lib/zookeeper-3.5.8.jar:/opt/zookeeper/lib/slf4j-api-1.7.25.jar' org.apache.auth.AuthenticationProvider nifi:<passwd> 

[zk: nifi1-5:2181,nifi2-5:2181,nifi3-5:2181(CONNECTED) 4] getAcl /nifi
'digest,'nifi:the-passwd-digest'
: cdrwa


after starting up Nifi, doing and ls /nifi, the znode is empty.
[zk: nifi1-5:2181,nifi2-5:2181,nifi3-5:2181(CONNECTED) 4] ls /nifi
[]

Seems like we can't write the leaders or components value under the /nifi znode.

 
Looking at the nifi-app log

2020-07-06 16:05:46,554 INFO [main-SendThread(xx.xxx.x.xx:2181)] org.apache.zookeeper.Login Client successfully logged in.
2020-07-06 16:05:46,556 INFO [main-SendThread(xx.xxx.x.xx:2181)] o.a.zookeeper.client.ZooKeeperSaslClient Client will use DIGEST-MD5 as SASL mechanism.
2020-07-06 16:05:46,900 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2020-07-06 16:05:47,347 INFO [main-EventThread] o.a.c.framework.imps.EnsembleTracker New config event received: {server.1=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181, version=0, server.3=xx.xxx.x.xx:2888:3888:participant;0.0.0.0:2181, server.2=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181}
2020-07-06 16:05:47,354 INFO [main-EventThread] o.a.c.framework.imps.EnsembleTracker New config event received: {server.1=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181, version=0, server.3=xx.xxx.x.xx:2888:3888:participant;0.0.0.0:2181, server.2=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181}
2020-07-06 16:05:47,357 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
2020-07-06 16:05:47,364 DEBUG [main] org.apache.zookeeper.ZooKeeper Closing session: 0x3002a05b0c60006
2020-07-06 16:05:47,469 INFO [main/ org.apache.zookeeper.ZooKeeper Session: 0x3002a05b0c60006 closed



Any ideas on what configuration I could be missing or have wrong?  I have a jaas.conf file in the $NIFI_HOME/conf directory and have a java.arg.18--Djava.security.auth.login.config=<path to jaas.conf file>

One question I have, in the jaas.conf file, I put the passwd in there and not the digest I believe...I understand this would be passed around cleartext, but this is just for testing purposes currently....

Nifi 1.11.4
external zookeeper 3.5.8

Regards,

Dano

Reply | Threaded
Open this post in threaded view
|

Re: zookeeper SASL/Digest authentication....

danoyoung
Correct, the leader seems to work, but not the components it seems..... Is there some additional config setting I might be missing?



<stateManagement>
   <local-provider>
      <id>local-provider</id>
      <class>org.apache.nifi.controller.state.providers.local.WriteAheadLocalStateProvider</class>
      <property name="Directory">/opt/nifi-configuration-resources/state/local</property>
      <property name="Always Sync">false</property>
      <property name="Partitions">16</property>
      <property name="Checkpoint Interval">2 mins</property>
   </local-provider>
   <cluster-provider>
      <id>zk-provider</id>
      <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class>
      <property name="Root Node">/nifi</property>
      <property name="Session Timeout">30 seconds</property>
      <property name="Access Control">CreatorOnly</property>
      <property name="Connect String">xx.xxx.x.xxx:2181,xx.xxx.x.xxx:2181,xx.xxx.x.xxx:2181</property>
   </cluster-provider>
</stateManagement>




2020-07-06 18:25:04,830 ERROR [Timer-Driven Process Thread-3] o.a.n.p.standard.GenerateTableFetch GenerateTableFetch[id=2a056fd7-b63c-33b4-a5c4-bf767c1a2983] GenerateTableFetch[id=2a056fd7-b63c-33b4-a5c4-bf767c1a2983] failed to update State Manager, observed maximum values will not be recorded. Also, any generated SQL statements may be duplicated.: java.io.IOException: Failed to set cluster-wide state in ZooKeeper for component with ID 2a056fd7-b63c-33b4-a5c4-bf767c1a2983
java.io.IOException: Failed to set cluster-wide state in ZooKeeper for component with ID 2a056fd7-b63c-33b4-a5c4-bf767c1a2983
        at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:343)
        at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:283)
        at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:228)
        at org.apache.nifi.controller.state.manager.StandardStateManagerProvider$1.setState(StandardStateManagerProvider.java:298)
        at org.apache.nifi.controller.state.StandardStateManager.setState(StandardStateManager.java:79)
        at org.apache.nifi.controller.lifecycle.TaskTerminationAwareStateManager.setState(TaskTerminationAwareStateManager.java:64)
        at org.apache.nifi.processors.standard.GenerateTableFetch.onTrigger(GenerateTableFetch.java:555)
        at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
        at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
        at org.apache.nifi.controller.scheduling.QuartzSchedulingAgent$2.run(QuartzSchedulingAgent.java:151)
        at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /nifi/components/2a056fd7-b63c-33b4-a5c4-bf767c1a2983
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:128)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
        at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1538)
        at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.createNode(ZooKeeperStateProvider.java:360)
        at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:321)
        ... 17 common frames omitted




On Mon, Jul 6, 2020 at 11:59 AM Bryan Bende <[hidden email]> wrote:
You set <property name="Access Control">CreatorOnly</property> in the ZK state manager ?

On Mon, Jul 6, 2020 at 1:40 PM dan young <[hidden email]> wrote:
Fat fingered... any insight into this error when the GenerateTableFectch?

Failed to set cluster-wide state in Zookeeper...
...
...

Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /nifi/components/2a056fd7-b63c-33b4-a5c4-bf767c1a2983
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:128)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
        at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1538)
        at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.createNode(ZooKeeperStateProvider.java:360)
        at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:321)
        ... 17 common frames omitted


On Mon, Jul 6, 2020 at 11:39 AM dan young <[hidden email]> wrote:
Hello Bryan,

Making some progress....any insight into this error with GenerateTableFectch processor?


On Mon, Jul 6, 2020 at 10:47 AM Bryan Bende <[hidden email]> wrote:
Have you configured this in nifi.properties?
nifi.zookeeper.auth.type=sasl


On Mon, Jul 6, 2020 at 12:43 PM dan young <[hidden email]> wrote:
Hello,

And a follow up on this, if I delete the znode in zookeeper, the leaders is written to the /nifi znode, but the ACL is open, 'world';'anyone....  I do have the Access COntrol set to CreatorOnly in the state-management.xml.  So one question, is the CreatorOnly only supported when we run in kerberos env?

Dano

On Mon, Jul 6, 2020 at 10:36 AM dan young <[hidden email]> wrote:
Hello everyone,

I'm trying to configure the zookeeper state provider in  NiFi to use the Access Policy of CreatorOnly vs Open using DIGEST vs Kerberos.   I believe I've setup zookeeper correctly for this, and partly Nifi, but when I startup nifi cluster, we seem to get stuck with the following:

2020-07-06 16:06:20,826 WARN [Clustering Tasks Thread-1] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:06:35,920 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:06:50,923 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:07:06,071 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected

I can see the znode in zookeeper, and it appears to at least have the correct permissions.  I created this znode in the CLI:

addauth digest nifi:<passwd>
create /nifi data digest:nifi<passwd digest>:cdrwa

The digest was generated via:

java -cp '/op/zookeeper/lib/zookeeper-3.5.8.jar:/opt/zookeeper/lib/slf4j-api-1.7.25.jar' org.apache.auth.AuthenticationProvider nifi:<passwd> 

[zk: nifi1-5:2181,nifi2-5:2181,nifi3-5:2181(CONNECTED) 4] getAcl /nifi
'digest,'nifi:the-passwd-digest'
: cdrwa


after starting up Nifi, doing and ls /nifi, the znode is empty.
[zk: nifi1-5:2181,nifi2-5:2181,nifi3-5:2181(CONNECTED) 4] ls /nifi
[]

Seems like we can't write the leaders or components value under the /nifi znode.

 
Looking at the nifi-app log

2020-07-06 16:05:46,554 INFO [main-SendThread(xx.xxx.x.xx:2181)] org.apache.zookeeper.Login Client successfully logged in.
2020-07-06 16:05:46,556 INFO [main-SendThread(xx.xxx.x.xx:2181)] o.a.zookeeper.client.ZooKeeperSaslClient Client will use DIGEST-MD5 as SASL mechanism.
2020-07-06 16:05:46,900 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2020-07-06 16:05:47,347 INFO [main-EventThread] o.a.c.framework.imps.EnsembleTracker New config event received: {server.1=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181, version=0, server.3=xx.xxx.x.xx:2888:3888:participant;0.0.0.0:2181, server.2=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181}
2020-07-06 16:05:47,354 INFO [main-EventThread] o.a.c.framework.imps.EnsembleTracker New config event received: {server.1=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181, version=0, server.3=xx.xxx.x.xx:2888:3888:participant;0.0.0.0:2181, server.2=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181}
2020-07-06 16:05:47,357 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
2020-07-06 16:05:47,364 DEBUG [main] org.apache.zookeeper.ZooKeeper Closing session: 0x3002a05b0c60006
2020-07-06 16:05:47,469 INFO [main/ org.apache.zookeeper.ZooKeeper Session: 0x3002a05b0c60006 closed



Any ideas on what configuration I could be missing or have wrong?  I have a jaas.conf file in the $NIFI_HOME/conf directory and have a java.arg.18--Djava.security.auth.login.config=<path to jaas.conf file>

One question I have, in the jaas.conf file, I put the passwd in there and not the digest I believe...I understand this would be passed around cleartext, but this is just for testing purposes currently....

Nifi 1.11.4
external zookeeper 3.5.8

Regards,

Dano

Reply | Threaded
Open this post in threaded view
|

Re: zookeeper SASL/Digest authentication....

Bryan Bende
So right now the /nifi z-node exists with the ACL you showed earlier for 'digest,'nifi:the-passwd-digest' , but then '/nifi/components' doesn't exist yet?

The one difference from a code perspective is that /nifi and the cluster nodes are created by Curator, and the state provider is done using plain ZK client code, although no reason why that should work.

I'm no ZK expert, but the code that is causing the error is a call to "create(path, data, acls, CreateMode.PERSISTENT)" where "acls" is a list with one element of  Ids.CREATOR_ALL_ACL which has
/**
 * This Id is only usable to set ACLs. It will get substituted with the
 * Id's the client authenticated with.
 */
public final Id AUTH_IDS = new Id("auth", "");

Any ZK client code should be seeing the same JAAS entry you configured, so not sure how it could be authenticating as different identities.


On Mon, Jul 6, 2020 at 2:27 PM dan young <[hidden email]> wrote:
Correct, the leader seems to work, but not the components it seems..... Is there some additional config setting I might be missing?



<stateManagement>
   <local-provider>
      <id>local-provider</id>
      <class>org.apache.nifi.controller.state.providers.local.WriteAheadLocalStateProvider</class>
      <property name="Directory">/opt/nifi-configuration-resources/state/local</property>
      <property name="Always Sync">false</property>
      <property name="Partitions">16</property>
      <property name="Checkpoint Interval">2 mins</property>
   </local-provider>
   <cluster-provider>
      <id>zk-provider</id>
      <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class>
      <property name="Root Node">/nifi</property>
      <property name="Session Timeout">30 seconds</property>
      <property name="Access Control">CreatorOnly</property>
      <property name="Connect String">xx.xxx.x.xxx:2181,xx.xxx.x.xxx:2181,xx.xxx.x.xxx:2181</property>
   </cluster-provider>
</stateManagement>




2020-07-06 18:25:04,830 ERROR [Timer-Driven Process Thread-3] o.a.n.p.standard.GenerateTableFetch GenerateTableFetch[id=2a056fd7-b63c-33b4-a5c4-bf767c1a2983] GenerateTableFetch[id=2a056fd7-b63c-33b4-a5c4-bf767c1a2983] failed to update State Manager, observed maximum values will not be recorded. Also, any generated SQL statements may be duplicated.: java.io.IOException: Failed to set cluster-wide state in ZooKeeper for component with ID 2a056fd7-b63c-33b4-a5c4-bf767c1a2983
java.io.IOException: Failed to set cluster-wide state in ZooKeeper for component with ID 2a056fd7-b63c-33b4-a5c4-bf767c1a2983
        at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:343)
        at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:283)
        at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:228)
        at org.apache.nifi.controller.state.manager.StandardStateManagerProvider$1.setState(StandardStateManagerProvider.java:298)
        at org.apache.nifi.controller.state.StandardStateManager.setState(StandardStateManager.java:79)
        at org.apache.nifi.controller.lifecycle.TaskTerminationAwareStateManager.setState(TaskTerminationAwareStateManager.java:64)
        at org.apache.nifi.processors.standard.GenerateTableFetch.onTrigger(GenerateTableFetch.java:555)
        at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
        at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
        at org.apache.nifi.controller.scheduling.QuartzSchedulingAgent$2.run(QuartzSchedulingAgent.java:151)
        at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /nifi/components/2a056fd7-b63c-33b4-a5c4-bf767c1a2983
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:128)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
        at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1538)
        at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.createNode(ZooKeeperStateProvider.java:360)
        at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:321)
        ... 17 common frames omitted




On Mon, Jul 6, 2020 at 11:59 AM Bryan Bende <[hidden email]> wrote:
You set <property name="Access Control">CreatorOnly</property> in the ZK state manager ?

On Mon, Jul 6, 2020 at 1:40 PM dan young <[hidden email]> wrote:
Fat fingered... any insight into this error when the GenerateTableFectch?

Failed to set cluster-wide state in Zookeeper...
...
...

Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /nifi/components/2a056fd7-b63c-33b4-a5c4-bf767c1a2983
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:128)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
        at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1538)
        at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.createNode(ZooKeeperStateProvider.java:360)
        at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:321)
        ... 17 common frames omitted


On Mon, Jul 6, 2020 at 11:39 AM dan young <[hidden email]> wrote:
Hello Bryan,

Making some progress....any insight into this error with GenerateTableFectch processor?


On Mon, Jul 6, 2020 at 10:47 AM Bryan Bende <[hidden email]> wrote:
Have you configured this in nifi.properties?
nifi.zookeeper.auth.type=sasl


On Mon, Jul 6, 2020 at 12:43 PM dan young <[hidden email]> wrote:
Hello,

And a follow up on this, if I delete the znode in zookeeper, the leaders is written to the /nifi znode, but the ACL is open, 'world';'anyone....  I do have the Access COntrol set to CreatorOnly in the state-management.xml.  So one question, is the CreatorOnly only supported when we run in kerberos env?

Dano

On Mon, Jul 6, 2020 at 10:36 AM dan young <[hidden email]> wrote:
Hello everyone,

I'm trying to configure the zookeeper state provider in  NiFi to use the Access Policy of CreatorOnly vs Open using DIGEST vs Kerberos.   I believe I've setup zookeeper correctly for this, and partly Nifi, but when I startup nifi cluster, we seem to get stuck with the following:

2020-07-06 16:06:20,826 WARN [Clustering Tasks Thread-1] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:06:35,920 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:06:50,923 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:07:06,071 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected

I can see the znode in zookeeper, and it appears to at least have the correct permissions.  I created this znode in the CLI:

addauth digest nifi:<passwd>
create /nifi data digest:nifi<passwd digest>:cdrwa

The digest was generated via:

java -cp '/op/zookeeper/lib/zookeeper-3.5.8.jar:/opt/zookeeper/lib/slf4j-api-1.7.25.jar' org.apache.auth.AuthenticationProvider nifi:<passwd> 

[zk: nifi1-5:2181,nifi2-5:2181,nifi3-5:2181(CONNECTED) 4] getAcl /nifi
'digest,'nifi:the-passwd-digest'
: cdrwa


after starting up Nifi, doing and ls /nifi, the znode is empty.
[zk: nifi1-5:2181,nifi2-5:2181,nifi3-5:2181(CONNECTED) 4] ls /nifi
[]

Seems like we can't write the leaders or components value under the /nifi znode.

 
Looking at the nifi-app log

2020-07-06 16:05:46,554 INFO [main-SendThread(xx.xxx.x.xx:2181)] org.apache.zookeeper.Login Client successfully logged in.
2020-07-06 16:05:46,556 INFO [main-SendThread(xx.xxx.x.xx:2181)] o.a.zookeeper.client.ZooKeeperSaslClient Client will use DIGEST-MD5 as SASL mechanism.
2020-07-06 16:05:46,900 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2020-07-06 16:05:47,347 INFO [main-EventThread] o.a.c.framework.imps.EnsembleTracker New config event received: {server.1=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181, version=0, server.3=xx.xxx.x.xx:2888:3888:participant;0.0.0.0:2181, server.2=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181}
2020-07-06 16:05:47,354 INFO [main-EventThread] o.a.c.framework.imps.EnsembleTracker New config event received: {server.1=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181, version=0, server.3=xx.xxx.x.xx:2888:3888:participant;0.0.0.0:2181, server.2=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181}
2020-07-06 16:05:47,357 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
2020-07-06 16:05:47,364 DEBUG [main] org.apache.zookeeper.ZooKeeper Closing session: 0x3002a05b0c60006
2020-07-06 16:05:47,469 INFO [main/ org.apache.zookeeper.ZooKeeper Session: 0x3002a05b0c60006 closed



Any ideas on what configuration I could be missing or have wrong?  I have a jaas.conf file in the $NIFI_HOME/conf directory and have a java.arg.18--Djava.security.auth.login.config=<path to jaas.conf file>

One question I have, in the jaas.conf file, I put the passwd in there and not the digest I believe...I understand this would be passed around cleartext, but this is just for testing purposes currently....

Nifi 1.11.4
external zookeeper 3.5.8

Regards,

Dano

Reply | Threaded
Open this post in threaded view
|

Re: zookeeper SASL/Digest authentication....

Bryan Bende
This seems like exactly the scenario we have, but unfortunately no resolution...

On ZK mailing list, same issue posted and says after restarting ZK it somehow works now...

This answer also makes me wonder about whether we'd have to do something special when knowing that digest is being used...



On Mon, Jul 6, 2020 at 3:06 PM Bryan Bende <[hidden email]> wrote:
So right now the /nifi z-node exists with the ACL you showed earlier for 'digest,'nifi:the-passwd-digest' , but then '/nifi/components' doesn't exist yet?

The one difference from a code perspective is that /nifi and the cluster nodes are created by Curator, and the state provider is done using plain ZK client code, although no reason why that should work.

I'm no ZK expert, but the code that is causing the error is a call to "create(path, data, acls, CreateMode.PERSISTENT)" where "acls" is a list with one element of  Ids.CREATOR_ALL_ACL which has
/**
 * This Id is only usable to set ACLs. It will get substituted with the
 * Id's the client authenticated with.
 */
public final Id AUTH_IDS = new Id("auth", "");

Any ZK client code should be seeing the same JAAS entry you configured, so not sure how it could be authenticating as different identities.


On Mon, Jul 6, 2020 at 2:27 PM dan young <[hidden email]> wrote:
Correct, the leader seems to work, but not the components it seems..... Is there some additional config setting I might be missing?



<stateManagement>
   <local-provider>
      <id>local-provider</id>
      <class>org.apache.nifi.controller.state.providers.local.WriteAheadLocalStateProvider</class>
      <property name="Directory">/opt/nifi-configuration-resources/state/local</property>
      <property name="Always Sync">false</property>
      <property name="Partitions">16</property>
      <property name="Checkpoint Interval">2 mins</property>
   </local-provider>
   <cluster-provider>
      <id>zk-provider</id>
      <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class>
      <property name="Root Node">/nifi</property>
      <property name="Session Timeout">30 seconds</property>
      <property name="Access Control">CreatorOnly</property>
      <property name="Connect String">xx.xxx.x.xxx:2181,xx.xxx.x.xxx:2181,xx.xxx.x.xxx:2181</property>
   </cluster-provider>
</stateManagement>




2020-07-06 18:25:04,830 ERROR [Timer-Driven Process Thread-3] o.a.n.p.standard.GenerateTableFetch GenerateTableFetch[id=2a056fd7-b63c-33b4-a5c4-bf767c1a2983] GenerateTableFetch[id=2a056fd7-b63c-33b4-a5c4-bf767c1a2983] failed to update State Manager, observed maximum values will not be recorded. Also, any generated SQL statements may be duplicated.: java.io.IOException: Failed to set cluster-wide state in ZooKeeper for component with ID 2a056fd7-b63c-33b4-a5c4-bf767c1a2983
java.io.IOException: Failed to set cluster-wide state in ZooKeeper for component with ID 2a056fd7-b63c-33b4-a5c4-bf767c1a2983
        at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:343)
        at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:283)
        at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:228)
        at org.apache.nifi.controller.state.manager.StandardStateManagerProvider$1.setState(StandardStateManagerProvider.java:298)
        at org.apache.nifi.controller.state.StandardStateManager.setState(StandardStateManager.java:79)
        at org.apache.nifi.controller.lifecycle.TaskTerminationAwareStateManager.setState(TaskTerminationAwareStateManager.java:64)
        at org.apache.nifi.processors.standard.GenerateTableFetch.onTrigger(GenerateTableFetch.java:555)
        at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
        at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
        at org.apache.nifi.controller.scheduling.QuartzSchedulingAgent$2.run(QuartzSchedulingAgent.java:151)
        at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /nifi/components/2a056fd7-b63c-33b4-a5c4-bf767c1a2983
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:128)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
        at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1538)
        at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.createNode(ZooKeeperStateProvider.java:360)
        at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:321)
        ... 17 common frames omitted




On Mon, Jul 6, 2020 at 11:59 AM Bryan Bende <[hidden email]> wrote:
You set <property name="Access Control">CreatorOnly</property> in the ZK state manager ?

On Mon, Jul 6, 2020 at 1:40 PM dan young <[hidden email]> wrote:
Fat fingered... any insight into this error when the GenerateTableFectch?

Failed to set cluster-wide state in Zookeeper...
...
...

Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /nifi/components/2a056fd7-b63c-33b4-a5c4-bf767c1a2983
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:128)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
        at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1538)
        at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.createNode(ZooKeeperStateProvider.java:360)
        at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:321)
        ... 17 common frames omitted


On Mon, Jul 6, 2020 at 11:39 AM dan young <[hidden email]> wrote:
Hello Bryan,

Making some progress....any insight into this error with GenerateTableFectch processor?


On Mon, Jul 6, 2020 at 10:47 AM Bryan Bende <[hidden email]> wrote:
Have you configured this in nifi.properties?
nifi.zookeeper.auth.type=sasl


On Mon, Jul 6, 2020 at 12:43 PM dan young <[hidden email]> wrote:
Hello,

And a follow up on this, if I delete the znode in zookeeper, the leaders is written to the /nifi znode, but the ACL is open, 'world';'anyone....  I do have the Access COntrol set to CreatorOnly in the state-management.xml.  So one question, is the CreatorOnly only supported when we run in kerberos env?

Dano

On Mon, Jul 6, 2020 at 10:36 AM dan young <[hidden email]> wrote:
Hello everyone,

I'm trying to configure the zookeeper state provider in  NiFi to use the Access Policy of CreatorOnly vs Open using DIGEST vs Kerberos.   I believe I've setup zookeeper correctly for this, and partly Nifi, but when I startup nifi cluster, we seem to get stuck with the following:

2020-07-06 16:06:20,826 WARN [Clustering Tasks Thread-1] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:06:35,920 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:06:50,923 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:07:06,071 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected

I can see the znode in zookeeper, and it appears to at least have the correct permissions.  I created this znode in the CLI:

addauth digest nifi:<passwd>
create /nifi data digest:nifi<passwd digest>:cdrwa

The digest was generated via:

java -cp '/op/zookeeper/lib/zookeeper-3.5.8.jar:/opt/zookeeper/lib/slf4j-api-1.7.25.jar' org.apache.auth.AuthenticationProvider nifi:<passwd> 

[zk: nifi1-5:2181,nifi2-5:2181,nifi3-5:2181(CONNECTED) 4] getAcl /nifi
'digest,'nifi:the-passwd-digest'
: cdrwa


after starting up Nifi, doing and ls /nifi, the znode is empty.
[zk: nifi1-5:2181,nifi2-5:2181,nifi3-5:2181(CONNECTED) 4] ls /nifi
[]

Seems like we can't write the leaders or components value under the /nifi znode.

 
Looking at the nifi-app log

2020-07-06 16:05:46,554 INFO [main-SendThread(xx.xxx.x.xx:2181)] org.apache.zookeeper.Login Client successfully logged in.
2020-07-06 16:05:46,556 INFO [main-SendThread(xx.xxx.x.xx:2181)] o.a.zookeeper.client.ZooKeeperSaslClient Client will use DIGEST-MD5 as SASL mechanism.
2020-07-06 16:05:46,900 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2020-07-06 16:05:47,347 INFO [main-EventThread] o.a.c.framework.imps.EnsembleTracker New config event received: {server.1=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181, version=0, server.3=xx.xxx.x.xx:2888:3888:participant;0.0.0.0:2181, server.2=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181}
2020-07-06 16:05:47,354 INFO [main-EventThread] o.a.c.framework.imps.EnsembleTracker New config event received: {server.1=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181, version=0, server.3=xx.xxx.x.xx:2888:3888:participant;0.0.0.0:2181, server.2=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181}
2020-07-06 16:05:47,357 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
2020-07-06 16:05:47,364 DEBUG [main] org.apache.zookeeper.ZooKeeper Closing session: 0x3002a05b0c60006
2020-07-06 16:05:47,469 INFO [main/ org.apache.zookeeper.ZooKeeper Session: 0x3002a05b0c60006 closed



Any ideas on what configuration I could be missing or have wrong?  I have a jaas.conf file in the $NIFI_HOME/conf directory and have a java.arg.18--Djava.security.auth.login.config=<path to jaas.conf file>

One question I have, in the jaas.conf file, I put the passwd in there and not the digest I believe...I understand this would be passed around cleartext, but this is just for testing purposes currently....

Nifi 1.11.4
external zookeeper 3.5.8

Regards,

Dano

Reply | Threaded
Open this post in threaded view
|

Re: zookeeper SASL/Digest authentication....

danoyoung
In reply to this post by Bryan Bende
"So right now the /nifi z-node exists with the ACL you showed earlier for 'digest,'nifi:the-passwd-digest' , but then '/nifi/components' doesn't exist yet?" correct....we see the leaders, and under that there's the Cluster Coordinator and the Primary Node children.  When I start the GenerateTableFetch, that's when we get the error.  On other clusters we're running  (with Open) we see the component id/UUID under the /nifi/components/<the-UUID>   I can do a get on them and see the table and the Max-value columns value.



On Mon, Jul 6, 2020 at 1:06 PM Bryan Bende <[hidden email]> wrote:
So right now the /nifi z-node exists with the ACL you showed earlier for 'digest,'nifi:the-passwd-digest' , but then '/nifi/components' doesn't exist yet?

The one difference from a code perspective is that /nifi and the cluster nodes are created by Curator, and the state provider is done using plain ZK client code, although no reason why that should work.

I'm no ZK expert, but the code that is causing the error is a call to "create(path, data, acls, CreateMode.PERSISTENT)" where "acls" is a list with one element of  Ids.CREATOR_ALL_ACL which has
/**
 * This Id is only usable to set ACLs. It will get substituted with the
 * Id's the client authenticated with.
 */
public final Id AUTH_IDS = new Id("auth", "");

Any ZK client code should be seeing the same JAAS entry you configured, so not sure how it could be authenticating as different identities.


On Mon, Jul 6, 2020 at 2:27 PM dan young <[hidden email]> wrote:
Correct, the leader seems to work, but not the components it seems..... Is there some additional config setting I might be missing?



<stateManagement>
   <local-provider>
      <id>local-provider</id>
      <class>org.apache.nifi.controller.state.providers.local.WriteAheadLocalStateProvider</class>
      <property name="Directory">/opt/nifi-configuration-resources/state/local</property>
      <property name="Always Sync">false</property>
      <property name="Partitions">16</property>
      <property name="Checkpoint Interval">2 mins</property>
   </local-provider>
   <cluster-provider>
      <id>zk-provider</id>
      <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class>
      <property name="Root Node">/nifi</property>
      <property name="Session Timeout">30 seconds</property>
      <property name="Access Control">CreatorOnly</property>
      <property name="Connect String">xx.xxx.x.xxx:2181,xx.xxx.x.xxx:2181,xx.xxx.x.xxx:2181</property>
   </cluster-provider>
</stateManagement>




2020-07-06 18:25:04,830 ERROR [Timer-Driven Process Thread-3] o.a.n.p.standard.GenerateTableFetch GenerateTableFetch[id=2a056fd7-b63c-33b4-a5c4-bf767c1a2983] GenerateTableFetch[id=2a056fd7-b63c-33b4-a5c4-bf767c1a2983] failed to update State Manager, observed maximum values will not be recorded. Also, any generated SQL statements may be duplicated.: java.io.IOException: Failed to set cluster-wide state in ZooKeeper for component with ID 2a056fd7-b63c-33b4-a5c4-bf767c1a2983
java.io.IOException: Failed to set cluster-wide state in ZooKeeper for component with ID 2a056fd7-b63c-33b4-a5c4-bf767c1a2983
        at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:343)
        at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:283)
        at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:228)
        at org.apache.nifi.controller.state.manager.StandardStateManagerProvider$1.setState(StandardStateManagerProvider.java:298)
        at org.apache.nifi.controller.state.StandardStateManager.setState(StandardStateManager.java:79)
        at org.apache.nifi.controller.lifecycle.TaskTerminationAwareStateManager.setState(TaskTerminationAwareStateManager.java:64)
        at org.apache.nifi.processors.standard.GenerateTableFetch.onTrigger(GenerateTableFetch.java:555)
        at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
        at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
        at org.apache.nifi.controller.scheduling.QuartzSchedulingAgent$2.run(QuartzSchedulingAgent.java:151)
        at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /nifi/components/2a056fd7-b63c-33b4-a5c4-bf767c1a2983
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:128)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
        at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1538)
        at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.createNode(ZooKeeperStateProvider.java:360)
        at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:321)
        ... 17 common frames omitted




On Mon, Jul 6, 2020 at 11:59 AM Bryan Bende <[hidden email]> wrote:
You set <property name="Access Control">CreatorOnly</property> in the ZK state manager ?

On Mon, Jul 6, 2020 at 1:40 PM dan young <[hidden email]> wrote:
Fat fingered... any insight into this error when the GenerateTableFectch?

Failed to set cluster-wide state in Zookeeper...
...
...

Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /nifi/components/2a056fd7-b63c-33b4-a5c4-bf767c1a2983
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:128)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
        at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1538)
        at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.createNode(ZooKeeperStateProvider.java:360)
        at org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider.setState(ZooKeeperStateProvider.java:321)
        ... 17 common frames omitted


On Mon, Jul 6, 2020 at 11:39 AM dan young <[hidden email]> wrote:
Hello Bryan,

Making some progress....any insight into this error with GenerateTableFectch processor?


On Mon, Jul 6, 2020 at 10:47 AM Bryan Bende <[hidden email]> wrote:
Have you configured this in nifi.properties?
nifi.zookeeper.auth.type=sasl


On Mon, Jul 6, 2020 at 12:43 PM dan young <[hidden email]> wrote:
Hello,

And a follow up on this, if I delete the znode in zookeeper, the leaders is written to the /nifi znode, but the ACL is open, 'world';'anyone....  I do have the Access COntrol set to CreatorOnly in the state-management.xml.  So one question, is the CreatorOnly only supported when we run in kerberos env?

Dano

On Mon, Jul 6, 2020 at 10:36 AM dan young <[hidden email]> wrote:
Hello everyone,

I'm trying to configure the zookeeper state provider in  NiFi to use the Access Policy of CreatorOnly vs Open using DIGEST vs Kerberos.   I believe I've setup zookeeper correctly for this, and partly Nifi, but when I startup nifi cluster, we seem to get stuck with the following:

2020-07-06 16:06:20,826 WARN [Clustering Tasks Thread-1] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:06:35,920 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:06:50,923 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected
2020-07-06 16:07:06,071 WARN [Clustering Tasks Thread-2] o.apache.nifi.controller.FlowController Failed to send heartbeat due to: org.apache.nifi.cluster.protocol.ProtocolException: Cannot send heartbeat because there is no Cluster Coordinator currently elected

I can see the znode in zookeeper, and it appears to at least have the correct permissions.  I created this znode in the CLI:

addauth digest nifi:<passwd>
create /nifi data digest:nifi<passwd digest>:cdrwa

The digest was generated via:

java -cp '/op/zookeeper/lib/zookeeper-3.5.8.jar:/opt/zookeeper/lib/slf4j-api-1.7.25.jar' org.apache.auth.AuthenticationProvider nifi:<passwd> 

[zk: nifi1-5:2181,nifi2-5:2181,nifi3-5:2181(CONNECTED) 4] getAcl /nifi
'digest,'nifi:the-passwd-digest'
: cdrwa


after starting up Nifi, doing and ls /nifi, the znode is empty.
[zk: nifi1-5:2181,nifi2-5:2181,nifi3-5:2181(CONNECTED) 4] ls /nifi
[]

Seems like we can't write the leaders or components value under the /nifi znode.

 
Looking at the nifi-app log

2020-07-06 16:05:46,554 INFO [main-SendThread(xx.xxx.x.xx:2181)] org.apache.zookeeper.Login Client successfully logged in.
2020-07-06 16:05:46,556 INFO [main-SendThread(xx.xxx.x.xx:2181)] o.a.zookeeper.client.ZooKeeperSaslClient Client will use DIGEST-MD5 as SASL mechanism.
2020-07-06 16:05:46,900 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2020-07-06 16:05:47,347 INFO [main-EventThread] o.a.c.framework.imps.EnsembleTracker New config event received: {server.1=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181, version=0, server.3=xx.xxx.x.xx:2888:3888:participant;0.0.0.0:2181, server.2=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181}
2020-07-06 16:05:47,354 INFO [main-EventThread] o.a.c.framework.imps.EnsembleTracker New config event received: {server.1=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181, version=0, server.3=xx.xxx.x.xx:2888:3888:participant;0.0.0.0:2181, server.2=xx.xxx.x.xxx:2888:3888:participant;0.0.0.0:2181}
2020-07-06 16:05:47,357 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
2020-07-06 16:05:47,364 DEBUG [main] org.apache.zookeeper.ZooKeeper Closing session: 0x3002a05b0c60006
2020-07-06 16:05:47,469 INFO [main/ org.apache.zookeeper.ZooKeeper Session: 0x3002a05b0c60006 closed



Any ideas on what configuration I could be missing or have wrong?  I have a jaas.conf file in the $NIFI_HOME/conf directory and have a java.arg.18--Djava.security.auth.login.config=<path to jaas.conf file>

One question I have, in the jaas.conf file, I put the passwd in there and not the digest I believe...I understand this would be passed around cleartext, but this is just for testing purposes currently....

Nifi 1.11.4
external zookeeper 3.5.8

Regards,

Dano