3 node of nifi generating 3 different flow files

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

3 node of nifi generating 3 different flow files

sanjeet rath
Hi ,

I got a issue while starting a 3 node nifi cluster. nifi-app.log is showing exception that" local flow file is differet than cluster flow"
The issue is i got 3 different flow files in each node.'rootGroupid' of each flow file is different.

Previous i was facing no issue in creating 3 node clister.but this time ,i made a change here  during creation the clusters i have encripted the flow file with using nifi toolkit.(encript-config.sh)

 Encript-config.sh -f /nifi/flow.xml.gz -n nifi/nifi.properties --bootstrapconf /nifi/bootstrap.conf -s password -x

Please note that there is are no flows are availavle in canvas.its a new nifi cluster.

Could you please help me to understand what i am doing wrong?

Thanks a lot in advance for helping me.

Regards,
Sanjeet
Reply | Threaded
Open this post in threaded view
|

Re: 3 node of nifi generating 3 different flow files

Andy LoPresto
You do not need to manually run this command, as it migrates the encryption key used for sensitive processor properties (e.g. database passwords in the flow) that are stored in your flow.xml.gz file. You only need this command when you have a cluster which has been using one encryption key to secure these values and you want to migrate to a new key. 

When starting a new cluster, set the nifi.sensitive.props.key value to the desired value on all cluster nodes, and NiFi will automatically encrypt and decrypt the sensitive processor properties with it.  


Andy LoPresto
[hidden email]
[hidden email]
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Jun 18, 2020, at 3:55 AM, sanjeet rath <[hidden email]> wrote:

Hi ,

I got a issue while starting a 3 node nifi cluster. nifi-app.log is showing exception that" local flow file is differet than cluster flow"
The issue is i got 3 different flow files in each node.'rootGroupid' of each flow file is different.

Previous i was facing no issue in creating 3 node clister.but this time ,i made a change here  during creation the clusters i have encripted the flow file with using nifi toolkit.(encript-config.sh)

 Encript-config.sh -f /nifi/flow.xml.gz -n nifi/nifi.properties --bootstrapconf /nifi/bootstrap.conf -s password -x

Please note that there is are no flows are availavle in canvas.its a new nifi cluster.

Could you please help me to understand what i am doing wrong?

Thanks a lot in advance for helping me.

Regards,
Sanjeet

Reply | Threaded
Open this post in threaded view
|

Re: 3 node of nifi generating 3 different flow files

sanjeet rath
Thanks, Andy for your quick response.
I did it as you suggested, now nifi is using its default encryption.

But still gettting the error, in nifi-app.log is showing exception that" local flow is differet than cluster flow"
As per my analysis, The issue is it got 3 different flow.xml.gz files in each node.The 'rootGroupid' of each flow.xml.gz file is different.
As per my understanding ,the flow.xml.gz fie should be same accross all nodes.so i copied flow.xml.gz file from one node to other two nodes to make it sync.but still getting same error.
The zookeeper is connected properly to all nodes.

Thanks,
Sanjeet


On Thu, 18 Jun 2020, 10:55 pm Andy LoPresto, <[hidden email]> wrote:
You do not need to manually run this command, as it migrates the encryption key used for sensitive processor properties (e.g. database passwords in the flow) that are stored in your flow.xml.gz file. You only need this command when you have a cluster which has been using one encryption key to secure these values and you want to migrate to a new key. 

When starting a new cluster, set the nifi.sensitive.props.key value to the desired value on all cluster nodes, and NiFi will automatically encrypt and decrypt the sensitive processor properties with it.  


Andy LoPresto
[hidden email]
[hidden email]
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Jun 18, 2020, at 3:55 AM, sanjeet rath <[hidden email]> wrote:

Hi ,

I got a issue while starting a 3 node nifi cluster. nifi-app.log is showing exception that" local flow file is differet than cluster flow"
The issue is i got 3 different flow files in each node.'rootGroupid' of each flow file is different.

Previous i was facing no issue in creating 3 node clister.but this time ,i made a change here  during creation the clusters i have encripted the flow file with using nifi toolkit.(encript-config.sh)

 Encript-config.sh -f /nifi/flow.xml.gz -n nifi/nifi.properties --bootstrapconf /nifi/bootstrap.conf -s password -x

Please note that there is are no flows are availavle in canvas.its a new nifi cluster.

Could you please help me to understand what i am doing wrong?

Thanks a lot in advance for helping me.

Regards,
Sanjeet

Reply | Threaded
Open this post in threaded view
|

Re: 3 node of nifi generating 3 different flow files

Andy LoPresto
Sanjeet, 

If this is for a new cluster, you can delete the flow.xml.gz file from all nodes and restart NiFi. When the nodes start up again, they will create the new flow definition file on each node respectively with the synced root process group ID. 

Andy LoPresto
[hidden email]
[hidden email]
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Jun 18, 2020, at 4:44 PM, sanjeet rath <[hidden email]> wrote:

Thanks, Andy for your quick response.
I did it as you suggested, now nifi is using its default encryption.

But still gettting the error, in nifi-app.log is showing exception that" local flow is differet than cluster flow"
As per my analysis, The issue is it got 3 different flow.xml.gz files in each node.The 'rootGroupid' of each flow.xml.gz file is different.
As per my understanding ,the flow.xml.gz fie should be same accross all nodes.so i copied flow.xml.gz file from one node to other two nodes to make it sync.but still getting same error.
The zookeeper is connected properly to all nodes.

Thanks,
Sanjeet


On Thu, 18 Jun 2020, 10:55 pm Andy LoPresto, <[hidden email]> wrote:
You do not need to manually run this command, as it migrates the encryption key used for sensitive processor properties (e.g. database passwords in the flow) that are stored in your flow.xml.gz file. You only need this command when you have a cluster which has been using one encryption key to secure these values and you want to migrate to a new key. 

When starting a new cluster, set the nifi.sensitive.props.key value to the desired value on all cluster nodes, and NiFi will automatically encrypt and decrypt the sensitive processor properties with it.  


Andy LoPresto
[hidden email]
[hidden email]
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Jun 18, 2020, at 3:55 AM, sanjeet rath <[hidden email]> wrote:

Hi ,

I got a issue while starting a 3 node nifi cluster. nifi-app.log is showing exception that" local flow file is differet than cluster flow"
The issue is i got 3 different flow files in each node.'rootGroupid' of each flow file is different.

Previous i was facing no issue in creating 3 node clister.but this time ,i made a change here  during creation the clusters i have encripted the flow file with using nifi toolkit.(encript-config.sh)

 Encript-config.sh -f /nifi/flow.xml.gz -n nifi/nifi.properties --bootstrapconf /nifi/bootstrap.conf -s password -x

Please note that there is are no flows are availavle in canvas.its a new nifi cluster.

Could you please help me to understand what i am doing wrong?

Thanks a lot in advance for helping me.

Regards,
Sanjeet


Reply | Threaded
Open this post in threaded view
|

Re: 3 node of nifi generating 3 different flow files

sanjeet rath
Hi Andy,
Already tried by deleting flow.xml.gz ,users.xml,authorization.xml file of 3 nodes.
But still getting the same error.
in nifi-app.log is showing exception that" local flow is differet than cluster flow"
Also new flow.xml.gz file is having different 'rootGroupid' on each node
Thanks.
Sanjeet

On Fri, 19 Jun 2020, 5:16 am Andy LoPresto, <[hidden email]> wrote:
Sanjeet, 

If this is for a new cluster, you can delete the flow.xml.gz file from all nodes and restart NiFi. When the nodes start up again, they will create the new flow definition file on each node respectively with the synced root process group ID. 

Andy LoPresto
[hidden email]
[hidden email]
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Jun 18, 2020, at 4:44 PM, sanjeet rath <[hidden email]> wrote:

Thanks, Andy for your quick response.
I did it as you suggested, now nifi is using its default encryption.

But still gettting the error, in nifi-app.log is showing exception that" local flow is differet than cluster flow"
As per my analysis, The issue is it got 3 different flow.xml.gz files in each node.The 'rootGroupid' of each flow.xml.gz file is different.
As per my understanding ,the flow.xml.gz fie should be same accross all nodes.so i copied flow.xml.gz file from one node to other two nodes to make it sync.but still getting same error.
The zookeeper is connected properly to all nodes.

Thanks,
Sanjeet


On Thu, 18 Jun 2020, 10:55 pm Andy LoPresto, <[hidden email]> wrote:
You do not need to manually run this command, as it migrates the encryption key used for sensitive processor properties (e.g. database passwords in the flow) that are stored in your flow.xml.gz file. You only need this command when you have a cluster which has been using one encryption key to secure these values and you want to migrate to a new key. 

When starting a new cluster, set the nifi.sensitive.props.key value to the desired value on all cluster nodes, and NiFi will automatically encrypt and decrypt the sensitive processor properties with it.  


Andy LoPresto
[hidden email]
[hidden email]
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Jun 18, 2020, at 3:55 AM, sanjeet rath <[hidden email]> wrote:

Hi ,

I got a issue while starting a 3 node nifi cluster. nifi-app.log is showing exception that" local flow file is differet than cluster flow"
The issue is i got 3 different flow files in each node.'rootGroupid' of each flow file is different.

Previous i was facing no issue in creating 3 node clister.but this time ,i made a change here  during creation the clusters i have encripted the flow file with using nifi toolkit.(encript-config.sh)

 Encript-config.sh -f /nifi/flow.xml.gz -n nifi/nifi.properties --bootstrapconf /nifi/bootstrap.conf -s password -x

Please note that there is are no flows are availavle in canvas.its a new nifi cluster.

Could you please help me to understand what i am doing wrong?

Thanks a lot in advance for helping me.

Regards,
Sanjeet


Reply | Threaded
Open this post in threaded view
|

Re: 3 node of nifi generating 3 different flow files

Andy LoPresto
Sanjeet, 

Did you stop every node in the cluster before deleting these files? Can you share the actual output of the log files?

Andy LoPresto
[hidden email]
[hidden email]
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Jun 18, 2020, at 5:15 PM, sanjeet rath <[hidden email]> wrote:

Hi Andy,
Already tried by deleting flow.xml.gz ,users.xml,authorization.xml file of 3 nodes.
But still getting the same error.
in nifi-app.log is showing exception that" local flow is differet than cluster flow"
Also new flow.xml.gz file is having different 'rootGroupid' on each node
Thanks.
Sanjeet

On Fri, 19 Jun 2020, 5:16 am Andy LoPresto, <[hidden email]> wrote:
Sanjeet, 

If this is for a new cluster, you can delete the flow.xml.gz file from all nodes and restart NiFi. When the nodes start up again, they will create the new flow definition file on each node respectively with the synced root process group ID. 

Andy LoPresto
[hidden email]
[hidden email]
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Jun 18, 2020, at 4:44 PM, sanjeet rath <[hidden email]> wrote:

Thanks, Andy for your quick response.
I did it as you suggested, now nifi is using its default encryption.

But still gettting the error, in nifi-app.log is showing exception that" local flow is differet than cluster flow"
As per my analysis, The issue is it got 3 different flow.xml.gz files in each node.The 'rootGroupid' of each flow.xml.gz file is different.
As per my understanding ,the flow.xml.gz fie should be same accross all nodes.so i copied flow.xml.gz file from one node to other two nodes to make it sync.but still getting same error.
The zookeeper is connected properly to all nodes.

Thanks,
Sanjeet


On Thu, 18 Jun 2020, 10:55 pm Andy LoPresto, <[hidden email]> wrote:
You do not need to manually run this command, as it migrates the encryption key used for sensitive processor properties (e.g. database passwords in the flow) that are stored in your flow.xml.gz file. You only need this command when you have a cluster which has been using one encryption key to secure these values and you want to migrate to a new key. 

When starting a new cluster, set the nifi.sensitive.props.key value to the desired value on all cluster nodes, and NiFi will automatically encrypt and decrypt the sensitive processor properties with it.  


Andy LoPresto
[hidden email]
[hidden email]
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Jun 18, 2020, at 3:55 AM, sanjeet rath <[hidden email]> wrote:

Hi ,

I got a issue while starting a 3 node nifi cluster. nifi-app.log is showing exception that" local flow file is differet than cluster flow"
The issue is i got 3 different flow files in each node.'rootGroupid' of each flow file is different.

Previous i was facing no issue in creating 3 node clister.but this time ,i made a change here  during creation the clusters i have encripted the flow file with using nifi toolkit.(encript-config.sh)

 Encript-config.sh -f /nifi/flow.xml.gz -n nifi/nifi.properties --bootstrapconf /nifi/bootstrap.conf -s password -x

Please note that there is are no flows are availavle in canvas.its a new nifi cluster.

Could you please help me to understand what i am doing wrong?

Thanks a lot in advance for helping me.

Regards,
Sanjeet