NiFi 1.11.4 Upgrade

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

NiFi 1.11.4 Upgrade

Sri Harsha Chavali

Hi Users,

 

I'm here with another question regarding the upgrade from NiFi-1.9.2 to NiFi-1.11.4. After testing a different instance of NiFi-1.11.4 and getting comfortable with the features, we have decided to upgrade our existing NiFi instance to 1.11.4. All of the process was smooth and I was able to get secure NiFi-1.11.4 up and running and then I get this message after logging into NiFi. Note that the authentication which uses LDAP has succeeded but then I get the screen which says:

 

An unexpected error has occurred. Please check the logs for additional details.

 

I copied users.xml, authorizations.xml as well as database_repository files from previous install to new installation. I have a separate location for other configurations like authorizers.xml, login-identity-providers.xml, state-management.xml, authorizers.xml,  flow.xml.gz and also the keystore and trustore files. I pointed the latest NiFi instance to the common config before starting it. Also, provenance, content and flowfile repositories are configured separately and I pointed new install to the existing locations.


Given below are the error messages in logs after authentication is successful. My UserId is unable to access the root process group and basically nothing can be accessed in NiFi.

 

nifi-user.log:

 

2020-07-14 13:41:18,996 INFO [NiFi Web Server-75] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for <UserName>

2020-07-14 13:41:19,064 ERROR [NiFi Web Server-36] o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: java.lang.NullPointerException. Returning Internal Server Error response.

java.lang.NullPointerException: null

        at org.apache.nifi.util.FlowDifferenceFilters.isNewRelationshipAutoTerminatedAndDefaulted(FlowDifferenceFilters.java:203)

        at org.apache.nifi.groups.StandardProcessGroup.lambda$getModifications$84(StandardProcessGroup.java:4911)

        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)

        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)

        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)

        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)

        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)

        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)

        at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1556)

        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)

        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)…..

 

nifi-app.log:

2020-07-14 13:41:52,706 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@6a2d0632 checkpointed with 0 Records and 0 Swap Files in 5 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 2 millis), max Transaction ID -1

2020-07-14 13:41:53,299 ERROR [Timer-Driven Process Thread-7] o.apache.nifi.controller.FlowController Failed to capture component stats for Stats History

java.lang.NullPointerException: null

        at org.apache.nifi.util.FlowDifferenceFilters.isNewRelationshipAutoTerminatedAndDefaulted(FlowDifferenceFilters.java:203)

        at org.apache.nifi.groups.StandardProcessGroup.lambda$getModifications$84(StandardProcessGroup.java:4911)……


One thing to note is that the existing NiFi instance is installed a system service and we start and stop it using a sudo user. As this is the first time using the new instance, I didn't change the system service or init.d configuration and started nifi user. Also, the previous and current installs both have the below property in bootstrap.conf

rus.as=nifi

Could you please guide me what I'm doing wrong and what needs to be done? I'm hoping this is a simple authorization issue which can be fixed with a small change.

Thank you,
Harsha

Sent from Outlook

Reply | Threaded
Open this post in threaded view
|

Re: NiFi 1.11.4 Upgrade

Bryan Bende
Hello,

Unfortunately this was a bug found after 1.11.4 and has been fixed in master, but is not in a release yet.


The community is hoping to release 1.12.0 soon though.

Thanks,

Bryan


On Tue, Jul 14, 2020 at 10:15 PM Sri Harsha Chavali <[hidden email]> wrote:

Hi Users,

 

I'm here with another question regarding the upgrade from NiFi-1.9.2 to NiFi-1.11.4. After testing a different instance of NiFi-1.11.4 and getting comfortable with the features, we have decided to upgrade our existing NiFi instance to 1.11.4. All of the process was smooth and I was able to get secure NiFi-1.11.4 up and running and then I get this message after logging into NiFi. Note that the authentication which uses LDAP has succeeded but then I get the screen which says:

 

An unexpected error has occurred. Please check the logs for additional details.

 

I copied users.xml, authorizations.xml as well as database_repository files from previous install to new installation. I have a separate location for other configurations like authorizers.xml, login-identity-providers.xml, state-management.xml, authorizers.xml,  flow.xml.gz and also the keystore and trustore files. I pointed the latest NiFi instance to the common config before starting it. Also, provenance, content and flowfile repositories are configured separately and I pointed new install to the existing locations.


Given below are the error messages in logs after authentication is successful. My UserId is unable to access the root process group and basically nothing can be accessed in NiFi.

 

nifi-user.log:

 

2020-07-14 13:41:18,996 INFO [NiFi Web Server-75] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for <UserName>

2020-07-14 13:41:19,064 ERROR [NiFi Web Server-36] o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: java.lang.NullPointerException. Returning Internal Server Error response.

java.lang.NullPointerException: null

        at org.apache.nifi.util.FlowDifferenceFilters.isNewRelationshipAutoTerminatedAndDefaulted(FlowDifferenceFilters.java:203)

        at org.apache.nifi.groups.StandardProcessGroup.lambda$getModifications$84(StandardProcessGroup.java:4911)

        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)

        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)

        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)

        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)

        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)

        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)

        at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1556)

        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)

        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)…..

 

nifi-app.log:

2020-07-14 13:41:52,706 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@6a2d0632 checkpointed with 0 Records and 0 Swap Files in 5 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 2 millis), max Transaction ID -1

2020-07-14 13:41:53,299 ERROR [Timer-Driven Process Thread-7] o.apache.nifi.controller.FlowController Failed to capture component stats for Stats History

java.lang.NullPointerException: null

        at org.apache.nifi.util.FlowDifferenceFilters.isNewRelationshipAutoTerminatedAndDefaulted(FlowDifferenceFilters.java:203)

        at org.apache.nifi.groups.StandardProcessGroup.lambda$getModifications$84(StandardProcessGroup.java:4911)……


One thing to note is that the existing NiFi instance is installed a system service and we start and stop it using a sudo user. As this is the first time using the new instance, I didn't change the system service or init.d configuration and started nifi user. Also, the previous and current installs both have the below property in bootstrap.conf

rus.as=nifi

Could you please guide me what I'm doing wrong and what needs to be done? I'm hoping this is a simple authorization issue which can be fixed with a small change.

Thank you,
Harsha

Sent from Outlook

Reply | Threaded
Open this post in threaded view
|

Re: NiFi 1.11.4 Upgrade

Sri Harsha Chavali
Hi Bryan,

Thank you for sharing that. We did some testing of 1.11.4 on a server without versioned flows and liked all new features in 1.11.4 but unfortunately are not in a position to upgrade to 1.11.4 on our actual nifi servers. The features we are mainly looking for are the lifecycle methods on InvokeScriptedProcessor (https://issues.apache.org/jira/browse/NIFI-2215) and some other features. In order to get these what do you suggest is the best and stable version we can upgrade to from 1.9.2? We are really heavy on usage of custom processors. I know it's suggested to build our own full-blown processors but we really like the ease of use and code change using InvokeScriptedProcessor. 

Also, any suggestion to find all the versioned flows where we have NULL values in autoTerminatedRelationships and fixing them? We would like to see if they can be fixed on our side and still go with 1.11.4 deployment.
 
Thank you,
Harsha

Sent from Outlook


From: Bryan Bende <[hidden email]>
Sent: Wednesday, July 15, 2020 9:06 AM
To: [hidden email] <[hidden email]>
Subject: Re: NiFi 1.11.4 Upgrade
 
Hello,

Unfortunately this was a bug found after 1.11.4 and has been fixed in master, but is not in a release yet.


The community is hoping to release 1.12.0 soon though.

Thanks,

Bryan


On Tue, Jul 14, 2020 at 10:15 PM Sri Harsha Chavali <[hidden email]> wrote:

Hi Users,

 

I'm here with another question regarding the upgrade from NiFi-1.9.2 to NiFi-1.11.4. After testing a different instance of NiFi-1.11.4 and getting comfortable with the features, we have decided to upgrade our existing NiFi instance to 1.11.4. All of the process was smooth and I was able to get secure NiFi-1.11.4 up and running and then I get this message after logging into NiFi. Note that the authentication which uses LDAP has succeeded but then I get the screen which says:

 

An unexpected error has occurred. Please check the logs for additional details.

 

I copied users.xml, authorizations.xml as well as database_repository files from previous install to new installation. I have a separate location for other configurations like authorizers.xml, login-identity-providers.xml, state-management.xml, authorizers.xml,  flow.xml.gz and also the keystore and trustore files. I pointed the latest NiFi instance to the common config before starting it. Also, provenance, content and flowfile repositories are configured separately and I pointed new install to the existing locations.


Given below are the error messages in logs after authentication is successful. My UserId is unable to access the root process group and basically nothing can be accessed in NiFi.

 

nifi-user.log:

 

2020-07-14 13:41:18,996 INFO [NiFi Web Server-75] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for <UserName>

2020-07-14 13:41:19,064 ERROR [NiFi Web Server-36] o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: java.lang.NullPointerException. Returning Internal Server Error response.

java.lang.NullPointerException: null

        at org.apache.nifi.util.FlowDifferenceFilters.isNewRelationshipAutoTerminatedAndDefaulted(FlowDifferenceFilters.java:203)

        at org.apache.nifi.groups.StandardProcessGroup.lambda$getModifications$84(StandardProcessGroup.java:4911)

        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)

        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)

        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)

        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)

        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)

        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)

        at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1556)

        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)

        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)…..

 

nifi-app.log:

2020-07-14 13:41:52,706 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@6a2d0632 checkpointed with 0 Records and 0 Swap Files in 5 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 2 millis), max Transaction ID -1

2020-07-14 13:41:53,299 ERROR [Timer-Driven Process Thread-7] o.apache.nifi.controller.FlowController Failed to capture component stats for Stats History

java.lang.NullPointerException: null

        at org.apache.nifi.util.FlowDifferenceFilters.isNewRelationshipAutoTerminatedAndDefaulted(FlowDifferenceFilters.java:203)

        at org.apache.nifi.groups.StandardProcessGroup.lambda$getModifications$84(StandardProcessGroup.java:4911)……


One thing to note is that the existing NiFi instance is installed a system service and we start and stop it using a sudo user. As this is the first time using the new instance, I didn't change the system service or init.d configuration and started nifi user. Also, the previous and current installs both have the below property in bootstrap.conf

rus.as=nifi

Could you please guide me what I'm doing wrong and what needs to be done? I'm hoping this is a simple authorization issue which can be fixed with a small change.

Thank you,
Harsha

Sent from Outlook

Reply | Threaded
Open this post in threaded view
|

Re: NiFi 1.11.4 Upgrade

Bryan Bende
Since NIFI-2215 has a fix version of 1.10.0, you could try going to that to get the scripted processor improvements, but its possible NIFI-7460 also affects the 1.10.0 release, I don't know without trying it out.

On Wed, Jul 15, 2020 at 5:09 PM Sri Harsha Chavali <[hidden email]> wrote:
Hi Bryan,

Thank you for sharing that. We did some testing of 1.11.4 on a server without versioned flows and liked all new features in 1.11.4 but unfortunately are not in a position to upgrade to 1.11.4 on our actual nifi servers. The features we are mainly looking for are the lifecycle methods on InvokeScriptedProcessor (https://issues.apache.org/jira/browse/NIFI-2215) and some other features. In order to get these what do you suggest is the best and stable version we can upgrade to from 1.9.2? We are really heavy on usage of custom processors. I know it's suggested to build our own full-blown processors but we really like the ease of use and code change using InvokeScriptedProcessor. 

Also, any suggestion to find all the versioned flows where we have NULL values in autoTerminatedRelationships and fixing them? We would like to see if they can be fixed on our side and still go with 1.11.4 deployment.
 
Thank you,
Harsha

Sent from Outlook


From: Bryan Bende <[hidden email]>
Sent: Wednesday, July 15, 2020 9:06 AM
To: [hidden email] <[hidden email]>
Subject: Re: NiFi 1.11.4 Upgrade
 
Hello,

Unfortunately this was a bug found after 1.11.4 and has been fixed in master, but is not in a release yet.


The community is hoping to release 1.12.0 soon though.

Thanks,

Bryan


On Tue, Jul 14, 2020 at 10:15 PM Sri Harsha Chavali <[hidden email]> wrote:

Hi Users,

 

I'm here with another question regarding the upgrade from NiFi-1.9.2 to NiFi-1.11.4. After testing a different instance of NiFi-1.11.4 and getting comfortable with the features, we have decided to upgrade our existing NiFi instance to 1.11.4. All of the process was smooth and I was able to get secure NiFi-1.11.4 up and running and then I get this message after logging into NiFi. Note that the authentication which uses LDAP has succeeded but then I get the screen which says:

 

An unexpected error has occurred. Please check the logs for additional details.

 

I copied users.xml, authorizations.xml as well as database_repository files from previous install to new installation. I have a separate location for other configurations like authorizers.xml, login-identity-providers.xml, state-management.xml, authorizers.xml,  flow.xml.gz and also the keystore and trustore files. I pointed the latest NiFi instance to the common config before starting it. Also, provenance, content and flowfile repositories are configured separately and I pointed new install to the existing locations.


Given below are the error messages in logs after authentication is successful. My UserId is unable to access the root process group and basically nothing can be accessed in NiFi.

 

nifi-user.log:

 

2020-07-14 13:41:18,996 INFO [NiFi Web Server-75] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for <UserName>

2020-07-14 13:41:19,064 ERROR [NiFi Web Server-36] o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: java.lang.NullPointerException. Returning Internal Server Error response.

java.lang.NullPointerException: null

        at org.apache.nifi.util.FlowDifferenceFilters.isNewRelationshipAutoTerminatedAndDefaulted(FlowDifferenceFilters.java:203)

        at org.apache.nifi.groups.StandardProcessGroup.lambda$getModifications$84(StandardProcessGroup.java:4911)

        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)

        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)

        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)

        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)

        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)

        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)

        at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1556)

        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)

        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)…..

 

nifi-app.log:

2020-07-14 13:41:52,706 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@6a2d0632 checkpointed with 0 Records and 0 Swap Files in 5 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 2 millis), max Transaction ID -1

2020-07-14 13:41:53,299 ERROR [Timer-Driven Process Thread-7] o.apache.nifi.controller.FlowController Failed to capture component stats for Stats History

java.lang.NullPointerException: null

        at org.apache.nifi.util.FlowDifferenceFilters.isNewRelationshipAutoTerminatedAndDefaulted(FlowDifferenceFilters.java:203)

        at org.apache.nifi.groups.StandardProcessGroup.lambda$getModifications$84(StandardProcessGroup.java:4911)……


One thing to note is that the existing NiFi instance is installed a system service and we start and stop it using a sudo user. As this is the first time using the new instance, I didn't change the system service or init.d configuration and started nifi user. Also, the previous and current installs both have the below property in bootstrap.conf

rus.as=nifi

Could you please guide me what I'm doing wrong and what needs to be done? I'm hoping this is a simple authorization issue which can be fixed with a small change.

Thank you,
Harsha

Sent from Outlook