Hive_1_1 Processors and Controllers Missing in NiFi 1.11.4

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Hive_1_1 Processors and Controllers Missing in NiFi 1.11.4

Sri Harsha Chavali
Hi All,

We recently upgraded from nifi 1.9.2 to 1.11.4 and are facing an issue with Hive 1_1 processors. When we were in 1.9.2 we manually deployed the "nifi-hive_1_1-nar-1.9.2.nar" file (per suggestion in email chain https://lists.apache.org/list.html?users@...:lte=5y:SELECTHIVE_1_1_QL%20-%20NiFi%201.9.0%20Missing) in the extensions folder. Now after upgrading to 1.11.4 we notice that the nar file (nifi-hive_1_1-nar-1.11.4.nar) is missing in the tarball again so we had to manually add it again to the extensions folder. 

The issue is not with getting and adding the nar file to extensions as nifi automatically unpacks it and makes processors and controllers available. The issue is that we need to redo all the controllers and processors if we download the latest nifi-hive_1_1-nar-1.11.4.nar file. If we use the old nifi-hive_1_1-nar-1.9.2.nar we get the below error. The rest of the controllers/processors are automatically upgraded or are backward compatible. Could you please let me know if we need to redo all the processors or if there is a easy way to fix this issue by eliminating all the redo steps?





Thank you,
Harsha

Sent from Outlook


hive_1_1.png (90K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Hive_1_1 Processors and Controllers Missing in NiFi 1.11.4

Matt Burgess-2
Harsha,

There are two NARs associated with Hive components, nifi-hive-services-api-nar which has the Hive1_1ConnectionPool service (actually an interface, but that's under the hood), and the nifi-hive1_1-nar which has the processors that declare themselves as users of that interface (and the actual implementations). Did you keep the old Hive 1.1 NAR in the lib/ folder when you upgraded? I would think if you removed the 1.9.2 NARs and used just the 1.11.4 NARs, NiFi would only find the newer ones and automatically upgrade the existing processors and controller services, but I haven't tried it myself. I suspect if they are both present then NiFi will try to match the version of the processors to the controller services and they won't match. Either you'd need both 1.9.2 NARs (the services-api one and the processors one) or neither, I recommend the latter so the components get upgraded to the latest version and that they match. If you're doing a hot reload by dropping the NARs into the dynamic loading location, you'll probably want to remove the old ones entirely, put the new ones in lib/, and restart NiFi.

Regards,
Matt



On Tue, Jul 7, 2020 at 6:27 PM Sri Harsha Chavali <[hidden email]> wrote:
Hi All,

We recently upgraded from nifi 1.9.2 to 1.11.4 and are facing an issue with Hive 1_1 processors. When we were in 1.9.2 we manually deployed the "nifi-hive_1_1-nar-1.9.2.nar" file (per suggestion in email chain https://lists.apache.org/list.html?users@...:lte=5y:SELECTHIVE_1_1_QL%20-%20NiFi%201.9.0%20Missing) in the extensions folder. Now after upgrading to 1.11.4 we notice that the nar file (nifi-hive_1_1-nar-1.11.4.nar) is missing in the tarball again so we had to manually add it again to the extensions folder. 

The issue is not with getting and adding the nar file to extensions as nifi automatically unpacks it and makes processors and controllers available. The issue is that we need to redo all the controllers and processors if we download the latest nifi-hive_1_1-nar-1.11.4.nar file. If we use the old nifi-hive_1_1-nar-1.9.2.nar we get the below error. The rest of the controllers/processors are automatically upgraded or are backward compatible. Could you please let me know if we need to redo all the processors or if there is a easy way to fix this issue by eliminating all the redo steps?





Thank you,
Harsha

Sent from Outlook

Reply | Threaded
Open this post in threaded view
|

Re: Hive_1_1 Processors and Controllers Missing in NiFi 1.11.4

Sri Harsha Chavali
Hi Matt,

Restarting NiFi did the trick. I removed the old nifi-hive_1_1-nar-1.9.2.nar from the extensions directory and "restarted" NiFi and all the processors and controllers automatically picked the newer versions. Earlier I removed the old version and placed the new version of the nar in extensions folder but never restarted NiFi (assumed NiFi would dynamically drop the deleted nars like how it's dynamically loading) 😄

But one thing to note:
When I check the lib directory in the latest release I don't find the nifi-hive-1_1* at all. All we see is the below nars, which I assume are not 1_1 versions. So the 1_1 nars are completely missing in lib directory. We need to manually load them again. 



Thank you,
Harsha

Sent from Outlook


From: Matt Burgess <[hidden email]>
Sent: Tuesday, July 7, 2020 7:05 PM
To: [hidden email] <[hidden email]>
Subject: Re: Hive_1_1 Processors and Controllers Missing in NiFi 1.11.4
 
Harsha,

There are two NARs associated with Hive components, nifi-hive-services-api-nar which has the Hive1_1ConnectionPool service (actually an interface, but that's under the hood), and the nifi-hive1_1-nar which has the processors that declare themselves as users of that interface (and the actual implementations). Did you keep the old Hive 1.1 NAR in the lib/ folder when you upgraded? I would think if you removed the 1.9.2 NARs and used just the 1.11.4 NARs, NiFi would only find the newer ones and automatically upgrade the existing processors and controller services, but I haven't tried it myself. I suspect if they are both present then NiFi will try to match the version of the processors to the controller services and they won't match. Either you'd need both 1.9.2 NARs (the services-api one and the processors one) or neither, I recommend the latter so the components get upgraded to the latest version and that they match. If you're doing a hot reload by dropping the NARs into the dynamic loading location, you'll probably want to remove the old ones entirely, put the new ones in lib/, and restart NiFi.

Regards,
Matt



On Tue, Jul 7, 2020 at 6:27 PM Sri Harsha Chavali <[hidden email]> wrote:
Hi All,

We recently upgraded from nifi 1.9.2 to 1.11.4 and are facing an issue with Hive 1_1 processors. When we were in 1.9.2 we manually deployed the "nifi-hive_1_1-nar-1.9.2.nar" file (per suggestion in email chain https://lists.apache.org/list.html?users@...:lte=5y:SELECTHIVE_1_1_QL%20-%20NiFi%201.9.0%20Missing) in the extensions folder. Now after upgrading to 1.11.4 we notice that the nar file (nifi-hive_1_1-nar-1.11.4.nar) is missing in the tarball again so we had to manually add it again to the extensions folder. 

The issue is not with getting and adding the nar file to extensions as nifi automatically unpacks it and makes processors and controllers available. The issue is that we need to redo all the controllers and processors if we download the latest nifi-hive_1_1-nar-1.11.4.nar file. If we use the old nifi-hive_1_1-nar-1.9.2.nar we get the below error. The rest of the controllers/processors are automatically upgraded or are backward compatible. Could you please let me know if we need to redo all the processors or if there is a easy way to fix this issue by eliminating all the redo steps?





Thank you,
Harsha

Sent from Outlook

Reply | Threaded
Open this post in threaded view
|

Re: Hive_1_1 Processors and Controllers Missing in NiFi 1.11.4

Matt Burgess-2
Harsha,

The nifi-hive-services-api-nar is included with the NiFi assembly/release and includes the definition of what the Hive1_1ConnectionPool should be, but the actual component is in the nifi-hive1_1-nar, which is not included with the NiFi release (due to space concerns) and must be manually downloaded.

Regards,
Matt


On Tue, Jul 7, 2020 at 9:01 PM Sri Harsha Chavali <[hidden email]> wrote:
Hi Matt,

Restarting NiFi did the trick. I removed the old nifi-hive_1_1-nar-1.9.2.nar from the extensions directory and "restarted" NiFi and all the processors and controllers automatically picked the newer versions. Earlier I removed the old version and placed the new version of the nar in extensions folder but never restarted NiFi (assumed NiFi would dynamically drop the deleted nars like how it's dynamically loading) 😄

But one thing to note:
When I check the lib directory in the latest release I don't find the nifi-hive-1_1* at all. All we see is the below nars, which I assume are not 1_1 versions. So the 1_1 nars are completely missing in lib directory. We need to manually load them again. 



Thank you,
Harsha

Sent from Outlook


From: Matt Burgess <[hidden email]>
Sent: Tuesday, July 7, 2020 7:05 PM
To: [hidden email] <[hidden email]>
Subject: Re: Hive_1_1 Processors and Controllers Missing in NiFi 1.11.4
 
Harsha,

There are two NARs associated with Hive components, nifi-hive-services-api-nar which has the Hive1_1ConnectionPool service (actually an interface, but that's under the hood), and the nifi-hive1_1-nar which has the processors that declare themselves as users of that interface (and the actual implementations). Did you keep the old Hive 1.1 NAR in the lib/ folder when you upgraded? I would think if you removed the 1.9.2 NARs and used just the 1.11.4 NARs, NiFi would only find the newer ones and automatically upgrade the existing processors and controller services, but I haven't tried it myself. I suspect if they are both present then NiFi will try to match the version of the processors to the controller services and they won't match. Either you'd need both 1.9.2 NARs (the services-api one and the processors one) or neither, I recommend the latter so the components get upgraded to the latest version and that they match. If you're doing a hot reload by dropping the NARs into the dynamic loading location, you'll probably want to remove the old ones entirely, put the new ones in lib/, and restart NiFi.

Regards,
Matt



On Tue, Jul 7, 2020 at 6:27 PM Sri Harsha Chavali <[hidden email]> wrote:
Hi All,

We recently upgraded from nifi 1.9.2 to 1.11.4 and are facing an issue with Hive 1_1 processors. When we were in 1.9.2 we manually deployed the "nifi-hive_1_1-nar-1.9.2.nar" file (per suggestion in email chain https://lists.apache.org/list.html?users@...:lte=5y:SELECTHIVE_1_1_QL%20-%20NiFi%201.9.0%20Missing) in the extensions folder. Now after upgrading to 1.11.4 we notice that the nar file (nifi-hive_1_1-nar-1.11.4.nar) is missing in the tarball again so we had to manually add it again to the extensions folder. 

The issue is not with getting and adding the nar file to extensions as nifi automatically unpacks it and makes processors and controllers available. The issue is that we need to redo all the controllers and processors if we download the latest nifi-hive_1_1-nar-1.11.4.nar file. If we use the old nifi-hive_1_1-nar-1.9.2.nar we get the below error. The rest of the controllers/processors are automatically upgraded or are backward compatible. Could you please let me know if we need to redo all the processors or if there is a easy way to fix this issue by eliminating all the redo steps?





Thank you,
Harsha

Sent from Outlook

Reply | Threaded
Open this post in threaded view
|

Re: Hive_1_1 Processors and Controllers Missing in NiFi 1.11.4

Sri Harsha Chavali
Thanks for the inputs Matt! Got it. I think I now understand why we had the dependency issue when we had the old hive_1_1. We downloaded the latest version and have it as a to-do item for every upgrade going forward. 

Thank you,
Harsha

Sent from Outlook


From: Matt Burgess <[hidden email]>
Sent: Tuesday, July 7, 2020 9:13 PM
To: [hidden email] <[hidden email]>
Subject: Re: Hive_1_1 Processors and Controllers Missing in NiFi 1.11.4
 
Harsha,

The nifi-hive-services-api-nar is included with the NiFi assembly/release and includes the definition of what the Hive1_1ConnectionPool should be, but the actual component is in the nifi-hive1_1-nar, which is not included with the NiFi release (due to space concerns) and must be manually downloaded.

Regards,
Matt


On Tue, Jul 7, 2020 at 9:01 PM Sri Harsha Chavali <[hidden email]> wrote:
Hi Matt,

Restarting NiFi did the trick. I removed the old nifi-hive_1_1-nar-1.9.2.nar from the extensions directory and "restarted" NiFi and all the processors and controllers automatically picked the newer versions. Earlier I removed the old version and placed the new version of the nar in extensions folder but never restarted NiFi (assumed NiFi would dynamically drop the deleted nars like how it's dynamically loading) 😄

But one thing to note:
When I check the lib directory in the latest release I don't find the nifi-hive-1_1* at all. All we see is the below nars, which I assume are not 1_1 versions. So the 1_1 nars are completely missing in lib directory. We need to manually load them again. 



Thank you,
Harsha

Sent from Outlook


From: Matt Burgess <[hidden email]>
Sent: Tuesday, July 7, 2020 7:05 PM
To: [hidden email] <[hidden email]>
Subject: Re: Hive_1_1 Processors and Controllers Missing in NiFi 1.11.4
 
Harsha,

There are two NARs associated with Hive components, nifi-hive-services-api-nar which has the Hive1_1ConnectionPool service (actually an interface, but that's under the hood), and the nifi-hive1_1-nar which has the processors that declare themselves as users of that interface (and the actual implementations). Did you keep the old Hive 1.1 NAR in the lib/ folder when you upgraded? I would think if you removed the 1.9.2 NARs and used just the 1.11.4 NARs, NiFi would only find the newer ones and automatically upgrade the existing processors and controller services, but I haven't tried it myself. I suspect if they are both present then NiFi will try to match the version of the processors to the controller services and they won't match. Either you'd need both 1.9.2 NARs (the services-api one and the processors one) or neither, I recommend the latter so the components get upgraded to the latest version and that they match. If you're doing a hot reload by dropping the NARs into the dynamic loading location, you'll probably want to remove the old ones entirely, put the new ones in lib/, and restart NiFi.

Regards,
Matt



On Tue, Jul 7, 2020 at 6:27 PM Sri Harsha Chavali <[hidden email]> wrote:
Hi All,

We recently upgraded from nifi 1.9.2 to 1.11.4 and are facing an issue with Hive 1_1 processors. When we were in 1.9.2 we manually deployed the "nifi-hive_1_1-nar-1.9.2.nar" file (per suggestion in email chain https://lists.apache.org/list.html?users@...:lte=5y:SELECTHIVE_1_1_QL%20-%20NiFi%201.9.0%20Missing) in the extensions folder. Now after upgrading to 1.11.4 we notice that the nar file (nifi-hive_1_1-nar-1.11.4.nar) is missing in the tarball again so we had to manually add it again to the extensions folder. 

The issue is not with getting and adding the nar file to extensions as nifi automatically unpacks it and makes processors and controllers available. The issue is that we need to redo all the controllers and processors if we download the latest nifi-hive_1_1-nar-1.11.4.nar file. If we use the old nifi-hive_1_1-nar-1.9.2.nar we get the below error. The rest of the controllers/processors are automatically upgraded or are backward compatible. Could you please let me know if we need to redo all the processors or if there is a easy way to fix this issue by eliminating all the redo steps?





Thank you,
Harsha

Sent from Outlook