NiFi registry safe upgrade path from 0.1.0

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

NiFi registry safe upgrade path from 0.1.0

Rosso, Roland

Hi all,

We are upgrading NiFi from 1.9.2 to 1.11.4. Our 1.9.2 uses NiFi Registry 0.1.0 still and we have some catching up to do.

After reading the migration guidance for NiFi registry, I thought the prudent approach would be to upgrade to 0.2.0, then up from there.

I cannot find the 0.2.0 install binaries; I found the source code but would rather not compile it.

Could you tell me if there is a safe path from Registry 0.1.0 to 0.5.0 (or even 0.6.0)? We would likely keep H2 initially and secure it.

We’re on Cloudera CDH 6.2.

 

Thank you

Roland Rosso

 

This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law or may constitute as attorney work product. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, notify us immediately by telephone and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you.
Reply | Threaded
Open this post in threaded view
|

Re: NiFi registry safe upgrade path from 0.1.0

Bryan Bende
Hello,

Older releases can always be found in the apache archive:


Thanks,

Bryan

On Wed, Jul 8, 2020 at 2:14 PM Rosso, Roland <[hidden email]> wrote:

Hi all,

We are upgrading NiFi from 1.9.2 to 1.11.4. Our 1.9.2 uses NiFi Registry 0.1.0 still and we have some catching up to do.

After reading the migration guidance for NiFi registry, I thought the prudent approach would be to upgrade to 0.2.0, then up from there.

I cannot find the 0.2.0 install binaries; I found the source code but would rather not compile it.

Could you tell me if there is a safe path from Registry 0.1.0 to 0.5.0 (or even 0.6.0)? We would likely keep H2 initially and secure it.

We’re on Cloudera CDH 6.2.

 

Thank you

Roland Rosso

 

This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law or may constitute as attorney work product. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, notify us immediately by telephone and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you.
Reply | Threaded
Open this post in threaded view
|

RE: [EXTERNAL] Re: NiFi registry safe upgrade path from 0.1.0

Rosso, Roland

Thank you Bryan. Missed this link.

Would you have a recommended path to upgrade from 0.1.0 keeping H2?

Thank you

Roland

 

From: Bryan Bende <[hidden email]>
Sent: Wednesday, July 8, 2020 3:10 PM
To: [hidden email]
Subject: [EXTERNAL] Re: NiFi registry safe upgrade path from 0.1.0

 

Hello,

 

Older releases can always be found in the apache archive:

 

 

Thanks,

 

Bryan

 

On Wed, Jul 8, 2020 at 2:14 PM Rosso, Roland <[hidden email]> wrote:

Hi all,

We are upgrading NiFi from 1.9.2 to 1.11.4. Our 1.9.2 uses NiFi Registry 0.1.0 still and we have some catching up to do.

After reading the migration guidance for NiFi registry, I thought the prudent approach would be to upgrade to 0.2.0, then up from there.

I cannot find the 0.2.0 install binaries; I found the source code but would rather not compile it.

Could you tell me if there is a safe path from Registry 0.1.0 to 0.5.0 (or even 0.6.0)? We would likely keep H2 initially and secure it.

We’re on Cloudera CDH 6.2.

 

Thank you

Roland Rosso

 

This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law or may constitute as attorney work product. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, notify us immediately by telephone and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you.

This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law or may constitute as attorney work product. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, notify us immediately by telephone and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you.
Reply | Threaded
Open this post in threaded view
|

Re: [EXTERNAL] Re: NiFi registry safe upgrade path from 0.1.0

Bryan Bende
I'm pretty sure you can go directly from 0.1.0 to 0.6.0, there is no requirement to switch from H2 to Postgres, that was just an example in the migration for people that wanted it.

The change that was relevant was between 0.1.0 and 0.2.0, the database properties were refactored, and even if you are using H2, it will migrate from the old H2 to a new one, so just make sure you follow that guidance...

"The 0.2.0 release introduces new database configuration in nifi-registry.properties which provides more flexible options. When upgrading from 0.1.0, ensure that nifi.registry.db.directory points to the location of the existing H2 database to ensure it is migrated to the new database. Please consult the Database Properties section of the Admin Guide for further details of these properties."


On Wed, Jul 8, 2020 at 3:22 PM Rosso, Roland <[hidden email]> wrote:

Thank you Bryan. Missed this link.

Would you have a recommended path to upgrade from 0.1.0 keeping H2?

Thank you

Roland

 

From: Bryan Bende <[hidden email]>
Sent: Wednesday, July 8, 2020 3:10 PM
To: [hidden email]
Subject: [EXTERNAL] Re: NiFi registry safe upgrade path from 0.1.0

 

Hello,

 

Older releases can always be found in the apache archive:

 

 

Thanks,

 

Bryan

 

On Wed, Jul 8, 2020 at 2:14 PM Rosso, Roland <[hidden email]> wrote:

Hi all,

We are upgrading NiFi from 1.9.2 to 1.11.4. Our 1.9.2 uses NiFi Registry 0.1.0 still and we have some catching up to do.

After reading the migration guidance for NiFi registry, I thought the prudent approach would be to upgrade to 0.2.0, then up from there.

I cannot find the 0.2.0 install binaries; I found the source code but would rather not compile it.

Could you tell me if there is a safe path from Registry 0.1.0 to 0.5.0 (or even 0.6.0)? We would likely keep H2 initially and secure it.

We’re on Cloudera CDH 6.2.

 

Thank you

Roland Rosso

 

This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law or may constitute as attorney work product. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, notify us immediately by telephone and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you.

This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law or may constitute as attorney work product. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, notify us immediately by telephone and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you.
Reply | Threaded
Open this post in threaded view
|

RE: [EXTERNAL] Re: NiFi registry safe upgrade path from 0.1.0

Rosso, Roland

Hi Bryan,

I was able to go from 0.1.0 to 0.6.0. I think the database upgraded fine but the ‘new’ H2 database nifi-registry-primary.mv.db is only about 1/10 the size of the old one.

I also had to copy the flow_storage directory from 0.1.0 to 0.6.0 (did a test upgrade in parallel), but it all appears to work fine after I did that. DB size still has me concerned though.

I may have fumbled something in the steps and if you can point it out, that’d be great.

Thank you.

Roland

From: Bryan Bende <[hidden email]>
Sent: Wednesday, July 8, 2020 4:04 PM
To: [hidden email]
Subject: Re: [EXTERNAL] Re: NiFi registry safe upgrade path from 0.1.0

 

I'm pretty sure you can go directly from 0.1.0 to 0.6.0, there is no requirement to switch from H2 to Postgres, that was just an example in the migration for people that wanted it.

 

The change that was relevant was between 0.1.0 and 0.2.0, the database properties were refactored, and even if you are using H2, it will migrate from the old H2 to a new one, so just make sure you follow that guidance...

 

"The 0.2.0 release introduces new database configuration in nifi-registry.properties which provides more flexible options. When upgrading from 0.1.0, ensure that nifi.registry.db.directory points to the location of the existing H2 database to ensure it is migrated to the new database. Please consult the Database Properties section of the Admin Guide for further details of these properties."

 

 

On Wed, Jul 8, 2020 at 3:22 PM Rosso, Roland <[hidden email]> wrote:

Thank you Bryan. Missed this link.

Would you have a recommended path to upgrade from 0.1.0 keeping H2?

Thank you

Roland

 

From: Bryan Bende <[hidden email]>
Sent: Wednesday, July 8, 2020 3:10 PM
To: [hidden email]
Subject: [EXTERNAL] Re: NiFi registry safe upgrade path from 0.1.0

 

Hello,

 

Older releases can always be found in the apache archive:

 

 

Thanks,

 

Bryan

 

On Wed, Jul 8, 2020 at 2:14 PM Rosso, Roland <[hidden email]> wrote:

Hi all,

We are upgrading NiFi from 1.9.2 to 1.11.4. Our 1.9.2 uses NiFi Registry 0.1.0 still and we have some catching up to do.

After reading the migration guidance for NiFi registry, I thought the prudent approach would be to upgrade to 0.2.0, then up from there.

I cannot find the 0.2.0 install binaries; I found the source code but would rather not compile it.

Could you tell me if there is a safe path from Registry 0.1.0 to 0.5.0 (or even 0.6.0)? We would likely keep H2 initially and secure it.

We’re on Cloudera CDH 6.2.

 

Thank you

Roland Rosso

 

This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law or may constitute as attorney work product. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, notify us immediately by telephone and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you.

This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law or may constitute as attorney work product. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, notify us immediately by telephone and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you.

This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law or may constitute as attorney work product. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, notify us immediately by telephone and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you.
Reply | Threaded
Open this post in threaded view
|

Re: [EXTERNAL] Re: NiFi registry safe upgrade path from 0.1.0

Bryan Bende
Do you have all the same buckets and flows in the registry UI? If so then I would think it worked correctly.

You should have populated both sets of these properties to migrate the old H2 to the new H2...

# legacy database properties, used to migrate data from original DB to new DB below
# NOTE: Users upgrading from 0.1.0 should leave these populated, but new installs after 0.1.0 should leave these empty
nifi.registry.db.directory=
nifi.registry.db.url.append=

# database properties
nifi.registry.db.url=jdbc:h2:./database/nifi-registry-primary;AUTOCOMMIT=OFF;DB_CLOSE_ON_EXIT=FALSE;LOCK_MODE=3;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE
nifi.registry.db.driver.class=org.h2.Driver


On Fri, Jul 10, 2020 at 1:53 PM Rosso, Roland <[hidden email]> wrote:

Hi Bryan,

I was able to go from 0.1.0 to 0.6.0. I think the database upgraded fine but the ‘new’ H2 database nifi-registry-primary.mv.db is only about 1/10 the size of the old one.

I also had to copy the flow_storage directory from 0.1.0 to 0.6.0 (did a test upgrade in parallel), but it all appears to work fine after I did that. DB size still has me concerned though.

I may have fumbled something in the steps and if you can point it out, that’d be great.

Thank you.

Roland

From: Bryan Bende <[hidden email]>
Sent: Wednesday, July 8, 2020 4:04 PM
To: [hidden email]
Subject: Re: [EXTERNAL] Re: NiFi registry safe upgrade path from 0.1.0

 

I'm pretty sure you can go directly from 0.1.0 to 0.6.0, there is no requirement to switch from H2 to Postgres, that was just an example in the migration for people that wanted it.

 

The change that was relevant was between 0.1.0 and 0.2.0, the database properties were refactored, and even if you are using H2, it will migrate from the old H2 to a new one, so just make sure you follow that guidance...

 

"The 0.2.0 release introduces new database configuration in nifi-registry.properties which provides more flexible options. When upgrading from 0.1.0, ensure that nifi.registry.db.directory points to the location of the existing H2 database to ensure it is migrated to the new database. Please consult the Database Properties section of the Admin Guide for further details of these properties."

 

 

On Wed, Jul 8, 2020 at 3:22 PM Rosso, Roland <[hidden email]> wrote:

Thank you Bryan. Missed this link.

Would you have a recommended path to upgrade from 0.1.0 keeping H2?

Thank you

Roland

 

From: Bryan Bende <[hidden email]>
Sent: Wednesday, July 8, 2020 3:10 PM
To: [hidden email]
Subject: [EXTERNAL] Re: NiFi registry safe upgrade path from 0.1.0

 

Hello,

 

Older releases can always be found in the apache archive:

 

 

Thanks,

 

Bryan

 

On Wed, Jul 8, 2020 at 2:14 PM Rosso, Roland <[hidden email]> wrote:

Hi all,

We are upgrading NiFi from 1.9.2 to 1.11.4. Our 1.9.2 uses NiFi Registry 0.1.0 still and we have some catching up to do.

After reading the migration guidance for NiFi registry, I thought the prudent approach would be to upgrade to 0.2.0, then up from there.

I cannot find the 0.2.0 install binaries; I found the source code but would rather not compile it.

Could you tell me if there is a safe path from Registry 0.1.0 to 0.5.0 (or even 0.6.0)? We would likely keep H2 initially and secure it.

We’re on Cloudera CDH 6.2.

 

Thank you

Roland Rosso

 

This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law or may constitute as attorney work product. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, notify us immediately by telephone and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you.

This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law or may constitute as attorney work product. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, notify us immediately by telephone and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you.

This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law or may constitute as attorney work product. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, notify us immediately by telephone and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you.
Reply | Threaded
Open this post in threaded view
|

RE: [EXTERNAL] Re: NiFi registry safe upgrade path from 0.1.0

Rosso, Roland

Yes sir,

Same buckets and flows are available. From the start on 0.6.0, I was able to see the buckets in Registry and via NiFi à Import. However, I did have to copy over the flow_storage directory to be able to import a flow (otherwise it would show the process group id and say unavailable, something to that effect.

I followed the directions as you have below, point to old, checked it created the new nifi-registry-primary.mv.db. Since then, I’ve done a couple of Registry stop/start, and removed the ‘old’ db entries from nifi-registry.properties, since we’re on 0.6.0 now.

All seems ok, but the old db shows as 2.4M, the new is merely 152K.

 

From: Bryan Bende <[hidden email]>
Sent: Friday, July 10, 2020 2:35 PM
To: [hidden email]
Subject: Re: [EXTERNAL] Re: NiFi registry safe upgrade path from 0.1.0

 

Do you have all the same buckets and flows in the registry UI? If so then I would think it worked correctly.

 

You should have populated both sets of these properties to migrate the old H2 to the new H2...

 

# legacy database properties, used to migrate data from original DB to new DB below
# NOTE: Users upgrading from 0.1.0 should leave these populated, but new installs after 0.1.0 should leave these empty
nifi.registry.db.directory=
nifi.registry.db.url.append=

# database properties
nifi.registry.db.url=jdbc:h2:./database/nifi-registry-primary;AUTOCOMMIT=OFF;DB_CLOSE_ON_EXIT=FALSE;LOCK_MODE=3;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE
nifi.registry.db.driver.class=org.h2.Driver

 

 

On Fri, Jul 10, 2020 at 1:53 PM Rosso, Roland <[hidden email]> wrote:

Hi Bryan,

I was able to go from 0.1.0 to 0.6.0. I think the database upgraded fine but the ‘new’ H2 database nifi-registry-primary.mv.db is only about 1/10 the size of the old one.

I also had to copy the flow_storage directory from 0.1.0 to 0.6.0 (did a test upgrade in parallel), but it all appears to work fine after I did that. DB size still has me concerned though.

I may have fumbled something in the steps and if you can point it out, that’d be great.

Thank you.

Roland

From: Bryan Bende <[hidden email]>
Sent: Wednesday, July 8, 2020 4:04 PM
To: [hidden email]
Subject: Re: [EXTERNAL] Re: NiFi registry safe upgrade path from 0.1.0

 

I'm pretty sure you can go directly from 0.1.0 to 0.6.0, there is no requirement to switch from H2 to Postgres, that was just an example in the migration for people that wanted it.

 

The change that was relevant was between 0.1.0 and 0.2.0, the database properties were refactored, and even if you are using H2, it will migrate from the old H2 to a new one, so just make sure you follow that guidance...

 

"The 0.2.0 release introduces new database configuration in nifi-registry.properties which provides more flexible options. When upgrading from 0.1.0, ensure that nifi.registry.db.directory points to the location of the existing H2 database to ensure it is migrated to the new database. Please consult the Database Properties section of the Admin Guide for further details of these properties."

 

 

On Wed, Jul 8, 2020 at 3:22 PM Rosso, Roland <[hidden email]> wrote:

Thank you Bryan. Missed this link.

Would you have a recommended path to upgrade from 0.1.0 keeping H2?

Thank you

Roland

 

From: Bryan Bende <[hidden email]>
Sent: Wednesday, July 8, 2020 3:10 PM
To: [hidden email]
Subject: [EXTERNAL] Re: NiFi registry safe upgrade path from 0.1.0

 

Hello,

 

Older releases can always be found in the apache archive:

 

 

Thanks,

 

Bryan

 

On Wed, Jul 8, 2020 at 2:14 PM Rosso, Roland <[hidden email]> wrote:

Hi all,

We are upgrading NiFi from 1.9.2 to 1.11.4. Our 1.9.2 uses NiFi Registry 0.1.0 still and we have some catching up to do.

After reading the migration guidance for NiFi registry, I thought the prudent approach would be to upgrade to 0.2.0, then up from there.

I cannot find the 0.2.0 install binaries; I found the source code but would rather not compile it.

Could you tell me if there is a safe path from Registry 0.1.0 to 0.5.0 (or even 0.6.0)? We would likely keep H2 initially and secure it.

We’re on Cloudera CDH 6.2.

 

Thank you

Roland Rosso

 

This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law or may constitute as attorney work product. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, notify us immediately by telephone and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you.

This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law or may constitute as attorney work product. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, notify us immediately by telephone and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you.

This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law or may constitute as attorney work product. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, notify us immediately by telephone and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you.

This message (including any attachments) is intended only for the use of the individual or entity to which it is addressed and may contain information that is non-public, proprietary, privileged, confidential, and exempt from disclosure under applicable law or may constitute as attorney work product. If you are not the intended recipient, you are hereby notified that any use, dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, notify us immediately by telephone and (i) destroy this message if a facsimile or (ii) delete this message immediately if this is an electronic communication. Thank you.