Nifi - how to achieve a concurrent development and CI/CD

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Nifi - how to achieve a concurrent development and CI/CD

Michal Slama (DHL IT Services), external

Hello,

 

may I ask you for recommendations for development and CI/CD in NiFi?

Pls let me describe our situation…I am a developer from DHL currently working on project including NiFi. It is part of our core and it is responsible for handling incoming data streams, data transormation and put it into various elastic search indexes (and queues).

 

Up to now development was quite straightforward as only one developer did it. But we have extended our team recently and now we face problems how to correctly maintain development for more developers working in parallel and then CI/CD of it as we have classical dev/test/uat/project env. structure.

 

Pls do you have any recommadation how to achive it? For now in general is enought.  Its good to mention that currently we work with NiFi version 1.8…tried to upgrade to 1.9. but some of components failed so the upgrade was postponed. But with new features in 1.10. and 1.11. we head to uprade to these versions.

 

Maybe if we can arrange a call it would be great!

 

With regards,

Michal Sláma

 



PRG Disclaimer This message is from DHL Information Services (Europe) s.r.o. and may contain confidential business information. It is intended solely for the use of the individual to whom it is addressed. If you are not the intended recipient please contact the sender and delete this message and any attachment from your system. Unauthorized publication, use, dissemination, forwarding, printing or copying of this email and its attachments is strictly prohibited.
Reply | Threaded
Open this post in threaded view
|

Re: Nifi - how to achieve a concurrent development and CI/CD

Jorge Machado
Hi, 

Managing xml is always hard I think. Last time I need to do something similar we used https://nifi.apache.org/registry.html
Works pretty well It was already 2 Years ago. Maybe now there is something better 

On 14. May 2020, at 15:57, Michal Slama (DHL IT Services), external <[hidden email]> wrote:

Hello,
 
may I ask you for recommendations for development and CI/CD in NiFi?
Pls let me describe our situation…I am a developer from DHL currently working on project including NiFi. It is part of our core and it is responsible for handling incoming data streams, data transormation and put it into various elastic search indexes (and queues).
 
Up to now development was quite straightforward as only one developer did it. But we have extended our team recently and now we face problems how to correctly maintain development for more developers working in parallel and then CI/CD of it as we have classical dev/test/uat/project env. structure.
 
Pls do you have any recommadation how to achive it? For now in general is enought.  Its good to mention that currently we work with NiFi version 1.8…tried to upgrade to 1.9. but some of components failed so the upgrade was postponed. But with new features in 1.10. and 1.11. we head to uprade to these versions.
 
Maybe if we can arrange a call it would be great!
 
With regards,
Michal Sláma
 


This message is from DHL Information Services (Europe) s.r.o. and may contain confidential business information. It is intended solely for the use of the individual to whom it is addressed. If you are not the intended recipient please contact the sender and delete this message and any attachment from your system. Unauthorized publication, use, dissemination, forwarding, printing or copying of this email and its attachments is strictly prohibited.

Reply | Threaded
Open this post in threaded view
|

Re: Nifi - how to achieve a concurrent development and CI/CD

Eric Secules
Hi Michal,

I'm also using a single registry for development and production. It doesn't help with collaborating on the same process group as there is way for it to reconcile merge conflicts. Instead, the registry will earn you that you're about to overwrite someone else's changes. Another pain of concurrent development is there's no concept of a PR and no visual diff of your local changes making review difficult. I've backed our registry up to git I've set up a CI pipeline in Azure Devops which runs our tests every time a new version of a process group is checked in. It's better than nothing but I'd rather nifi implemented git flow. Developing as a team on nifi and nifi registry is like your whole team developing and pushing directly to master.


-Eric

On Thu., May 14, 2020, 7:02 a.m. Jorge Machado, <[hidden email]> wrote:
Hi, 

Managing xml is always hard I think. Last time I need to do something similar we used https://nifi.apache.org/registry.html
Works pretty well It was already 2 Years ago. Maybe now there is something better 

On 14. May 2020, at 15:57, Michal Slama (DHL IT Services), external <[hidden email]> wrote:

Hello,
 
may I ask you for recommendations for development and CI/CD in NiFi?
Pls let me describe our situation…I am a developer from DHL currently working on project including NiFi. It is part of our core and it is responsible for handling incoming data streams, data transormation and put it into various elastic search indexes (and queues).
 
Up to now development was quite straightforward as only one developer did it. But we have extended our team recently and now we face problems how to correctly maintain development for more developers working in parallel and then CI/CD of it as we have classical dev/test/uat/project env. structure.
 
Pls do you have any recommadation how to achive it? For now in general is enought.  Its good to mention that currently we work with NiFi version 1.8…tried to upgrade to 1.9. but some of components failed so the upgrade was postponed. But with new features in 1.10. and 1.11. we head to uprade to these versions.
 
Maybe if we can arrange a call it would be great!
 
With regards,
Michal Sláma
 


This message is from DHL Information Services (Europe) s.r.o. and may contain confidential business information. It is intended solely for the use of the individual to whom it is addressed. If you are not the intended recipient please contact the sender and delete this message and any attachment from your system. Unauthorized publication, use, dissemination, forwarding, printing or copying of this email and its attachments is strictly prohibited.

Reply | Threaded
Open this post in threaded view
|

Re: Nifi - how to achieve a concurrent development and CI/CD

Ami Goldenberg
Hi Eric,
Would love to know, what kind of tests do you write/run for NiFi? We were just researching this topic.

On Thu, May 14, 2020 at 6:38 PM Eric Secules <[hidden email]> wrote:
Hi Michal,

I'm also using a single registry for development and production. It doesn't help with collaborating on the same process group as there is way for it to reconcile merge conflicts. Instead, the registry will earn you that you're about to overwrite someone else's changes. Another pain of concurrent development is there's no concept of a PR and no visual diff of your local changes making review difficult. I've backed our registry up to git I've set up a CI pipeline in Azure Devops which runs our tests every time a new version of a process group is checked in. It's better than nothing but I'd rather nifi implemented git flow. Developing as a team on nifi and nifi registry is like your whole team developing and pushing directly to master.


-Eric

On Thu., May 14, 2020, 7:02 a.m. Jorge Machado, <[hidden email]> wrote:
Hi, 

Managing xml is always hard I think. Last time I need to do something similar we used https://nifi.apache.org/registry.html
Works pretty well It was already 2 Years ago. Maybe now there is something better 

On 14. May 2020, at 15:57, Michal Slama (DHL IT Services), external <[hidden email]> wrote:

Hello,
 
may I ask you for recommendations for development and CI/CD in NiFi?
Pls let me describe our situation…I am a developer from DHL currently working on project including NiFi. It is part of our core and it is responsible for handling incoming data streams, data transormation and put it into various elastic search indexes (and queues).
 
Up to now development was quite straightforward as only one developer did it. But we have extended our team recently and now we face problems how to correctly maintain development for more developers working in parallel and then CI/CD of it as we have classical dev/test/uat/project env. structure.
 
Pls do you have any recommadation how to achive it? For now in general is enought.  Its good to mention that currently we work with NiFi version 1.8…tried to upgrade to 1.9. but some of components failed so the upgrade was postponed. But with new features in 1.10. and 1.11. we head to uprade to these versions.
 
Maybe if we can arrange a call it would be great!
 
With regards,
Michal Sláma
 


This message is from DHL Information Services (Europe) s.r.o. and may contain confidential business information. It is intended solely for the use of the individual to whom it is addressed. If you are not the intended recipient please contact the sender and delete this message and any attachment from your system. Unauthorized publication, use, dissemination, forwarding, printing or copying of this email and its attachments is strictly prohibited.

Reply | Threaded
Open this post in threaded view
|

Re: Nifi - how to achieve a concurrent development and CI/CD

Jorge Machado
I think we could improve Nifi by hooking up to GitHub somehow. So that we don’t need the registry… 

On 14. May 2020, at 18:49, Ami Goldenberg <[hidden email]> wrote:

Hi Eric,
Would love to know, what kind of tests do you write/run for NiFi? We were just researching this topic.

On Thu, May 14, 2020 at 6:38 PM Eric Secules <[hidden email]> wrote:
Hi Michal,

I'm also using a single registry for development and production. It doesn't help with collaborating on the same process group as there is way for it to reconcile merge conflicts. Instead, the registry will earn you that you're about to overwrite someone else's changes. Another pain of concurrent development is there's no concept of a PR and no visual diff of your local changes making review difficult. I've backed our registry up to git I've set up a CI pipeline in Azure Devops which runs our tests every time a new version of a process group is checked in. It's better than nothing but I'd rather nifi implemented git flow. Developing as a team on nifi and nifi registry is like your whole team developing and pushing directly to master.


-Eric

On Thu., May 14, 2020, 7:02 a.m. Jorge Machado, <[hidden email]> wrote:
Hi, 

Managing xml is always hard I think. Last time I need to do something similar we used https://nifi.apache.org/registry.html
Works pretty well It was already 2 Years ago. Maybe now there is something better 

On 14. May 2020, at 15:57, Michal Slama (DHL IT Services), external <[hidden email]> wrote:

Hello,
 
may I ask you for recommendations for development and CI/CD in NiFi?
Pls let me describe our situation…I am a developer from DHL currently working on project including NiFi. It is part of our core and it is responsible for handling incoming data streams, data transormation and put it into various elastic search indexes (and queues).
 
Up to now development was quite straightforward as only one developer did it. But we have extended our team recently and now we face problems how to correctly maintain development for more developers working in parallel and then CI/CD of it as we have classical dev/test/uat/project env. structure.
 
Pls do you have any recommadation how to achive it? For now in general is enought.  Its good to mention that currently we work with NiFi version 1.8…tried to upgrade to 1.9. but some of components failed so the upgrade was postponed. But with new features in 1.10. and 1.11. we head to uprade to these versions.
 
Maybe if we can arrange a call it would be great!
 
With regards,
Michal Sláma
 


This message is from DHL Information Services (Europe) s.r.o. and may contain confidential business information. It is intended solely for the use of the individual to whom it is addressed. If you are not the intended recipient please contact the sender and delete this message and any attachment from your system. Unauthorized publication, use, dissemination, forwarding, printing or copying of this email and its attachments is strictly prohibited.


Reply | Threaded
Open this post in threaded view
|

Re: Nifi - how to achieve a concurrent development and CI/CD

Eric Secules
Hi Ami,

I'm testing our NiFi application using a blackbox approach. We have a test harness that stages input files and waits for the corresponding outputs, then validates the content of the output. NiFi runs in a docker container (see https://hub.docker.com/r/apache/nifi/) and you can automate starting it up and orchestrating a test harness for your application in any competent CI/CD pipeline.

Jorge, I don't understand your comment. NiFi registry can be synced to a git repo as a backup mechanism, but you can't simply make branches, work and merge them back, because even the smallest change in a flow will create a massive diff in the Registry's backup git repo. It would be great if NiFi had a way of highlighting the differences between flow versions on the canvas and allow you to choose which elements to keep and which to discard when resolving a merge conflict.

-Eric

On Thu, May 14, 2020 at 12:00 PM Jorge Machado <[hidden email]> wrote:
I think we could improve Nifi by hooking up to GitHub somehow. So that we don’t need the registry… 

On 14. May 2020, at 18:49, Ami Goldenberg <[hidden email]> wrote:

Hi Eric,
Would love to know, what kind of tests do you write/run for NiFi? We were just researching this topic.

On Thu, May 14, 2020 at 6:38 PM Eric Secules <[hidden email]> wrote:
Hi Michal,

I'm also using a single registry for development and production. It doesn't help with collaborating on the same process group as there is way for it to reconcile merge conflicts. Instead, the registry will earn you that you're about to overwrite someone else's changes. Another pain of concurrent development is there's no concept of a PR and no visual diff of your local changes making review difficult. I've backed our registry up to git I've set up a CI pipeline in Azure Devops which runs our tests every time a new version of a process group is checked in. It's better than nothing but I'd rather nifi implemented git flow. Developing as a team on nifi and nifi registry is like your whole team developing and pushing directly to master.


-Eric

On Thu., May 14, 2020, 7:02 a.m. Jorge Machado, <[hidden email]> wrote:
Hi, 

Managing xml is always hard I think. Last time I need to do something similar we used https://nifi.apache.org/registry.html
Works pretty well It was already 2 Years ago. Maybe now there is something better 

On 14. May 2020, at 15:57, Michal Slama (DHL IT Services), external <[hidden email]> wrote:

Hello,
 
may I ask you for recommendations for development and CI/CD in NiFi?
Pls let me describe our situation…I am a developer from DHL currently working on project including NiFi. It is part of our core and it is responsible for handling incoming data streams, data transormation and put it into various elastic search indexes (and queues).
 
Up to now development was quite straightforward as only one developer did it. But we have extended our team recently and now we face problems how to correctly maintain development for more developers working in parallel and then CI/CD of it as we have classical dev/test/uat/project env. structure.
 
Pls do you have any recommadation how to achive it? For now in general is enought.  Its good to mention that currently we work with NiFi version 1.8…tried to upgrade to 1.9. but some of components failed so the upgrade was postponed. But with new features in 1.10. and 1.11. we head to uprade to these versions.
 
Maybe if we can arrange a call it would be great!
 
With regards,
Michal Sláma
 


This message is from DHL Information Services (Europe) s.r.o. and may contain confidential business information. It is intended solely for the use of the individual to whom it is addressed. If you are not the intended recipient please contact the sender and delete this message and any attachment from your system. Unauthorized publication, use, dissemination, forwarding, printing or copying of this email and its attachments is strictly prohibited.


Reply | Threaded
Open this post in threaded view
|

Re: Nifi - how to achieve a concurrent development and CI/CD

Edward Armes
Hi Eric,

So I looked into this September last year. I found that a variant of master driven development was the best course of action, for collaboration on a single Nifi flow.

As far as I can tell the git integration deliberately designed to be as simple a s possible, the down side of this is that git functionality like branching just doesn't work with the git integration.

I briefly looked into seeing if I could sidestep a few of the issues by changing how the flow version information was stored, but I ran into several road blocks around adding support for history re-writes.......


Edward


On Fri, 15 May 2020, 01:17 Eric Secules, <[hidden email]> wrote:
Hi Ami,

I'm testing our NiFi application using a blackbox approach. We have a test harness that stages input files and waits for the corresponding outputs, then validates the content of the output. NiFi runs in a docker container (see https://hub.docker.com/r/apache/nifi/) and you can automate starting it up and orchestrating a test harness for your application in any competent CI/CD pipeline.

Jorge, I don't understand your comment. NiFi registry can be synced to a git repo as a backup mechanism, but you can't simply make branches, work and merge them back, because even the smallest change in a flow will create a massive diff in the Registry's backup git repo. It would be great if NiFi had a way of highlighting the differences between flow versions on the canvas and allow you to choose which elements to keep and which to discard when resolving a merge conflict.

-Eric

On Thu, May 14, 2020 at 12:00 PM Jorge Machado <[hidden email]> wrote:
I think we could improve Nifi by hooking up to GitHub somehow. So that we don’t need the registry… 

On 14. May 2020, at 18:49, Ami Goldenberg <[hidden email]> wrote:

Hi Eric,
Would love to know, what kind of tests do you write/run for NiFi? We were just researching this topic.

On Thu, May 14, 2020 at 6:38 PM Eric Secules <[hidden email]> wrote:
Hi Michal,

I'm also using a single registry for development and production. It doesn't help with collaborating on the same process group as there is way for it to reconcile merge conflicts. Instead, the registry will earn you that you're about to overwrite someone else's changes. Another pain of concurrent development is there's no concept of a PR and no visual diff of your local changes making review difficult. I've backed our registry up to git I've set up a CI pipeline in Azure Devops which runs our tests every time a new version of a process group is checked in. It's better than nothing but I'd rather nifi implemented git flow. Developing as a team on nifi and nifi registry is like your whole team developing and pushing directly to master.


-Eric

On Thu., May 14, 2020, 7:02 a.m. Jorge Machado, <[hidden email]> wrote:
Hi, 

Managing xml is always hard I think. Last time I need to do something similar we used https://nifi.apache.org/registry.html
Works pretty well It was already 2 Years ago. Maybe now there is something better 

On 14. May 2020, at 15:57, Michal Slama (DHL IT Services), external <[hidden email]> wrote:

Hello,
 
may I ask you for recommendations for development and CI/CD in NiFi?
Pls let me describe our situation…I am a developer from DHL currently working on project including NiFi. It is part of our core and it is responsible for handling incoming data streams, data transormation and put it into various elastic search indexes (and queues).
 
Up to now development was quite straightforward as only one developer did it. But we have extended our team recently and now we face problems how to correctly maintain development for more developers working in parallel and then CI/CD of it as we have classical dev/test/uat/project env. structure.
 
Pls do you have any recommadation how to achive it? For now in general is enought.  Its good to mention that currently we work with NiFi version 1.8…tried to upgrade to 1.9. but some of components failed so the upgrade was postponed. But with new features in 1.10. and 1.11. we head to uprade to these versions.
 
Maybe if we can arrange a call it would be great!
 
With regards,
Michal Sláma
 


This message is from DHL Information Services (Europe) s.r.o. and may contain confidential business information. It is intended solely for the use of the individual to whom it is addressed. If you are not the intended recipient please contact the sender and delete this message and any attachment from your system. Unauthorized publication, use, dissemination, forwarding, printing or copying of this email and its attachments is strictly prohibited.