NiFi Swagger/OpenAPI Processor

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

NiFi Swagger/OpenAPI Processor

David Hoffer
I'm looking for a Swagger/OpenAPI v3 processor where I can use NiFi as an embedded REST server per the Swagger/OpenAPI schema/spec.

I realize that NiFi has the HandleHttpRequest and HandleHttpResponse but that appears to be either a global (accept everything) or specific for one endpoint/action.  I want a dynamic REST service that exposes all the endpoints/actions that are defined in the schema/spec and nothing else.

If this is not something that someone has done with NiFi already is there a way I could embed my own REST service inside of a generic NiFi process/etc?  E.g. can NiFi host some arbitrary code?

The reason I'd like to do this is because we already have NiFi flows for this same 'business functionality' but now I also need a new REST service for it that might not be connected to the existing flow but I'd like all the logic in one place...so NiFi to host everything.

Can someone point me in the right direction?

-Dave
Reply | Threaded
Open this post in threaded view
|

Re: NiFi Swagger/OpenAPI Processor

Otto Fowler
If nobody has done this you can look at:

https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html
https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide#ContributorGuide-providingCodeOrDocumentationContributionProvidingcodeordocumentationcontributions

On May 11, 2020 at 17:47:10, David Hoffer ([hidden email]) wrote:

I'm looking for a Swagger/OpenAPI v3 processor where I can use NiFi as an embedded REST server per the Swagger/OpenAPI schema/spec.

I realize that NiFi has the HandleHttpRequest and HandleHttpResponse but that appears to be either a global (accept everything) or specific for one endpoint/action.  I want a dynamic REST service that exposes all the endpoints/actions that are defined in the schema/spec and nothing else.

If this is not something that someone has done with NiFi already is there a way I could embed my own REST service inside of a generic NiFi process/etc?  E.g. can NiFi host some arbitrary code?

The reason I'd like to do this is because we already have NiFi flows for this same 'business functionality' but now I also need a new REST service for it that might not be connected to the existing flow but I'd like all the logic in one place...so NiFi to host everything.

Can someone point me in the right direction?

-Dave
Reply | Threaded
Open this post in threaded view
|

Re: NiFi Swagger/OpenAPI Processor

Ami Goldenberg
Maybe put an nginx in front that only passes through the requests you need? Or some other reverse proxy. 

I'm not affiliated with AWS but know their API gateway has this functionality 


On Tue, May 12, 2020, 04:20 Otto Fowler <[hidden email]> wrote:
If nobody has done this you can look at:

https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html
https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide#ContributorGuide-providingCodeOrDocumentationContributionProvidingcodeordocumentationcontributions

On May 11, 2020 at 17:47:10, David Hoffer ([hidden email]) wrote:

I'm looking for a Swagger/OpenAPI v3 processor where I can use NiFi as an embedded REST server per the Swagger/OpenAPI schema/spec.

I realize that NiFi has the HandleHttpRequest and HandleHttpResponse but that appears to be either a global (accept everything) or specific for one endpoint/action.  I want a dynamic REST service that exposes all the endpoints/actions that are defined in the schema/spec and nothing else.

If this is not something that someone has done with NiFi already is there a way I could embed my own REST service inside of a generic NiFi process/etc?  E.g. can NiFi host some arbitrary code?

The reason I'd like to do this is because we already have NiFi flows for this same 'business functionality' but now I also need a new REST service for it that might not be connected to the existing flow but I'd like all the logic in one place...so NiFi to host everything.

Can someone point me in the right direction?

-Dave
Reply | Threaded
Open this post in threaded view
|

Re: NiFi Swagger/OpenAPI Processor

Mark Payne
Dave,

You may also want to consider using RouteOnAttribute. The HandleHttpRequest processor adds attributes for just about everything in the request. You can easily use those to route the requests not matching the path/method you want to a HandleHttpResponse that responds with a 404 or 405. Now that said, you’d have to configure those methods/paths that are allowed in the RouteOnAttribute rather than providing a swagger spec, but it may be worth considering. 

Thanks
-Mark

Sent from my iPhone

On May 11, 2020, at 11:02 PM, Ami Goldenberg <[hidden email]> wrote:


Maybe put an nginx in front that only passes through the requests you need? Or some other reverse proxy. 

I'm not affiliated with AWS but know their API gateway has this functionality 


On Tue, May 12, 2020, 04:20 Otto Fowler <[hidden email]> wrote:
If nobody has done this you can look at:

https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html
<a href="https://cwiki.apache.org/confluence/display/NIFI/Contributor&#43;Guide#ContributorGuide-providingCodeOrDocumentationContributionProvidingcodeordocumentationcontributions" rel="noreferrer noreferrer" target="_blank">https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide#ContributorGuide-providingCodeOrDocumentationContributionProvidingcodeordocumentationcontributions

On May 11, 2020 at 17:47:10, David Hoffer ([hidden email]) wrote:

I'm looking for a Swagger/OpenAPI v3 processor where I can use NiFi as an embedded REST server per the Swagger/OpenAPI schema/spec.

I realize that NiFi has the HandleHttpRequest and HandleHttpResponse but that appears to be either a global (accept everything) or specific for one endpoint/action.  I want a dynamic REST service that exposes all the endpoints/actions that are defined in the schema/spec and nothing else.

If this is not something that someone has done with NiFi already is there a way I could embed my own REST service inside of a generic NiFi process/etc?  E.g. can NiFi host some arbitrary code?

The reason I'd like to do this is because we already have NiFi flows for this same 'business functionality' but now I also need a new REST service for it that might not be connected to the existing flow but I'd like all the logic in one place...so NiFi to host everything.

Can someone point me in the right direction?

-Dave
Reply | Threaded
Open this post in threaded view
|

Re: NiFi Swagger/OpenAPI Processor

David Hoffer
Thanks for the suggestions.  I'm trying to find a way to handle this in a dynamic way based on the OpenAPI spec.  Obviously I need to have a connected processor that can then generate the correct response for the request.  You mentioned that that the HandleHttpRequest  processor adds attributes for just about everything in the request.  Is this documented somewhere?  Perhaps I can use that info in my custom processor to handle correctly.  Ideally I want the HandleHttpRequest processor to add the OpenAPI metadata as attributes as well.  Perhaps I could extend the HandleHttpRequest to do that?

My OpenAPI spec might grow to dozens of operations/actions so I don't want to have to manually keep the NiFi flow up-to-date for each change.  I understand I will have to code the behavior for each operation/action.

So basically what I need is an OpenApiHandleHttpRequest processor that is fully OpenAPI aware where I can provide the behavior in one location.

Also I need NiFi to host the static Swagger UI as well.  This is just a folder of HTML/JS.  I'd want OpenApiHandleHttpRequest to handle that as well.

Thanks,
-Dave

On Mon, May 11, 2020 at 9:37 PM Mark Payne <[hidden email]> wrote:
Dave,

You may also want to consider using RouteOnAttribute. The HandleHttpRequest processor adds attributes for just about everything in the request. You can easily use those to route the requests not matching the path/method you want to a HandleHttpResponse that responds with a 404 or 405. Now that said, you’d have to configure those methods/paths that are allowed in the RouteOnAttribute rather than providing a swagger spec, but it may be worth considering. 

Thanks
-Mark

Sent from my iPhone

On May 11, 2020, at 11:02 PM, Ami Goldenberg <[hidden email]> wrote:


Maybe put an nginx in front that only passes through the requests you need? Or some other reverse proxy. 

I'm not affiliated with AWS but know their API gateway has this functionality 


On Tue, May 12, 2020, 04:20 Otto Fowler <[hidden email]> wrote:
If nobody has done this you can look at:

https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html
https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide#ContributorGuide-providingCodeOrDocumentationContributionProvidingcodeordocumentationcontributions

On May 11, 2020 at 17:47:10, David Hoffer ([hidden email]) wrote:

I'm looking for a Swagger/OpenAPI v3 processor where I can use NiFi as an embedded REST server per the Swagger/OpenAPI schema/spec.

I realize that NiFi has the HandleHttpRequest and HandleHttpResponse but that appears to be either a global (accept everything) or specific for one endpoint/action.  I want a dynamic REST service that exposes all the endpoints/actions that are defined in the schema/spec and nothing else.

If this is not something that someone has done with NiFi already is there a way I could embed my own REST service inside of a generic NiFi process/etc?  E.g. can NiFi host some arbitrary code?

The reason I'd like to do this is because we already have NiFi flows for this same 'business functionality' but now I also need a new REST service for it that might not be connected to the existing flow but I'd like all the logic in one place...so NiFi to host everything.

Can someone point me in the right direction?

-Dave
Reply | Threaded
Open this post in threaded view
|

Re: NiFi Swagger/OpenAPI Processor

Mark Payne
David,

Fair enough.

The attributes that it adds are documented. If you add the processor to your NiFi canvas and right-click on it and View Usage, you will get the processor’s documentation that indicates each of the attributes that are added along with an explanation of it. Or you could go to [1] for the docs on the latest version of that processor and see them there, under the “Writes Attributes” section.

I would caution you, though, that in general processors are not meant to be used as base classes that can be extended. You may have trouble trying to extend from that processor. It may make more sense to create your own or else create a processor that uses the attributes provided.

Hope this helps!
-Mark


[1] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.11.4/org.apache.nifi.processors.standard.HandleHttpRequest/index.html

On May 12, 2020, at 9:29 AM, David Hoffer <[hidden email]> wrote:

Thanks for the suggestions.  I'm trying to find a way to handle this in a dynamic way based on the OpenAPI spec.  Obviously I need to have a connected processor that can then generate the correct response for the request.  You mentioned that that the HandleHttpRequest  processor adds attributes for just about everything in the request.  Is this documented somewhere?  Perhaps I can use that info in my custom processor to handle correctly.  Ideally I want the HandleHttpRequest processor to add the OpenAPI metadata as attributes as well.  Perhaps I could extend the HandleHttpRequest to do that?

My OpenAPI spec might grow to dozens of operations/actions so I don't want to have to manually keep the NiFi flow up-to-date for each change.  I understand I will have to code the behavior for each operation/action.

So basically what I need is an OpenApiHandleHttpRequest processor that is fully OpenAPI aware where I can provide the behavior in one location.

Also I need NiFi to host the static Swagger UI as well.  This is just a folder of HTML/JS.  I'd want OpenApiHandleHttpRequest to handle that as well.

Thanks,
-Dave

On Mon, May 11, 2020 at 9:37 PM Mark Payne <[hidden email]> wrote:
Dave,

You may also want to consider using RouteOnAttribute. The HandleHttpRequest processor adds attributes for just about everything in the request. You can easily use those to route the requests not matching the path/method you want to a HandleHttpResponse that responds with a 404 or 405. Now that said, you’d have to configure those methods/paths that are allowed in the RouteOnAttribute rather than providing a swagger spec, but it may be worth considering. 

Thanks
-Mark

Sent from my iPhone

On May 11, 2020, at 11:02 PM, Ami Goldenberg <[hidden email]> wrote:


Maybe put an nginx in front that only passes through the requests you need? Or some other reverse proxy. 

I'm not affiliated with AWS but know their API gateway has this functionality 


On Tue, May 12, 2020, 04:20 Otto Fowler <[hidden email]> wrote:
If nobody has done this you can look at:

https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html
<a href="https://cwiki.apache.org/confluence/display/NIFI/Contributor&#43;Guide#ContributorGuide-providingCodeOrDocumentationContributionProvidingcodeordocumentationcontributions" rel="noreferrer noreferrer" target="_blank" class="">https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide#ContributorGuide-providingCodeOrDocumentationContributionProvidingcodeordocumentationcontributions

On May 11, 2020 at 17:47:10, David Hoffer ([hidden email]) wrote:

I'm looking for a Swagger/OpenAPI v3 processor where I can use NiFi as an embedded REST server per the Swagger/OpenAPI schema/spec.

I realize that NiFi has the HandleHttpRequest and HandleHttpResponse but that appears to be either a global (accept everything) or specific for one endpoint/action.  I want a dynamic REST service that exposes all the endpoints/actions that are defined in the schema/spec and nothing else.

If this is not something that someone has done with NiFi already is there a way I could embed my own REST service inside of a generic NiFi process/etc?  E.g. can NiFi host some arbitrary code?

The reason I'd like to do this is because we already have NiFi flows for this same 'business functionality' but now I also need a new REST service for it that might not be connected to the existing flow but I'd like all the logic in one place...so NiFi to host everything.

Can someone point me in the right direction?

-Dave