Session state in cluster HandleHttpRequest and HandleHttpReponse

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Session state in cluster HandleHttpRequest and HandleHttpReponse

Jeremy Pemberton-Pigott
Hi,

I have a cluster of 3 nodes and the incoming request on one node's HandleHttpRequest may be replied to by a different node's HandleHttpResponse, in between there is a Spark streaming job process.  Is there any example how to do that, maybe with DistributeMapCacheService?  So that I can still reply back to the client with the necessary data in the response.

Jeremy
Reply | Threaded
Open this post in threaded view
|

Re: Session state in cluster HandleHttpRequest and HandleHttpReponse

Peter Turcsanyi-2
Hi Jeremy,

I don't think you can accept the request in one node and send back the response from another node. 
There is an open HTTP connection between the client and the NiFi node while the HandleHttpRequest -> ... -> HandleHttpResponse flow is running.
Even if we passed the request/response context object between the NiFi nodes via a distributed cache, it would not be possible to send back the HTTP response to a client that originally connected and sent the request to another node. At least I don't know any solution to it in/outside NiFi.

Best,
Peter

On Tue, Jun 30, 2020 at 6:04 AM Jeremy Pemberton-Pigott <[hidden email]> wrote:
Hi,

I have a cluster of 3 nodes and the incoming request on one node's HandleHttpRequest may be replied to by a different node's HandleHttpResponse, in between there is a Spark streaming job process.  Is there any example how to do that, maybe with DistributeMapCacheService?  So that I can still reply back to the client with the necessary data in the response.

Jeremy
Reply | Threaded
Open this post in threaded view
|

Re: Session state in cluster HandleHttpRequest and HandleHttpReponse

Jeremy Pemberton-Pigott
Thanks Peter that makes sense. I'll try a wait/notify using an identifier for the node in the Spark messages being monitored so that the same node will receive the reply from Spark and respond to the client that initiated the connection. 

Regards,

Jeremy


On 30 Jun 2020, at 22:41, Peter Turcsanyi <[hidden email]> wrote:


Hi Jeremy,

I don't think you can accept the request in one node and send back the response from another node. 
There is an open HTTP connection between the client and the NiFi node while the HandleHttpRequest -> ... -> HandleHttpResponse flow is running.
Even if we passed the request/response context object between the NiFi nodes via a distributed cache, it would not be possible to send back the HTTP response to a client that originally connected and sent the request to another node. At least I don't know any solution to it in/outside NiFi.

Best,
Peter

On Tue, Jun 30, 2020 at 6:04 AM Jeremy Pemberton-Pigott <[hidden email]> wrote:
Hi,

I have a cluster of 3 nodes and the incoming request on one node's HandleHttpRequest may be replied to by a different node's HandleHttpResponse, in between there is a Spark streaming job process.  Is there any example how to do that, maybe with DistributeMapCacheService?  So that I can still reply back to the client with the necessary data in the response.

Jeremy