OIDC Redirect loop

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

OIDC Redirect loop

Ami Goldenberg
Hi,

We are trying to deploy NiFi on kubernetes after successfully using it for a while.
The issue we are having is that every time we enter our nifi URL it will redirect us to Google and once we sign in we just get redirected again.

The error I see on users.log is:
o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<JWT token>) GET https://XXX.XXX.XXXX/nifi-api/flow/current-user (source ip: 172.32.34.99) 
2020-04-25T19:48:06.256605759Z 2020-04-25 19:48:05,983 ERROR [NiFi Web Server-16] o.a.nifi.web.security.jwt.JwtService There was an error validating the JWT
2020-04-25T19:48:06.256610178Z 2020-04-25 19:48:05,983 ERROR [NiFi Web Server-16] o.a.nifi.web.security.jwt.JwtService Unable to validate the access token.
2020-04-25T19:48:06.256613727Z Caused by: JWT signature does not match locally computed signature. JWT validity cannot be asserted and should not be trusted.
2020-04-25T19:48:06.256617124Z 2020-04-25 19:48:05,984 WARN [NiFi Web Server-16] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api:Unable to validate the access token.

We're trying to follow practices from blogs and pvillard's repo:
Our set up is as such:
  • OIDC provider is Google
  • TLS-toolkit running in server mode inside k8s
  • StatefulSet of 3 replicas
  • Zookeeper in K8s
  • Ingress that is set up to create a load balancer in AWS - with sticky sessions (based on cookie)
  • Service that is set up with sessionAffinity: ClientIP

Any idea which direction I should be checking next?anks!
Reply | Threaded
Open this post in threaded view
|

Re: OIDC Redirect loop

Edward Armes
Hi Ami,

Biased on the error you've got in the user log it looks like you've got a local trust issue. If you could tell us what you've already tried, someone might be able to help you a bit more.

Edward

On 27/04/2020 05:36, Ami Goldenberg wrote:
Hi,

We are trying to deploy NiFi on kubernetes after successfully using it for a while.
The issue we are having is that every time we enter our nifi URL it will redirect us to Google and once we sign in we just get redirected again.

The error I see on users.log is:
o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<JWT token>) GET https://XXX.XXX.XXXX/nifi-api/flow/current-user (source ip: 172.32.34.99) 
2020-04-25T19:48:06.256605759Z 2020-04-25 19:48:05,983 ERROR [NiFi Web Server-16] o.a.nifi.web.security.jwt.JwtService There was an error validating the JWT
2020-04-25T19:48:06.256610178Z 2020-04-25 19:48:05,983 ERROR [NiFi Web Server-16] o.a.nifi.web.security.jwt.JwtService Unable to validate the access token.
2020-04-25T19:48:06.256613727Z Caused by: JWT signature does not match locally computed signature. JWT validity cannot be asserted and should not be trusted.
2020-04-25T19:48:06.256617124Z 2020-04-25 19:48:05,984 WARN [NiFi Web Server-16] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api:Unable to validate the access token.

We're trying to follow practices from blogs and pvillard's repo:
Our set up is as such:
  • OIDC provider is Google
  • TLS-toolkit running in server mode inside k8s
  • StatefulSet of 3 replicas
  • Zookeeper in K8s
  • Ingress that is set up to create a load balancer in AWS - with sticky sessions (based on cookie)
  • Service that is set up with sessionAffinity: ClientIP

Any idea which direction I should be checking next?anks!


Reply | Threaded
Open this post in threaded view
|

Re: OIDC Redirect loop

Nathan Gough
In reply to this post by Ami Goldenberg
Hi Ami,

Just to confirm, the OAuth Client ID redirect URL in OIDC is set to "https://${nifi.hostname}:${nifi.port}/nifi-api/access/oidc/callback" and the NiFi property is set "nifi.security.user.oidc.discovery.url=https://accounts.google.com/.well-known/openid-configuration".

Nathan

On Mon, Apr 27, 2020 at 12:37 AM Ami Goldenberg <[hidden email]> wrote:
Hi,

We are trying to deploy NiFi on kubernetes after successfully using it for a while.
The issue we are having is that every time we enter our nifi URL it will redirect us to Google and once we sign in we just get redirected again.

The error I see on users.log is:
o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<JWT token>) GET https://XXX.XXX.XXXX/nifi-api/flow/current-user (source ip: 172.32.34.99) 
2020-04-25T19:48:06.256605759Z 2020-04-25 19:48:05,983 ERROR [NiFi Web Server-16] o.a.nifi.web.security.jwt.JwtService There was an error validating the JWT
2020-04-25T19:48:06.256610178Z 2020-04-25 19:48:05,983 ERROR [NiFi Web Server-16] o.a.nifi.web.security.jwt.JwtService Unable to validate the access token.
2020-04-25T19:48:06.256613727Z Caused by: JWT signature does not match locally computed signature. JWT validity cannot be asserted and should not be trusted.
2020-04-25T19:48:06.256617124Z 2020-04-25 19:48:05,984 WARN [NiFi Web Server-16] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api:Unable to validate the access token.

We're trying to follow practices from blogs and pvillard's repo:
Our set up is as such:
  • OIDC provider is Google
  • TLS-toolkit running in server mode inside k8s
  • StatefulSet of 3 replicas
  • Zookeeper in K8s
  • Ingress that is set up to create a load balancer in AWS - with sticky sessions (based on cookie)
  • Service that is set up with sessionAffinity: ClientIP

Any idea which direction I should be checking next?anks!
Reply | Threaded
Open this post in threaded view
|

Re: OIDC Redirect loop

Ami Goldenberg
Hi Nathan,
Indeed, that's the case

On Mon, Apr 27, 2020 at 5:57 PM Nathan Gough <[hidden email]> wrote:
Hi Ami,

Just to confirm, the OAuth Client ID redirect URL in OIDC is set to "https://${nifi.hostname}:${nifi.port}/nifi-api/access/oidc/callback" and the NiFi property is set "nifi.security.user.oidc.discovery.url=https://accounts.google.com/.well-known/openid-configuration".

Nathan

On Mon, Apr 27, 2020 at 12:37 AM Ami Goldenberg <[hidden email]> wrote:
Hi,

We are trying to deploy NiFi on kubernetes after successfully using it for a while.
The issue we are having is that every time we enter our nifi URL it will redirect us to Google and once we sign in we just get redirected again.

The error I see on users.log is:
o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<JWT token>) GET https://XXX.XXX.XXXX/nifi-api/flow/current-user (source ip: 172.32.34.99) 
2020-04-25T19:48:06.256605759Z 2020-04-25 19:48:05,983 ERROR [NiFi Web Server-16] o.a.nifi.web.security.jwt.JwtService There was an error validating the JWT
2020-04-25T19:48:06.256610178Z 2020-04-25 19:48:05,983 ERROR [NiFi Web Server-16] o.a.nifi.web.security.jwt.JwtService Unable to validate the access token.
2020-04-25T19:48:06.256613727Z Caused by: JWT signature does not match locally computed signature. JWT validity cannot be asserted and should not be trusted.
2020-04-25T19:48:06.256617124Z 2020-04-25 19:48:05,984 WARN [NiFi Web Server-16] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api:Unable to validate the access token.

We're trying to follow practices from blogs and pvillard's repo:
Our set up is as such:
  • OIDC provider is Google
  • TLS-toolkit running in server mode inside k8s
  • StatefulSet of 3 replicas
  • Zookeeper in K8s
  • Ingress that is set up to create a load balancer in AWS - with sticky sessions (based on cookie)
  • Service that is set up with sessionAffinity: ClientIP

Any idea which direction I should be checking next?anks!
Reply | Threaded
Open this post in threaded view
|

Re: OIDC Redirect loop

Andy LoPresto
Can you verify the initial redirect to OIDC and the callback are going to the same node in NiFi? I see your LB configs are set to sticky sessions, but it may be that if the callback is originating from the OIDC IDP server rather than the actual client IP, the session affinity is not being applied. Regardless, the error appears to indicate that the JWT provided in the request to NiFi isn’t able to be validated, which indicates that the key used to sign it isn’t present on that node, which is likely due to the request being sent to a node other than the one that signed it. 

Quick and easy way to validate this would be to change the stateful set # to 1 node and attempt the same sequence of operations. 


Andy LoPresto
[hidden email]
[hidden email]
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Apr 27, 2020, at 8:12 AM, Ami Goldenberg <[hidden email]> wrote:

Hi Nathan,
Indeed, that's the case

On Mon, Apr 27, 2020 at 5:57 PM Nathan Gough <[hidden email]> wrote:
Hi Ami,

Just to confirm, the OAuth Client ID redirect URL in OIDC is set to "<a href="https://${nifi.hostname}:${nifi.port}/nifi-api/access/oidc/callback" class="">https://${nifi.hostname}:${nifi.port}/nifi-api/access/oidc/callback" and the NiFi property is set "nifi.security.user.oidc.discovery.url=https://accounts.google.com/.well-known/openid-configuration".

Nathan

On Mon, Apr 27, 2020 at 12:37 AM Ami Goldenberg <[hidden email]> wrote:
Hi,

We are trying to deploy NiFi on kubernetes after successfully using it for a while.
The issue we are having is that every time we enter our nifi URL it will redirect us to Google and once we sign in we just get redirected again.

The error I see on users.log is:
o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<JWT token>) GET https://XXX.XXX.XXXX/nifi-api/flow/current-user (source ip: 172.32.34.99) 
2020-04-25T19:48:06.256605759Z 2020-04-25 19:48:05,983 ERROR [NiFi Web Server-16] o.a.nifi.web.security.jwt.JwtService There was an error validating the JWT
2020-04-25T19:48:06.256610178Z 2020-04-25 19:48:05,983 ERROR [NiFi Web Server-16] o.a.nifi.web.security.jwt.JwtService Unable to validate the access token.
2020-04-25T19:48:06.256613727Z Caused by: JWT signature does not match locally computed signature. JWT validity cannot be asserted and should not be trusted.
2020-04-25T19:48:06.256617124Z 2020-04-25 19:48:05,984 WARN [NiFi Web Server-16] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api:Unable to validate the access token.

We're trying to follow practices from blogs and pvillard's repo:
Our set up is as such:
  • OIDC provider is Google
  • TLS-toolkit running in server mode inside k8s
  • StatefulSet of 3 replicas
  • Zookeeper in K8s
  • Ingress that is set up to create a load balancer in AWS - with sticky sessions (based on cookie)
  • Service that is set up with sessionAffinity: ClientIP

Any idea which direction I should be checking next?anks!

Reply | Threaded
Open this post in threaded view
|

Re: OIDC Redirect loop

Wyllys Ingersoll
I have a very similar configuration and similar problem.  After authenticating with the OIDC server (Keycloak), I often get multiple failures in verifying the JWT from the nifi servers and have to reload the browser multiple times until it eventually hits the right one.  

On Mon, Apr 27, 2020 at 2:25 PM Andy LoPresto <[hidden email]> wrote:
Can you verify the initial redirect to OIDC and the callback are going to the same node in NiFi? I see your LB configs are set to sticky sessions, but it may be that if the callback is originating from the OIDC IDP server rather than the actual client IP, the session affinity is not being applied. Regardless, the error appears to indicate that the JWT provided in the request to NiFi isn’t able to be validated, which indicates that the key used to sign it isn’t present on that node, which is likely due to the request being sent to a node other than the one that signed it. 

Quick and easy way to validate this would be to change the stateful set # to 1 node and attempt the same sequence of operations. 


Andy LoPresto
[hidden email]
[hidden email]
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Apr 27, 2020, at 8:12 AM, Ami Goldenberg <[hidden email]> wrote:

Hi Nathan,
Indeed, that's the case

On Mon, Apr 27, 2020 at 5:57 PM Nathan Gough <[hidden email]> wrote:
Hi Ami,

Just to confirm, the OAuth Client ID redirect URL in OIDC is set to "<a href="https://$%7Bnifi.hostname%7D:$%7Bnifi.port%7D/nifi-api/access/oidc/callback" target="_blank">https://${nifi.hostname}:${nifi.port}/nifi-api/access/oidc/callback" and the NiFi property is set "nifi.security.user.oidc.discovery.url=https://accounts.google.com/.well-known/openid-configuration".

Nathan

On Mon, Apr 27, 2020 at 12:37 AM Ami Goldenberg <[hidden email]> wrote:
Hi,

We are trying to deploy NiFi on kubernetes after successfully using it for a while.
The issue we are having is that every time we enter our nifi URL it will redirect us to Google and once we sign in we just get redirected again.

The error I see on users.log is:
o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<JWT token>) GET https://XXX.XXX.XXXX/nifi-api/flow/current-user (source ip: 172.32.34.99) 
2020-04-25T19:48:06.256605759Z 2020-04-25 19:48:05,983 ERROR [NiFi Web Server-16] o.a.nifi.web.security.jwt.JwtService There was an error validating the JWT
2020-04-25T19:48:06.256610178Z 2020-04-25 19:48:05,983 ERROR [NiFi Web Server-16] o.a.nifi.web.security.jwt.JwtService Unable to validate the access token.
2020-04-25T19:48:06.256613727Z Caused by: JWT signature does not match locally computed signature. JWT validity cannot be asserted and should not be trusted.
2020-04-25T19:48:06.256617124Z 2020-04-25 19:48:05,984 WARN [NiFi Web Server-16] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api:Unable to validate the access token.

We're trying to follow practices from blogs and pvillard's repo:
Our set up is as such:
  • OIDC provider is Google
  • TLS-toolkit running in server mode inside k8s
  • StatefulSet of 3 replicas
  • Zookeeper in K8s
  • Ingress that is set up to create a load balancer in AWS - with sticky sessions (based on cookie)
  • Service that is set up with sessionAffinity: ClientIP

Any idea which direction I should be checking next?anks!

Reply | Threaded
Open this post in threaded view
|

Re: OIDC Redirect loop

Ami Goldenberg
In reply to this post by Andy LoPresto
Great idea Andy

I reduced the scale to 1 and it is still doing the same redirect loop.
I guess the load balancer is hitting a different node even if sticky is set up? Even if eventually the service does clientAffinity maybe the client IP is not taken correctly?

What are your thoughts?

On Mon, Apr 27, 2020 at 9:25 PM Andy LoPresto <[hidden email]> wrote:
Can you verify the initial redirect to OIDC and the callback are going to the same node in NiFi? I see your LB configs are set to sticky sessions, but it may be that if the callback is originating from the OIDC IDP server rather than the actual client IP, the session affinity is not being applied. Regardless, the error appears to indicate that the JWT provided in the request to NiFi isn’t able to be validated, which indicates that the key used to sign it isn’t present on that node, which is likely due to the request being sent to a node other than the one that signed it. 

Quick and easy way to validate this would be to change the stateful set # to 1 node and attempt the same sequence of operations. 


Andy LoPresto
[hidden email]
[hidden email]
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Apr 27, 2020, at 8:12 AM, Ami Goldenberg <[hidden email]> wrote:

Hi Nathan,
Indeed, that's the case

On Mon, Apr 27, 2020 at 5:57 PM Nathan Gough <[hidden email]> wrote:
Hi Ami,

Just to confirm, the OAuth Client ID redirect URL in OIDC is set to "<a href="https://$%7Bnifi.hostname%7D:$%7Bnifi.port%7D/nifi-api/access/oidc/callback" target="_blank">https://${nifi.hostname}:${nifi.port}/nifi-api/access/oidc/callback" and the NiFi property is set "nifi.security.user.oidc.discovery.url=https://accounts.google.com/.well-known/openid-configuration".

Nathan

On Mon, Apr 27, 2020 at 12:37 AM Ami Goldenberg <[hidden email]> wrote:
Hi,

We are trying to deploy NiFi on kubernetes after successfully using it for a while.
The issue we are having is that every time we enter our nifi URL it will redirect us to Google and once we sign in we just get redirected again.

The error I see on users.log is:
o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<JWT token>) GET https://XXX.XXX.XXXX/nifi-api/flow/current-user (source ip: 172.32.34.99) 
2020-04-25T19:48:06.256605759Z 2020-04-25 19:48:05,983 ERROR [NiFi Web Server-16] o.a.nifi.web.security.jwt.JwtService There was an error validating the JWT
2020-04-25T19:48:06.256610178Z 2020-04-25 19:48:05,983 ERROR [NiFi Web Server-16] o.a.nifi.web.security.jwt.JwtService Unable to validate the access token.
2020-04-25T19:48:06.256613727Z Caused by: JWT signature does not match locally computed signature. JWT validity cannot be asserted and should not be trusted.
2020-04-25T19:48:06.256617124Z 2020-04-25 19:48:05,984 WARN [NiFi Web Server-16] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api:Unable to validate the access token.

We're trying to follow practices from blogs and pvillard's repo:
Our set up is as such:
  • OIDC provider is Google
  • TLS-toolkit running in server mode inside k8s
  • StatefulSet of 3 replicas
  • Zookeeper in K8s
  • Ingress that is set up to create a load balancer in AWS - with sticky sessions (based on cookie)
  • Service that is set up with sessionAffinity: ClientIP

Any idea which direction I should be checking next?anks!

Reply | Threaded
Open this post in threaded view
|

Re: OIDC Redirect loop

Ami Goldenberg
Actually with a set of 1 this would not have mattered. I think the problem is not with the sticky sessions...

On Mon, Apr 27, 2020 at 9:43 PM Ami Goldenberg <[hidden email]> wrote:
Great idea Andy

I reduced the scale to 1 and it is still doing the same redirect loop.
I guess the load balancer is hitting a different node even if sticky is set up? Even if eventually the service does clientAffinity maybe the client IP is not taken correctly?

What are your thoughts?

On Mon, Apr 27, 2020 at 9:25 PM Andy LoPresto <[hidden email]> wrote:
Can you verify the initial redirect to OIDC and the callback are going to the same node in NiFi? I see your LB configs are set to sticky sessions, but it may be that if the callback is originating from the OIDC IDP server rather than the actual client IP, the session affinity is not being applied. Regardless, the error appears to indicate that the JWT provided in the request to NiFi isn’t able to be validated, which indicates that the key used to sign it isn’t present on that node, which is likely due to the request being sent to a node other than the one that signed it. 

Quick and easy way to validate this would be to change the stateful set # to 1 node and attempt the same sequence of operations. 


Andy LoPresto
[hidden email]
[hidden email]
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Apr 27, 2020, at 8:12 AM, Ami Goldenberg <[hidden email]> wrote:

Hi Nathan,
Indeed, that's the case

On Mon, Apr 27, 2020 at 5:57 PM Nathan Gough <[hidden email]> wrote:
Hi Ami,

Just to confirm, the OAuth Client ID redirect URL in OIDC is set to "<a href="https://$%7Bnifi.hostname%7D:$%7Bnifi.port%7D/nifi-api/access/oidc/callback" target="_blank">https://${nifi.hostname}:${nifi.port}/nifi-api/access/oidc/callback" and the NiFi property is set "nifi.security.user.oidc.discovery.url=https://accounts.google.com/.well-known/openid-configuration".

Nathan

On Mon, Apr 27, 2020 at 12:37 AM Ami Goldenberg <[hidden email]> wrote:
Hi,

We are trying to deploy NiFi on kubernetes after successfully using it for a while.
The issue we are having is that every time we enter our nifi URL it will redirect us to Google and once we sign in we just get redirected again.

The error I see on users.log is:
o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<JWT token>) GET https://XXX.XXX.XXXX/nifi-api/flow/current-user (source ip: 172.32.34.99) 
2020-04-25T19:48:06.256605759Z 2020-04-25 19:48:05,983 ERROR [NiFi Web Server-16] o.a.nifi.web.security.jwt.JwtService There was an error validating the JWT
2020-04-25T19:48:06.256610178Z 2020-04-25 19:48:05,983 ERROR [NiFi Web Server-16] o.a.nifi.web.security.jwt.JwtService Unable to validate the access token.
2020-04-25T19:48:06.256613727Z Caused by: JWT signature does not match locally computed signature. JWT validity cannot be asserted and should not be trusted.
2020-04-25T19:48:06.256617124Z 2020-04-25 19:48:05,984 WARN [NiFi Web Server-16] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api:Unable to validate the access token.

We're trying to follow practices from blogs and pvillard's repo:
Our set up is as such:
  • OIDC provider is Google
  • TLS-toolkit running in server mode inside k8s
  • StatefulSet of 3 replicas
  • Zookeeper in K8s
  • Ingress that is set up to create a load balancer in AWS - with sticky sessions (based on cookie)
  • Service that is set up with sessionAffinity: ClientIP

Any idea which direction I should be checking next?anks!

Reply | Threaded
Open this post in threaded view
|

Re: OIDC Redirect loop

Ami Goldenberg
Ok I have an update
I tried running a cluster without Kubernetes, on AWS and following the terraform configuration by pvillard here https://github.com/pvillard31/nifi-gcp-terraform/tree/master/gcp-cluster-secured-nifi-oidc
Got a tls-toolkit CA server, zookeeper server, 2 nodes and an AWS ALB with sticky sessions

With 1 and 2 nodes everything works fine
If I recreate the cluster with another node however, then it all breaks and I'm back to the redirect loop

Any idea what I'm missing here?
Reply | Threaded
Open this post in threaded view
|

Re: OIDC Redirect loop

Troy Melhase
Ami, can you post any of your logs/configuration?  I've been working
thru some of the OIDC related issues and might be able to provide
insight.

On Mon, May 4, 2020 at 7:57 AM Ami Goldenberg <[hidden email]> wrote:
>
> Ok I have an update
> I tried running a cluster without Kubernetes, on AWS and following the terraform configuration by pvillard here https://github.com/pvillard31/nifi-gcp-terraform/tree/master/gcp-cluster-secured-nifi-oidc
> Got a tls-toolkit CA server, zookeeper server, 2 nodes and an AWS ALB with sticky sessions
>
> With 1 and 2 nodes everything works fine
> If I recreate the cluster with another node however, then it all breaks and I'm back to the redirect loop
>
> Any idea what I'm missing here?
Reply | Threaded
Open this post in threaded view
|

Re: OIDC Redirect loop

Ami Goldenberg
Phew,
Eventually found out that my node hostnames and the node identities in authorizers.xml had a tiny mismatch
After making them identical it solved my issue 

Thanks to all who helped me!

On Mon, May 4, 2020 at 11:06 PM Troy Melhase <[hidden email]> wrote:
Ami, can you post any of your logs/configuration?  I've been working
thru some of the OIDC related issues and might be able to provide
insight.

On Mon, May 4, 2020 at 7:57 AM Ami Goldenberg <[hidden email]> wrote:
>
> Ok I have an update
> I tried running a cluster without Kubernetes, on AWS and following the terraform configuration by pvillard here https://github.com/pvillard31/nifi-gcp-terraform/tree/master/gcp-cluster-secured-nifi-oidc
> Got a tls-toolkit CA server, zookeeper server, 2 nodes and an AWS ALB with sticky sessions
>
> With 1 and 2 nodes everything works fine
> If I recreate the cluster with another node however, then it all breaks and I'm back to the redirect loop
>
> Any idea what I'm missing here?