publishmqtt with SSL

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

publishmqtt with SSL

Oxenberg, Jeff
Hey,

I’m trying to get NiFi to send mqtt messages to the Azure IoT Hub.  The IoT Hub uses SSL certificates, and I’m having trouble getting it working with the publishmqtt processor.  I create a StandardSSLContextService pointing the truststore at /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts.  I made sure (I think) that the chain was trusted by importing it manually into the cacerts:
openssl s_client -showcerts -connect gsetest.azure-devices.net:8883 </dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > msft.cert
keytool -import -noprompt -trustcacerts -alias azure -file msft.cert -keystore /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts -storepass changeit

When I start the processor, I immediately get the below error.  This all works when I do it manually outside of NiFi using mosquitto_pub, so I know that my various settings (username, password, etc) are correct.  Has anyone done something similar, or can anyone offer any help here?  

2017-08-08 17:20:28,570 ERROR [StandardProcessScheduler Thread-6] o.a.n.controller.StandardProcessorNode Failed to invoke @OnScheduled method due to java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.
java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.
        at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1480)
        at org.apache.nifi.controller.StandardProcessorNode.access$000(StandardProcessorNode.java:100)
        at org.apache.nifi.controller.StandardProcessorNode$1.run(StandardProcessorNode.java:1301)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:206)
        at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1463)
        ... 9 common frames omitted
Caused by: java.lang.reflect.InvocationTargetException: null
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)

Thanks,


Jeff Oxenberg
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: publishmqtt with SSL

Oxenberg, Jeff

Bumping this up as I’m still having an issue here; has anyone gotten publishmqtt working with SSL?

 

Jeff Oxenberg

 

From: Oxenberg, Jeff
Sent: Tuesday, August 08, 2017 8:33 PM
To: [hidden email]
Subject: publishmqtt with SSL

 

Hey,

 

I’m trying to get NiFi to send mqtt messages to the Azure IoT Hub.  The IoT Hub uses SSL certificates, and I’m having trouble getting it working with the publishmqtt processor.  I create a StandardSSLContextService pointing the truststore at /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts.  I made sure (I think) that the chain was trusted by importing it manually into the cacerts:

openssl s_client -showcerts -connect gsetest.azure-devices.net:8883 </dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > msft.cert

keytool -import -noprompt -trustcacerts -alias azure -file msft.cert -keystore /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts -storepass changeit

 

When I start the processor, I immediately get the below error.  This all works when I do it manually outside of NiFi using mosquitto_pub, so I know that my various settings (username, password, etc) are correct.  Has anyone done something similar, or can anyone offer any help here?  

 

2017-08-08 17:20:28,570 ERROR [StandardProcessScheduler Thread-6] o.a.n.controller.StandardProcessorNode Failed to invoke @OnScheduled method due to java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.

java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.

        at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1480)

        at org.apache.nifi.controller.StandardProcessorNode.access$000(StandardProcessorNode.java:100)

        at org.apache.nifi.controller.StandardProcessorNode$1.run(StandardProcessorNode.java:1301)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

Caused by: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException

        at java.util.concurrent.FutureTask.report(FutureTask.java:122)

        at java.util.concurrent.FutureTask.get(FutureTask.java:206)

        at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1463)

        ... 9 common frames omitted

Caused by: java.lang.reflect.InvocationTargetException: null

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

 

Thanks,

 

 

Jeff Oxenberg

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: publishmqtt with SSL

Andy LoPresto
Hi Jeff,

Sorry you are having issues with this. Can you provide a full nifi-app.log which includes all the stacktraces? If you can enable “java.arg.15=-Djavax.net.debug=ssl,handshake” in your conf/bootstrap.conf, please also include nifi-bootstrap.log as this will contain the JSSE SSL/TLS output. From your stacktrace, it does not appear that this is a specific SSL/TLS issue, but it may be exposed by code related to that, so I can take a look. 

Usually, "InvocationTargetException: null” means that a NullPointerException was generated when trying to invoke the method on a null object. If you can do a remote debug session, I would look at PublishMQTT:131 and check if an exception is being generated there (or catch Throwable on line 338 rather than specific MQTTException). 

Andy LoPresto
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Aug 16, 2017, at 3:55 PM, Oxenberg, Jeff <[hidden email]> wrote:

Bumping this up as I’m still having an issue here; has anyone gotten publishmqtt working with SSL?
 
Jeff Oxenberg
 
From: Oxenberg, Jeff 
Sent: Tuesday, August 08, 2017 8:33 PM
To: [hidden email]
Subject: publishmqtt with SSL
 
Hey,
 
I’m trying to get NiFi to send mqtt messages to the Azure IoT Hub.  The IoT Hub uses SSL certificates, and I’m having trouble getting it working with the publishmqtt processor.  I create a StandardSSLContextService pointing the truststore at /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts.  I made sure (I think) that the chain was trusted by importing it manually into the cacerts:
openssl s_client -showcerts -connect gsetest.azure-devices.net:8883 </dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > msft.cert
keytool -import -noprompt -trustcacerts -alias azure -file msft.cert -keystore /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts -storepass changeit
 
When I start the processor, I immediately get the below error.  This all works when I do it manually outside of NiFi using mosquitto_pub, so I know that my various settings (username, password, etc) are correct.  Has anyone done something similar, or can anyone offer any help here?  
 
2017-08-08 17:20:28,570 ERROR [StandardProcessScheduler Thread-6] o.a.n.controller.StandardProcessorNode Failed to invoke @OnScheduled method due to java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.
java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.
        at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1480)
        at org.apache.nifi.controller.StandardProcessorNode.access$000(StandardProcessorNode.java:100)
        at org.apache.nifi.controller.StandardProcessorNode$1.run(StandardProcessorNode.java:1301)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:206)
        at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1463)
        ... 9 common frames omitted
Caused by: java.lang.reflect.InvocationTargetException: null
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
 
Thanks,
 
 
Jeff Oxenberg


signature.asc (859 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: publishmqtt with SSL

Oxenberg, Jeff
Hey Andy,

Thanks for getting back to me.  I’ve linked to the log files below.  I do see in nifi-bootstrap.log that the cert is trusted but like you said it doesn’t look to be an SSL-specific issue.  I will work on a remote debug session and see if that gives me any additional clues.

2017-08-17 10:17:49,763 INFO [NiFi logging handler] org.apache.nifi.StdOut adding as trusted cert:
2017-08-17 10:17:49,763 INFO [NiFi logging handler] org.apache.nifi.StdOut   Subject: CN=*.azure-devices.net
2017-08-17 10:17:49,763 INFO [NiFi logging handler] org.apache.nifi.StdOut   Issuer:  CN=Microsoft IT SSL SHA2, OU=Microsoft IT, O=Microsoft Corporation, L=Redmond, ST=Washington, C=US
2017-08-17 10:17:49,763 INFO [NiFi logging handler] org.apache.nifi.StdOut   Algorithm: RSA; Serial number: 0x5a0008405e4aa32ff9d2f2377100000008405e
2017-08-17 10:17:49,763 INFO [NiFi logging handler] org.apache.nifi.StdOut   Valid from Thu May 11 20:25:52 CDT 2017 until Mon May 07 12:03:30 CDT 2018


Thanks,

Jeff Oxenberg

From: Andy LoPresto <[hidden email]>
Reply-To: <[hidden email]>
Date: Wednesday, August 16, 2017 at 3:05 PM
To: <[hidden email]>
Subject: Re: publishmqtt with SSL

Hi Jeff,

Sorry you are having issues with this. Can you provide a full nifi-app.log which includes all the stacktraces? If you can enable “java.arg.15=-Djavax.net.debug=ssl,handshake” in your conf/bootstrap.conf, please also include nifi-bootstrap.log as this will contain the JSSE SSL/TLS output. From your stacktrace, it does not appear that this is a specific SSL/TLS issue, but it may be exposed by code related to that, so I can take a look. 

Usually, "InvocationTargetException: null” means that a NullPointerException was generated when trying to invoke the method on a null object. If you can do a remote debug session, I would look at PublishMQTT:131 and check if an exception is being generated there (or catch Throwable on line 338 rather than specific MQTTException). 

Andy LoPresto
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Aug 16, 2017, at 3:55 PM, Oxenberg, Jeff <[hidden email]> wrote:

Bumping this up as I’m still having an issue here; has anyone gotten publishmqtt working with SSL?
 
Jeff Oxenberg
 
From: Oxenberg, Jeff 
Sent: Tuesday, August 08, 2017 8:33 PM
To: [hidden email]
Subject: publishmqtt with SSL
 
Hey,
 
I’m trying to get NiFi to send mqtt messages to the Azure IoT Hub.  The IoT Hub uses SSL certificates, and I’m having trouble getting it working with the publishmqtt processor.  I create a StandardSSLContextService pointing the truststore at /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts.  I made sure (I think) that the chain was trusted by importing it manually into the cacerts:
openssl s_client -showcerts -connect gsetest.azure-devices.net:8883 </dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > msft.cert
keytool -import -noprompt -trustcacerts -alias azure -file msft.cert -keystore /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts -storepass changeit
 
When I start the processor, I immediately get the below error.  This all works when I do it manually outside of NiFi using mosquitto_pub, so I know that my various settings (username, password, etc) are correct.  Has anyone done something similar, or can anyone offer any help here?  
 
2017-08-08 17:20:28,570 ERROR [StandardProcessScheduler Thread-6] o.a.n.controller.StandardProcessorNode Failed to invoke @OnScheduled method due to java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.
java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.
        at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1480)
        at org.apache.nifi.controller.StandardProcessorNode.access$000(StandardProcessorNode.java:100)
        at org.apache.nifi.controller.StandardProcessorNode$1.run(StandardProcessorNode.java:1301)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:206)
        at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1463)
        ... 9 common frames omitted
Caused by: java.lang.reflect.InvocationTargetException: null
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
 
Thanks,
 
 
Jeff Oxenberg

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: publishmqtt with SSL

Oxenberg, Jeff
I was able to get remote debugging working - I’m guessing sslClientProps should not be null in this case?


Thanks,

Jeff Oxenberg

From: Jeff Oxenberg <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Thursday, August 17, 2017 at 10:44 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: publishmqtt with SSL

Hey Andy,

Thanks for getting back to me.  I’ve linked to the log files below.  I do see in nifi-bootstrap.log that the cert is trusted but like you said it doesn’t look to be an SSL-specific issue.  I will work on a remote debug session and see if that gives me any additional clues.

2017-08-17 10:17:49,763 INFO [NiFi logging handler] org.apache.nifi.StdOut adding as trusted cert:
2017-08-17 10:17:49,763 INFO [NiFi logging handler] org.apache.nifi.StdOut   Subject: CN=*.azure-devices.net
2017-08-17 10:17:49,763 INFO [NiFi logging handler] org.apache.nifi.StdOut   Issuer:  CN=Microsoft IT SSL SHA2, OU=Microsoft IT, O=Microsoft Corporation, L=Redmond, ST=Washington, C=US
2017-08-17 10:17:49,763 INFO [NiFi logging handler] org.apache.nifi.StdOut   Algorithm: RSA; Serial number: 0x5a0008405e4aa32ff9d2f2377100000008405e
2017-08-17 10:17:49,763 INFO [NiFi logging handler] org.apache.nifi.StdOut   Valid from Thu May 11 20:25:52 CDT 2017 until Mon May 07 12:03:30 CDT 2018


Thanks,

Jeff Oxenberg

From: Andy LoPresto <[hidden email]>
Reply-To: <[hidden email]>
Date: Wednesday, August 16, 2017 at 3:05 PM
To: <[hidden email]>
Subject: Re: publishmqtt with SSL

Hi Jeff,

Sorry you are having issues with this. Can you provide a full nifi-app.log which includes all the stacktraces? If you can enable “java.arg.15=-Djavax.net.debug=ssl,handshake” in your conf/bootstrap.conf, please also include nifi-bootstrap.log as this will contain the JSSE SSL/TLS output. From your stacktrace, it does not appear that this is a specific SSL/TLS issue, but it may be exposed by code related to that, so I can take a look. 

Usually, "InvocationTargetException: null” means that a NullPointerException was generated when trying to invoke the method on a null object. If you can do a remote debug session, I would look at PublishMQTT:131 and check if an exception is being generated there (or catch Throwable on line 338 rather than specific MQTTException). 

Andy LoPresto
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Aug 16, 2017, at 3:55 PM, Oxenberg, Jeff <[hidden email]> wrote:

Bumping this up as I’m still having an issue here; has anyone gotten publishmqtt working with SSL?
 
Jeff Oxenberg
 
From: Oxenberg, Jeff 
Sent: Tuesday, August 08, 2017 8:33 PM
To: [hidden email]
Subject: publishmqtt with SSL
 
Hey,
 
I’m trying to get NiFi to send mqtt messages to the Azure IoT Hub.  The IoT Hub uses SSL certificates, and I’m having trouble getting it working with the publishmqtt processor.  I create a StandardSSLContextService pointing the truststore at /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts.  I made sure (I think) that the chain was trusted by importing it manually into the cacerts:
openssl s_client -showcerts -connect gsetest.azure-devices.net:8883 </dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > msft.cert
keytool -import -noprompt -trustcacerts -alias azure -file msft.cert -keystore /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts -storepass changeit
 
When I start the processor, I immediately get the below error.  This all works when I do it manually outside of NiFi using mosquitto_pub, so I know that my various settings (username, password, etc) are correct.  Has anyone done something similar, or can anyone offer any help here?  
 
2017-08-08 17:20:28,570 ERROR [StandardProcessScheduler Thread-6] o.a.n.controller.StandardProcessorNode Failed to invoke @OnScheduled method due to java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.
java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.
        at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1480)
        at org.apache.nifi.controller.StandardProcessorNode.access$000(StandardProcessorNode.java:100)
        at org.apache.nifi.controller.StandardProcessorNode$1.run(StandardProcessorNode.java:1301)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:206)
        at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1463)
        ... 9 common frames omitted
Caused by: java.lang.reflect.InvocationTargetException: null
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
 
Thanks,
 
 
Jeff Oxenberg

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: publishmqtt with SSL

Oxenberg, Jeff
I got it working.  I think transformSSLContextService [1] doesn’t like when some of the properties are null (in my case, I didn’t need any of the keystore properties as I only used the truststore).  This line [2] wasn’t getting executed as a result.  I worked around the issue by just creating a dummy keystore, but I will go back and try to fix it when I have more time.


Thanks,

Jeff Oxenberg

From: Jeff Oxenberg <[hidden email]>
Date: Thursday, August 17, 2017 at 5:27 PM
To: "[hidden email]" <[hidden email]>
Subject: Re: publishmqtt with SSL

I was able to get remote debugging working - I’m guessing sslClientProps should not be null in this case?


Thanks,

Jeff Oxenberg

From: Jeff Oxenberg <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Thursday, August 17, 2017 at 10:44 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: publishmqtt with SSL

Hey Andy,

Thanks for getting back to me.  I’ve linked to the log files below.  I do see in nifi-bootstrap.log that the cert is trusted but like you said it doesn’t look to be an SSL-specific issue.  I will work on a remote debug session and see if that gives me any additional clues.

2017-08-17 10:17:49,763 INFO [NiFi logging handler] org.apache.nifi.StdOut adding as trusted cert:
2017-08-17 10:17:49,763 INFO [NiFi logging handler] org.apache.nifi.StdOut   Subject: CN=*.azure-devices.net
2017-08-17 10:17:49,763 INFO [NiFi logging handler] org.apache.nifi.StdOut   Issuer:  CN=Microsoft IT SSL SHA2, OU=Microsoft IT, O=Microsoft Corporation, L=Redmond, ST=Washington, C=US
2017-08-17 10:17:49,763 INFO [NiFi logging handler] org.apache.nifi.StdOut   Algorithm: RSA; Serial number: 0x5a0008405e4aa32ff9d2f2377100000008405e
2017-08-17 10:17:49,763 INFO [NiFi logging handler] org.apache.nifi.StdOut   Valid from Thu May 11 20:25:52 CDT 2017 until Mon May 07 12:03:30 CDT 2018


Thanks,

Jeff Oxenberg

From: Andy LoPresto <[hidden email]>
Reply-To: <[hidden email]>
Date: Wednesday, August 16, 2017 at 3:05 PM
To: <[hidden email]>
Subject: Re: publishmqtt with SSL

Hi Jeff,

Sorry you are having issues with this. Can you provide a full nifi-app.log which includes all the stacktraces? If you can enable “java.arg.15=-Djavax.net.debug=ssl,handshake” in your conf/bootstrap.conf, please also include nifi-bootstrap.log as this will contain the JSSE SSL/TLS output. From your stacktrace, it does not appear that this is a specific SSL/TLS issue, but it may be exposed by code related to that, so I can take a look. 

Usually, "InvocationTargetException: null” means that a NullPointerException was generated when trying to invoke the method on a null object. If you can do a remote debug session, I would look at PublishMQTT:131 and check if an exception is being generated there (or catch Throwable on line 338 rather than specific MQTTException). 

Andy LoPresto
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Aug 16, 2017, at 3:55 PM, Oxenberg, Jeff <[hidden email]> wrote:

Bumping this up as I’m still having an issue here; has anyone gotten publishmqtt working with SSL?
 
Jeff Oxenberg
 
From: Oxenberg, Jeff 
Sent: Tuesday, August 08, 2017 8:33 PM
To: [hidden email]
Subject: publishmqtt with SSL
 
Hey,
 
I’m trying to get NiFi to send mqtt messages to the Azure IoT Hub.  The IoT Hub uses SSL certificates, and I’m having trouble getting it working with the publishmqtt processor.  I create a StandardSSLContextService pointing the truststore at /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts.  I made sure (I think) that the chain was trusted by importing it manually into the cacerts:
openssl s_client -showcerts -connect gsetest.azure-devices.net:8883 </dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > msft.cert
keytool -import -noprompt -trustcacerts -alias azure -file msft.cert -keystore /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts -storepass changeit
 
When I start the processor, I immediately get the below error.  This all works when I do it manually outside of NiFi using mosquitto_pub, so I know that my various settings (username, password, etc) are correct.  Has anyone done something similar, or can anyone offer any help here?  
 
2017-08-08 17:20:28,570 ERROR [StandardProcessScheduler Thread-6] o.a.n.controller.StandardProcessorNode Failed to invoke @OnScheduled method due to java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.
java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.
        at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1480)
        at org.apache.nifi.controller.StandardProcessorNode.access$000(StandardProcessorNode.java:100)
        at org.apache.nifi.controller.StandardProcessorNode$1.run(StandardProcessorNode.java:1301)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:206)
        at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1463)
        ... 9 common frames omitted
Caused by: java.lang.reflect.InvocationTargetException: null
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
 
Thanks,
 
 
Jeff Oxenberg

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: publishmqtt with SSL

Andy LoPresto-2
Jeff,

Glad to hear you got it working. That definitely shouldn't be happening as the keystore should only be required when doing mutual authentication, so thanks for detecting that. Either way, it should handle NPE better. 

I'm out of the country for a couple days but will try to address this on Monday if you can file a Jira. Thanks. 

Andy LoPresto
[hidden email]
[hidden email]
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69


On Aug 17, 2017, at 23:59, Oxenberg, Jeff <[hidden email]> wrote:

I got it working.  I think transformSSLContextService [1] doesn’t like when some of the properties are null (in my case, I didn’t need any of the keystore properties as I only used the truststore).  This line [2] wasn’t getting executed as a result.  I worked around the issue by just creating a dummy keystore, but I will go back and try to fix it when I have more time.


Thanks,

Jeff Oxenberg

From: Jeff Oxenberg <[hidden email]>
Date: Thursday, August 17, 2017 at 5:27 PM
To: "[hidden email]" <[hidden email]>
Subject: Re: publishmqtt with SSL

I was able to get remote debugging working - I’m guessing sslClientProps should not be null in this case?


Thanks,

Jeff Oxenberg

From: Jeff Oxenberg <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Thursday, August 17, 2017 at 10:44 AM
To: "[hidden email]" <[hidden email]>
Subject: Re: publishmqtt with SSL

Hey Andy,

Thanks for getting back to me.  I’ve linked to the log files below.  I do see in nifi-bootstrap.log that the cert is trusted but like you said it doesn’t look to be an SSL-specific issue.  I will work on a remote debug session and see if that gives me any additional clues.

2017-08-17 10:17:49,763 INFO [NiFi logging handler] org.apache.nifi.StdOut adding as trusted cert:
2017-08-17 10:17:49,763 INFO [NiFi logging handler] org.apache.nifi.StdOut   Subject: CN=*.azure-devices.net
2017-08-17 10:17:49,763 INFO [NiFi logging handler] org.apache.nifi.StdOut   Issuer:  CN=Microsoft IT SSL SHA2, OU=Microsoft IT, O=Microsoft Corporation, L=Redmond, ST=Washington, C=US
2017-08-17 10:17:49,763 INFO [NiFi logging handler] org.apache.nifi.StdOut   Algorithm: RSA; Serial number: 0x5a0008405e4aa32ff9d2f2377100000008405e
2017-08-17 10:17:49,763 INFO [NiFi logging handler] org.apache.nifi.StdOut   Valid from Thu May 11 20:25:52 CDT 2017 until Mon May 07 12:03:30 CDT 2018


Thanks,

Jeff Oxenberg

From: Andy LoPresto <[hidden email]>
Reply-To: <[hidden email]>
Date: Wednesday, August 16, 2017 at 3:05 PM
To: <[hidden email]>
Subject: Re: publishmqtt with SSL

Hi Jeff,

Sorry you are having issues with this. Can you provide a full nifi-app.log which includes all the stacktraces? If you can enable “java.arg.15=-Djavax.net.debug=ssl,handshake” in your conf/bootstrap.conf, please also include nifi-bootstrap.log as this will contain the JSSE SSL/TLS output. From your stacktrace, it does not appear that this is a specific SSL/TLS issue, but it may be exposed by code related to that, so I can take a look. 

Usually, "InvocationTargetException: null” means that a NullPointerException was generated when trying to invoke the method on a null object. If you can do a remote debug session, I would look at PublishMQTT:131 and check if an exception is being generated there (or catch Throwable on line 338 rather than specific MQTTException). 

Andy LoPresto
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Aug 16, 2017, at 3:55 PM, Oxenberg, Jeff <[hidden email]> wrote:

Bumping this up as I’m still having an issue here; has anyone gotten publishmqtt working with SSL?
 
Jeff Oxenberg
 
From: Oxenberg, Jeff 
Sent: Tuesday, August 08, 2017 8:33 PM
To: [hidden email]
Subject: publishmqtt with SSL
 
Hey,
 
I’m trying to get NiFi to send mqtt messages to the Azure IoT Hub.  The IoT Hub uses SSL certificates, and I’m having trouble getting it working with the publishmqtt processor.  I create a StandardSSLContextService pointing the truststore at /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts.  I made sure (I think) that the chain was trusted by importing it manually into the cacerts:
openssl s_client -showcerts -connect gsetest.azure-devices.net:8883 </dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > msft.cert
keytool -import -noprompt -trustcacerts -alias azure -file msft.cert -keystore /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts -storepass changeit
 
When I start the processor, I immediately get the below error.  This all works when I do it manually outside of NiFi using mosquitto_pub, so I know that my various settings (username, password, etc) are correct.  Has anyone done something similar, or can anyone offer any help here?  
 
2017-08-08 17:20:28,570 ERROR [StandardProcessScheduler Thread-6] o.a.n.controller.StandardProcessorNode Failed to invoke @OnScheduled method due to java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.
java.lang.RuntimeException: Failed while executing one of processor's OnScheduled task.
        at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1480)
        at org.apache.nifi.controller.StandardProcessorNode.access$000(StandardProcessorNode.java:100)
        at org.apache.nifi.controller.StandardProcessorNode$1.run(StandardProcessorNode.java:1301)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:206)
        at org.apache.nifi.controller.StandardProcessorNode.invokeTaskAsCancelableFuture(StandardProcessorNode.java:1463)
        ... 9 common frames omitted
Caused by: java.lang.reflect.InvocationTargetException: null
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
 
Thanks,
 
 
Jeff Oxenberg

Loading...