how to update nifi to the latest jolt version?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

how to update nifi to the latest jolt version?

Sebastian Lagemann
Hello,

I’m trying to use the latest jolt specification, especially the operation "modify-overwrite-beta“, introduced with jolt version 0.0.22. It seems that currently only version 0.0.21 is used by nifi (see nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/pom.xml). I tried to update to the latest jolt version (from 0.0.21 to 0.0.24) in the corresponding pom.xml and packaged the whole package (after I removed one failing test in nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/test/java/org/apache/nifi/cluster/coordination/http/StandardHttpResponseMapperSpec.groovy, not the way to go but need some fast results :-)).
Nifi is using the new .nar file but unfortunately the jolt processor is still complaining that the jolt specification is invalid (see below).

Does anyone has a hint/idea how to use the latest jolt version, I had already in mind of extracting the processor from the nifi standard bar bundle and create a new named processor for testing, but I guess there must be an easier solution?

The jolt spec I’m using:
[
  {
    "operation": "modify-overwrite",
    "spec": {
      "lastElementOfType": "=lastElement(@(1,type))"
    }
  },
  {
    "operation": "shift",
    "spec": {
      "value": "@(1,lastElementOfType)",
      "_meta": {
        "appId": "appId",
        "userId": "userId"
      }
    }
  }
]

The data I’m using:
  {
    "type":["user","profile","personalInfo", "firstName"],
    "value":"testname",
    "_meta":{"appId":"test","userId":"56c1614b677c35cc3f28fbd0"}
  }

The expected flow file content afterwards:
{
  "appId" : "test",
  "firstName" : "testname",
  "playerId" : "56c1614b677c35cc3f28fbd0"
}

The jolt demo from http://jolt-demo.appspot.com/#inception delivers the expected results.

Thanks,

Sebastian

Reply | Threaded
Open this post in threaded view
|

Re: how to update nifi to the latest jolt version?

Matt Gilman
Sebastian,

It appears the jolt library is being bundled into the WAR for that Processor's Advanced UI. Despite the UI not working with your changes, the Process should likely run ok. 

To fix the Advanced UI, you could simply bump the version there too [1] or you could define that dependency in the dependency management in the nifi-standard-bundle NAR. Then the dependency version would be set in a single location. Additionally, you could mark the dependency in the WAR as provided as the bundled dependencies in the standard NAR will be available to the Advanced UI at runtime.

$ find work -name "jolt-core*"
work/jetty/nifi-jolt-transform-json-ui-1.1.0-SNAPSHOT.war/webapp/WEB-INF/lib/jolt-core-0.0.21.jar
work/nar/extensions/nifi-standard-nar-1.1.0-SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/jolt-core-0.0.21.jar

Matt


On Thu, Nov 10, 2016 at 1:00 PM, Sebastian Lagemann <[hidden email]> wrote:
Hello,

I’m trying to use the latest jolt specification, especially the operation "modify-overwrite-beta“, introduced with jolt version 0.0.22. It seems that currently only version 0.0.21 is used by nifi (see nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/pom.xml). I tried to update to the latest jolt version (from 0.0.21 to 0.0.24) in the corresponding pom.xml and packaged the whole package (after I removed one failing test in nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/test/java/org/apache/nifi/cluster/coordination/http/StandardHttpResponseMapperSpec.groovy, not the way to go but need some fast results :-)).
Nifi is using the new .nar file but unfortunately the jolt processor is still complaining that the jolt specification is invalid (see below).

Does anyone has a hint/idea how to use the latest jolt version, I had already in mind of extracting the processor from the nifi standard bar bundle and create a new named processor for testing, but I guess there must be an easier solution?

The jolt spec I’m using:
[
  {
    "operation": "modify-overwrite",
    "spec": {
      "lastElementOfType": "=lastElement(@(1,type))"
    }
  },
  {
    "operation": "shift",
    "spec": {
      "value": "@(1,lastElementOfType)",
      "_meta": {
        "appId": "appId",
        "userId": "userId"
      }
    }
  }
]

The data I’m using:
  {
    "type":["user","profile","personalInfo", "firstName"],
    "value":"testname",
    "_meta":{"appId":"test","userId":"56c1614b677c35cc3f28fbd0"}
  }

The expected flow file content afterwards:
{
  "appId" : "test",
  "firstName" : "testname",
  "playerId" : "56c1614b677c35cc3f28fbd0"
}

The jolt demo from http://jolt-demo.appspot.com/#inception delivers the expected results.

Thanks,

Sebastian