Replacing a base64-encoded field in a JSON-document with its decoded/converted value

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Replacing a base64-encoded field in a JSON-document with its decoded/converted value

Myklebust, Bjørn Magnar

 

Hi.

I have a set of Json-files which contain a base64-coded field (Jsonpath to this field is $.data.content), and this field contains a XML-document.  Decoding the field works as expected, so does the conversion from xml to json,  and I'm able to write the content from this field to a file in a bucket in S3.  But what I would like to do is to be able to replace the coded value for this field in the original file with the decoded/converted value in stead of writing the decoded/converted value to file. And after replacing the json-value then I can write the updated Json-file to a new S3 bucket.

My process look like this at the moment, and works fine for getting the data to file, but it's missing the last part of replacing $.data.content with the decoded/converted data.

 

So how can I do the last part?

 

 

The EvaluedJsonPath looks like this:

 

 

The ReplaceText looks like this:

 

The Base64EncodeContent looks like this:

 

and finally, the CovertRecord looks like this:

 

 

 

This is a testfile for that I'm working with:

 

{

  "header": {

    "dokumentidentifikator": null,

    "dokumentidentifikatorV2": "dcff985b-c652-4085-b8f1-45a2f4b6d150",

    "revisjonsnummer": 1,

    "dokumentnavn": "Engangsavgiftfastsettelse:55TEST661122334455:44BIL1:2017-10-20",

    "dokumenttype": "SKATTEMELDING_ENGANGSAVGIFT",

    "dokumenttilstand": "OPPRETTET",

    "gyldig": true,

    "gjelderInntektsaar": 2017,

    "gjelderPeriode": "2017_10",

    "gjelderPart": {

      "partsnummer": 5544332211,

      "identifiseringstype": "MASKINELL",

      "identifikator": null

    },

    "opphavspart": {

      "partsnummer": 5544332211,

      "identifikator": null

    },

    "kildereferanse": {

      "kildesystem": "ENGANGSAVGIFTFASTSETTELSE",

      "gruppe": "",

      "referanse": "aef147fb-8ce8-43ef-833b-7aa3bac1ece0",

      "tidspunkt": "2018-01-16T13:28:02.49+01:00"

    }

  },

  "data": {

    "metadata": {

      "format": "ske:fastsetting:motorvogn:motorvognavgift:v1",

      "bytes": 4420,

      "mimeType": "application/xml",

      "sha1": "c0AowOsTdNdo6VufeSsZqTphc0Y="

    },

    "content": "PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiIHN0YW5kYWxvbmU9InllcyI/Pgo8bW90b3J2b2duYXZnaWZ0IHhtbG5zPSJza2U6ZmFzdHNldHRpbmc6bW90b3J2b2duOm1vdG9ydm9nbmF2Z2lmdDp2MSI+CiAgICA8YXZnaWZ0c2xpbmplPgogICAgICAgIDxhdmdpZnRzYmVsb2VwPjU0Mjg5Ni4wMDwvYXZnaWZ0c2JlbG9lcD4KICAgICAgICA8YXZnaWZ0c29wcGx5c25pbmc+CiAgICAgICAgICAgIDxzYWVyYXZnaWZ0VHlwZWtvZGU+QkI8L3NhZXJhdmdpZnRUeXBla29kZT4KICAgICAgICAgICAgPHNhZXJhdmdpZnRHcnVwcGVrb2RlPlg8L3NhZXJhdmdpZnRHcnVwcGVrb2RlPgogICAgICAgIDwvYXZnaWZ0c29wcGx5c25pbmc+CiAgICAgICAgPGF2Z2lmdHNkYXRvPjIwMTctMTAtMjA8L2F2Z2lmdHNkYXRvPgogICAgPC9hdmdpZnRzbGluamU+CiAgICA8YmV0YWxpbmdzaW5mb3JtYXNqb24+CiAgICAgICAgPGtpZG51bW1lcj4xMDEwMTAxMDEwMTA8L2tpZG51bW1lcj4KICAgICAgICA8Zm9yZmFsbHNkYXRvPjIwMTctMTAtMjA8L2ZvcmZhbGxzZGF0bz4KICAgICAgICA8ZmFrdHVyYWRhdG8+MjAxNy0xMC0yMDwvZmFrdHVyYWRhdG8+CiAgICAgICAgPHRvdGFsQXZnaWZ0c2JlbG9lcD41NDI4OTYuMDA8L3RvdGFsQXZnaWZ0c2JlbG9lcD4KICAgIDwvYmV0YWxpbmdzaW5mb3JtYXNqb24+CiAgICA8bW90b3J2b2duYXZnaWZ0c3R5cGU+ZW5nYW5nc2F2Z2lmdDwvbW90b3J2b2duYXZnaWZ0c3R5cGU+CiAgICA8dGlkc3N0ZW1wZWw+MjAxOC0wMS0xNiswMTowMDwvdGlkc3N0ZW1wZWw+CiAgICA8Z3J1bm5sYWdGb3JNb3RvcnZvZ25hdmdpZnQ+CiAgICAgICAgPGtqb2VyaW5nZW5zQXJ0PjEwPC9ram9lcmluZ2Vuc0FydD4KICAgICAgICA8a2pvZXJldG9leT4KICAgICAgICAgICAgPGVpZXJza2FwUmVnaXN0cmVydD4yMDE3LTEwLTIwPC9laWVyc2thcFJlZ2lzdHJlcnQ+CiAgICAgICAgICAgIDxmb2Vyc3RlUmVnaXN0cmVyaW5nc2Fhcj4yMDE3PC9mb2Vyc3RlUmVnaXN0cmVyaW5nc2Fhcj4KICAgICAgICAgICAgPGZvZXJzdGVSZWdpc3RyZXJpbmdzZGF0b0lOb3JnZT4yMDE3LTEwLTIwPC9mb2Vyc3RlUmVnaXN0cmVyaW5nc2RhdG9JTm9yZ2U+CiAgICAgICAgICAgIDxram9lcmV0b2V5Z3J1cHBlPjEwMTwva2pvZXJldG9leWdydXBwZT4KICAgICAgICAgICAgPGxlbmdkZT4zOTY0PC9sZW5nZGU+CiAgICAgICAgICAgIDxtb3RvcmVmZmVrdD45NjwvbW90b3JlZmZla3Q+CiAgICAgICAgICAgIDxzbGFndm9sdW0+MTQzPC9zbGFndm9sdW0+CiAgICAgICAgICAgIDxkcml2c3RvZmY+QkVOU0lOPC9kcml2c3RvZmY+CiAgICAgICAgICAgIDxlZ2VudmVrdD4xNTE5PC9lZ2VudmVrdD4KICAgICAgICAgICAgPGVpZXI+CiAgICAgICAgICAgICAgICA8Zm9lZHNlbHNFbGxlckRudW1tZXI+MTEyMjMzNDQ1NTwvZm9lZHNlbHNFbGxlckRudW1tZXI+CiAgICAgICAgICAgICAgICA8cGFydHNudW1tZXI+NTU0NDMzMjIxMTwvcGFydHNudW1tZXI+CiAgICAgICAgICAgICAgICA8bmF2bj5LTEFSQSBLVTwvbmF2bj4KICAgICAgICAgICAgPC9laWVyPgogICAgICAgICAgICA8dGlsbGF0dFRvdGFsdmVrdD4yMTY0PC90aWxsYXR0VG90YWx2ZWt0PgogICAgICAgICAgICA8aHlicmlkPm5laTwvaHlicmlkPgogICAgICAgICAgICA8Y28ydXRzbGlwcD4yNjg8L2NvMnV0c2xpcHA+CiAgICAgICAgICAgIDxub3h1dHNsaXBwPjU5LjQ8L25veHV0c2xpcHA+CiAgICAgICAgICAgIDxram9lcmV0b2V5aWRlbnRpZmlrYXRvcj4KICAgICAgICAgICAgICAgIDxram9lcmV0b2V5VW5pa0lkZW50aWZpa2F0b3I+QUJDREVGR0hJSjwva2pvZXJldG9leVVuaWtJZGVudGlmaWthdG9yPgogICAgICAgICAgICAgICAgPGtqZW5uZW1lcmtlPjQ0QklMMTwva2plbm5lbWVya2U+CiAgICAgICAgICAgICAgICA8dW5kZXJzdGVsbHNudW1tZXI+VU5ERVJTVEVMTDQ0PC91bmRlcnN0ZWxsc251bW1lcj4KICAgICAgICAgICAgPC9ram9lcmV0b2V5aWRlbnRpZmlrYXRvcj4KICAgICAgICA8L2tqb2VyZXRvZXk+CiAgICA8L2dydW5ubGFnRm9yTW90b3J2b2duYXZnaWZ0PgogICAgPGF2Z2lmdHNwbGlrdGlnPgogICAgICAgIDxmb2Vkc2Vsc0VsbGVyRG51bW1lcj4xMTIyMzM0NDU1PC9mb2Vkc2Vsc0VsbGVyRG51bW1lcj4KICAgICAgICA8cGFydHNudW1tZXI+NTU0NDMzMjIxMTwvcGFydHNudW1tZXI+CiAgICA8L2F2Z2lmdHNwbGlrdGlnPgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzI8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjQ3NTMzNy4yMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5FZ2VudmVrdDwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+NjA5NDUuNjQ8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+TW90b3JlZmZla3Q8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94PC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD40MjEzLjI0PC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlNsYWd2b2x1bTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzIgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+RWdlbnZla3QgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+TW90b3JlZmZla3QgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94IEZyYXRyZWtrPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4wLjAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlNsYWd2b2x1bSBGcmF0cmVrazwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzIgU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD40NzUzMzcuMjA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+RWdlbnZla3QgU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD42MDk0NS42NDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5Nb3RvcmVmZmVrdCBTdW08L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94IFN1bTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+NDIxMy4yNDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5TbGFndm9sdW0gU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4wLjAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlZyYWtwYW50PC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4yNDAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PkJydWtzZnJhZHJhZyAwJTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGZvcmhhYW5kc2JlcmVnbmluZz5mYWxzZTwvZm9yaGFhbmRzYmVyZWduaW5nPgo8L21vdG9ydm9nbmF2Z2lmdD4K"

  },

  "extension": null,

  "skjemaversjon": "v3_0"

}

 

 

 

 

Cheers,

Bjørn

 



Denne e-posten og eventuelle vedlegg er beregnet utelukkende for den institusjon eller person den er rettet til og kan være belagt med lovbestemt taushetsplikt. Dersom e-posten er feilsendt, vennligst slett den og kontakt Skatteetaten.
The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential information and may be legally protected from disclosure. If you are not the intended recipient of this message, please immediately delete the message and alert the Norwegian Tax Administration.
Reply | Threaded
Open this post in threaded view
|

Re: Replacing a base64-encoded field in a JSON-document with its decoded/converted value

Andy LoPresto
Hello Bjørn, 

If the size of the encoded XML document is small (under ~1 KB), you can extract the Base64-encoded value to a flowfile attribute using EvaluateJSONPath, perform the decoding using the base64Decode Expression Language function [1], and then replace it into the flowfile JSON content using ReplaceText (using some regex like "content": ".*" -> “content": ”${decodedXML}” where decodedXML is the name of the attribute you are using). 

If the XML content could be very large, this will negatively affect your performance, as attributes are stored directly in memory and handling large amounts of data will impact the heap. In this case, I would recommend writing a Groovy script in ExecuteScript processor to leverage Groovy’s very friendly JSON handling and extract the value, Base64 decode it, and replace it in a couple lines. 

Hope this helps. 



Andy LoPresto
[hidden email]
[hidden email]
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Jun 24, 2020, at 4:24 AM, Myklebust, Bjørn Magnar <[hidden email]> wrote:

 
Hi.
I have a set of Json-files which contain a base64-coded field (Jsonpath to this field is $.data.content), and this field contains a XML-document.  Decoding the field works as expected, so does the conversion from xml to json,  and I'm able to write the content from this field to a file in a bucket in S3.  But what I would like to do is to be able to replace the coded value for this field in the original file with the decoded/converted value in stead of writing the decoded/converted value to file. And after replacing the json-value then I can write the updated Json-file to a new S3 bucket.
My process look like this at the moment, and works fine for getting the data to file, but it's missing the last part of replacing $.data.content with the decoded/converted data.
 
So how can I do the last part?
 
<pastedImage.png>
 
The EvaluedJsonPath looks like this:
 

<EvaluateJsonPath.png>

 
The ReplaceText looks like this:
 

<ReplaceText.png>

The Base64EncodeContent looks like this:
 

<Base64Decode.png>

and finally, the CovertRecord looks like this:
 

<ConvertRecord.png>

 
 
This is a testfile for that I'm working with:
 
{
  "header": {
    "dokumentidentifikator": null,
    "dokumentidentifikatorV2": "dcff985b-c652-4085-b8f1-45a2f4b6d150",
    "revisjonsnummer": 1,
    "dokumentnavn": "Engangsavgiftfastsettelse:55TEST661122334455:44BIL1:2017-10-20",
    "dokumenttype": "SKATTEMELDING_ENGANGSAVGIFT",
    "dokumenttilstand": "OPPRETTET",
    "gyldig": true,
    "gjelderInntektsaar": 2017,
    "gjelderPeriode": "2017_10",
    "gjelderPart": {
      "partsnummer": 5544332211,
      "identifiseringstype": "MASKINELL",
      "identifikator": null
    },
    "opphavspart": {
      "partsnummer": 5544332211,
      "identifikator": null
    },
    "kildereferanse": {
      "kildesystem": "ENGANGSAVGIFTFASTSETTELSE",
      "gruppe": "",
      "referanse": "aef147fb-8ce8-43ef-833b-7aa3bac1ece0",
      "tidspunkt": "2018-01-16T13:28:02.49+01:00"
    }
  },
  "data": {
    "metadata": {
      "format": "ske:fastsetting:motorvogn:motorvognavgift:v1",
      "bytes": 4420,
      "mimeType": "application/xml",
      "sha1": "c0AowOsTdNdo6VufeSsZqTphc0Y="
    },
    "content": "PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiIHN0YW5kYWxvbmU9InllcyI/Pgo8bW90b3J2b2duYXZnaWZ0IHhtbG5zPSJza2U6ZmFzdHNldHRpbmc6bW90b3J2b2duOm1vdG9ydm9nbmF2Z2lmdDp2MSI+CiAgICA8YXZnaWZ0c2xpbmplPgogICAgICAgIDxhdmdpZnRzYmVsb2VwPjU0Mjg5Ni4wMDwvYXZnaWZ0c2JlbG9lcD4KICAgICAgICA8YXZnaWZ0c29wcGx5c25pbmc+CiAgICAgICAgICAgIDxzYWVyYXZnaWZ0VHlwZWtvZGU+QkI8L3NhZXJhdmdpZnRUeXBla29kZT4KICAgICAgICAgICAgPHNhZXJhdmdpZnRHcnVwcGVrb2RlPlg8L3NhZXJhdmdpZnRHcnVwcGVrb2RlPgogICAgICAgIDwvYXZnaWZ0c29wcGx5c25pbmc+CiAgICAgICAgPGF2Z2lmdHNkYXRvPjIwMTctMTAtMjA8L2F2Z2lmdHNkYXRvPgogICAgPC9hdmdpZnRzbGluamU+CiAgICA8YmV0YWxpbmdzaW5mb3JtYXNqb24+CiAgICAgICAgPGtpZG51bW1lcj4xMDEwMTAxMDEwMTA8L2tpZG51bW1lcj4KICAgICAgICA8Zm9yZmFsbHNkYXRvPjIwMTctMTAtMjA8L2ZvcmZhbGxzZGF0bz4KICAgICAgICA8ZmFrdHVyYWRhdG8+MjAxNy0xMC0yMDwvZmFrdHVyYWRhdG8+CiAgICAgICAgPHRvdGFsQXZnaWZ0c2JlbG9lcD41NDI4OTYuMDA8L3RvdGFsQXZnaWZ0c2JlbG9lcD4KICAgIDwvYmV0YWxpbmdzaW5mb3JtYXNqb24+CiAgICA8bW90b3J2b2duYXZnaWZ0c3R5cGU+ZW5nYW5nc2F2Z2lmdDwvbW90b3J2b2duYXZnaWZ0c3R5cGU+CiAgICA8dGlkc3N0ZW1wZWw+MjAxOC0wMS0xNiswMTowMDwvdGlkc3N0ZW1wZWw+CiAgICA8Z3J1bm5sYWdGb3JNb3RvcnZvZ25hdmdpZnQ+CiAgICAgICAgPGtqb2VyaW5nZW5zQXJ0PjEwPC9ram9lcmluZ2Vuc0FydD4KICAgICAgICA8a2pvZXJldG9leT4KICAgICAgICAgICAgPGVpZXJza2FwUmVnaXN0cmVydD4yMDE3LTEwLTIwPC9laWVyc2thcFJlZ2lzdHJlcnQ+CiAgICAgICAgICAgIDxmb2Vyc3RlUmVnaXN0cmVyaW5nc2Fhcj4yMDE3PC9mb2Vyc3RlUmVnaXN0cmVyaW5nc2Fhcj4KICAgICAgICAgICAgPGZvZXJzdGVSZWdpc3RyZXJpbmdzZGF0b0lOb3JnZT4yMDE3LTEwLTIwPC9mb2Vyc3RlUmVnaXN0cmVyaW5nc2RhdG9JTm9yZ2U+CiAgICAgICAgICAgIDxram9lcmV0b2V5Z3J1cHBlPjEwMTwva2pvZXJldG9leWdydXBwZT4KICAgICAgICAgICAgPGxlbmdkZT4zOTY0PC9sZW5nZGU+CiAgICAgICAgICAgIDxtb3RvcmVmZmVrdD45NjwvbW90b3JlZmZla3Q+CiAgICAgICAgICAgIDxzbGFndm9sdW0+MTQzPC9zbGFndm9sdW0+CiAgICAgICAgICAgIDxkcml2c3RvZmY+QkVOU0lOPC9kcml2c3RvZmY+CiAgICAgICAgICAgIDxlZ2VudmVrdD4xNTE5PC9lZ2VudmVrdD4KICAgICAgICAgICAgPGVpZXI+CiAgICAgICAgICAgICAgICA8Zm9lZHNlbHNFbGxlckRudW1tZXI+MTEyMjMzNDQ1NTwvZm9lZHNlbHNFbGxlckRudW1tZXI+CiAgICAgICAgICAgICAgICA8cGFydHNudW1tZXI+NTU0NDMzMjIxMTwvcGFydHNudW1tZXI+CiAgICAgICAgICAgICAgICA8bmF2bj5LTEFSQSBLVTwvbmF2bj4KICAgICAgICAgICAgPC9laWVyPgogICAgICAgICAgICA8dGlsbGF0dFRvdGFsdmVrdD4yMTY0PC90aWxsYXR0VG90YWx2ZWt0PgogICAgICAgICAgICA8aHlicmlkPm5laTwvaHlicmlkPgogICAgICAgICAgICA8Y28ydXRzbGlwcD4yNjg8L2NvMnV0c2xpcHA+CiAgICAgICAgICAgIDxub3h1dHNsaXBwPjU5LjQ8L25veHV0c2xpcHA+CiAgICAgICAgICAgIDxram9lcmV0b2V5aWRlbnRpZmlrYXRvcj4KICAgICAgICAgICAgICAgIDxram9lcmV0b2V5VW5pa0lkZW50aWZpa2F0b3I+QUJDREVGR0hJSjwva2pvZXJldG9leVVuaWtJZGVudGlmaWthdG9yPgogICAgICAgICAgICAgICAgPGtqZW5uZW1lcmtlPjQ0QklMMTwva2plbm5lbWVya2U+CiAgICAgICAgICAgICAgICA8dW5kZXJzdGVsbHNudW1tZXI+VU5ERVJTVEVMTDQ0PC91bmRlcnN0ZWxsc251bW1lcj4KICAgICAgICAgICAgPC9ram9lcmV0b2V5aWRlbnRpZmlrYXRvcj4KICAgICAgICA8L2tqb2VyZXRvZXk+CiAgICA8L2dydW5ubGFnRm9yTW90b3J2b2duYXZnaWZ0PgogICAgPGF2Z2lmdHNwbGlrdGlnPgogICAgICAgIDxmb2Vkc2Vsc0VsbGVyRG51bW1lcj4xMTIyMzM0NDU1PC9mb2Vkc2Vsc0VsbGVyRG51bW1lcj4KICAgICAgICA8cGFydHNudW1tZXI+NTU0NDMzMjIxMTwvcGFydHNudW1tZXI+CiAgICA8L2F2Z2lmdHNwbGlrdGlnPgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzI8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjQ3NTMzNy4yMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5FZ2VudmVrdDwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+NjA5NDUuNjQ8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+TW90b3JlZmZla3Q8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94PC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD40MjEzLjI0PC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlNsYWd2b2x1bTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzIgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+RWdlbnZla3QgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+TW90b3JlZmZla3QgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94IEZyYXRyZWtrPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4wLjAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlNsYWd2b2x1bSBGcmF0cmVrazwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzIgU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD40NzUzMzcuMjA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+RWdlbnZla3QgU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD42MDk0NS42NDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5Nb3RvcmVmZmVrdCBTdW08L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94IFN1bTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+NDIxMy4yNDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5TbGFndm9sdW0gU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4wLjAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlZyYWtwYW50PC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4yNDAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PkJydWtzZnJhZHJhZyAwJTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGZvcmhhYW5kc2JlcmVnbmluZz5mYWxzZTwvZm9yaGFhbmRzYmVyZWduaW5nPgo8L21vdG9ydm9nbmF2Z2lmdD4K"
  },
  "extension": null,
  "skjemaversjon": "v3_0"
}
 
 
 
 
Cheers,
Bjørn
 


Denne e-posten og eventuelle vedlegg er beregnet utelukkende for den institusjon eller person den er rettet til og kan være belagt med lovbestemt taushetsplikt. Dersom e-posten er feilsendt, vennligst slett den og kontakt Skatteetaten. 
The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential information and may be legally protected from disclosure. If you are not the intended recipient of this message, please immediately delete the message and alert the Norwegian Tax Administration. 

Reply | Threaded
Open this post in threaded view
|

SV: Replacing a base64-encoded field in a JSON-document with its decoded/converted value

Myklebust, Bjørn Magnar

Thanks Andy.

The XML-content is around 5 kB-ish.  But I also need to convert the XML to JSON before replacing it back into the original JSON-file.  Can this be done with e.g a ConvertAttribute before the ReplaceText?

 

Thanks,

Bjørn

 

 

 

Fra: Andy LoPresto <[hidden email]>
Sendt: onsdag 24. juni 2020 17:24
Til: [hidden email]
Emne: Re: Replacing a base64-encoded field in a JSON-document with its decoded/converted value

 

Hello Bjørn, 

 

If the size of the encoded XML document is small (under ~1 KB), you can extract the Base64-encoded value to a flowfile attribute using EvaluateJSONPath, perform the decoding using the base64Decode Expression Language function [1], and then replace it into the flowfile JSON content using ReplaceText (using some regex like "content": ".*" -> “content": ”${decodedXML}” where decodedXML is the name of the attribute you are using). 

 

If the XML content could be very large, this will negatively affect your performance, as attributes are stored directly in memory and handling large amounts of data will impact the heap. In this case, I would recommend writing a Groovy script in ExecuteScript processor to leverage Groovy’s very friendly JSON handling and extract the value, Base64 decode it, and replace it in a couple lines. 

 

Hope this helps. 

 

 

 

Andy LoPresto
[hidden email]
[hidden email]
He/Him

PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69



On Jun 24, 2020, at 4:24 AM, Myklebust, Bjørn Magnar <[hidden email]> wrote:

 

 

Hi.

I have a set of Json-files which contain a base64-coded field (Jsonpath to this field is $.data.content), and this field contains a XML-document.  Decoding the field works as expected, so does the conversion from xml to json,  and I'm able to write the content from this field to a file in a bucket in S3.  But what I would like to do is to be able to replace the coded value for this field in the original file with the decoded/converted value in stead of writing the decoded/converted value to file. And after replacing the json-value then I can write the updated Json-file to a new S3 bucket.

My process look like this at the moment, and works fine for getting the data to file, but it's missing the last part of replacing $.data.content with the decoded/converted data.

 

So how can I do the last part?

 

<pastedImage.png>

 

The EvaluedJsonPath looks like this:

 

<EvaluateJsonPath.png>

 

The ReplaceText looks like this:

 

<ReplaceText.png>

The Base64EncodeContent looks like this:

 

<Base64Decode.png>

and finally, the CovertRecord looks like this:

 

<ConvertRecord.png>

 

 

This is a testfile for that I'm working with:

 

{

  "header": {

    "dokumentidentifikator": null,

    "dokumentidentifikatorV2": "dcff985b-c652-4085-b8f1-45a2f4b6d150",

    "revisjonsnummer": 1,

    "dokumentnavn": "Engangsavgiftfastsettelse:55TEST661122334455:44BIL1:2017-10-20",

    "dokumenttype": "SKATTEMELDING_ENGANGSAVGIFT",

    "dokumenttilstand": "OPPRETTET",

    "gyldig": true,

    "gjelderInntektsaar": 2017,

    "gjelderPeriode": "2017_10",

    "gjelderPart": {

      "partsnummer": 5544332211,

      "identifiseringstype": "MASKINELL",

      "identifikator": null

    },

    "opphavspart": {

      "partsnummer": 5544332211,

      "identifikator": null

    },

    "kildereferanse": {

      "kildesystem": "ENGANGSAVGIFTFASTSETTELSE",

      "gruppe": "",

      "referanse": "aef147fb-8ce8-43ef-833b-7aa3bac1ece0",

      "tidspunkt": "2018-01-16T13:28:02.49+01:00"

    }

  },

  "data": {

    "metadata": {

      "format": "ske:fastsetting:motorvogn:motorvognavgift:v1",

      "bytes": 4420,

      "mimeType": "application/xml",

      "sha1": "c0AowOsTdNdo6VufeSsZqTphc0Y="

    },

    "content": "PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiIHN0YW5kYWxvbmU9InllcyI/Pgo8bW90b3J2b2duYXZnaWZ0IHhtbG5zPSJza2U6ZmFzdHNldHRpbmc6bW90b3J2b2duOm1vdG9ydm9nbmF2Z2lmdDp2MSI+CiAgICA8YXZnaWZ0c2xpbmplPgogICAgICAgIDxhdmdpZnRzYmVsb2VwPjU0Mjg5Ni4wMDwvYXZnaWZ0c2JlbG9lcD4KICAgICAgICA8YXZnaWZ0c29wcGx5c25pbmc+CiAgICAgICAgICAgIDxzYWVyYXZnaWZ0VHlwZWtvZGU+QkI8L3NhZXJhdmdpZnRUeXBla29kZT4KICAgICAgICAgICAgPHNhZXJhdmdpZnRHcnVwcGVrb2RlPlg8L3NhZXJhdmdpZnRHcnVwcGVrb2RlPgogICAgICAgIDwvYXZnaWZ0c29wcGx5c25pbmc+CiAgICAgICAgPGF2Z2lmdHNkYXRvPjIwMTctMTAtMjA8L2F2Z2lmdHNkYXRvPgogICAgPC9hdmdpZnRzbGluamU+CiAgICA8YmV0YWxpbmdzaW5mb3JtYXNqb24+CiAgICAgICAgPGtpZG51bW1lcj4xMDEwMTAxMDEwMTA8L2tpZG51bW1lcj4KICAgICAgICA8Zm9yZmFsbHNkYXRvPjIwMTctMTAtMjA8L2ZvcmZhbGxzZGF0bz4KICAgICAgICA8ZmFrdHVyYWRhdG8+MjAxNy0xMC0yMDwvZmFrdHVyYWRhdG8+CiAgICAgICAgPHRvdGFsQXZnaWZ0c2JlbG9lcD41NDI4OTYuMDA8L3RvdGFsQXZnaWZ0c2JlbG9lcD4KICAgIDwvYmV0YWxpbmdzaW5mb3JtYXNqb24+CiAgICA8bW90b3J2b2duYXZnaWZ0c3R5cGU+ZW5nYW5nc2F2Z2lmdDwvbW90b3J2b2duYXZnaWZ0c3R5cGU+CiAgICA8dGlkc3N0ZW1wZWw+MjAxOC0wMS0xNiswMTowMDwvdGlkc3N0ZW1wZWw+CiAgICA8Z3J1bm5sYWdGb3JNb3RvcnZvZ25hdmdpZnQ+CiAgICAgICAgPGtqb2VyaW5nZW5zQXJ0PjEwPC9ram9lcmluZ2Vuc0FydD4KICAgICAgICA8a2pvZXJldG9leT4KICAgICAgICAgICAgPGVpZXJza2FwUmVnaXN0cmVydD4yMDE3LTEwLTIwPC9laWVyc2thcFJlZ2lzdHJlcnQ+CiAgICAgICAgICAgIDxmb2Vyc3RlUmVnaXN0cmVyaW5nc2Fhcj4yMDE3PC9mb2Vyc3RlUmVnaXN0cmVyaW5nc2Fhcj4KICAgICAgICAgICAgPGZvZXJzdGVSZWdpc3RyZXJpbmdzZGF0b0lOb3JnZT4yMDE3LTEwLTIwPC9mb2Vyc3RlUmVnaXN0cmVyaW5nc2RhdG9JTm9yZ2U+CiAgICAgICAgICAgIDxram9lcmV0b2V5Z3J1cHBlPjEwMTwva2pvZXJldG9leWdydXBwZT4KICAgICAgICAgICAgPGxlbmdkZT4zOTY0PC9sZW5nZGU+CiAgICAgICAgICAgIDxtb3RvcmVmZmVrdD45NjwvbW90b3JlZmZla3Q+CiAgICAgICAgICAgIDxzbGFndm9sdW0+MTQzPC9zbGFndm9sdW0+CiAgICAgICAgICAgIDxkcml2c3RvZmY+QkVOU0lOPC9kcml2c3RvZmY+CiAgICAgICAgICAgIDxlZ2VudmVrdD4xNTE5PC9lZ2VudmVrdD4KICAgICAgICAgICAgPGVpZXI+CiAgICAgICAgICAgICAgICA8Zm9lZHNlbHNFbGxlckRudW1tZXI+MTEyMjMzNDQ1NTwvZm9lZHNlbHNFbGxlckRudW1tZXI+CiAgICAgICAgICAgICAgICA8cGFydHNudW1tZXI+NTU0NDMzMjIxMTwvcGFydHNudW1tZXI+CiAgICAgICAgICAgICAgICA8bmF2bj5LTEFSQSBLVTwvbmF2bj4KICAgICAgICAgICAgPC9laWVyPgogICAgICAgICAgICA8dGlsbGF0dFRvdGFsdmVrdD4yMTY0PC90aWxsYXR0VG90YWx2ZWt0PgogICAgICAgICAgICA8aHlicmlkPm5laTwvaHlicmlkPgogICAgICAgICAgICA8Y28ydXRzbGlwcD4yNjg8L2NvMnV0c2xpcHA+CiAgICAgICAgICAgIDxub3h1dHNsaXBwPjU5LjQ8L25veHV0c2xpcHA+CiAgICAgICAgICAgIDxram9lcmV0b2V5aWRlbnRpZmlrYXRvcj4KICAgICAgICAgICAgICAgIDxram9lcmV0b2V5VW5pa0lkZW50aWZpa2F0b3I+QUJDREVGR0hJSjwva2pvZXJldG9leVVuaWtJZGVudGlmaWthdG9yPgogICAgICAgICAgICAgICAgPGtqZW5uZW1lcmtlPjQ0QklMMTwva2plbm5lbWVya2U+CiAgICAgICAgICAgICAgICA8dW5kZXJzdGVsbHNudW1tZXI+VU5ERVJTVEVMTDQ0PC91bmRlcnN0ZWxsc251bW1lcj4KICAgICAgICAgICAgPC9ram9lcmV0b2V5aWRlbnRpZmlrYXRvcj4KICAgICAgICA8L2tqb2VyZXRvZXk+CiAgICA8L2dydW5ubGFnRm9yTW90b3J2b2duYXZnaWZ0PgogICAgPGF2Z2lmdHNwbGlrdGlnPgogICAgICAgIDxmb2Vkc2Vsc0VsbGVyRG51bW1lcj4xMTIyMzM0NDU1PC9mb2Vkc2Vsc0VsbGVyRG51bW1lcj4KICAgICAgICA8cGFydHNudW1tZXI+NTU0NDMzMjIxMTwvcGFydHNudW1tZXI+CiAgICA8L2F2Z2lmdHNwbGlrdGlnPgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzI8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjQ3NTMzNy4yMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5FZ2VudmVrdDwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+NjA5NDUuNjQ8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+TW90b3JlZmZla3Q8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94PC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD40MjEzLjI0PC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlNsYWd2b2x1bTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzIgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+RWdlbnZla3QgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+TW90b3JlZmZla3QgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94IEZyYXRyZWtrPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4wLjAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlNsYWd2b2x1bSBGcmF0cmVrazwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzIgU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD40NzUzMzcuMjA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+RWdlbnZla3QgU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD42MDk0NS42NDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5Nb3RvcmVmZmVrdCBTdW08L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94IFN1bTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+NDIxMy4yNDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5TbGFndm9sdW0gU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4wLjAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlZyYWtwYW50PC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4yNDAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PkJydWtzZnJhZHJhZyAwJTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGZvcmhhYW5kc2JlcmVnbmluZz5mYWxzZTwvZm9yaGFhbmRzYmVyZWduaW5nPgo8L21vdG9ydm9nbmF2Z2lmdD4K"

  },

  "extension": null,

  "skjemaversjon": "v3_0"

}

 

 

 

 

Cheers,

Bjørn

 

 


Denne e-posten og eventuelle vedlegg er beregnet utelukkende for den institusjon eller person den er rettet til og kan være belagt med lovbestemt taushetsplikt. Dersom e-posten er feilsendt, vennligst slett den og kontakt Skatteetaten. 
The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential information and may be legally protected from disclosure. If you are not the intended recipient of this message, please immediately delete the message and alert the Norwegian Tax Administration. 

 

Reply | Threaded
Open this post in threaded view
|

Re: Replacing a base64-encoded field in a JSON-document with its decoded/converted value

Andy LoPresto
Hi Bjørn,

No, XML to JSON conversion is not an Expression Language feature. You’ll need to either get this data into a flowfile as the complete content to perform the conversion with existing built-in tools, or add that step to your Groovy script. 

With that additional requirement, I think using the Groovy script to perform those steps in tandem is probably the most performant and logical approach here. 


Andy LoPresto
[hidden email]
[hidden email]
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Jun 24, 2020, at 11:25 PM, Myklebust, Bjørn Magnar <[hidden email]> wrote:

Thanks Andy.
The XML-content is around 5 kB-ish.  But I also need to convert the XML to JSON before replacing it back into the original JSON-file.  Can this be done with e.g a ConvertAttribute before the ReplaceText?
 
Thanks,
Bjørn
 
 
 
Fra: Andy LoPresto <[hidden email]> 
Sendt: onsdag 24. juni 2020 17:24
Til: [hidden email]
Emne: Re: Replacing a base64-encoded field in a JSON-document with its decoded/converted value
 
Hello Bjørn, 
 
If the size of the encoded XML document is small (under ~1 KB), you can extract the Base64-encoded value to a flowfile attribute using EvaluateJSONPath, perform the decoding using the base64Decode Expression Language function [1], and then replace it into the flowfile JSON content using ReplaceText (using some regex like "content": ".*" -> “content": ”${decodedXML}” where decodedXML is the name of the attribute you are using). 
 
If the XML content could be very large, this will negatively affect your performance, as attributes are stored directly in memory and handling large amounts of data will impact the heap. In this case, I would recommend writing a Groovy script in ExecuteScript processor to leverage Groovy’s very friendly JSON handling and extract the value, Base64 decode it, and replace it in a couple lines. 
 
Hope this helps. 
 
 
 
Andy LoPresto
[hidden email]
[hidden email]
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69


On Jun 24, 2020, at 4:24 AM, Myklebust, Bjørn Magnar <[hidden email]> wrote:
 
 
Hi.
I have a set of Json-files which contain a base64-coded field (Jsonpath to this field is $.data.content), and this field contains a XML-document.  Decoding the field works as expected, so does the conversion from xml to json,  and I'm able to write the content from this field to a file in a bucket in S3.  But what I would like to do is to be able to replace the coded value for this field in the original file with the decoded/converted value in stead of writing the decoded/converted value to file. And after replacing the json-value then I can write the updated Json-file to a new S3 bucket.
My process look like this at the moment, and works fine for getting the data to file, but it's missing the last part of replacing $.data.content with the decoded/converted data.
 
So how can I do the last part?
 
<pastedImage.png>
 
The EvaluedJsonPath looks like this:
 

<EvaluateJsonPath.png>

 
The ReplaceText looks like this:
 

<ReplaceText.png>

The Base64EncodeContent looks like this:
 

<Base64Decode.png>

and finally, the CovertRecord looks like this:
 

<ConvertRecord.png>

 
 
This is a testfile for that I'm working with:
 
{
  "header": {
    "dokumentidentifikator": null,
    "dokumentidentifikatorV2": "dcff985b-c652-4085-b8f1-45a2f4b6d150",
    "revisjonsnummer": 1,
    "dokumentnavn": "Engangsavgiftfastsettelse:55TEST661122334455:44BIL1:2017-10-20",
    "dokumenttype": "SKATTEMELDING_ENGANGSAVGIFT",
    "dokumenttilstand": "OPPRETTET",
    "gyldig": true,
    "gjelderInntektsaar": 2017,
    "gjelderPeriode": "2017_10",
    "gjelderPart": {
      "partsnummer": 5544332211,
      "identifiseringstype": "MASKINELL",
      "identifikator": null
    },
    "opphavspart": {
      "partsnummer": 5544332211,
      "identifikator": null
    },
    "kildereferanse": {
      "kildesystem": "ENGANGSAVGIFTFASTSETTELSE",
      "gruppe": "",
      "referanse": "aef147fb-8ce8-43ef-833b-7aa3bac1ece0",
      "tidspunkt": "2018-01-16T13:28:02.49+01:00"
    }
  },
  "data": {
    "metadata": {
      "format": "ske:fastsetting:motorvogn:motorvognavgift:v1",
      "bytes": 4420,
      "mimeType": "application/xml",
      "sha1": "c0AowOsTdNdo6VufeSsZqTphc0Y="
    },
    "content": "PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiIHN0YW5kYWxvbmU9InllcyI/Pgo8bW90b3J2b2duYXZnaWZ0IHhtbG5zPSJza2U6ZmFzdHNldHRpbmc6bW90b3J2b2duOm1vdG9ydm9nbmF2Z2lmdDp2MSI+CiAgICA8YXZnaWZ0c2xpbmplPgogICAgICAgIDxhdmdpZnRzYmVsb2VwPjU0Mjg5Ni4wMDwvYXZnaWZ0c2JlbG9lcD4KICAgICAgICA8YXZnaWZ0c29wcGx5c25pbmc+CiAgICAgICAgICAgIDxzYWVyYXZnaWZ0VHlwZWtvZGU+QkI8L3NhZXJhdmdpZnRUeXBla29kZT4KICAgICAgICAgICAgPHNhZXJhdmdpZnRHcnVwcGVrb2RlPlg8L3NhZXJhdmdpZnRHcnVwcGVrb2RlPgogICAgICAgIDwvYXZnaWZ0c29wcGx5c25pbmc+CiAgICAgICAgPGF2Z2lmdHNkYXRvPjIwMTctMTAtMjA8L2F2Z2lmdHNkYXRvPgogICAgPC9hdmdpZnRzbGluamU+CiAgICA8YmV0YWxpbmdzaW5mb3JtYXNqb24+CiAgICAgICAgPGtpZG51bW1lcj4xMDEwMTAxMDEwMTA8L2tpZG51bW1lcj4KICAgICAgICA8Zm9yZmFsbHNkYXRvPjIwMTctMTAtMjA8L2ZvcmZhbGxzZGF0bz4KICAgICAgICA8ZmFrdHVyYWRhdG8+MjAxNy0xMC0yMDwvZmFrdHVyYWRhdG8+CiAgICAgICAgPHRvdGFsQXZnaWZ0c2JlbG9lcD41NDI4OTYuMDA8L3RvdGFsQXZnaWZ0c2JlbG9lcD4KICAgIDwvYmV0YWxpbmdzaW5mb3JtYXNqb24+CiAgICA8bW90b3J2b2duYXZnaWZ0c3R5cGU+ZW5nYW5nc2F2Z2lmdDwvbW90b3J2b2duYXZnaWZ0c3R5cGU+CiAgICA8dGlkc3N0ZW1wZWw+MjAxOC0wMS0xNiswMTowMDwvdGlkc3N0ZW1wZWw+CiAgICA8Z3J1bm5sYWdGb3JNb3RvcnZvZ25hdmdpZnQ+CiAgICAgICAgPGtqb2VyaW5nZW5zQXJ0PjEwPC9ram9lcmluZ2Vuc0FydD4KICAgICAgICA8a2pvZXJldG9leT4KICAgICAgICAgICAgPGVpZXJza2FwUmVnaXN0cmVydD4yMDE3LTEwLTIwPC9laWVyc2thcFJlZ2lzdHJlcnQ+CiAgICAgICAgICAgIDxmb2Vyc3RlUmVnaXN0cmVyaW5nc2Fhcj4yMDE3PC9mb2Vyc3RlUmVnaXN0cmVyaW5nc2Fhcj4KICAgICAgICAgICAgPGZvZXJzdGVSZWdpc3RyZXJpbmdzZGF0b0lOb3JnZT4yMDE3LTEwLTIwPC9mb2Vyc3RlUmVnaXN0cmVyaW5nc2RhdG9JTm9yZ2U+CiAgICAgICAgICAgIDxram9lcmV0b2V5Z3J1cHBlPjEwMTwva2pvZXJldG9leWdydXBwZT4KICAgICAgICAgICAgPGxlbmdkZT4zOTY0PC9sZW5nZGU+CiAgICAgICAgICAgIDxtb3RvcmVmZmVrdD45NjwvbW90b3JlZmZla3Q+CiAgICAgICAgICAgIDxzbGFndm9sdW0+MTQzPC9zbGFndm9sdW0+CiAgICAgICAgICAgIDxkcml2c3RvZmY+QkVOU0lOPC9kcml2c3RvZmY+CiAgICAgICAgICAgIDxlZ2VudmVrdD4xNTE5PC9lZ2VudmVrdD4KICAgICAgICAgICAgPGVpZXI+CiAgICAgICAgICAgICAgICA8Zm9lZHNlbHNFbGxlckRudW1tZXI+MTEyMjMzNDQ1NTwvZm9lZHNlbHNFbGxlckRudW1tZXI+CiAgICAgICAgICAgICAgICA8cGFydHNudW1tZXI+NTU0NDMzMjIxMTwvcGFydHNudW1tZXI+CiAgICAgICAgICAgICAgICA8bmF2bj5LTEFSQSBLVTwvbmF2bj4KICAgICAgICAgICAgPC9laWVyPgogICAgICAgICAgICA8dGlsbGF0dFRvdGFsdmVrdD4yMTY0PC90aWxsYXR0VG90YWx2ZWt0PgogICAgICAgICAgICA8aHlicmlkPm5laTwvaHlicmlkPgogICAgICAgICAgICA8Y28ydXRzbGlwcD4yNjg8L2NvMnV0c2xpcHA+CiAgICAgICAgICAgIDxub3h1dHNsaXBwPjU5LjQ8L25veHV0c2xpcHA+CiAgICAgICAgICAgIDxram9lcmV0b2V5aWRlbnRpZmlrYXRvcj4KICAgICAgICAgICAgICAgIDxram9lcmV0b2V5VW5pa0lkZW50aWZpa2F0b3I+QUJDREVGR0hJSjwva2pvZXJldG9leVVuaWtJZGVudGlmaWthdG9yPgogICAgICAgICAgICAgICAgPGtqZW5uZW1lcmtlPjQ0QklMMTwva2plbm5lbWVya2U+CiAgICAgICAgICAgICAgICA8dW5kZXJzdGVsbHNudW1tZXI+VU5ERVJTVEVMTDQ0PC91bmRlcnN0ZWxsc251bW1lcj4KICAgICAgICAgICAgPC9ram9lcmV0b2V5aWRlbnRpZmlrYXRvcj4KICAgICAgICA8L2tqb2VyZXRvZXk+CiAgICA8L2dydW5ubGFnRm9yTW90b3J2b2duYXZnaWZ0PgogICAgPGF2Z2lmdHNwbGlrdGlnPgogICAgICAgIDxmb2Vkc2Vsc0VsbGVyRG51bW1lcj4xMTIyMzM0NDU1PC9mb2Vkc2Vsc0VsbGVyRG51bW1lcj4KICAgICAgICA8cGFydHNudW1tZXI+NTU0NDMzMjIxMTwvcGFydHNudW1tZXI+CiAgICA8L2F2Z2lmdHNwbGlrdGlnPgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzI8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjQ3NTMzNy4yMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5FZ2VudmVrdDwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+NjA5NDUuNjQ8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+TW90b3JlZmZla3Q8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94PC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD40MjEzLjI0PC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlNsYWd2b2x1bTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzIgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+RWdlbnZla3QgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+TW90b3JlZmZla3QgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94IEZyYXRyZWtrPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4wLjAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlNsYWd2b2x1bSBGcmF0cmVrazwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzIgU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD40NzUzMzcuMjA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+RWdlbnZla3QgU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD42MDk0NS42NDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5Nb3RvcmVmZmVrdCBTdW08L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94IFN1bTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+NDIxMy4yNDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5TbGFndm9sdW0gU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4wLjAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlZyYWtwYW50PC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4yNDAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PkJydWtzZnJhZHJhZyAwJTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGZvcmhhYW5kc2JlcmVnbmluZz5mYWxzZTwvZm9yaGFhbmRzYmVyZWduaW5nPgo8L21vdG9ydm9nbmF2Z2lmdD4K"
  },
  "extension": null,
  "skjemaversjon": "v3_0"
}
 
 
 
 
Cheers,
Bjørn
 
 

Denne e-posten og eventuelle vedlegg er beregnet utelukkende for den institusjon eller person den er rettet til og kan være belagt med lovbestemt taushetsplikt. Dersom e-posten er feilsendt, vennligst slett den og kontakt Skatteetaten. 
The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential information and may be legally protected from disclosure. If you are not the intended recipient of this message, please immediately delete the message and alert the Norwegian Tax Administration.

Reply | Threaded
Open this post in threaded view
|

SV: Replacing a base64-encoded field in a JSON-document with its decoded/converted value

Myklebust, Bjørn Magnar

Ok, I see.

Thanks, Andy.

 

Bjørn

 

Fra: Andy LoPresto <[hidden email]>
Sendt: torsdag 25. juni 2020 19:20
Til: [hidden email]
Emne: Re: Replacing a base64-encoded field in a JSON-document with its decoded/converted value

 

Hi Bjørn,

 

No, XML to JSON conversion is not an Expression Language feature. You’ll need to either get this data into a flowfile as the complete content to perform the conversion with existing built-in tools, or add that step to your Groovy script. 

 

With that additional requirement, I think using the Groovy script to perform those steps in tandem is probably the most performant and logical approach here. 

 

 

Andy LoPresto
[hidden email]
[hidden email]
He/Him

PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69



On Jun 24, 2020, at 11:25 PM, Myklebust, Bjørn Magnar <[hidden email]> wrote:

 

Thanks Andy.

The XML-content is around 5 kB-ish.  But I also need to convert the XML to JSON before replacing it back into the original JSON-file.  Can this be done with e.g a ConvertAttribute before the ReplaceText?

 

Thanks,

Bjørn

 

 

 

Fra: Andy LoPresto <[hidden email]> 
Sendt: onsdag 24. juni 2020 17:24
Til: [hidden email]
Emne: Re: Replacing a base64-encoded field in a JSON-document with its decoded/converted value

 

Hello Bjørn, 

 

If the size of the encoded XML document is small (under ~1 KB), you can extract the Base64-encoded value to a flowfile attribute using EvaluateJSONPath, perform the decoding using the base64Decode Expression Language function [1], and then replace it into the flowfile JSON content using ReplaceText (using some regex like "content": ".*" -> “content": ”${decodedXML}” where decodedXML is the name of the attribute you are using). 

 

If the XML content could be very large, this will negatively affect your performance, as attributes are stored directly in memory and handling large amounts of data will impact the heap. In this case, I would recommend writing a Groovy script in ExecuteScript processor to leverage Groovy’s very friendly JSON handling and extract the value, Base64 decode it, and replace it in a couple lines. 

 

Hope this helps. 

 

 

 

Andy LoPresto
[hidden email]
[hidden email]
He/Him

PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69




On Jun 24, 2020, at 4:24 AM, Myklebust, Bjørn Magnar <[hidden email]> wrote:

 

 

Hi.

I have a set of Json-files which contain a base64-coded field (Jsonpath to this field is $.data.content), and this field contains a XML-document.  Decoding the field works as expected, so does the conversion from xml to json,  and I'm able to write the content from this field to a file in a bucket in S3.  But what I would like to do is to be able to replace the coded value for this field in the original file with the decoded/converted value in stead of writing the decoded/converted value to file. And after replacing the json-value then I can write the updated Json-file to a new S3 bucket.

My process look like this at the moment, and works fine for getting the data to file, but it's missing the last part of replacing $.data.content with the decoded/converted data.

 

So how can I do the last part?

 

<pastedImage.png>

 

The EvaluedJsonPath looks like this:

 

<EvaluateJsonPath.png>

 

The ReplaceText looks like this:

 

<ReplaceText.png>

The Base64EncodeContent looks like this:

 

<Base64Decode.png>

and finally, the CovertRecord looks like this:

 

<ConvertRecord.png>

 

 

This is a testfile for that I'm working with:

 

{

  "header": {

    "dokumentidentifikator": null,

    "dokumentidentifikatorV2": "dcff985b-c652-4085-b8f1-45a2f4b6d150",

    "revisjonsnummer": 1,

    "dokumentnavn": "Engangsavgiftfastsettelse:55TEST661122334455:44BIL1:2017-10-20",

    "dokumenttype": "SKATTEMELDING_ENGANGSAVGIFT",

    "dokumenttilstand": "OPPRETTET",

    "gyldig": true,

    "gjelderInntektsaar": 2017,

    "gjelderPeriode": "2017_10",

    "gjelderPart": {

      "partsnummer": 5544332211,

      "identifiseringstype": "MASKINELL",

      "identifikator": null

    },

    "opphavspart": {

      "partsnummer": 5544332211,

      "identifikator": null

    },

    "kildereferanse": {

      "kildesystem": "ENGANGSAVGIFTFASTSETTELSE",

      "gruppe": "",

      "referanse": "aef147fb-8ce8-43ef-833b-7aa3bac1ece0",

      "tidspunkt": "2018-01-16T13:28:02.49+01:00"

    }

  },

  "data": {

    "metadata": {

      "format": "ske:fastsetting:motorvogn:motorvognavgift:v1",

      "bytes": 4420,

      "mimeType": "application/xml",

      "sha1": "c0AowOsTdNdo6VufeSsZqTphc0Y="

    },

    "content": "PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiIHN0YW5kYWxvbmU9InllcyI/Pgo8bW90b3J2b2duYXZnaWZ0IHhtbG5zPSJza2U6ZmFzdHNldHRpbmc6bW90b3J2b2duOm1vdG9ydm9nbmF2Z2lmdDp2MSI+CiAgICA8YXZnaWZ0c2xpbmplPgogICAgICAgIDxhdmdpZnRzYmVsb2VwPjU0Mjg5Ni4wMDwvYXZnaWZ0c2JlbG9lcD4KICAgICAgICA8YXZnaWZ0c29wcGx5c25pbmc+CiAgICAgICAgICAgIDxzYWVyYXZnaWZ0VHlwZWtvZGU+QkI8L3NhZXJhdmdpZnRUeXBla29kZT4KICAgICAgICAgICAgPHNhZXJhdmdpZnRHcnVwcGVrb2RlPlg8L3NhZXJhdmdpZnRHcnVwcGVrb2RlPgogICAgICAgIDwvYXZnaWZ0c29wcGx5c25pbmc+CiAgICAgICAgPGF2Z2lmdHNkYXRvPjIwMTctMTAtMjA8L2F2Z2lmdHNkYXRvPgogICAgPC9hdmdpZnRzbGluamU+CiAgICA8YmV0YWxpbmdzaW5mb3JtYXNqb24+CiAgICAgICAgPGtpZG51bW1lcj4xMDEwMTAxMDEwMTA8L2tpZG51bW1lcj4KICAgICAgICA8Zm9yZmFsbHNkYXRvPjIwMTctMTAtMjA8L2ZvcmZhbGxzZGF0bz4KICAgICAgICA8ZmFrdHVyYWRhdG8+MjAxNy0xMC0yMDwvZmFrdHVyYWRhdG8+CiAgICAgICAgPHRvdGFsQXZnaWZ0c2JlbG9lcD41NDI4OTYuMDA8L3RvdGFsQXZnaWZ0c2JlbG9lcD4KICAgIDwvYmV0YWxpbmdzaW5mb3JtYXNqb24+CiAgICA8bW90b3J2b2duYXZnaWZ0c3R5cGU+ZW5nYW5nc2F2Z2lmdDwvbW90b3J2b2duYXZnaWZ0c3R5cGU+CiAgICA8dGlkc3N0ZW1wZWw+MjAxOC0wMS0xNiswMTowMDwvdGlkc3N0ZW1wZWw+CiAgICA8Z3J1bm5sYWdGb3JNb3RvcnZvZ25hdmdpZnQ+CiAgICAgICAgPGtqb2VyaW5nZW5zQXJ0PjEwPC9ram9lcmluZ2Vuc0FydD4KICAgICAgICA8a2pvZXJldG9leT4KICAgICAgICAgICAgPGVpZXJza2FwUmVnaXN0cmVydD4yMDE3LTEwLTIwPC9laWVyc2thcFJlZ2lzdHJlcnQ+CiAgICAgICAgICAgIDxmb2Vyc3RlUmVnaXN0cmVyaW5nc2Fhcj4yMDE3PC9mb2Vyc3RlUmVnaXN0cmVyaW5nc2Fhcj4KICAgICAgICAgICAgPGZvZXJzdGVSZWdpc3RyZXJpbmdzZGF0b0lOb3JnZT4yMDE3LTEwLTIwPC9mb2Vyc3RlUmVnaXN0cmVyaW5nc2RhdG9JTm9yZ2U+CiAgICAgICAgICAgIDxram9lcmV0b2V5Z3J1cHBlPjEwMTwva2pvZXJldG9leWdydXBwZT4KICAgICAgICAgICAgPGxlbmdkZT4zOTY0PC9sZW5nZGU+CiAgICAgICAgICAgIDxtb3RvcmVmZmVrdD45NjwvbW90b3JlZmZla3Q+CiAgICAgICAgICAgIDxzbGFndm9sdW0+MTQzPC9zbGFndm9sdW0+CiAgICAgICAgICAgIDxkcml2c3RvZmY+QkVOU0lOPC9kcml2c3RvZmY+CiAgICAgICAgICAgIDxlZ2VudmVrdD4xNTE5PC9lZ2VudmVrdD4KICAgICAgICAgICAgPGVpZXI+CiAgICAgICAgICAgICAgICA8Zm9lZHNlbHNFbGxlckRudW1tZXI+MTEyMjMzNDQ1NTwvZm9lZHNlbHNFbGxlckRudW1tZXI+CiAgICAgICAgICAgICAgICA8cGFydHNudW1tZXI+NTU0NDMzMjIxMTwvcGFydHNudW1tZXI+CiAgICAgICAgICAgICAgICA8bmF2bj5LTEFSQSBLVTwvbmF2bj4KICAgICAgICAgICAgPC9laWVyPgogICAgICAgICAgICA8dGlsbGF0dFRvdGFsdmVrdD4yMTY0PC90aWxsYXR0VG90YWx2ZWt0PgogICAgICAgICAgICA8aHlicmlkPm5laTwvaHlicmlkPgogICAgICAgICAgICA8Y28ydXRzbGlwcD4yNjg8L2NvMnV0c2xpcHA+CiAgICAgICAgICAgIDxub3h1dHNsaXBwPjU5LjQ8L25veHV0c2xpcHA+CiAgICAgICAgICAgIDxram9lcmV0b2V5aWRlbnRpZmlrYXRvcj4KICAgICAgICAgICAgICAgIDxram9lcmV0b2V5VW5pa0lkZW50aWZpa2F0b3I+QUJDREVGR0hJSjwva2pvZXJldG9leVVuaWtJZGVudGlmaWthdG9yPgogICAgICAgICAgICAgICAgPGtqZW5uZW1lcmtlPjQ0QklMMTwva2plbm5lbWVya2U+CiAgICAgICAgICAgICAgICA8dW5kZXJzdGVsbHNudW1tZXI+VU5ERVJTVEVMTDQ0PC91bmRlcnN0ZWxsc251bW1lcj4KICAgICAgICAgICAgPC9ram9lcmV0b2V5aWRlbnRpZmlrYXRvcj4KICAgICAgICA8L2tqb2VyZXRvZXk+CiAgICA8L2dydW5ubGFnRm9yTW90b3J2b2duYXZnaWZ0PgogICAgPGF2Z2lmdHNwbGlrdGlnPgogICAgICAgIDxmb2Vkc2Vsc0VsbGVyRG51bW1lcj4xMTIyMzM0NDU1PC9mb2Vkc2Vsc0VsbGVyRG51bW1lcj4KICAgICAgICA8cGFydHNudW1tZXI+NTU0NDMzMjIxMTwvcGFydHNudW1tZXI+CiAgICA8L2F2Z2lmdHNwbGlrdGlnPgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzI8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjQ3NTMzNy4yMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5FZ2VudmVrdDwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+NjA5NDUuNjQ8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+TW90b3JlZmZla3Q8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94PC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD40MjEzLjI0PC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlNsYWd2b2x1bTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzIgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+RWdlbnZla3QgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+TW90b3JlZmZla3QgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94IEZyYXRyZWtrPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4wLjAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlNsYWd2b2x1bSBGcmF0cmVrazwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzIgU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD40NzUzMzcuMjA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+RWdlbnZla3QgU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD42MDk0NS42NDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5Nb3RvcmVmZmVrdCBTdW08L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94IFN1bTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+NDIxMy4yNDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5TbGFndm9sdW0gU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4wLjAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlZyYWtwYW50PC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4yNDAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PkJydWtzZnJhZHJhZyAwJTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGZvcmhhYW5kc2JlcmVnbmluZz5mYWxzZTwvZm9yaGFhbmRzYmVyZWduaW5nPgo8L21vdG9ydm9nbmF2Z2lmdD4K"

  },

  "extension": null,

  "skjemaversjon": "v3_0"

}

 

 

 

 

Cheers,

Bjørn

 

 


Denne e-posten og eventuelle vedlegg er beregnet utelukkende for den institusjon eller person den er rettet til og kan være belagt med lovbestemt taushetsplikt. Dersom e-posten er feilsendt, vennligst slett den og kontakt Skatteetaten. 
The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential information and may be legally protected from disclosure. If you are not the intended recipient of this message, please immediately delete the message and alert the Norwegian Tax Administration.

 

Reply | Threaded
Open this post in threaded view
|

Re: Replacing a base64-encoded field in a JSON-document with its decoded/converted value

Pierre Villard
Hi,

I think that would be a perfect use case for NIFI-7572 [1] once there is a solution for it.


Le ven. 26 juin 2020 à 08:20, Myklebust, Bjørn Magnar <[hidden email]> a écrit :

Ok, I see.

Thanks, Andy.

 

Bjørn

 

Fra: Andy LoPresto <[hidden email]>
Sendt: torsdag 25. juni 2020 19:20
Til: [hidden email]
Emne: Re: Replacing a base64-encoded field in a JSON-document with its decoded/converted value

 

Hi Bjørn,

 

No, XML to JSON conversion is not an Expression Language feature. You’ll need to either get this data into a flowfile as the complete content to perform the conversion with existing built-in tools, or add that step to your Groovy script. 

 

With that additional requirement, I think using the Groovy script to perform those steps in tandem is probably the most performant and logical approach here. 

 

 

Andy LoPresto
[hidden email]
[hidden email]
He/Him

PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69



On Jun 24, 2020, at 11:25 PM, Myklebust, Bjørn Magnar <[hidden email]> wrote:

 

Thanks Andy.

The XML-content is around 5 kB-ish.  But I also need to convert the XML to JSON before replacing it back into the original JSON-file.  Can this be done with e.g a ConvertAttribute before the ReplaceText?

 

Thanks,

Bjørn

 

 

 

Fra: Andy LoPresto <[hidden email]> 
Sendt: onsdag 24. juni 2020 17:24
Til: [hidden email]
Emne: Re: Replacing a base64-encoded field in a JSON-document with its decoded/converted value

 

Hello Bjørn, 

 

If the size of the encoded XML document is small (under ~1 KB), you can extract the Base64-encoded value to a flowfile attribute using EvaluateJSONPath, perform the decoding using the base64Decode Expression Language function [1], and then replace it into the flowfile JSON content using ReplaceText (using some regex like "content": ".*" -> “content": ”${decodedXML}” where decodedXML is the name of the attribute you are using). 

 

If the XML content could be very large, this will negatively affect your performance, as attributes are stored directly in memory and handling large amounts of data will impact the heap. In this case, I would recommend writing a Groovy script in ExecuteScript processor to leverage Groovy’s very friendly JSON handling and extract the value, Base64 decode it, and replace it in a couple lines. 

 

Hope this helps. 

 

 

 

Andy LoPresto
[hidden email]
[hidden email]
He/Him

PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69




On Jun 24, 2020, at 4:24 AM, Myklebust, Bjørn Magnar <[hidden email]> wrote:

 

 

Hi.

I have a set of Json-files which contain a base64-coded field (Jsonpath to this field is $.data.content), and this field contains a XML-document.  Decoding the field works as expected, so does the conversion from xml to json,  and I'm able to write the content from this field to a file in a bucket in S3.  But what I would like to do is to be able to replace the coded value for this field in the original file with the decoded/converted value in stead of writing the decoded/converted value to file. And after replacing the json-value then I can write the updated Json-file to a new S3 bucket.

My process look like this at the moment, and works fine for getting the data to file, but it's missing the last part of replacing $.data.content with the decoded/converted data.

 

So how can I do the last part?

 

<pastedImage.png>

 

The EvaluedJsonPath looks like this:

 

<EvaluateJsonPath.png>

 

The ReplaceText looks like this:

 

<ReplaceText.png>

The Base64EncodeContent looks like this:

 

<Base64Decode.png>

and finally, the CovertRecord looks like this:

 

<ConvertRecord.png>

 

 

This is a testfile for that I'm working with:

 

{

  "header": {

    "dokumentidentifikator": null,

    "dokumentidentifikatorV2": "dcff985b-c652-4085-b8f1-45a2f4b6d150",

    "revisjonsnummer": 1,

    "dokumentnavn": "Engangsavgiftfastsettelse:55TEST661122334455:44BIL1:2017-10-20",

    "dokumenttype": "SKATTEMELDING_ENGANGSAVGIFT",

    "dokumenttilstand": "OPPRETTET",

    "gyldig": true,

    "gjelderInntektsaar": 2017,

    "gjelderPeriode": "2017_10",

    "gjelderPart": {

      "partsnummer": 5544332211,

      "identifiseringstype": "MASKINELL",

      "identifikator": null

    },

    "opphavspart": {

      "partsnummer": 5544332211,

      "identifikator": null

    },

    "kildereferanse": {

      "kildesystem": "ENGANGSAVGIFTFASTSETTELSE",

      "gruppe": "",

      "referanse": "aef147fb-8ce8-43ef-833b-7aa3bac1ece0",

      "tidspunkt": "2018-01-16T13:28:02.49+01:00"

    }

  },

  "data": {

    "metadata": {

      "format": "ske:fastsetting:motorvogn:motorvognavgift:v1",

      "bytes": 4420,

      "mimeType": "application/xml",

      "sha1": "c0AowOsTdNdo6VufeSsZqTphc0Y="

    },

    "content": "PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiIHN0YW5kYWxvbmU9InllcyI/Pgo8bW90b3J2b2duYXZnaWZ0IHhtbG5zPSJza2U6ZmFzdHNldHRpbmc6bW90b3J2b2duOm1vdG9ydm9nbmF2Z2lmdDp2MSI+CiAgICA8YXZnaWZ0c2xpbmplPgogICAgICAgIDxhdmdpZnRzYmVsb2VwPjU0Mjg5Ni4wMDwvYXZnaWZ0c2JlbG9lcD4KICAgICAgICA8YXZnaWZ0c29wcGx5c25pbmc+CiAgICAgICAgICAgIDxzYWVyYXZnaWZ0VHlwZWtvZGU+QkI8L3NhZXJhdmdpZnRUeXBla29kZT4KICAgICAgICAgICAgPHNhZXJhdmdpZnRHcnVwcGVrb2RlPlg8L3NhZXJhdmdpZnRHcnVwcGVrb2RlPgogICAgICAgIDwvYXZnaWZ0c29wcGx5c25pbmc+CiAgICAgICAgPGF2Z2lmdHNkYXRvPjIwMTctMTAtMjA8L2F2Z2lmdHNkYXRvPgogICAgPC9hdmdpZnRzbGluamU+CiAgICA8YmV0YWxpbmdzaW5mb3JtYXNqb24+CiAgICAgICAgPGtpZG51bW1lcj4xMDEwMTAxMDEwMTA8L2tpZG51bW1lcj4KICAgICAgICA8Zm9yZmFsbHNkYXRvPjIwMTctMTAtMjA8L2ZvcmZhbGxzZGF0bz4KICAgICAgICA8ZmFrdHVyYWRhdG8+MjAxNy0xMC0yMDwvZmFrdHVyYWRhdG8+CiAgICAgICAgPHRvdGFsQXZnaWZ0c2JlbG9lcD41NDI4OTYuMDA8L3RvdGFsQXZnaWZ0c2JlbG9lcD4KICAgIDwvYmV0YWxpbmdzaW5mb3JtYXNqb24+CiAgICA8bW90b3J2b2duYXZnaWZ0c3R5cGU+ZW5nYW5nc2F2Z2lmdDwvbW90b3J2b2duYXZnaWZ0c3R5cGU+CiAgICA8dGlkc3N0ZW1wZWw+MjAxOC0wMS0xNiswMTowMDwvdGlkc3N0ZW1wZWw+CiAgICA8Z3J1bm5sYWdGb3JNb3RvcnZvZ25hdmdpZnQ+CiAgICAgICAgPGtqb2VyaW5nZW5zQXJ0PjEwPC9ram9lcmluZ2Vuc0FydD4KICAgICAgICA8a2pvZXJldG9leT4KICAgICAgICAgICAgPGVpZXJza2FwUmVnaXN0cmVydD4yMDE3LTEwLTIwPC9laWVyc2thcFJlZ2lzdHJlcnQ+CiAgICAgICAgICAgIDxmb2Vyc3RlUmVnaXN0cmVyaW5nc2Fhcj4yMDE3PC9mb2Vyc3RlUmVnaXN0cmVyaW5nc2Fhcj4KICAgICAgICAgICAgPGZvZXJzdGVSZWdpc3RyZXJpbmdzZGF0b0lOb3JnZT4yMDE3LTEwLTIwPC9mb2Vyc3RlUmVnaXN0cmVyaW5nc2RhdG9JTm9yZ2U+CiAgICAgICAgICAgIDxram9lcmV0b2V5Z3J1cHBlPjEwMTwva2pvZXJldG9leWdydXBwZT4KICAgICAgICAgICAgPGxlbmdkZT4zOTY0PC9sZW5nZGU+CiAgICAgICAgICAgIDxtb3RvcmVmZmVrdD45NjwvbW90b3JlZmZla3Q+CiAgICAgICAgICAgIDxzbGFndm9sdW0+MTQzPC9zbGFndm9sdW0+CiAgICAgICAgICAgIDxkcml2c3RvZmY+QkVOU0lOPC9kcml2c3RvZmY+CiAgICAgICAgICAgIDxlZ2VudmVrdD4xNTE5PC9lZ2VudmVrdD4KICAgICAgICAgICAgPGVpZXI+CiAgICAgICAgICAgICAgICA8Zm9lZHNlbHNFbGxlckRudW1tZXI+MTEyMjMzNDQ1NTwvZm9lZHNlbHNFbGxlckRudW1tZXI+CiAgICAgICAgICAgICAgICA8cGFydHNudW1tZXI+NTU0NDMzMjIxMTwvcGFydHNudW1tZXI+CiAgICAgICAgICAgICAgICA8bmF2bj5LTEFSQSBLVTwvbmF2bj4KICAgICAgICAgICAgPC9laWVyPgogICAgICAgICAgICA8dGlsbGF0dFRvdGFsdmVrdD4yMTY0PC90aWxsYXR0VG90YWx2ZWt0PgogICAgICAgICAgICA8aHlicmlkPm5laTwvaHlicmlkPgogICAgICAgICAgICA8Y28ydXRzbGlwcD4yNjg8L2NvMnV0c2xpcHA+CiAgICAgICAgICAgIDxub3h1dHNsaXBwPjU5LjQ8L25veHV0c2xpcHA+CiAgICAgICAgICAgIDxram9lcmV0b2V5aWRlbnRpZmlrYXRvcj4KICAgICAgICAgICAgICAgIDxram9lcmV0b2V5VW5pa0lkZW50aWZpa2F0b3I+QUJDREVGR0hJSjwva2pvZXJldG9leVVuaWtJZGVudGlmaWthdG9yPgogICAgICAgICAgICAgICAgPGtqZW5uZW1lcmtlPjQ0QklMMTwva2plbm5lbWVya2U+CiAgICAgICAgICAgICAgICA8dW5kZXJzdGVsbHNudW1tZXI+VU5ERVJTVEVMTDQ0PC91bmRlcnN0ZWxsc251bW1lcj4KICAgICAgICAgICAgPC9ram9lcmV0b2V5aWRlbnRpZmlrYXRvcj4KICAgICAgICA8L2tqb2VyZXRvZXk+CiAgICA8L2dydW5ubGFnRm9yTW90b3J2b2duYXZnaWZ0PgogICAgPGF2Z2lmdHNwbGlrdGlnPgogICAgICAgIDxmb2Vkc2Vsc0VsbGVyRG51bW1lcj4xMTIyMzM0NDU1PC9mb2Vkc2Vsc0VsbGVyRG51bW1lcj4KICAgICAgICA8cGFydHNudW1tZXI+NTU0NDMzMjIxMTwvcGFydHNudW1tZXI+CiAgICA8L2F2Z2lmdHNwbGlrdGlnPgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzI8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjQ3NTMzNy4yMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5FZ2VudmVrdDwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+NjA5NDUuNjQ8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+TW90b3JlZmZla3Q8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94PC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD40MjEzLjI0PC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlNsYWd2b2x1bTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzIgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+RWdlbnZla3QgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+TW90b3JlZmZla3QgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94IEZyYXRyZWtrPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4wLjAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlNsYWd2b2x1bSBGcmF0cmVrazwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzIgU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD40NzUzMzcuMjA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+RWdlbnZla3QgU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD42MDk0NS42NDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5Nb3RvcmVmZmVrdCBTdW08L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94IFN1bTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+NDIxMy4yNDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5TbGFndm9sdW0gU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4wLjAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlZyYWtwYW50PC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4yNDAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PkJydWtzZnJhZHJhZyAwJTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGZvcmhhYW5kc2JlcmVnbmluZz5mYWxzZTwvZm9yaGFhbmRzYmVyZWduaW5nPgo8L21vdG9ydm9nbmF2Z2lmdD4K"

  },

  "extension": null,

  "skjemaversjon": "v3_0"

}

 

 

 

 

Cheers,

Bjørn

 

 


Denne e-posten og eventuelle vedlegg er beregnet utelukkende for den institusjon eller person den er rettet til og kan være belagt med lovbestemt taushetsplikt. Dersom e-posten er feilsendt, vennligst slett den og kontakt Skatteetaten. 
The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential information and may be legally protected from disclosure. If you are not the intended recipient of this message, please immediately delete the message and alert the Norwegian Tax Administration.

 

Reply | Threaded
Open this post in threaded view
|

SV: Replacing a base64-encoded field in a JSON-document with its decoded/converted value

Myklebust, Bjørn Magnar
In reply to this post by Andy LoPresto

Andy, just a quick followup on this.

 

I wanted to test a groovy-script with this code (not finished by far yet), and the script is placed in the Script Body part of an ExecuteGroovyScript-process in NiFi:

 

 

import org.json.JSONObject

import org.json.XML

import org.apache.commons.io.IOUtils

import java.nio.charset.*

 

def flowFile = session.get()

if (!flowFile) return

 

flowFile = session.write(flowFile,

  {inputStream, outputStream ->

      def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)

      def xmlJSONObj = XML.toJSONObject(text);

      def json = xmlJSONObj.toString();

      outputStream.write(json.getBytes(StandardCharsets.UTF_8))

  } as StreamCallback)

 

session.transfer(flowFile, ExecuteScript.REL_SUCCESS)

 

But when trying to run this I get the message «unable to resolve class org.json.JSONObject @ line 1»

I have downloaded the jar file from this site:  https://repo1.maven.org/maven2/org/json/json/20200518/json-20200518.jar

And placed it in my nifi/lib-directory.

And the content of this jar you can see in the enclosed png-picture.

 

Do I need to set a value for the property Additional Classpath when the jar-file is stored in the lib-directory?

 

Thanks,

Bjørn

 

 

Fra: Andy LoPresto <[hidden email]>
Sendt: torsdag 25. juni 2020 19:20
Til: [hidden email]
Emne: Re: Replacing a base64-encoded field in a JSON-document with its decoded/converted value

 

Hi Bjørn,

 

No, XML to JSON conversion is not an Expression Language feature. You’ll need to either get this data into a flowfile as the complete content to perform the conversion with existing built-in tools, or add that step to your Groovy script. 

 

With that additional requirement, I think using the Groovy script to perform those steps in tandem is probably the most performant and logical approach here. 

 

 

Andy LoPresto
[hidden email]
[hidden email]
He/Him

PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69



On Jun 24, 2020, at 11:25 PM, Myklebust, Bjørn Magnar <[hidden email]> wrote:

 

Thanks Andy.

The XML-content is around 5 kB-ish.  But I also need to convert the XML to JSON before replacing it back into the original JSON-file.  Can this be done with e.g a ConvertAttribute before the ReplaceText?

 

Thanks,

Bjørn

 

 

 

Fra: Andy LoPresto <[hidden email]> 
Sendt: onsdag 24. juni 2020 17:24
Til: 
[hidden email]
Emne: Re: Replacing a base64-encoded field in a JSON-document with its decoded/converted value

 

Hello Bjørn, 

 

If the size of the encoded XML document is small (under ~1 KB), you can extract the Base64-encoded value to a flowfile attribute using EvaluateJSONPath, perform the decoding using the base64Decode Expression Language function [1], and then replace it into the flowfile JSON content using ReplaceText (using some regex like "content": ".*" -> “content": ”${decodedXML}” where decodedXML is the name of the attribute you are using). 

 

If the XML content could be very large, this will negatively affect your performance, as attributes are stored directly in memory and handling large amounts of data will impact the heap. In this case, I would recommend writing a Groovy script in ExecuteScript processor to leverage Groovy’s very friendly JSON handling and extract the value, Base64 decode it, and replace it in a couple lines. 

 

Hope this helps. 

 

 

 

Andy LoPresto
[hidden email]
[hidden email]
He/Him

PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69




On Jun 24, 2020, at 4:24 AM, Myklebust, Bjørn Magnar <[hidden email]> wrote:

 

 

Hi.

I have a set of Json-files which contain a base64-coded field (Jsonpath to this field is $.data.content), and this field contains a XML-document.  Decoding the field works as expected, so does the conversion from xml to json,  and I'm able to write the content from this field to a file in a bucket in S3.  But what I would like to do is to be able to replace the coded value for this field in the original file with the decoded/converted value in stead of writing the decoded/converted value to file. And after replacing the json-value then I can write the updated Json-file to a new S3 bucket.

My process look like this at the moment, and works fine for getting the data to file, but it's missing the last part of replacing $.data.content with the decoded/converted data.

 

So how can I do the last part?

 

<pastedImage.png>

 

The EvaluedJsonPath looks like this:

 

<EvaluateJsonPath.png>

 

The ReplaceText looks like this:

 

<ReplaceText.png>

The Base64EncodeContent looks like this:

 

<Base64Decode.png>

and finally, the CovertRecord looks like this:

 

<ConvertRecord.png>

 

 

This is a testfile for that I'm working with:

 

{

  "header": {

    "dokumentidentifikator": null,

    "dokumentidentifikatorV2": "dcff985b-c652-4085-b8f1-45a2f4b6d150",

    "revisjonsnummer": 1,

    "dokumentnavn": "Engangsavgiftfastsettelse:55TEST661122334455:44BIL1:2017-10-20",

    "dokumenttype": "SKATTEMELDING_ENGANGSAVGIFT",

    "dokumenttilstand": "OPPRETTET",

    "gyldig": true,

    "gjelderInntektsaar": 2017,

    "gjelderPeriode": "2017_10",

    "gjelderPart": {

      "partsnummer": 5544332211,

      "identifiseringstype": "MASKINELL",

      "identifikator": null

    },

    "opphavspart": {

      "partsnummer": 5544332211,

      "identifikator": null

    },

    "kildereferanse": {

      "kildesystem": "ENGANGSAVGIFTFASTSETTELSE",

      "gruppe": "",

      "referanse": "aef147fb-8ce8-43ef-833b-7aa3bac1ece0",

      "tidspunkt": "2018-01-16T13:28:02.49+01:00"

    }

  },

  "data": {

    "metadata": {

      "format": "ske:fastsetting:motorvogn:motorvognavgift:v1",

      "bytes": 4420,

      "mimeType": "application/xml",

      "sha1": "c0AowOsTdNdo6VufeSsZqTphc0Y="

    },

    "content": "PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiIHN0YW5kYWxvbmU9InllcyI/Pgo8bW90b3J2b2duYXZnaWZ0IHhtbG5zPSJza2U6ZmFzdHNldHRpbmc6bW90b3J2b2duOm1vdG9ydm9nbmF2Z2lmdDp2MSI+CiAgICA8YXZnaWZ0c2xpbmplPgogICAgICAgIDxhdmdpZnRzYmVsb2VwPjU0Mjg5Ni4wMDwvYXZnaWZ0c2JlbG9lcD4KICAgICAgICA8YXZnaWZ0c29wcGx5c25pbmc+CiAgICAgICAgICAgIDxzYWVyYXZnaWZ0VHlwZWtvZGU+QkI8L3NhZXJhdmdpZnRUeXBla29kZT4KICAgICAgICAgICAgPHNhZXJhdmdpZnRHcnVwcGVrb2RlPlg8L3NhZXJhdmdpZnRHcnVwcGVrb2RlPgogICAgICAgIDwvYXZnaWZ0c29wcGx5c25pbmc+CiAgICAgICAgPGF2Z2lmdHNkYXRvPjIwMTctMTAtMjA8L2F2Z2lmdHNkYXRvPgogICAgPC9hdmdpZnRzbGluamU+CiAgICA8YmV0YWxpbmdzaW5mb3JtYXNqb24+CiAgICAgICAgPGtpZG51bW1lcj4xMDEwMTAxMDEwMTA8L2tpZG51bW1lcj4KICAgICAgICA8Zm9yZmFsbHNkYXRvPjIwMTctMTAtMjA8L2ZvcmZhbGxzZGF0bz4KICAgICAgICA8ZmFrdHVyYWRhdG8+MjAxNy0xMC0yMDwvZmFrdHVyYWRhdG8+CiAgICAgICAgPHRvdGFsQXZnaWZ0c2JlbG9lcD41NDI4OTYuMDA8L3RvdGFsQXZnaWZ0c2JlbG9lcD4KICAgIDwvYmV0YWxpbmdzaW5mb3JtYXNqb24+CiAgICA8bW90b3J2b2duYXZnaWZ0c3R5cGU+ZW5nYW5nc2F2Z2lmdDwvbW90b3J2b2duYXZnaWZ0c3R5cGU+CiAgICA8dGlkc3N0ZW1wZWw+MjAxOC0wMS0xNiswMTowMDwvdGlkc3N0ZW1wZWw+CiAgICA8Z3J1bm5sYWdGb3JNb3RvcnZvZ25hdmdpZnQ+CiAgICAgICAgPGtqb2VyaW5nZW5zQXJ0PjEwPC9ram9lcmluZ2Vuc0FydD4KICAgICAgICA8a2pvZXJldG9leT4KICAgICAgICAgICAgPGVpZXJza2FwUmVnaXN0cmVydD4yMDE3LTEwLTIwPC9laWVyc2thcFJlZ2lzdHJlcnQ+CiAgICAgICAgICAgIDxmb2Vyc3RlUmVnaXN0cmVyaW5nc2Fhcj4yMDE3PC9mb2Vyc3RlUmVnaXN0cmVyaW5nc2Fhcj4KICAgICAgICAgICAgPGZvZXJzdGVSZWdpc3RyZXJpbmdzZGF0b0lOb3JnZT4yMDE3LTEwLTIwPC9mb2Vyc3RlUmVnaXN0cmVyaW5nc2RhdG9JTm9yZ2U+CiAgICAgICAgICAgIDxram9lcmV0b2V5Z3J1cHBlPjEwMTwva2pvZXJldG9leWdydXBwZT4KICAgICAgICAgICAgPGxlbmdkZT4zOTY0PC9sZW5nZGU+CiAgICAgICAgICAgIDxtb3RvcmVmZmVrdD45NjwvbW90b3JlZmZla3Q+CiAgICAgICAgICAgIDxzbGFndm9sdW0+MTQzPC9zbGFndm9sdW0+CiAgICAgICAgICAgIDxkcml2c3RvZmY+QkVOU0lOPC9kcml2c3RvZmY+CiAgICAgICAgICAgIDxlZ2VudmVrdD4xNTE5PC9lZ2VudmVrdD4KICAgICAgICAgICAgPGVpZXI+CiAgICAgICAgICAgICAgICA8Zm9lZHNlbHNFbGxlckRudW1tZXI+MTEyMjMzNDQ1NTwvZm9lZHNlbHNFbGxlckRudW1tZXI+CiAgICAgICAgICAgICAgICA8cGFydHNudW1tZXI+NTU0NDMzMjIxMTwvcGFydHNudW1tZXI+CiAgICAgICAgICAgICAgICA8bmF2bj5LTEFSQSBLVTwvbmF2bj4KICAgICAgICAgICAgPC9laWVyPgogICAgICAgICAgICA8dGlsbGF0dFRvdGFsdmVrdD4yMTY0PC90aWxsYXR0VG90YWx2ZWt0PgogICAgICAgICAgICA8aHlicmlkPm5laTwvaHlicmlkPgogICAgICAgICAgICA8Y28ydXRzbGlwcD4yNjg8L2NvMnV0c2xpcHA+CiAgICAgICAgICAgIDxub3h1dHNsaXBwPjU5LjQ8L25veHV0c2xpcHA+CiAgICAgICAgICAgIDxram9lcmV0b2V5aWRlbnRpZmlrYXRvcj4KICAgICAgICAgICAgICAgIDxram9lcmV0b2V5VW5pa0lkZW50aWZpa2F0b3I+QUJDREVGR0hJSjwva2pvZXJldG9leVVuaWtJZGVudGlmaWthdG9yPgogICAgICAgICAgICAgICAgPGtqZW5uZW1lcmtlPjQ0QklMMTwva2plbm5lbWVya2U+CiAgICAgICAgICAgICAgICA8dW5kZXJzdGVsbHNudW1tZXI+VU5ERVJTVEVMTDQ0PC91bmRlcnN0ZWxsc251bW1lcj4KICAgICAgICAgICAgPC9ram9lcmV0b2V5aWRlbnRpZmlrYXRvcj4KICAgICAgICA8L2tqb2VyZXRvZXk+CiAgICA8L2dydW5ubGFnRm9yTW90b3J2b2duYXZnaWZ0PgogICAgPGF2Z2lmdHNwbGlrdGlnPgogICAgICAgIDxmb2Vkc2Vsc0VsbGVyRG51bW1lcj4xMTIyMzM0NDU1PC9mb2Vkc2Vsc0VsbGVyRG51bW1lcj4KICAgICAgICA8cGFydHNudW1tZXI+NTU0NDMzMjIxMTwvcGFydHNudW1tZXI+CiAgICA8L2F2Z2lmdHNwbGlrdGlnPgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzI8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjQ3NTMzNy4yMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5FZ2VudmVrdDwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+NjA5NDUuNjQ8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+TW90b3JlZmZla3Q8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94PC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD40MjEzLjI0PC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlNsYWd2b2x1bTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzIgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+RWdlbnZla3QgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+TW90b3JlZmZla3QgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94IEZyYXRyZWtrPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4wLjAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlNsYWd2b2x1bSBGcmF0cmVrazwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzIgU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD40NzUzMzcuMjA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+RWdlbnZla3QgU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD42MDk0NS42NDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5Nb3RvcmVmZmVrdCBTdW08L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94IFN1bTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+NDIxMy4yNDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5TbGFndm9sdW0gU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4wLjAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlZyYWtwYW50PC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4yNDAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PkJydWtzZnJhZHJhZyAwJTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGZvcmhhYW5kc2JlcmVnbmluZz5mYWxzZTwvZm9yaGFhbmRzYmVyZWduaW5nPgo8L21vdG9ydm9nbmF2Z2lmdD4K"

  },

  "extension": null,

  "skjemaversjon": "v3_0"

}

 

 

 

 

Cheers,

Bjørn

 

 


Denne e-posten og eventuelle vedlegg er beregnet utelukkende for den institusjon eller person den er rettet til og kan være belagt med lovbestemt taushetsplikt. Dersom e-posten er feilsendt, vennligst slett den og kontakt Skatteetaten. 
The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential information and may be legally protected from disclosure. If you are not the intended recipient of this message, please immediately delete the message and alert the Norwegian Tax Administration.

 


org.json.JSONObject.png (193K) Download Attachment
executegroovyscript.png (114K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Replacing a base64-encoded field in a JSON-document with its decoded/converted value

Andy LoPresto
You should not need to explicitly set the additional module directory to cover that location. Is there a reason you can’t use the native Groovy JSON [1] parsing? That way you don’t have to download any additional libraries. 


Andy LoPresto
[hidden email]
[hidden email]
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Jun 29, 2020, at 7:41 AM, Myklebust, Bjørn Magnar <[hidden email]> wrote:

Andy, just a quick followup on this.
 
I wanted to test a groovy-script with this code (not finished by far yet), and the script is placed in the Script Body part of an ExecuteGroovyScript-process in NiFi:
 
 
import org.json.JSONObject
import org.json.XML
import org.apache.commons.io.IOUtils
import java.nio.charset.*
 
def flowFile = session.get()
if (!flowFile) return
 
flowFile = session.write(flowFile,
  {inputStream, outputStream ->
      def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
      def xmlJSONObj = XML.toJSONObject(text);
      def json = xmlJSONObj.toString();
      outputStream.write(json.getBytes(StandardCharsets.UTF_8))
  } as StreamCallback)
 
session.transfer(flowFile, ExecuteScript.REL_SUCCESS)
 
But when trying to run this I get the message «unable to resolve class org.json.JSONObject @ line 1»
I have downloaded the jar file from this site:  https://repo1.maven.org/maven2/org/json/json/20200518/json-20200518.jar
And placed it in my nifi/lib-directory.
And the content of this jar you can see in the enclosed png-picture.
 
Do I need to set a value for the property Additional Classpath when the jar-file is stored in the lib-directory?
 
Thanks,
Bjørn
 
 
Fra: Andy LoPresto <[hidden email]> 
Sendt: torsdag 25. juni 2020 19:20
Til: [hidden email]
Emne: Re: Replacing a base64-encoded field in a JSON-document with its decoded/converted value
 
Hi Bjørn,
 
No, XML to JSON conversion is not an Expression Language feature. You’ll need to either get this data into a flowfile as the complete content to perform the conversion with existing built-in tools, or add that step to your Groovy script. 
 
With that additional requirement, I think using the Groovy script to perform those steps in tandem is probably the most performant and logical approach here. 
 
 
Andy LoPresto
[hidden email]
[hidden email]
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69


On Jun 24, 2020, at 11:25 PM, Myklebust, Bjørn Magnar <[hidden email]> wrote:
 
Thanks Andy.
The XML-content is around 5 kB-ish.  But I also need to convert the XML to JSON before replacing it back into the original JSON-file.  Can this be done with e.g a ConvertAttribute before the ReplaceText?
 
Thanks,
Bjørn
 
 
 
Fra: Andy LoPresto <[hidden email]> 
Sendt: onsdag 24. juni 2020 17:24
Til: 
[hidden email]
Emne: Re: Replacing a base64-encoded field in a JSON-document with its decoded/converted value
 
Hello Bjørn, 
 
If the size of the encoded XML document is small (under ~1 KB), you can extract the Base64-encoded value to a flowfile attribute using EvaluateJSONPath, perform the decoding using the base64Decode Expression Language function [1], and then replace it into the flowfile JSON content using ReplaceText (using some regex like "content": ".*" -> “content": ”${decodedXML}” where decodedXML is the name of the attribute you are using). 
 
If the XML content could be very large, this will negatively affect your performance, as attributes are stored directly in memory and handling large amounts of data will impact the heap. In this case, I would recommend writing a Groovy script in ExecuteScript processor to leverage Groovy’s very friendly JSON handling and extract the value, Base64 decode it, and replace it in a couple lines. 
 
Hope this helps. 
 
 
 
Andy LoPresto
[hidden email]
[hidden email]
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69



On Jun 24, 2020, at 4:24 AM, Myklebust, Bjørn Magnar <[hidden email]> wrote:
 
 
Hi.
I have a set of Json-files which contain a base64-coded field (Jsonpath to this field is $.data.content), and this field contains a XML-document.  Decoding the field works as expected, so does the conversion from xml to json,  and I'm able to write the content from this field to a file in a bucket in S3.  But what I would like to do is to be able to replace the coded value for this field in the original file with the decoded/converted value in stead of writing the decoded/converted value to file. And after replacing the json-value then I can write the updated Json-file to a new S3 bucket.
My process look like this at the moment, and works fine for getting the data to file, but it's missing the last part of replacing $.data.content with the decoded/converted data.
 
So how can I do the last part?
 
<pastedImage.png>
 
The EvaluedJsonPath looks like this:
 

<EvaluateJsonPath.png>

 
The ReplaceText looks like this:
 

<ReplaceText.png>

The Base64EncodeContent looks like this:
 

<Base64Decode.png>

and finally, the CovertRecord looks like this:
 

<ConvertRecord.png>

 
 
This is a testfile for that I'm working with:
 
{
  "header": {
    "dokumentidentifikator": null,
    "dokumentidentifikatorV2": "dcff985b-c652-4085-b8f1-45a2f4b6d150",
    "revisjonsnummer": 1,
    "dokumentnavn": "Engangsavgiftfastsettelse:55TEST661122334455:44BIL1:2017-10-20",
    "dokumenttype": "SKATTEMELDING_ENGANGSAVGIFT",
    "dokumenttilstand": "OPPRETTET",
    "gyldig": true,
    "gjelderInntektsaar": 2017,
    "gjelderPeriode": "2017_10",
    "gjelderPart": {
      "partsnummer": 5544332211,
      "identifiseringstype": "MASKINELL",
      "identifikator": null
    },
    "opphavspart": {
      "partsnummer": 5544332211,
      "identifikator": null
    },
    "kildereferanse": {
      "kildesystem": "ENGANGSAVGIFTFASTSETTELSE",
      "gruppe": "",
      "referanse": "aef147fb-8ce8-43ef-833b-7aa3bac1ece0",
      "tidspunkt": "2018-01-16T13:28:02.49+01:00"
    }
  },
  "data": {
    "metadata": {
      "format": "ske:fastsetting:motorvogn:motorvognavgift:v1",
      "bytes": 4420,
      "mimeType": "application/xml",
      "sha1": "c0AowOsTdNdo6VufeSsZqTphc0Y="
    },
    "content": "PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiIHN0YW5kYWxvbmU9InllcyI/Pgo8bW90b3J2b2duYXZnaWZ0IHhtbG5zPSJza2U6ZmFzdHNldHRpbmc6bW90b3J2b2duOm1vdG9ydm9nbmF2Z2lmdDp2MSI+CiAgICA8YXZnaWZ0c2xpbmplPgogICAgICAgIDxhdmdpZnRzYmVsb2VwPjU0Mjg5Ni4wMDwvYXZnaWZ0c2JlbG9lcD4KICAgICAgICA8YXZnaWZ0c29wcGx5c25pbmc+CiAgICAgICAgICAgIDxzYWVyYXZnaWZ0VHlwZWtvZGU+QkI8L3NhZXJhdmdpZnRUeXBla29kZT4KICAgICAgICAgICAgPHNhZXJhdmdpZnRHcnVwcGVrb2RlPlg8L3NhZXJhdmdpZnRHcnVwcGVrb2RlPgogICAgICAgIDwvYXZnaWZ0c29wcGx5c25pbmc+CiAgICAgICAgPGF2Z2lmdHNkYXRvPjIwMTctMTAtMjA8L2F2Z2lmdHNkYXRvPgogICAgPC9hdmdpZnRzbGluamU+CiAgICA8YmV0YWxpbmdzaW5mb3JtYXNqb24+CiAgICAgICAgPGtpZG51bW1lcj4xMDEwMTAxMDEwMTA8L2tpZG51bW1lcj4KICAgICAgICA8Zm9yZmFsbHNkYXRvPjIwMTctMTAtMjA8L2ZvcmZhbGxzZGF0bz4KICAgICAgICA8ZmFrdHVyYWRhdG8+MjAxNy0xMC0yMDwvZmFrdHVyYWRhdG8+CiAgICAgICAgPHRvdGFsQXZnaWZ0c2JlbG9lcD41NDI4OTYuMDA8L3RvdGFsQXZnaWZ0c2JlbG9lcD4KICAgIDwvYmV0YWxpbmdzaW5mb3JtYXNqb24+CiAgICA8bW90b3J2b2duYXZnaWZ0c3R5cGU+ZW5nYW5nc2F2Z2lmdDwvbW90b3J2b2duYXZnaWZ0c3R5cGU+CiAgICA8dGlkc3N0ZW1wZWw+MjAxOC0wMS0xNiswMTowMDwvdGlkc3N0ZW1wZWw+CiAgICA8Z3J1bm5sYWdGb3JNb3RvcnZvZ25hdmdpZnQ+CiAgICAgICAgPGtqb2VyaW5nZW5zQXJ0PjEwPC9ram9lcmluZ2Vuc0FydD4KICAgICAgICA8a2pvZXJldG9leT4KICAgICAgICAgICAgPGVpZXJza2FwUmVnaXN0cmVydD4yMDE3LTEwLTIwPC9laWVyc2thcFJlZ2lzdHJlcnQ+CiAgICAgICAgICAgIDxmb2Vyc3RlUmVnaXN0cmVyaW5nc2Fhcj4yMDE3PC9mb2Vyc3RlUmVnaXN0cmVyaW5nc2Fhcj4KICAgICAgICAgICAgPGZvZXJzdGVSZWdpc3RyZXJpbmdzZGF0b0lOb3JnZT4yMDE3LTEwLTIwPC9mb2Vyc3RlUmVnaXN0cmVyaW5nc2RhdG9JTm9yZ2U+CiAgICAgICAgICAgIDxram9lcmV0b2V5Z3J1cHBlPjEwMTwva2pvZXJldG9leWdydXBwZT4KICAgICAgICAgICAgPGxlbmdkZT4zOTY0PC9sZW5nZGU+CiAgICAgICAgICAgIDxtb3RvcmVmZmVrdD45NjwvbW90b3JlZmZla3Q+CiAgICAgICAgICAgIDxzbGFndm9sdW0+MTQzPC9zbGFndm9sdW0+CiAgICAgICAgICAgIDxkcml2c3RvZmY+QkVOU0lOPC9kcml2c3RvZmY+CiAgICAgICAgICAgIDxlZ2VudmVrdD4xNTE5PC9lZ2VudmVrdD4KICAgICAgICAgICAgPGVpZXI+CiAgICAgICAgICAgICAgICA8Zm9lZHNlbHNFbGxlckRudW1tZXI+MTEyMjMzNDQ1NTwvZm9lZHNlbHNFbGxlckRudW1tZXI+CiAgICAgICAgICAgICAgICA8cGFydHNudW1tZXI+NTU0NDMzMjIxMTwvcGFydHNudW1tZXI+CiAgICAgICAgICAgICAgICA8bmF2bj5LTEFSQSBLVTwvbmF2bj4KICAgICAgICAgICAgPC9laWVyPgogICAgICAgICAgICA8dGlsbGF0dFRvdGFsdmVrdD4yMTY0PC90aWxsYXR0VG90YWx2ZWt0PgogICAgICAgICAgICA8aHlicmlkPm5laTwvaHlicmlkPgogICAgICAgICAgICA8Y28ydXRzbGlwcD4yNjg8L2NvMnV0c2xpcHA+CiAgICAgICAgICAgIDxub3h1dHNsaXBwPjU5LjQ8L25veHV0c2xpcHA+CiAgICAgICAgICAgIDxram9lcmV0b2V5aWRlbnRpZmlrYXRvcj4KICAgICAgICAgICAgICAgIDxram9lcmV0b2V5VW5pa0lkZW50aWZpa2F0b3I+QUJDREVGR0hJSjwva2pvZXJldG9leVVuaWtJZGVudGlmaWthdG9yPgogICAgICAgICAgICAgICAgPGtqZW5uZW1lcmtlPjQ0QklMMTwva2plbm5lbWVya2U+CiAgICAgICAgICAgICAgICA8dW5kZXJzdGVsbHNudW1tZXI+VU5ERVJTVEVMTDQ0PC91bmRlcnN0ZWxsc251bW1lcj4KICAgICAgICAgICAgPC9ram9lcmV0b2V5aWRlbnRpZmlrYXRvcj4KICAgICAgICA8L2tqb2VyZXRvZXk+CiAgICA8L2dydW5ubGFnRm9yTW90b3J2b2duYXZnaWZ0PgogICAgPGF2Z2lmdHNwbGlrdGlnPgogICAgICAgIDxmb2Vkc2Vsc0VsbGVyRG51bW1lcj4xMTIyMzM0NDU1PC9mb2Vkc2Vsc0VsbGVyRG51bW1lcj4KICAgICAgICA8cGFydHNudW1tZXI+NTU0NDMzMjIxMTwvcGFydHNudW1tZXI+CiAgICA8L2F2Z2lmdHNwbGlrdGlnPgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzI8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjQ3NTMzNy4yMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5FZ2VudmVrdDwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+NjA5NDUuNjQ8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+TW90b3JlZmZla3Q8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94PC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD40MjEzLjI0PC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlNsYWd2b2x1bTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzIgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+RWdlbnZla3QgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+TW90b3JlZmZla3QgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94IEZyYXRyZWtrPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4wLjAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlNsYWd2b2x1bSBGcmF0cmVrazwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzIgU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD40NzUzMzcuMjA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+RWdlbnZla3QgU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD42MDk0NS42NDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5Nb3RvcmVmZmVrdCBTdW08L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94IFN1bTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+NDIxMy4yNDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5TbGFndm9sdW0gU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4wLjAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlZyYWtwYW50PC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4yNDAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PkJydWtzZnJhZHJhZyAwJTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGZvcmhhYW5kc2JlcmVnbmluZz5mYWxzZTwvZm9yaGFhbmRzYmVyZWduaW5nPgo8L21vdG9ydm9nbmF2Z2lmdD4K"
  },
  "extension": null,
  "skjemaversjon": "v3_0"
}
 
 
 
 
Cheers,
Bjørn
 
 

Denne e-posten og eventuelle vedlegg er beregnet utelukkende for den institusjon eller person den er rettet til og kan være belagt med lovbestemt taushetsplikt. Dersom e-posten er feilsendt, vennligst slett den og kontakt Skatteetaten. 
The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential information and may be legally protected from disclosure. If you are not the intended recipient of this message, please immediately delete the message and alert the Norwegian Tax Administration.
 
<org.json.JSONObject.png><executegroovyscript.png>

Reply | Threaded
Open this post in threaded view
|

SV: Replacing a base64-encoded field in a JSON-document with its decoded/converted value

Myklebust, Bjørn Magnar

Thanks, Andy.

No, I’m sure there is no reason for that – it’s just that I’m fairly new to NiFi and don’t know it too well yet.

 

Thanks,

Bjørn

 

 

Fra: Andy LoPresto <[hidden email]>
Sendt: tirsdag 30. juni 2020 18:37
Til: [hidden email]
Emne: Re: Replacing a base64-encoded field in a JSON-document with its decoded/converted value

 

You should not need to explicitly set the additional module directory to cover that location. Is there a reason you can’t use the native Groovy JSON [1] parsing? That way you don’t have to download any additional libraries. 

 

 

Andy LoPresto
[hidden email]
[hidden email]
He/Him

PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69



On Jun 29, 2020, at 7:41 AM, Myklebust, Bjørn Magnar <[hidden email]> wrote:

 

Andy, just a quick followup on this.

 

I wanted to test a groovy-script with this code (not finished by far yet), and the script is placed in the Script Body part of an ExecuteGroovyScript-process in NiFi:

 

 

import org.json.JSONObject

import org.json.XML

import org.apache.commons.io.IOUtils

import java.nio.charset.*

 

def flowFile = session.get()

if (!flowFile) return

 

flowFile = session.write(flowFile,

  {inputStream, outputStream ->

      def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)

      def xmlJSONObj = XML.toJSONObject(text);

      def json = xmlJSONObj.toString();

      outputStream.write(json.getBytes(StandardCharsets.UTF_8))

  } as StreamCallback)

 

session.transfer(flowFile, ExecuteScript.REL_SUCCESS)

 

But when trying to run this I get the message «unable to resolve class org.json.JSONObject @ line 1»

I have downloaded the jar file from this site:  https://repo1.maven.org/maven2/org/json/json/20200518/json-20200518.jar

And placed it in my nifi/lib-directory.

And the content of this jar you can see in the enclosed png-picture.

 

Do I need to set a value for the property Additional Classpath when the jar-file is stored in the lib-directory?

 

Thanks,

Bjørn

 

 

Fra: Andy LoPresto <[hidden email]> 
Sendt: torsdag 25. juni 2020 19:20
Til: [hidden email]
Emne: Re: Replacing a base64-encoded field in a JSON-document with its decoded/converted value

 

Hi Bjørn,

 

No, XML to JSON conversion is not an Expression Language feature. You’ll need to either get this data into a flowfile as the complete content to perform the conversion with existing built-in tools, or add that step to your Groovy script. 

 

With that additional requirement, I think using the Groovy script to perform those steps in tandem is probably the most performant and logical approach here. 

 

 

Andy LoPresto
[hidden email]
[hidden email]
He/Him

PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69




On Jun 24, 2020, at 11:25 PM, Myklebust, Bjørn Magnar <[hidden email]> wrote:

 

Thanks Andy.

The XML-content is around 5 kB-ish.  But I also need to convert the XML to JSON before replacing it back into the original JSON-file.  Can this be done with e.g a ConvertAttribute before the ReplaceText?

 

Thanks,

Bjørn

 

 

 

Fra: Andy LoPresto <[hidden email]> 
Sendt: onsdag 24. juni 2020 17:24
Til: 
[hidden email]
Emne: Re: Replacing a base64-encoded field in a JSON-document with its decoded/converted value

 

Hello Bjørn, 

 

If the size of the encoded XML document is small (under ~1 KB), you can extract the Base64-encoded value to a flowfile attribute using EvaluateJSONPath, perform the decoding using the base64Decode Expression Language function [1], and then replace it into the flowfile JSON content using ReplaceText (using some regex like "content": ".*" -> “content": ”${decodedXML}” where decodedXML is the name of the attribute you are using). 

 

If the XML content could be very large, this will negatively affect your performance, as attributes are stored directly in memory and handling large amounts of data will impact the heap. In this case, I would recommend writing a Groovy script in ExecuteScript processor to leverage Groovy’s very friendly JSON handling and extract the value, Base64 decode it, and replace it in a couple lines. 

 

Hope this helps. 

 

 

 

Andy LoPresto
[hidden email]
[hidden email]
He/Him

PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69





On Jun 24, 2020, at 4:24 AM, Myklebust, Bjørn Magnar <[hidden email]> wrote:

 

 

Hi.

I have a set of Json-files which contain a base64-coded field (Jsonpath to this field is $.data.content), and this field contains a XML-document.  Decoding the field works as expected, so does the conversion from xml to json,  and I'm able to write the content from this field to a file in a bucket in S3.  But what I would like to do is to be able to replace the coded value for this field in the original file with the decoded/converted value in stead of writing the decoded/converted value to file. And after replacing the json-value then I can write the updated Json-file to a new S3 bucket.

My process look like this at the moment, and works fine for getting the data to file, but it's missing the last part of replacing $.data.content with the decoded/converted data.

 

So how can I do the last part?

 

<pastedImage.png>

 

The EvaluedJsonPath looks like this:

 

<EvaluateJsonPath.png>

 

The ReplaceText looks like this:

 

<ReplaceText.png>

The Base64EncodeContent looks like this:

 

<Base64Decode.png>

and finally, the CovertRecord looks like this:

 

<ConvertRecord.png>

 

 

This is a testfile for that I'm working with:

 

{

  "header": {

    "dokumentidentifikator": null,

    "dokumentidentifikatorV2": "dcff985b-c652-4085-b8f1-45a2f4b6d150",

    "revisjonsnummer": 1,

    "dokumentnavn": "Engangsavgiftfastsettelse:55TEST661122334455:44BIL1:2017-10-20",

    "dokumenttype": "SKATTEMELDING_ENGANGSAVGIFT",

    "dokumenttilstand": "OPPRETTET",

    "gyldig": true,

    "gjelderInntektsaar": 2017,

    "gjelderPeriode": "2017_10",

    "gjelderPart": {

      "partsnummer": 5544332211,

      "identifiseringstype": "MASKINELL",

      "identifikator": null

    },

    "opphavspart": {

      "partsnummer": 5544332211,

      "identifikator": null

    },

    "kildereferanse": {

      "kildesystem": "ENGANGSAVGIFTFASTSETTELSE",

      "gruppe": "",

      "referanse": "aef147fb-8ce8-43ef-833b-7aa3bac1ece0",

      "tidspunkt": "2018-01-16T13:28:02.49+01:00"

    }

  },

  "data": {

    "metadata": {

      "format": "ske:fastsetting:motorvogn:motorvognavgift:v1",

      "bytes": 4420,

      "mimeType": "application/xml",

      "sha1": "c0AowOsTdNdo6VufeSsZqTphc0Y="

    },

    "content": "PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiIHN0YW5kYWxvbmU9InllcyI/Pgo8bW90b3J2b2duYXZnaWZ0IHhtbG5zPSJza2U6ZmFzdHNldHRpbmc6bW90b3J2b2duOm1vdG9ydm9nbmF2Z2lmdDp2MSI+CiAgICA8YXZnaWZ0c2xpbmplPgogICAgICAgIDxhdmdpZnRzYmVsb2VwPjU0Mjg5Ni4wMDwvYXZnaWZ0c2JlbG9lcD4KICAgICAgICA8YXZnaWZ0c29wcGx5c25pbmc+CiAgICAgICAgICAgIDxzYWVyYXZnaWZ0VHlwZWtvZGU+QkI8L3NhZXJhdmdpZnRUeXBla29kZT4KICAgICAgICAgICAgPHNhZXJhdmdpZnRHcnVwcGVrb2RlPlg8L3NhZXJhdmdpZnRHcnVwcGVrb2RlPgogICAgICAgIDwvYXZnaWZ0c29wcGx5c25pbmc+CiAgICAgICAgPGF2Z2lmdHNkYXRvPjIwMTctMTAtMjA8L2F2Z2lmdHNkYXRvPgogICAgPC9hdmdpZnRzbGluamU+CiAgICA8YmV0YWxpbmdzaW5mb3JtYXNqb24+CiAgICAgICAgPGtpZG51bW1lcj4xMDEwMTAxMDEwMTA8L2tpZG51bW1lcj4KICAgICAgICA8Zm9yZmFsbHNkYXRvPjIwMTctMTAtMjA8L2ZvcmZhbGxzZGF0bz4KICAgICAgICA8ZmFrdHVyYWRhdG8+MjAxNy0xMC0yMDwvZmFrdHVyYWRhdG8+CiAgICAgICAgPHRvdGFsQXZnaWZ0c2JlbG9lcD41NDI4OTYuMDA8L3RvdGFsQXZnaWZ0c2JlbG9lcD4KICAgIDwvYmV0YWxpbmdzaW5mb3JtYXNqb24+CiAgICA8bW90b3J2b2duYXZnaWZ0c3R5cGU+ZW5nYW5nc2F2Z2lmdDwvbW90b3J2b2duYXZnaWZ0c3R5cGU+CiAgICA8dGlkc3N0ZW1wZWw+MjAxOC0wMS0xNiswMTowMDwvdGlkc3N0ZW1wZWw+CiAgICA8Z3J1bm5sYWdGb3JNb3RvcnZvZ25hdmdpZnQ+CiAgICAgICAgPGtqb2VyaW5nZW5zQXJ0PjEwPC9ram9lcmluZ2Vuc0FydD4KICAgICAgICA8a2pvZXJldG9leT4KICAgICAgICAgICAgPGVpZXJza2FwUmVnaXN0cmVydD4yMDE3LTEwLTIwPC9laWVyc2thcFJlZ2lzdHJlcnQ+CiAgICAgICAgICAgIDxmb2Vyc3RlUmVnaXN0cmVyaW5nc2Fhcj4yMDE3PC9mb2Vyc3RlUmVnaXN0cmVyaW5nc2Fhcj4KICAgICAgICAgICAgPGZvZXJzdGVSZWdpc3RyZXJpbmdzZGF0b0lOb3JnZT4yMDE3LTEwLTIwPC9mb2Vyc3RlUmVnaXN0cmVyaW5nc2RhdG9JTm9yZ2U+CiAgICAgICAgICAgIDxram9lcmV0b2V5Z3J1cHBlPjEwMTwva2pvZXJldG9leWdydXBwZT4KICAgICAgICAgICAgPGxlbmdkZT4zOTY0PC9sZW5nZGU+CiAgICAgICAgICAgIDxtb3RvcmVmZmVrdD45NjwvbW90b3JlZmZla3Q+CiAgICAgICAgICAgIDxzbGFndm9sdW0+MTQzPC9zbGFndm9sdW0+CiAgICAgICAgICAgIDxkcml2c3RvZmY+QkVOU0lOPC9kcml2c3RvZmY+CiAgICAgICAgICAgIDxlZ2VudmVrdD4xNTE5PC9lZ2VudmVrdD4KICAgICAgICAgICAgPGVpZXI+CiAgICAgICAgICAgICAgICA8Zm9lZHNlbHNFbGxlckRudW1tZXI+MTEyMjMzNDQ1NTwvZm9lZHNlbHNFbGxlckRudW1tZXI+CiAgICAgICAgICAgICAgICA8cGFydHNudW1tZXI+NTU0NDMzMjIxMTwvcGFydHNudW1tZXI+CiAgICAgICAgICAgICAgICA8bmF2bj5LTEFSQSBLVTwvbmF2bj4KICAgICAgICAgICAgPC9laWVyPgogICAgICAgICAgICA8dGlsbGF0dFRvdGFsdmVrdD4yMTY0PC90aWxsYXR0VG90YWx2ZWt0PgogICAgICAgICAgICA8aHlicmlkPm5laTwvaHlicmlkPgogICAgICAgICAgICA8Y28ydXRzbGlwcD4yNjg8L2NvMnV0c2xpcHA+CiAgICAgICAgICAgIDxub3h1dHNsaXBwPjU5LjQ8L25veHV0c2xpcHA+CiAgICAgICAgICAgIDxram9lcmV0b2V5aWRlbnRpZmlrYXRvcj4KICAgICAgICAgICAgICAgIDxram9lcmV0b2V5VW5pa0lkZW50aWZpa2F0b3I+QUJDREVGR0hJSjwva2pvZXJldG9leVVuaWtJZGVudGlmaWthdG9yPgogICAgICAgICAgICAgICAgPGtqZW5uZW1lcmtlPjQ0QklMMTwva2plbm5lbWVya2U+CiAgICAgICAgICAgICAgICA8dW5kZXJzdGVsbHNudW1tZXI+VU5ERVJTVEVMTDQ0PC91bmRlcnN0ZWxsc251bW1lcj4KICAgICAgICAgICAgPC9ram9lcmV0b2V5aWRlbnRpZmlrYXRvcj4KICAgICAgICA8L2tqb2VyZXRvZXk+CiAgICA8L2dydW5ubGFnRm9yTW90b3J2b2duYXZnaWZ0PgogICAgPGF2Z2lmdHNwbGlrdGlnPgogICAgICAgIDxmb2Vkc2Vsc0VsbGVyRG51bW1lcj4xMTIyMzM0NDU1PC9mb2Vkc2Vsc0VsbGVyRG51bW1lcj4KICAgICAgICA8cGFydHNudW1tZXI+NTU0NDMzMjIxMTwvcGFydHNudW1tZXI+CiAgICA8L2F2Z2lmdHNwbGlrdGlnPgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzI8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjQ3NTMzNy4yMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5FZ2VudmVrdDwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+NjA5NDUuNjQ8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+TW90b3JlZmZla3Q8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94PC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD40MjEzLjI0PC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlNsYWd2b2x1bTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzIgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+RWdlbnZla3QgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+TW90b3JlZmZla3QgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94IEZyYXRyZWtrPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4wLjAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlNsYWd2b2x1bSBGcmF0cmVrazwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzIgU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD40NzUzMzcuMjA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+RWdlbnZla3QgU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD42MDk0NS42NDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5Nb3RvcmVmZmVrdCBTdW08L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94IFN1bTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+NDIxMy4yNDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5TbGFndm9sdW0gU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4wLjAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlZyYWtwYW50PC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4yNDAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PkJydWtzZnJhZHJhZyAwJTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGZvcmhhYW5kc2JlcmVnbmluZz5mYWxzZTwvZm9yaGFhbmRzYmVyZWduaW5nPgo8L21vdG9ydm9nbmF2Z2lmdD4K"

  },

  "extension": null,

  "skjemaversjon": "v3_0"

}

 

 

 

 

Cheers,

Bjørn

 

 


Denne e-posten og eventuelle vedlegg er beregnet utelukkende for den institusjon eller person den er rettet til og kan være belagt med lovbestemt taushetsplikt. Dersom e-posten er feilsendt, vennligst slett den og kontakt Skatteetaten. 
The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential information and may be legally protected from disclosure. If you are not the intended recipient of this message, please immediately delete the message and alert the Norwegian Tax Administration.

 

<org.json.JSONObject.png><executegroovyscript.png>

 

Reply | Threaded
Open this post in threaded view
|

SV: Replacing a base64-encoded field in a JSON-document with its decoded/converted value

Myklebust, Bjørn Magnar

Thanks for the suggestion on using native Groovy JSON parsing, it is now running without any additional downloaded libraries. But I still don't get the required result.

My flow now looks like this.  Two parts where I try to do similar things.

  1. The left column of processes: ListS3, FetchS3Object, RoutOnAttribute and ExecuteGroovyScript
  2. ...RoutOnAttribute, EvaluateJsonPath, UpdateAttribute, ReplaceText and ExecuteGroovyScript 



The common processes just retrieves the files from S3 as you know.

The ExecuteGroovyScrip from the first part is like this.  I'm still using a sample JSON-file similar to the on at the bottom of my first post below:

import groovy.json.JsonSlurper
import org.json.JSONObject
import org.json.XML
import org.apache.commons.io.IOUtils
import java.nio.charset.StandardCharsets

def flowFile = session.get()
if (!flowFile) return

def slurper = new JsonSlurper()

flowFile = session.write(flowFile,
  {inputStream, outputStream ->

      def row = slurper.parseText(IOUtils.toString(inputStream, StandardCharsets.UTF_8))

      def content = row.data.content
      def decoded = content.decodeBase64()    
      def jsondata = XML.toJSONObject(decoded.toString())

      //row.data.content = jsondata

      outputStream.write(decoded)    //.getBytes(StandardCharsets.UTF_8))

  } as StreamCallback)

session.transfer(flowFile, REL_SUCCESS)

The content in the decoded-attribute is now the XML-file as I expected:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<motorvognavgift xmlns="ske:fastsetting:motorvogn:motorvognavgift:v1">
    <avgiftslinje>
        <avgiftsbeloep>449665.00</avgiftsbeloep>
        <avgiftsopplysning>
            <saeravgiftTypekode>EM</saeravgiftTypekode>
            <saeravgiftGruppekode>A</saeravgiftGruppekode>
        </avgiftsopplysning>
        <avgiftsdato>2017-01-01</avgiftsdato>
    </avgiftslinje>
    <betalingsinformasjon>
        <kidnummer>7413283630</kidnummer>
        <forfallsdato>2017-01-01</forfallsdato>
        <fakturadato>2017-01-01</fakturadato>
        <totalAvgiftsbeloep>449665.00</totalAvgiftsbeloep>
    </betalingsinformasjon>
    <motorvognavgiftstype>engangsavgift</motorvognavgiftstype>
    <tidsstempel>2017-12-19+01:00</tidsstempel>
    <grunnlagForMotorvognavgift>
        <kjoeringensArt>10</kjoeringensArt>
        <kjoeretoey>
            <foersteRegistreringsaar>2017</foersteRegistreringsaar>
            <foersteRegistreringsdatoINorge>2017-01-01</foersteRegistreringsdatoINorge>
            <foersteRegistreringsdatoUtland>2016-12-19</foersteRegistreringsdatoUtland>
            <kjoeretoeygruppe>101</kjoeretoeygruppe>
            <lengde>1461</lengde>
            <motoreffekt>368</motoreffekt>
            <slagvolum>135</slagvolum>
            <drivstoff>BENSIN</drivstoff>
            <egenvekt>1535</egenvekt>
            <eier>
                <foedselsEllerDnummer>05075530201</foedselsEllerDnummer>
                <partsnummer>9000000004370</partsnummer>
                <navn>TONJE INGVALDSEN</navn>
            </eier>
            <tillattTotalvekt>2500</tillattTotalvekt>
            <melder>
                <foedselsEllerDnummer>21044743008</foedselsEllerDnummer>
                <partsnummer>9000001840810</partsnummer>
            </melder>
            <co2utslipp>161</co2utslipp>
            <noxutslipp>3475.0</noxutslipp>
            <kjoeretoeyidentifikator>
                <kjoeretoeyUnikIdentifikator>ELGRUUDN5L</kjoeretoeyUnikIdentifikator>
                <kjennemerke>ZB36363</kjennemerke>
                <understellsnummer>ZB363633690Testklie</understellsnummer>
            </kjoeretoeyidentifikator>
        </kjoeretoey>
    </grunnlagForMotorvognavgift>
    <avgiftspliktig>
        <foedselsEllerDnummer>21044743008</foedselsEllerDnummer>
        <partsnummer>9000001840810</partsnummer>
    </avgiftspliktig>
    <avgiftskomponent>
        <komponent>Co2</komponent>
        <beloep>135997.58</beloep>
    </avgiftskomponent>
    <avgiftskomponent>
        <komponent>Egenvekt</komponent>
        <beloep>64786.60</beloep>
    </avgiftskomponent>
    <avgiftskomponent>
        <komponent>Motoreffekt</komponent>
        <beloep>0.00</beloep>
    </avgiftskomponent>
    <avgiftskomponent>
        <komponent>NOx</komponent>
        <beloep>246481.75</beloep>
    </avgiftskomponent>
    <avgiftskomponent>
        <komponent>Slagvolum</komponent>
        <beloep>0.00</beloep>
    </avgiftskomponent>
    <avgiftskomponent>
        <komponent>Co2 Fratrekk</komponent>
        <beloep>0.00</beloep>
    </avgiftskomponent>
    <avgiftskomponent>
        <komponent>Egenvekt Fratrekk</komponent>
        <beloep>0.00</beloep>
    </avgiftskomponent>
    <avgiftskomponent>
        <komponent>Motoreffekt Fratrekk</komponent>
        <beloep>0.00</beloep>
    </avgiftskomponent>
    <avgiftskomponent>
        <komponent>NOx Fratrekk</komponent>
        <beloep>0.00</beloep>
    </avgiftskomponent>
    <avgiftskomponent>
        <komponent>Slagvolum Fratrekk</komponent>
        <beloep>0.00</beloep>
    </avgiftskomponent>
    <avgiftskomponent>
        <komponent>Co2 Sum</komponent>
        <beloep>135997.58</beloep>
    </avgiftskomponent>
    <avgiftskomponent>
        <komponent>Egenvekt Sum</komponent>
        <beloep>64786.60</beloep>
    </avgiftskomponent>
    <avgiftskomponent>
        <komponent>Motoreffekt Sum</komponent>
        <beloep>0.00</beloep>
    </avgiftskomponent>
    <avgiftskomponent>
        <komponent>NOx Sum</komponent>
        <beloep>246481.75</beloep>
    </avgiftskomponent>
    <avgiftskomponent>
        <komponent>Slagvolum Sum</komponent>
        <beloep>0.00</beloep>
    </avgiftskomponent>
    <avgiftskomponent>
        <komponent>Vrakpant</komponent>
        <beloep>2400</beloep>
    </avgiftskomponent>
    <forhaandsberegning>false</forhaandsberegning>
</motorvognavgift>

But I'm not able to convert this XML to JSON.
What I'm trying to do here is first decode the value in row.data.content, and this is working. Next, I want to take the resulting XML-file from the decoding and convert this to JSON (not working as expected).
Finally I'm trying to replace the original value in row.data.content with the new JSON-file and then return the updated flowFile.

I would appreciate any suggestion on how to do this.



In the second part of the NiFi process I use EvaluateJsonPath to put the data.content into a variable, 
then decode this 

then replace:

before running this groovy scrpt:

import org.apache.commons.io.IOUtils
import java.nio.charset.StandardCharsets
import org.json.XML

def flowFile = session.get()
if (!flowFile) return

flowFile = session.write(flowFile,
  {inputStream, outputStream ->
      def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
      def xmlJSONObj = XML.toJSONObject(text);
      def json = xmlJSONObj.toString();
      outputStream.write(json.getBytes(StandardCharsets.UTF_8))
  } as StreamCallback)

session.transfer(flowFile, REL_SUCCESS)


Here I do get the JSON output from the encoded field, but because of the RaplaceText I have now longer access to the original/main flowFile and thus cannot put this back into the original JSON-file.  This part is mainly for testing the second groovy-script.

{
  "motorvognavgift" : {
    "xmlns" : "ske:fastsetting:motorvogn:motorvognavgift:v1",
    "betalingsinformasjon" : {
      "fakturadato" : "2017-01-01",
      "totalAvgiftsbeloep" : 449665,
      "kidnummer" : 7413283630,
      "forfallsdato" : "2017-01-01"
    },
    "grunnlagForMotorvognavgift" : {
      "kjoeringensArt" : 10,
      "kjoeretoey" : {
        "noxutslipp" : 3475,
        "slagvolum" : 135,
        "foersteRegistreringsaar" : 2017,
        "drivstoff" : "BENSIN",
        "co2utslipp" : 161,
        "foersteRegistreringsdatoINorge" : "2017-01-01",
        "tillattTotalvekt" : 2500,
        "kjoeretoeygruppe" : 101,
        "motoreffekt" : 368,
        "egenvekt" : 1535,
        "foersteRegistreringsdatoUtland" : "2016-12-19",
        "kjoeretoeyidentifikator" : {
          "understellsnummer" : "ZB363633690Testklie",
          "kjennemerke" : "ZB36363",
          "kjoeretoeyUnikIdentifikator" : "ELGRUUDN5L"
        },
        "lengde" : 1461,
        "eier" : {
          "navn" : "TONJE INGVALDSEN",
          "partsnummer" : 9000000004370,
          "foedselsEllerDnummer" : "05075530201"
        },
        "melder" : {
          "partsnummer" : 9000001840810,
          "foedselsEllerDnummer" : 21044743008
        }
      }
    },
    "avgiftspliktig" : {
      "partsnummer" : 9000001840810,
      "foedselsEllerDnummer" : 21044743008
    },
    "forhaandsberegning" : false,
    "motorvognavgiftstype" : "engangsavgift",
    "avgiftskomponent" : [ {
      "komponent" : "Co2",
      "beloep" : 135997.58
    }, {
      "komponent" : "Egenvekt",
      "beloep" : 64786.6
    }, {
      "komponent" : "Motoreffekt",
      "beloep" : 0
    }, {
      "komponent" : "NOx",
      "beloep" : 246481.75
    }, {
      "komponent" : "Slagvolum",
      "beloep" : 0
    }, {
      "komponent" : "Co2 Fratrekk",
      "beloep" : 0
    }, {
      "komponent" : "Egenvekt Fratrekk",
      "beloep" : 0
    }, {
      "komponent" : "Motoreffekt Fratrekk",
      "beloep" : 0
    }, {
      "komponent" : "NOx Fratrekk",
      "beloep" : 0
    }, {
      "komponent" : "Slagvolum Fratrekk",
      "beloep" : 0
    }, {
      "komponent" : "Co2 Sum",
      "beloep" : 135997.58
    }, {
      "komponent" : "Egenvekt Sum",
      "beloep" : 64786.6
    }, {
      "komponent" : "Motoreffekt Sum",
      "beloep" : 0
    }, {
      "komponent" : "NOx Sum",
      "beloep" : 246481.75
    }, {
      "komponent" : "Slagvolum Sum",
      "beloep" : 0
    }, {
      "komponent" : "Vrakpant",
      "beloep" : 2400
    } ],
    "avgiftslinje" : {
      "avgiftsopplysning" : {
        "saeravgiftTypekode" : "EM",
        "saeravgiftGruppekode" : "A"
      },
      "avgiftsdato" : "2017-01-01",
      "avgiftsbeloep" : 449665
    },
    "tidsstempel" : "2017-12-19+01:00"
  }
}




Cheers
Bjørn



Fra: Myklebust, Bjørn Magnar <[hidden email]>
Sendt: 1. juli 2020 08:54
Til: [hidden email]
Emne: SV: Replacing a base64-encoded field in a JSON-document with its decoded/converted value
 

Thanks, Andy.

No, I’m sure there is no reason for that – it’s just that I’m fairly new to NiFi and don’t know it too well yet.

 

Thanks,

Bjørn

 

 

Fra: Andy LoPresto <[hidden email]>
Sendt: tirsdag 30. juni 2020 18:37
Til: [hidden email]
Emne: Re: Replacing a base64-encoded field in a JSON-document with its decoded/converted value

 

You should not need to explicitly set the additional module directory to cover that location. Is there a reason you can’t use the native Groovy JSON [1] parsing? That way you don’t have to download any additional libraries. 

 

 

Andy LoPresto
[hidden email]
[hidden email]
He/Him

PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69



On Jun 29, 2020, at 7:41 AM, Myklebust, Bjørn Magnar <[hidden email]> wrote:

 

Andy, just a quick followup on this.

 

I wanted to test a groovy-script with this code (not finished by far yet), and the script is placed in the Script Body part of an ExecuteGroovyScript-process in NiFi:

 

 

import org.json.JSONObject

import org.json.XML

import org.apache.commons.io.IOUtils

import java.nio.charset.*

 

def flowFile = session.get()

if (!flowFile) return

 

flowFile = session.write(flowFile,

  {inputStream, outputStream ->

      def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)

      def xmlJSONObj = XML.toJSONObject(text);

      def json = xmlJSONObj.toString();

      outputStream.write(json.getBytes(StandardCharsets.UTF_8))

  } as StreamCallback)

 

session.transfer(flowFile, ExecuteScript.REL_SUCCESS)

 

But when trying to run this I get the message «unable to resolve class org.json.JSONObject @ line 1»

I have downloaded the jar file from this site:  https://repo1.maven.org/maven2/org/json/json/20200518/json-20200518.jar

And placed it in my nifi/lib-directory.

And the content of this jar you can see in the enclosed png-picture.

 

Do I need to set a value for the property Additional Classpath when the jar-file is stored in the lib-directory?

 

Thanks,

Bjørn

 

 

Fra: Andy LoPresto <[hidden email]> 
Sendt: torsdag 25. juni 2020 19:20
Til: [hidden email]
Emne: Re: Replacing a base64-encoded field in a JSON-document with its decoded/converted value

 

Hi Bjørn,

 

No, XML to JSON conversion is not an Expression Language feature. You’ll need to either get this data into a flowfile as the complete content to perform the conversion with existing built-in tools, or add that step to your Groovy script. 

 

With that additional requirement, I think using the Groovy script to perform those steps in tandem is probably the most performant and logical approach here. 

 

 

Andy LoPresto
[hidden email]
[hidden email]
He/Him

PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69




On Jun 24, 2020, at 11:25 PM, Myklebust, Bjørn Magnar <[hidden email]> wrote:

 

Thanks Andy.

The XML-content is around 5 kB-ish.  But I also need to convert the XML to JSON before replacing it back into the original JSON-file.  Can this be done with e.g a ConvertAttribute before the ReplaceText?

 

Thanks,

Bjørn

 

 

 

Fra: Andy LoPresto <[hidden email]> 
Sendt: onsdag 24. juni 2020 17:24
Til: 
[hidden email]
Emne: Re: Replacing a base64-encoded field in a JSON-document with its decoded/converted value

 

Hello Bjørn, 

 

If the size of the encoded XML document is small (under ~1 KB), you can extract the Base64-encoded value to a flowfile attribute using EvaluateJSONPath, perform the decoding using the base64Decode Expression Language function [1], and then replace it into the flowfile JSON content using ReplaceText (using some regex like "content": ".*" -> “content": ”${decodedXML}” where decodedXML is the name of the attribute you are using). 

 

If the XML content could be very large, this will negatively affect your performance, as attributes are stored directly in memory and handling large amounts of data will impact the heap. In this case, I would recommend writing a Groovy script in ExecuteScript processor to leverage Groovy’s very friendly JSON handling and extract the value, Base64 decode it, and replace it in a couple lines. 

 

Hope this helps. 

 

 

 

Andy LoPresto
[hidden email]
[hidden email]
He/Him

PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69





On Jun 24, 2020, at 4:24 AM, Myklebust, Bjørn Magnar <[hidden email]> wrote:

 

 

Hi.

I have a set of Json-files which contain a base64-coded field (Jsonpath to this field is $.data.content), and this field contains a XML-document.  Decoding the field works as expected, so does the conversion from xml to json,  and I'm able to write the content from this field to a file in a bucket in S3.  But what I would like to do is to be able to replace the coded value for this field in the original file with the decoded/converted value in stead of writing the decoded/converted value to file. And after replacing the json-value then I can write the updated Json-file to a new S3 bucket.

My process look like this at the moment, and works fine for getting the data to file, but it's missing the last part of replacing $.data.content with the decoded/converted data.

 

So how can I do the last part?

 

<pastedImage.png>

 

The EvaluedJsonPath looks like this:

 

<EvaluateJsonPath.png>

 

The ReplaceText looks like this:

 

<ReplaceText.png>

The Base64EncodeContent looks like this:

 

<Base64Decode.png>

and finally, the CovertRecord looks like this:

 

<ConvertRecord.png>

 

 

This is a testfile for that I'm working with:

 

{

  "header": {

    "dokumentidentifikator": null,

    "dokumentidentifikatorV2": "dcff985b-c652-4085-b8f1-45a2f4b6d150",

    "revisjonsnummer": 1,

    "dokumentnavn": "Engangsavgiftfastsettelse:55TEST661122334455:44BIL1:2017-10-20",

    "dokumenttype": "SKATTEMELDING_ENGANGSAVGIFT",

    "dokumenttilstand": "OPPRETTET",

    "gyldig": true,

    "gjelderInntektsaar": 2017,

    "gjelderPeriode": "2017_10",

    "gjelderPart": {

      "partsnummer": 5544332211,

      "identifiseringstype": "MASKINELL",

      "identifikator": null

    },

    "opphavspart": {

      "partsnummer": 5544332211,

      "identifikator": null

    },

    "kildereferanse": {

      "kildesystem": "ENGANGSAVGIFTFASTSETTELSE",

      "gruppe": "",

      "referanse": "aef147fb-8ce8-43ef-833b-7aa3bac1ece0",

      "tidspunkt": "2018-01-16T13:28:02.49+01:00"

    }

  },

  "data": {

    "metadata": {

      "format": "ske:fastsetting:motorvogn:motorvognavgift:v1",

      "bytes": 4420,

      "mimeType": "application/xml",

      "sha1": "c0AowOsTdNdo6VufeSsZqTphc0Y="

    },

    "content": "PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiIHN0YW5kYWxvbmU9InllcyI/Pgo8bW90b3J2b2duYXZnaWZ0IHhtbG5zPSJza2U6ZmFzdHNldHRpbmc6bW90b3J2b2duOm1vdG9ydm9nbmF2Z2lmdDp2MSI+CiAgICA8YXZnaWZ0c2xpbmplPgogICAgICAgIDxhdmdpZnRzYmVsb2VwPjU0Mjg5Ni4wMDwvYXZnaWZ0c2JlbG9lcD4KICAgICAgICA8YXZnaWZ0c29wcGx5c25pbmc+CiAgICAgICAgICAgIDxzYWVyYXZnaWZ0VHlwZWtvZGU+QkI8L3NhZXJhdmdpZnRUeXBla29kZT4KICAgICAgICAgICAgPHNhZXJhdmdpZnRHcnVwcGVrb2RlPlg8L3NhZXJhdmdpZnRHcnVwcGVrb2RlPgogICAgICAgIDwvYXZnaWZ0c29wcGx5c25pbmc+CiAgICAgICAgPGF2Z2lmdHNkYXRvPjIwMTctMTAtMjA8L2F2Z2lmdHNkYXRvPgogICAgPC9hdmdpZnRzbGluamU+CiAgICA8YmV0YWxpbmdzaW5mb3JtYXNqb24+CiAgICAgICAgPGtpZG51bW1lcj4xMDEwMTAxMDEwMTA8L2tpZG51bW1lcj4KICAgICAgICA8Zm9yZmFsbHNkYXRvPjIwMTctMTAtMjA8L2ZvcmZhbGxzZGF0bz4KICAgICAgICA8ZmFrdHVyYWRhdG8+MjAxNy0xMC0yMDwvZmFrdHVyYWRhdG8+CiAgICAgICAgPHRvdGFsQXZnaWZ0c2JlbG9lcD41NDI4OTYuMDA8L3RvdGFsQXZnaWZ0c2JlbG9lcD4KICAgIDwvYmV0YWxpbmdzaW5mb3JtYXNqb24+CiAgICA8bW90b3J2b2duYXZnaWZ0c3R5cGU+ZW5nYW5nc2F2Z2lmdDwvbW90b3J2b2duYXZnaWZ0c3R5cGU+CiAgICA8dGlkc3N0ZW1wZWw+MjAxOC0wMS0xNiswMTowMDwvdGlkc3N0ZW1wZWw+CiAgICA8Z3J1bm5sYWdGb3JNb3RvcnZvZ25hdmdpZnQ+CiAgICAgICAgPGtqb2VyaW5nZW5zQXJ0PjEwPC9ram9lcmluZ2Vuc0FydD4KICAgICAgICA8a2pvZXJldG9leT4KICAgICAgICAgICAgPGVpZXJza2FwUmVnaXN0cmVydD4yMDE3LTEwLTIwPC9laWVyc2thcFJlZ2lzdHJlcnQ+CiAgICAgICAgICAgIDxmb2Vyc3RlUmVnaXN0cmVyaW5nc2Fhcj4yMDE3PC9mb2Vyc3RlUmVnaXN0cmVyaW5nc2Fhcj4KICAgICAgICAgICAgPGZvZXJzdGVSZWdpc3RyZXJpbmdzZGF0b0lOb3JnZT4yMDE3LTEwLTIwPC9mb2Vyc3RlUmVnaXN0cmVyaW5nc2RhdG9JTm9yZ2U+CiAgICAgICAgICAgIDxram9lcmV0b2V5Z3J1cHBlPjEwMTwva2pvZXJldG9leWdydXBwZT4KICAgICAgICAgICAgPGxlbmdkZT4zOTY0PC9sZW5nZGU+CiAgICAgICAgICAgIDxtb3RvcmVmZmVrdD45NjwvbW90b3JlZmZla3Q+CiAgICAgICAgICAgIDxzbGFndm9sdW0+MTQzPC9zbGFndm9sdW0+CiAgICAgICAgICAgIDxkcml2c3RvZmY+QkVOU0lOPC9kcml2c3RvZmY+CiAgICAgICAgICAgIDxlZ2VudmVrdD4xNTE5PC9lZ2VudmVrdD4KICAgICAgICAgICAgPGVpZXI+CiAgICAgICAgICAgICAgICA8Zm9lZHNlbHNFbGxlckRudW1tZXI+MTEyMjMzNDQ1NTwvZm9lZHNlbHNFbGxlckRudW1tZXI+CiAgICAgICAgICAgICAgICA8cGFydHNudW1tZXI+NTU0NDMzMjIxMTwvcGFydHNudW1tZXI+CiAgICAgICAgICAgICAgICA8bmF2bj5LTEFSQSBLVTwvbmF2bj4KICAgICAgICAgICAgPC9laWVyPgogICAgICAgICAgICA8dGlsbGF0dFRvdGFsdmVrdD4yMTY0PC90aWxsYXR0VG90YWx2ZWt0PgogICAgICAgICAgICA8aHlicmlkPm5laTwvaHlicmlkPgogICAgICAgICAgICA8Y28ydXRzbGlwcD4yNjg8L2NvMnV0c2xpcHA+CiAgICAgICAgICAgIDxub3h1dHNsaXBwPjU5LjQ8L25veHV0c2xpcHA+CiAgICAgICAgICAgIDxram9lcmV0b2V5aWRlbnRpZmlrYXRvcj4KICAgICAgICAgICAgICAgIDxram9lcmV0b2V5VW5pa0lkZW50aWZpa2F0b3I+QUJDREVGR0hJSjwva2pvZXJldG9leVVuaWtJZGVudGlmaWthdG9yPgogICAgICAgICAgICAgICAgPGtqZW5uZW1lcmtlPjQ0QklMMTwva2plbm5lbWVya2U+CiAgICAgICAgICAgICAgICA8dW5kZXJzdGVsbHNudW1tZXI+VU5ERVJTVEVMTDQ0PC91bmRlcnN0ZWxsc251bW1lcj4KICAgICAgICAgICAgPC9ram9lcmV0b2V5aWRlbnRpZmlrYXRvcj4KICAgICAgICA8L2tqb2VyZXRvZXk+CiAgICA8L2dydW5ubGFnRm9yTW90b3J2b2duYXZnaWZ0PgogICAgPGF2Z2lmdHNwbGlrdGlnPgogICAgICAgIDxmb2Vkc2Vsc0VsbGVyRG51bW1lcj4xMTIyMzM0NDU1PC9mb2Vkc2Vsc0VsbGVyRG51bW1lcj4KICAgICAgICA8cGFydHNudW1tZXI+NTU0NDMzMjIxMTwvcGFydHNudW1tZXI+CiAgICA8L2F2Z2lmdHNwbGlrdGlnPgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzI8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjQ3NTMzNy4yMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5FZ2VudmVrdDwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+NjA5NDUuNjQ8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+TW90b3JlZmZla3Q8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94PC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD40MjEzLjI0PC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlNsYWd2b2x1bTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzIgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+RWdlbnZla3QgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+TW90b3JlZmZla3QgRnJhdHJla2s8L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94IEZyYXRyZWtrPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4wLjAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlNsYWd2b2x1bSBGcmF0cmVrazwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5DbzIgU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD40NzUzMzcuMjA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+RWdlbnZla3QgU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD42MDk0NS42NDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5Nb3RvcmVmZmVrdCBTdW08L2tvbXBvbmVudD4KICAgICAgICA8YmVsb2VwPjAuMDA8L2JlbG9lcD4KICAgIDwvYXZnaWZ0c2tvbXBvbmVudD4KICAgIDxhdmdpZnRza29tcG9uZW50PgogICAgICAgIDxrb21wb25lbnQ+Tk94IFN1bTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+NDIxMy4yNDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGF2Z2lmdHNrb21wb25lbnQ+CiAgICAgICAgPGtvbXBvbmVudD5TbGFndm9sdW0gU3VtPC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4wLjAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PlZyYWtwYW50PC9rb21wb25lbnQ+CiAgICAgICAgPGJlbG9lcD4yNDAwPC9iZWxvZXA+CiAgICA8L2F2Z2lmdHNrb21wb25lbnQ+CiAgICA8YXZnaWZ0c2tvbXBvbmVudD4KICAgICAgICA8a29tcG9uZW50PkJydWtzZnJhZHJhZyAwJTwva29tcG9uZW50PgogICAgICAgIDxiZWxvZXA+MC4wMDwvYmVsb2VwPgogICAgPC9hdmdpZnRza29tcG9uZW50PgogICAgPGZvcmhhYW5kc2JlcmVnbmluZz5mYWxzZTwvZm9yaGFhbmRzYmVyZWduaW5nPgo8L21vdG9ydm9nbmF2Z2lmdD4K"

  },

  "extension": null,

  "skjemaversjon": "v3_0"

}

 

 

 

 

Cheers,

Bjørn

 

 


Denne e-posten og eventuelle vedlegg er beregnet utelukkende for den institusjon eller person den er rettet til og kan være belagt med lovbestemt taushetsplikt. Dersom e-posten er feilsendt, vennligst slett den og kontakt Skatteetaten. 
The contents of this email message and any attachments are intended solely for the addressee(s) and may contain confidential information and may be legally protected from disclosure. If you are not the intended recipient of this message, please immediately delete the message and alert the Norwegian Tax Administration.

 

<org.json.JSONObject.png><executegroovyscript.png>