Combine Attributes & Content

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Combine Attributes & Content

Dweep Sharma
Hi All,

Does anyone have a strategy to combine Attributes & Content into a single Flowfile?

AttributesToJson seems like the best way convert all attributes to content -Json but I 

lose the original content. 

Basically, I have a HandleHTTPRequest Processor and headers and body must be combined,

currently Headers are stored as Attributes in a flowfile 

-Dweep


::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------

The contents of this e-mail and any attachments are confidential and intended for the named recipient(s) only.E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents(with or without referred errors) shall therefore not attach any liability on the originator or redBus.com. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of redBus.com. Any form of reproduction, dissemination, copying, disclosure, modification,distribution and / or publication of this message without the prior written consent of authorized representative of redbus.com is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.Before opening any email and/or attachments, please check them for viruses and other defects.
Reply | Threaded
Open this post in threaded view
|

Re: Combine Attributes & Content

Matt Burgess-2
Dweep,

Depending on how complex the content JSON is, you might be able to use
ReplaceText to smuggle the attributes into the text, but this can be
tricky as you need to match on the opening JSON and the rest, and then
replace it with the opening JSON, the attributes, then the rest in
order to preserve valid JSON.

You may also be able to use UpdateRecord to add fields to your
content, I believe if the Replacement Value Strategy property is set
to "Literal Value" you can use Expression Language which may be able
access flow file attributes.

An alternative is to use a scripted processor if you're comfortable
with a scripting language such as Groovy, as it can "slurp" JSON into
a Plain Old Groovy Object (POGO), then you can add fields and
serialize back out to JSON. I have a somewhat similar example here
[1].

Regards,
Matt

[1] http://funnifi.blogspot.com/2016/02/executescript-json-to-json-conversion.html

On Tue, May 19, 2020 at 10:46 AM Dweep Sharma <[hidden email]> wrote:

>
> Hi All,
>
> Does anyone have a strategy to combine Attributes & Content into a single Flowfile?
>
> AttributesToJson seems like the best way convert all attributes to content -Json but I
>
> lose the original content.
>
> Basically, I have a HandleHTTPRequest Processor and headers and body must be combined,
>
> currently Headers are stored as Attributes in a flowfile
>
> -Dweep
>
>
> ::DISCLAIMER::
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> The contents of this e-mail and any attachments are confidential and intended for the named recipient(s) only.E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents(with or without referred errors) shall therefore not attach any liability on the originator or redBus.com. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of redBus.com. Any form of reproduction, dissemination, copying, disclosure, modification,distribution and / or publication of this message without the prior written consent of authorized representative of redbus.com is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.Before opening any email and/or attachments, please check them for viruses and other defects.
Reply | Threaded
Open this post in threaded view
|

Re: Combine Attributes & Content

Dweep Sharma
Thanks Matt, UpdateRecord works however, all headers must be added manually (unlike AttributesToJson). Since it is one time effort, it will do for now. 

-Dweep

On Tue, May 19, 2020 at 9:16 PM Matt Burgess <[hidden email]> wrote:
Dweep,

Depending on how complex the content JSON is, you might be able to use
ReplaceText to smuggle the attributes into the text, but this can be
tricky as you need to match on the opening JSON and the rest, and then
replace it with the opening JSON, the attributes, then the rest in
order to preserve valid JSON.

You may also be able to use UpdateRecord to add fields to your
content, I believe if the Replacement Value Strategy property is set
to "Literal Value" you can use Expression Language which may be able
access flow file attributes.

An alternative is to use a scripted processor if you're comfortable
with a scripting language such as Groovy, as it can "slurp" JSON into
a Plain Old Groovy Object (POGO), then you can add fields and
serialize back out to JSON. I have a somewhat similar example here
[1].

Regards,
Matt

[1] http://funnifi.blogspot.com/2016/02/executescript-json-to-json-conversion.html

On Tue, May 19, 2020 at 10:46 AM Dweep Sharma <[hidden email]> wrote:
>
> Hi All,
>
> Does anyone have a strategy to combine Attributes & Content into a single Flowfile?
>
> AttributesToJson seems like the best way convert all attributes to content -Json but I
>
> lose the original content.
>
> Basically, I have a HandleHTTPRequest Processor and headers and body must be combined,
>
> currently Headers are stored as Attributes in a flowfile
>
> -Dweep
>
>
> ::DISCLAIMER::
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> The contents of this e-mail and any attachments are confidential and intended for the named recipient(s) only.E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents(with or without referred errors) shall therefore not attach any liability on the originator or redBus.com. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of redBus.com. Any form of reproduction, dissemination, copying, disclosure, modification,distribution and / or publication of this message without the prior written consent of authorized representative of redbus.com is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.Before opening any email and/or attachments, please check them for viruses and other defects.

::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------

The contents of this e-mail and any attachments are confidential and intended for the named recipient(s) only.E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents(with or without referred errors) shall therefore not attach any liability on the originator or redBus.com. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of redBus.com. Any form of reproduction, dissemination, copying, disclosure, modification,distribution and / or publication of this message without the prior written consent of authorized representative of redbus.com is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.Before opening any email and/or attachments, please check them for viruses and other defects.
Reply | Threaded
Open this post in threaded view
|

Re: Combine Attributes & Content

Edward Armes
Hi Sharma,

An alternative could be, duplicate the FlowFile from the HandleHTTPRequest Processor have one wait while the other goes through an AttributesToJSON and stores the JSON in a lookup cache. The other FlowFile then grabs that json from the lookup cache and then use ReplaceText processor mode to add that specific attribute into the content.

Edward

On Thu, May 21, 2020 at 10:11 AM Dweep Sharma <[hidden email]> wrote:
Thanks Matt, UpdateRecord works however, all headers must be added manually (unlike AttributesToJson). Since it is one time effort, it will do for now. 

-Dweep

On Tue, May 19, 2020 at 9:16 PM Matt Burgess <[hidden email]> wrote:
Dweep,

Depending on how complex the content JSON is, you might be able to use
ReplaceText to smuggle the attributes into the text, but this can be
tricky as you need to match on the opening JSON and the rest, and then
replace it with the opening JSON, the attributes, then the rest in
order to preserve valid JSON.

You may also be able to use UpdateRecord to add fields to your
content, I believe if the Replacement Value Strategy property is set
to "Literal Value" you can use Expression Language which may be able
access flow file attributes.

An alternative is to use a scripted processor if you're comfortable
with a scripting language such as Groovy, as it can "slurp" JSON into
a Plain Old Groovy Object (POGO), then you can add fields and
serialize back out to JSON. I have a somewhat similar example here
[1].

Regards,
Matt

[1] http://funnifi.blogspot.com/2016/02/executescript-json-to-json-conversion.html

On Tue, May 19, 2020 at 10:46 AM Dweep Sharma <[hidden email]> wrote:
>
> Hi All,
>
> Does anyone have a strategy to combine Attributes & Content into a single Flowfile?
>
> AttributesToJson seems like the best way convert all attributes to content -Json but I
>
> lose the original content.
>
> Basically, I have a HandleHTTPRequest Processor and headers and body must be combined,
>
> currently Headers are stored as Attributes in a flowfile
>
> -Dweep
>
>
> ::DISCLAIMER::
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> The contents of this e-mail and any attachments are confidential and intended for the named recipient(s) only.E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents(with or without referred errors) shall therefore not attach any liability on the originator or redBus.com. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of redBus.com. Any form of reproduction, dissemination, copying, disclosure, modification,distribution and / or publication of this message without the prior written consent of authorized representative of redbus.com is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.Before opening any email and/or attachments, please check them for viruses and other defects.

::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------

The contents of this e-mail and any attachments are confidential and intended for the named recipient(s) only.E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents(with or without referred errors) shall therefore not attach any liability on the originator or redBus.com. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of redBus.com. Any form of reproduction, dissemination, copying, disclosure, modification,distribution and / or publication of this message without the prior written consent of authorized representative of redbus.com is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.Before opening any email and/or attachments, please check them for viruses and other defects.
Reply | Threaded
Open this post in threaded view
|

Re: Combine Attributes & Content

Dweep Sharma
Thanks, I will try this approach as well. I can combine this with some other flows/processes I have in mind. 

On Thu, May 21, 2020 at 3:00 PM Edward Armes <[hidden email]> wrote:
Hi Sharma,

An alternative could be, duplicate the FlowFile from the HandleHTTPRequest Processor have one wait while the other goes through an AttributesToJSON and stores the JSON in a lookup cache. The other FlowFile then grabs that json from the lookup cache and then use ReplaceText processor mode to add that specific attribute into the content.

Edward

On Thu, May 21, 2020 at 10:11 AM Dweep Sharma <[hidden email]> wrote:
Thanks Matt, UpdateRecord works however, all headers must be added manually (unlike AttributesToJson). Since it is one time effort, it will do for now. 

-Dweep

On Tue, May 19, 2020 at 9:16 PM Matt Burgess <[hidden email]> wrote:
Dweep,

Depending on how complex the content JSON is, you might be able to use
ReplaceText to smuggle the attributes into the text, but this can be
tricky as you need to match on the opening JSON and the rest, and then
replace it with the opening JSON, the attributes, then the rest in
order to preserve valid JSON.

You may also be able to use UpdateRecord to add fields to your
content, I believe if the Replacement Value Strategy property is set
to "Literal Value" you can use Expression Language which may be able
access flow file attributes.

An alternative is to use a scripted processor if you're comfortable
with a scripting language such as Groovy, as it can "slurp" JSON into
a Plain Old Groovy Object (POGO), then you can add fields and
serialize back out to JSON. I have a somewhat similar example here
[1].

Regards,
Matt

[1] http://funnifi.blogspot.com/2016/02/executescript-json-to-json-conversion.html

On Tue, May 19, 2020 at 10:46 AM Dweep Sharma <[hidden email]> wrote:
>
> Hi All,
>
> Does anyone have a strategy to combine Attributes & Content into a single Flowfile?
>
> AttributesToJson seems like the best way convert all attributes to content -Json but I
>
> lose the original content.
>
> Basically, I have a HandleHTTPRequest Processor and headers and body must be combined,
>
> currently Headers are stored as Attributes in a flowfile
>
> -Dweep
>
>
> ::DISCLAIMER::
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> The contents of this e-mail and any attachments are confidential and intended for the named recipient(s) only.E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents(with or without referred errors) shall therefore not attach any liability on the originator or redBus.com. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of redBus.com. Any form of reproduction, dissemination, copying, disclosure, modification,distribution and / or publication of this message without the prior written consent of authorized representative of redbus.com is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.Before opening any email and/or attachments, please check them for viruses and other defects.

::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------

The contents of this e-mail and any attachments are confidential and intended for the named recipient(s) only.E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents(with or without referred errors) shall therefore not attach any liability on the originator or redBus.com. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of redBus.com. Any form of reproduction, dissemination, copying, disclosure, modification,distribution and / or publication of this message without the prior written consent of authorized representative of redbus.com is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.Before opening any email and/or attachments, please check them for viruses and other defects.

::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------

The contents of this e-mail and any attachments are confidential and intended for the named recipient(s) only.E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents(with or without referred errors) shall therefore not attach any liability on the originator or redBus.com. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of redBus.com. Any form of reproduction, dissemination, copying, disclosure, modification,distribution and / or publication of this message without the prior written consent of authorized representative of redbus.com is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.Before opening any email and/or attachments, please check them for viruses and other defects.