How to reference attribute by variable name

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

How to reference attribute by variable name

James McMahon
Hello. I currently have a python script in which I set values in a dictionary from flowfile attributes, referencing them directly by name. I then create a json string, which I can send to an AMQP broker as a message. Something like this:
result['absolute.path'] = flowFile.getAttribute('absolute.path')
result['filename'] = flowFile.getAttribute('filename')
result['myAttr'] = flowFile.getAttribute('myAttr')
# Serialize the object to JSON...
json_str = json.dumps(result)
# Replace flowfile payload with this string representation of my json...
outputStream.write(unicode(json_str).encode('utf-8'))

I need to generalize this. I want to create attribute properties in my ExecuteScript processor from which I call this python, each property being a value I want to include in my json message.

Assuming I create an attribute property in the processor called draftPick, and I set it to "Mahomes, KC". I read it into the script something like this:
thisPick =  name of the property field .evaluateAttributeExpressions().getValue()

My first question: can I reference that incoming value in this expression directly by variable reference without quoting as above, and is there a way I can auto detect the property name to use as my key value in my dictionary?
result[name of the property field] = flowFile.getAttribute(thisPick)

My second question: is there a way to generalize this so that it dynamically identifies and handles any variable number of properties and values defined in my ExecuteScript processor? This would be the ideal solution, because then I can have one python script that dynamically handles any number of  properties I set in the processor. I would do that to grab whichever subset of attributes I happened to want to send to my AMQP as a message at that time.

Jim
Reply | Threaded
Open this post in threaded view
|

Re: How to reference attribute by variable name

Mark Payne
Jim,

This isn’t a direct answer to your question. I don’t remember whether or not ExecuteScript provides access directly to the ProcessContext or not - if it does, then that provides methods for getting all properties and determining whether or not the property is “dynamic” (aka user-defined).

But given the description of this script, why not simply use AttributesToJSON? It already provides a property for supplying a comma-separated list of attributes and converts them into a JSON payload.

Thanks
-Mark


On Jun 10, 2020, at 10:37 AM, James McMahon <[hidden email]> wrote:

Hello. I currently have a python script in which I set values in a dictionary from flowfile attributes, referencing them directly by name. I then create a json string, which I can send to an AMQP broker as a message. Something like this:
result['absolute.path'] = flowFile.getAttribute('absolute.path')
result['filename'] = flowFile.getAttribute('filename')
result['myAttr'] = flowFile.getAttribute('myAttr')
# Serialize the object to JSON...
json_str = json.dumps(result)
# Replace flowfile payload with this string representation of my json...
outputStream.write(unicode(json_str).encode('utf-8'))

I need to generalize this. I want to create attribute properties in my ExecuteScript processor from which I call this python, each property being a value I want to include in my json message.

Assuming I create an attribute property in the processor called draftPick, and I set it to "Mahomes, KC". I read it into the script something like this:
thisPick =  name of the property field .evaluateAttributeExpressions().getValue()

My first question: can I reference that incoming value in this expression directly by variable reference without quoting as above, and is there a way I can auto detect the property name to use as my key value in my dictionary?
result[name of the property field] = flowFile.getAttribute(thisPick)

My second question: is there a way to generalize this so that it dynamically identifies and handles any variable number of properties and values defined in my ExecuteScript processor? This would be the ideal solution, because then I can have one python script that dynamically handles any number of  properties I set in the processor. I would do that to grab whichever subset of attributes I happened to want to send to my AMQP as a message at that time.

Jim

Reply | Threaded
Open this post in threaded view
|

Re: How to reference attribute by variable name

James McMahon
Mark, where can I find the methods that associate with ProcessContext? I can give it a try. I'm not smart enough to know how to tell whether a processor gives us access to ProcessContext.

AttributesToJSON is an excellent "out of the box" solution when the values exist as attributes - I can then reference those in that comma-separated list of attributes to include in the JSON (alternatively it can include all attributes, which you already know). It wasn't clear to me, though, that a property I define in that same processor would be available to it to reference in property "Attributes List" in the same processor where it is defined as a property itself. Something like this:
Attributes List ......................... filename,absolute.path,newPropertyInProcessor
.
.
.
newPropertyInProcessor .......... aValue

I'll test that. That would be simpler if I can reference new processor properties in the Attributes list. Thanks Mark.

On Wed, Jun 10, 2020 at 10:47 AM Mark Payne <[hidden email]> wrote:
Jim,

This isn’t a direct answer to your question. I don’t remember whether or not ExecuteScript provides access directly to the ProcessContext or not - if it does, then that provides methods for getting all properties and determining whether or not the property is “dynamic” (aka user-defined).

But given the description of this script, why not simply use AttributesToJSON? It already provides a property for supplying a comma-separated list of attributes and converts them into a JSON payload.

Thanks
-Mark


On Jun 10, 2020, at 10:37 AM, James McMahon <[hidden email]> wrote:

Hello. I currently have a python script in which I set values in a dictionary from flowfile attributes, referencing them directly by name. I then create a json string, which I can send to an AMQP broker as a message. Something like this:
result['absolute.path'] = flowFile.getAttribute('absolute.path')
result['filename'] = flowFile.getAttribute('filename')
result['myAttr'] = flowFile.getAttribute('myAttr')
# Serialize the object to JSON...
json_str = json.dumps(result)
# Replace flowfile payload with this string representation of my json...
outputStream.write(unicode(json_str).encode('utf-8'))

I need to generalize this. I want to create attribute properties in my ExecuteScript processor from which I call this python, each property being a value I want to include in my json message.

Assuming I create an attribute property in the processor called draftPick, and I set it to "Mahomes, KC". I read it into the script something like this:
thisPick =  name of the property field .evaluateAttributeExpressions().getValue()

My first question: can I reference that incoming value in this expression directly by variable reference without quoting as above, and is there a way I can auto detect the property name to use as my key value in my dictionary?
result[name of the property field] = flowFile.getAttribute(thisPick)

My second question: is there a way to generalize this so that it dynamically identifies and handles any variable number of properties and values defined in my ExecuteScript processor? This would be the ideal solution, because then I can have one python script that dynamically handles any number of  properties I set in the processor. I would do that to grab whichever subset of attributes I happened to want to send to my AMQP as a message at that time.

Jim

Reply | Threaded
Open this post in threaded view
|

Re: How to reference attribute by variable name

Mark Payne
Unfortunately, it doesn’t appear that the processor documents the variables that it provides. But it does provide access to the context, as the “context” variable:

bindings.put("session", session);
bindings.put("context", context);
bindings.put("log", log);
bindings.put("REL_SUCCESS", REL_SUCCESS);
bindings.put("REL_FAILURE", REL_FAILURE);
The JavaDocs for ProcessContext [1] can then be used to determine what’s available.

Thanks
-Mark


On Jun 10, 2020, at 11:02 AM, James McMahon <[hidden email]> wrote:

Mark, where can I find the methods that associate with ProcessContext? I can give it a try. I'm not smart enough to know how to tell whether a processor gives us access to ProcessContext.

AttributesToJSON is an excellent "out of the box" solution when the values exist as attributes - I can then reference those in that comma-separated list of attributes to include in the JSON (alternatively it can include all attributes, which you already know). It wasn't clear to me, though, that a property I define in that same processor would be available to it to reference in property "Attributes List" in the same processor where it is defined as a property itself. Something like this:
Attributes List ......................... filename,absolute.path,newPropertyInProcessor
.
.
.
newPropertyInProcessor .......... aValue

I'll test that. That would be simpler if I can reference new processor properties in the Attributes list. Thanks Mark.

On Wed, Jun 10, 2020 at 10:47 AM Mark Payne <[hidden email]> wrote:
Jim,

This isn’t a direct answer to your question. I don’t remember whether or not ExecuteScript provides access directly to the ProcessContext or not - if it does, then that provides methods for getting all properties and determining whether or not the property is “dynamic” (aka user-defined).

But given the description of this script, why not simply use AttributesToJSON? It already provides a property for supplying a comma-separated list of attributes and converts them into a JSON payload.

Thanks
-Mark


On Jun 10, 2020, at 10:37 AM, James McMahon <[hidden email]> wrote:

Hello. I currently have a python script in which I set values in a dictionary from flowfile attributes, referencing them directly by name. I then create a json string, which I can send to an AMQP broker as a message. Something like this:
result['absolute.path'] = flowFile.getAttribute('absolute.path')
result['filename'] = flowFile.getAttribute('filename')
result['myAttr'] = flowFile.getAttribute('myAttr')
# Serialize the object to JSON...
json_str = json.dumps(result)
# Replace flowfile payload with this string representation of my json...
outputStream.write(unicode(json_str).encode('utf-8'))

I need to generalize this. I want to create attribute properties in my ExecuteScript processor from which I call this python, each property being a value I want to include in my json message.

Assuming I create an attribute property in the processor called draftPick, and I set it to "Mahomes, KC". I read it into the script something like this:
thisPick =  name of the property field .evaluateAttributeExpressions().getValue()

My first question: can I reference that incoming value in this expression directly by variable reference without quoting as above, and is there a way I can auto detect the property name to use as my key value in my dictionary?
result[name of the property field] = flowFile.getAttribute(thisPick)

My second question: is there a way to generalize this so that it dynamically identifies and handles any variable number of properties and values defined in my ExecuteScript processor? This would be the ideal solution, because then I can have one python script that dynamically handles any number of  properties I set in the processor. I would do that to grab whichever subset of attributes I happened to want to send to my AMQP as a message at that time.

Jim


Reply | Threaded
Open this post in threaded view
|

Re: How to reference attribute by variable name

James McMahon
Very good. I will look into that more.
One way to solve this, I think, is for me to define anything I want to use in an updateAttribute processor just prior to, and then I can reference in Attributes List in AttributesToJSON (saving the JSON as flowfile content for send to RabbitMQ).
Thanks for your thoughts and comments Mark. They are very helpful.

On Wed, Jun 10, 2020 at 11:06 AM Mark Payne <[hidden email]> wrote:
Unfortunately, it doesn’t appear that the processor documents the variables that it provides. But it does provide access to the context, as the “context” variable:

bindings.put("session", session);
bindings.put("context", context);
bindings.put("log", log);
bindings.put("REL_SUCCESS", REL_SUCCESS);
bindings.put("REL_FAILURE", REL_FAILURE);
The JavaDocs for ProcessContext [1] can then be used to determine what’s available.

Thanks
-Mark


On Jun 10, 2020, at 11:02 AM, James McMahon <[hidden email]> wrote:

Mark, where can I find the methods that associate with ProcessContext? I can give it a try. I'm not smart enough to know how to tell whether a processor gives us access to ProcessContext.

AttributesToJSON is an excellent "out of the box" solution when the values exist as attributes - I can then reference those in that comma-separated list of attributes to include in the JSON (alternatively it can include all attributes, which you already know). It wasn't clear to me, though, that a property I define in that same processor would be available to it to reference in property "Attributes List" in the same processor where it is defined as a property itself. Something like this:
Attributes List ......................... filename,absolute.path,newPropertyInProcessor
.
.
.
newPropertyInProcessor .......... aValue

I'll test that. That would be simpler if I can reference new processor properties in the Attributes list. Thanks Mark.

On Wed, Jun 10, 2020 at 10:47 AM Mark Payne <[hidden email]> wrote:
Jim,

This isn’t a direct answer to your question. I don’t remember whether or not ExecuteScript provides access directly to the ProcessContext or not - if it does, then that provides methods for getting all properties and determining whether or not the property is “dynamic” (aka user-defined).

But given the description of this script, why not simply use AttributesToJSON? It already provides a property for supplying a comma-separated list of attributes and converts them into a JSON payload.

Thanks
-Mark


On Jun 10, 2020, at 10:37 AM, James McMahon <[hidden email]> wrote:

Hello. I currently have a python script in which I set values in a dictionary from flowfile attributes, referencing them directly by name. I then create a json string, which I can send to an AMQP broker as a message. Something like this:
result['absolute.path'] = flowFile.getAttribute('absolute.path')
result['filename'] = flowFile.getAttribute('filename')
result['myAttr'] = flowFile.getAttribute('myAttr')
# Serialize the object to JSON...
json_str = json.dumps(result)
# Replace flowfile payload with this string representation of my json...
outputStream.write(unicode(json_str).encode('utf-8'))

I need to generalize this. I want to create attribute properties in my ExecuteScript processor from which I call this python, each property being a value I want to include in my json message.

Assuming I create an attribute property in the processor called draftPick, and I set it to "Mahomes, KC". I read it into the script something like this:
thisPick =  name of the property field .evaluateAttributeExpressions().getValue()

My first question: can I reference that incoming value in this expression directly by variable reference without quoting as above, and is there a way I can auto detect the property name to use as my key value in my dictionary?
result[name of the property field] = flowFile.getAttribute(thisPick)

My second question: is there a way to generalize this so that it dynamically identifies and handles any variable number of properties and values defined in my ExecuteScript processor? This would be the ideal solution, because then I can have one python script that dynamically handles any number of  properties I set in the processor. I would do that to grab whichever subset of attributes I happened to want to send to my AMQP as a message at that time.

Jim