Get Ignite Processor - Upstream Connections is invalid

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Get Ignite Processor - Upstream Connections is invalid

Deepak Kumar
Hi,
I have a cache with data stored in ignite. I am using Get Ignite Processor to get Cache but i am getting the following error "Upstream Connections is invalid because Processor requires an upstream connection but currently has none."
 
1. Please correct me if i am wrong do we need Upstream Connection since its a get tag processor and already we have data in ignite cache. 
2. Or if its mandatory to use Upstream Connections which processor can i use in connection with Get Ignite Processor.
3. Can i get more information on property Ignite Cache Entry Identifier like what value i need to set here.


Thanks & Regards

Deepak

Reply | Threaded
Open this post in threaded view
|

Re: Get Ignite Processor - Upstream Connections is invalid

Bryan Bende
Hello,

I think the description of Cache Entry Identifier might be inaccurate because it says ""A FlowFile attribute, or attribute expression used for determining Ignite cache key for the Flow File content" but in the code it does this:

String key = context.getProperty(IGNITE_CACHE_ENTRY_KEY).evaluateAttributeExpressions(flowFile).getValue();

So the Cache Entry Identifier looks like the name of the key to retrieve, which could be a static value to retrieve the same key over and over, or could be expression language to get the key dynamically from an incoming flow file. 
So lets say in your cache you have key = foo with value = bar, you could have a Flow File with an attribute named cache.key and the value set to foo, and in GetIgnite you can set Cache Entry Identifier to ${cache.key}, or you could just put foo directly into the processor property. 

As far as requiring an incoming Flow File, you are right that this processor should be be called FetchIgnite since "Get" processors are usually source processors. It looks like there would need to be a corresponding ListIgnite that could produce Flow Files for each of the keys available that could then be fed into GetIgnite, but currently that doesn't exist. I don't know anything about Ignite so I don't know if that is possible or not.

-Bryan
 

On Thu, Nov 3, 2016 at 9:32 AM, Deepak Kumar <[hidden email]> wrote:
Hi,
I have a cache with data stored in ignite. I am using Get Ignite Processor to get Cache but i am getting the following error "Upstream Connections is invalid because Processor requires an upstream connection but currently has none."
 
1. Please correct me if i am wrong do we need Upstream Connection since its a get tag processor and already we have data in ignite cache. 
2. Or if its mandatory to use Upstream Connections which processor can i use in connection with Get Ignite Processor.
3. Can i get more information on property Ignite Cache Entry Identifier like what value i need to set here.


Thanks & Regards

Deepak


Reply | Threaded
Open this post in threaded view
|

Re: Get Ignite Processor - Upstream Connections is invalid

Pierre Villard
Hi,

Bryan is correct, the idea here is to have the Ignite cache key (linked to the content to retrieve from Ignite cache) based on an incoming flow file attribute and that is why the processor requires an upstream connection. I understand it might be confusing regarding the name of the processor but I assume the author of the processor didn't see a use case where it would be necessary to get the value of one key over and over. If this is what you need, then you can go with a GenerateFlowFile and UpdateAttribute processors as the upstream part of the GetIgnite processor.

Hope this helps a little.


2016-11-03 15:05 GMT+01:00 Bryan Bende <[hidden email]>:
Hello,

I think the description of Cache Entry Identifier might be inaccurate because it says ""A FlowFile attribute, or attribute expression used for determining Ignite cache key for the Flow File content" but in the code it does this:

String key = context.getProperty(IGNITE_CACHE_ENTRY_KEY).evaluateAttributeExpressions(flowFile).getValue();

So the Cache Entry Identifier looks like the name of the key to retrieve, which could be a static value to retrieve the same key over and over, or could be expression language to get the key dynamically from an incoming flow file. 
So lets say in your cache you have key = foo with value = bar, you could have a Flow File with an attribute named cache.key and the value set to foo, and in GetIgnite you can set Cache Entry Identifier to ${cache.key}, or you could just put foo directly into the processor property. 

As far as requiring an incoming Flow File, you are right that this processor should be be called FetchIgnite since "Get" processors are usually source processors. It looks like there would need to be a corresponding ListIgnite that could produce Flow Files for each of the keys available that could then be fed into GetIgnite, but currently that doesn't exist. I don't know anything about Ignite so I don't know if that is possible or not.

-Bryan
 

On Thu, Nov 3, 2016 at 9:32 AM, Deepak Kumar <[hidden email]> wrote:
Hi,
I have a cache with data stored in ignite. I am using Get Ignite Processor to get Cache but i am getting the following error "Upstream Connections is invalid because Processor requires an upstream connection but currently has none."
 
1. Please correct me if i am wrong do we need Upstream Connection since its a get tag processor and already we have data in ignite cache. 
2. Or if its mandatory to use Upstream Connections which processor can i use in connection with Get Ignite Processor.
3. Can i get more information on property Ignite Cache Entry Identifier like what value i need to set here.


Thanks & Regards

Deepak



Reply | Threaded
Open this post in threaded view
|

Re: Get Ignite Processor - Upstream Connections is invalid

Deepak Kumar
Hi,
Thank you Bryan and Pierre for your valuable suggestion. I had a use case like where i have data in ignite and i need to get that data in nifi and able to process it. 

Thanks & Regards

Deepak


On Thursday, November 3, 2016 9:21 PM, Pierre Villard <[hidden email]> wrote:


Hi,

Bryan is correct, the idea here is to have the Ignite cache key (linked to the content to retrieve from Ignite cache) based on an incoming flow file attribute and that is why the processor requires an upstream connection. I understand it might be confusing regarding the name of the processor but I assume the author of the processor didn't see a use case where it would be necessary to get the value of one key over and over. If this is what you need, then you can go with a GenerateFlowFile and UpdateAttribute processors as the upstream part of the GetIgnite processor.

Hope this helps a little.


2016-11-03 15:05 GMT+01:00 Bryan Bende <[hidden email]>:
Hello,

I think the description of Cache Entry Identifier might be inaccurate because it says ""A FlowFile attribute, or attribute expression used for determining Ignite cache key for the Flow File content" but in the code it does this:

String key = context.getProperty(IGNITE_ CACHE_ENTRY_KEY). evaluateAttributeExpressions( flowFile).getValue();

So the Cache Entry Identifier looks like the name of the key to retrieve, which could be a static value to retrieve the same key over and over, or could be expression language to get the key dynamically from an incoming flow file. 
So lets say in your cache you have key = foo with value = bar, you could have a Flow File with an attribute named cache.key and the value set to foo, and in GetIgnite you can set Cache Entry Identifier to ${cache.key}, or you could just put foo directly into the processor property. 

As far as requiring an incoming Flow File, you are right that this processor should be be called FetchIgnite since "Get" processors are usually source processors. It looks like there would need to be a corresponding ListIgnite that could produce Flow Files for each of the keys available that could then be fed into GetIgnite, but currently that doesn't exist. I don't know anything about Ignite so I don't know if that is possible or not.

-Bryan
 

On Thu, Nov 3, 2016 at 9:32 AM, Deepak Kumar <[hidden email]> wrote:
Hi,
I have a cache with data stored in ignite. I am using Get Ignite Processor to get Cache but i am getting the following error "Upstream Connections is invalid because Processor requires an upstream connection but currently has none."
 
1. Please correct me if i am wrong do we need Upstream Connection since its a get tag processor and already we have data in ignite cache. 
2. Or if its mandatory to use Upstream Connections which processor can i use in connection with Get Ignite Processor.
3. Can i get more information on property Ignite Cache Entry Identifier like what value i need to set here.


Thanks & Regards

Deepak