GetFile Usage

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

GetFile Usage

KhajaAsmath Mohammed
Hi,

I have a question regarding how the getfile runs in the background. Lets say I have 1000 files in the directory, when I use getfile will all 1000 files are read once and loaded into queue or is it sequentially by file?

My requirement is to load all files in queue at a time so that I can come up with approach of dependency management.

Thanks,
Asmath
Reply | Threaded
Open this post in threaded view
|

Re: GetFile Usage

Luis Carmona
Hi Asmath,

I have used GetFile several times, and you can control what you mention with one of the parameters of the Processor. Check the attached image. 

I haven't gone that far like 1.000 files, but I guess it is not a problem depending on the size of each file versus the available resources of your server.

LC



El vie, 22-05-2020 a las 03:56 -0500, KhajaAsmath Mohammed escribió:
Hi,

I have a question regarding how the getfile runs in the background. Lets say I have 1000 files in the directory, when I use getfile will all 1000 files are read once and loaded into queue or is it sequentially by file?

My requirement is to load all files in queue at a time so that I can come up with approach of dependency management.

Thanks,
Asmath
Reply | Threaded
Open this post in threaded view
|

Re: GetFile Usage

Edward Armes
Just to note, I believe that this is the maximum number of files in a given window before the processor chooses to yield.

The processor could finish before reaching that limit if told to by the scheduling framework.

It's worth having a quick look in the source code to see how and when the processor releases FlowFiles to the queue in relation to the number files streamed to the content repository.

Edward

On Fri, 22 May 2020, 16:13 Luis, <[hidden email]> wrote:
Hi Asmath,

I have used GetFile several times, and you can control what you mention with one of the parameters of the Processor. Check the attached image. 

I haven't gone that far like 1.000 files, but I guess it is not a problem depending on the size of each file versus the available resources of your server.

LC



El vie, 22-05-2020 a las 03:56 -0500, KhajaAsmath Mohammed escribió:
Hi,

I have a question regarding how the getfile runs in the background. Lets say I have 1000 files in the directory, when I use getfile will all 1000 files are read once and loaded into queue or is it sequentially by file?

My requirement is to load all files in queue at a time so that I can come up with approach of dependency management.

Thanks,
Asmath

Imagen-QURUK0.png (200K) Download Attachment