Loading set-of files to tables

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Loading set-of files to tables

Samarendra Sahoo
Hi all,
We have a need where we get a list containing a set of source files and target table names. Would like to load the files in parallel (as much as possible). The list need to be picked up from the database and it can change for each of the runs.

Could you pls suggest an approach for this.

FYI, we are able to get this done sequentially by updating the parameter context with the source file and target tables. 

Thanks a lot
Reply | Threaded
Open this post in threaded view
|

Re: Loading set-of files to tables

Bryan Bende
Hello,

Can you describe the flow and processors that you used to make it work with parameter context?

It will help people better understand what you are trying to do and how to make it more dynamic.

Thanks,

Bryan

On Tue, Jul 14, 2020 at 9:09 AM Samarendra Sahoo <[hidden email]> wrote:
Hi all,
We have a need where we get a list containing a set of source files and target table names. Would like to load the files in parallel (as much as possible). The list need to be picked up from the database and it can change for each of the runs.

Could you pls suggest an approach for this.

FYI, we are able to get this done sequentially by updating the parameter context with the source file and target tables. 

Thanks a lot
Reply | Threaded
Open this post in threaded view
|

Re: Loading set-of files to tables

Samarendra Sahoo

Thanks a lot Bryan. Below is what I ve done so far.


1. Set the source files and target tables mapping.

 

 

2.Create separate flow files. Extract data and load it to target table.

 

 



On Tue, Jul 14, 2020 at 7:23 PM Bryan Bende <[hidden email]> wrote:
Hello,

Can you describe the flow and processors that you used to make it work with parameter context?

It will help people better understand what you are trying to do and how to make it more dynamic.

Thanks,

Bryan

On Tue, Jul 14, 2020 at 9:09 AM Samarendra Sahoo <[hidden email]> wrote:
Hi all,
We have a need where we get a list containing a set of source files and target table names. Would like to load the files in parallel (as much as possible). The list need to be picked up from the database and it can change for each of the runs.

Could you pls suggest an approach for this.

FYI, we are able to get this done sequentially by updating the parameter context with the source file and target tables. 

Thanks a lot

image007.jpg (50K) Download Attachment