PutHiveStreaming has only one concurrent task

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

PutHiveStreaming has only one concurrent task

Olav Jordens

Hi,

 

I have noticed that it is not possible to change the number of concurrent tasks on the Scheduling tab of the PutHiveStreaming processor from the default value of 1. Is there a reason for this, or is it a bug? What would be the consequence if I try to get around this by using a DistributeLoad processor followed by a sequence of PutHiveStreaming processors, which would all be streaming into the same Hive table?

 

Thanks,

Olav

 



Olav Jordens
Senior ETL Developer
Two Degrees Mobile Limited
===========================
(M) 022 620 2429
(P) 09 919 7000
www.2degreesmobile.co.nz

Two Degrees Mobile Limited | 47-49 George Street | Newmarket | Auckland | New Zealand |
PO Box 8355 | Symonds Street | Auckland 1150 | New Zealand | Fax +64 9 919 7001


   


Disclaimer
The e-mail and any files transmitted with it are confidential and may contain privileged or copyright information. If you are not the intended recipient you must not copy, distribute, or use this e-mail or the information contained in it for any purpose other than to notify us of the error. If you have received this message in error, please notify the sender immediately, by email or phone (+64 9 919 7000) and delete this email from your system. Any views expressed in this message are those of the individual sender, except where the sender specifically states them to be the views of Two Degrees Mobile Limited. We do not guarantee that this material is free from viruses or any other defects although due care has been taken to minimize the risk




Reply | Threaded
Open this post in threaded view
|

Re: PutHiveStreaming has only one concurrent task

Joe Witt
Olav,

I'll let someone more familiar with that processor comment on the best usage pattern but will say that for any processor which does not offer controls on the number of concurrent tasks then it means that processor is meant to run serially by design.

Thanks
Joe

On Sun, Nov 6, 2016 at 7:06 PM, Olav Jordens <[hidden email]> wrote:

Hi,

 

I have noticed that it is not possible to change the number of concurrent tasks on the Scheduling tab of the PutHiveStreaming processor from the default value of 1. Is there a reason for this, or is it a bug? What would be the consequence if I try to get around this by using a DistributeLoad processor followed by a sequence of PutHiveStreaming processors, which would all be streaming into the same Hive table?

 

Thanks,

Olav

 



Olav Jordens
Senior ETL Developer
Two Degrees Mobile Limited
===========================
(M) 022 620 2429
(P) 09 919 7000
www.2degreesmobile.co.nz

Two Degrees Mobile Limited | 47-49 George Street | Newmarket | Auckland | New Zealand |
PO Box 8355 | Symonds Street | Auckland 1150 | New Zealand | Fax <a href="tel:%2B64%209%20919%207001" value="+6499197001" target="_blank">+64 9 919 7001


   


Disclaimer
The e-mail and any files transmitted with it are confidential and may contain privileged or copyright information. If you are not the intended recipient you must not copy, distribute, or use this e-mail or the information contained in it for any purpose other than to notify us of the error. If you have received this message in error, please notify the sender immediately, by email or phone <a href="tel:%28%2B64%209%20919%207000" value="+6499197000" target="_blank">(+64 9 919 7000) and delete this email from your system. Any views expressed in this message are those of the individual sender, except where the sender specifically states them to be the views of Two Degrees Mobile Limited. We do not guarantee that this material is free from viruses or any other defects although due care has been taken to minimize the risk





Reply | Threaded
Open this post in threaded view
|

Re: PutHiveStreaming has only one concurrent task

Matt Burgess-2
Olav,

I contributed that processor, and for the life of me I can't remember why it is forcing serial execution. My guess is that it was a limitation of the version of the Hive Streaming libraries. I will try it with multiple concurrent tasks to see if it works, and if so, I will write a Jira to support concurrency for this processor. In either case I will reply here with my findings.

Regards,
Matt

On Sun, Nov 6, 2016 at 7:18 PM, Joe Witt <[hidden email]> wrote:
Olav,

I'll let someone more familiar with that processor comment on the best usage pattern but will say that for any processor which does not offer controls on the number of concurrent tasks then it means that processor is meant to run serially by design.

Thanks
Joe

On Sun, Nov 6, 2016 at 7:06 PM, Olav Jordens <[hidden email]> wrote:

Hi,

 

I have noticed that it is not possible to change the number of concurrent tasks on the Scheduling tab of the PutHiveStreaming processor from the default value of 1. Is there a reason for this, or is it a bug? What would be the consequence if I try to get around this by using a DistributeLoad processor followed by a sequence of PutHiveStreaming processors, which would all be streaming into the same Hive table?

 

Thanks,

Olav

 



Olav Jordens
Senior ETL Developer
Two Degrees Mobile Limited
===========================
(M) 022 620 2429
(P) 09 919 7000
www.2degreesmobile.co.nz

Two Degrees Mobile Limited | 47-49 George Street | Newmarket | Auckland | New Zealand |
PO Box 8355 | Symonds Street | Auckland 1150 | New Zealand | Fax <a href="tel:%2B64%209%20919%207001" value="+6499197001" target="_blank">+64 9 919 7001


   


Disclaimer
The e-mail and any files transmitted with it are confidential and may contain privileged or copyright information. If you are not the intended recipient you must not copy, distribute, or use this e-mail or the information contained in it for any purpose other than to notify us of the error. If you have received this message in error, please notify the sender immediately, by email or phone <a href="tel:%28%2B64%209%20919%207000" value="+6499197000" target="_blank">(+64 9 919 7000) and delete this email from your system. Any views expressed in this message are those of the individual sender, except where the sender specifically states them to be the views of Two Degrees Mobile Limited. We do not guarantee that this material is free from viruses or any other defects although due care has been taken to minimize the risk






Reply | Threaded
Open this post in threaded view
|

RE: PutHiveStreaming has only one concurrent task

Olav Jordens

Hi Matt – Thanks for the prompt reply. This processor is perfect for our needs – great job!

Olav

 

From: Matt Burgess [mailto:[hidden email]]
Sent: Tuesday, 8 November 2016 6:08 a.m.
To: [hidden email]
Subject: Re: PutHiveStreaming has only one concurrent task

 

Olav,

 

I contributed that processor, and for the life of me I can't remember why it is forcing serial execution. My guess is that it was a limitation of the version of the Hive Streaming libraries. I will try it with multiple concurrent tasks to see if it works, and if so, I will write a Jira to support concurrency for this processor. In either case I will reply here with my findings.

 

Regards,

Matt

 

On Sun, Nov 6, 2016 at 7:18 PM, Joe Witt <[hidden email]> wrote:

Olav,

 

I'll let someone more familiar with that processor comment on the best usage pattern but will say that for any processor which does not offer controls on the number of concurrent tasks then it means that processor is meant to run serially by design.

 

Thanks
Joe

 

On Sun, Nov 6, 2016 at 7:06 PM, Olav Jordens <[hidden email]> wrote:

Hi,

 

I have noticed that it is not possible to change the number of concurrent tasks on the Scheduling tab of the PutHiveStreaming processor from the default value of 1. Is there a reason for this, or is it a bug? What would be the consequence if I try to get around this by using a DistributeLoad processor followed by a sequence of PutHiveStreaming processors, which would all be streaming into the same Hive table?

 

Thanks,

Olav

 



Olav Jordens
Senior ETL Developer
Two Degrees Mobile Limited
===========================
(M) 022 620 2429
(P) 09 919 7000
www.2degreesmobile.co.nz

Two Degrees Mobile Limited | 47-49 George Street | Newmarket | Auckland | New Zealand |
PO Box 8355 | Symonds Street | Auckland 1150 | New Zealand | Fax <a href="tel:%2B64%209%20919%207001" target="_blank"> +64 9 919 7001


   


Disclaimer
The e-mail and any files transmitted with it are confidential and may contain privileged or copyright information. If you are not the intended recipient you must not copy, distribute, or use this e-mail or the information contained in it for any purpose other than to notify us of the error. If you have received this message in error, please notify the sender immediately, by email or phone <a href="tel:%28%2B64%209%20919%207000" target="_blank">(+64 9 919 7000) and delete this email from your system. Any views expressed in this message are those of the individual sender, except where the sender specifically states them to be the views of Two Degrees Mobile Limited. We do not guarantee that this material is free from viruses or any other defects although due care has been taken to minimize the risk