Apache Beam Runner for a single machine in Production
up vote
0
down vote
favorite
From what I've read from Apache Beam's documentation, the Direct Runner should be used as a runner to test/debug your pipeline code.
Thing is, my use-case consists of big and (sometimes) small datasets that should be processed in batches. I want to reuse the same pipeline code for both types of datasets.
I don't think paralellism would be beneficial for writing small datasets (2000/3000 records) in a SQL database most of the times. If I were to use Cloud Dataflow with small datasets, I would have an overhead related to VM startup time, since it isn't possible to use a single dedicated VM in Dataflow.
In this context, I thought of using the Direct Runner to deal with the small datasets. Would this be a bad decision on a production environment? Or is there a runner more suitable for this that I don't know about?
Also, does Direct Runner creates threads for ParDo transforms automatically? If yes, is there an argument/option to specify the maximum number of threads the runner should work with?
google-cloud-dataflow apache-beam asked Nov 12 at 0:40
d4nielfr4nco
8017
add a comment |
up vote
0
down vote
favorite
From what I've read from Apache Beam's documentation, the Direct Runner should be used as a runner to test/debug your pipeline code.
Thing is, my use-case consists of big and (sometimes) small datasets that should be processed in batches. I want to reuse the same pipeline code for both types of datasets.
I don't think paralellism would be beneficial for writing small datasets (2000/3000 records) in a SQL database most of the times. If I were to use Cloud Dataflow with small datasets, I would have an overhead related to VM startup time, since it isn't possible to use a single dedicated VM in Dataflow.
In this context, I thought of using the Direct Runner to deal with the small datasets. Would this be a bad decision on a production environment? Or is there a runner more suitable for this that I don't know about?
Also, does Direct Runner creates threads for ParDo transforms automatically? If yes, is there an argument/option to specify the maximum number of threads the runner should work with?
google-cloud-dataflow apache-beam asked Nov 12 at 0:40
d4nielfr4nco
8017
From my experience Direct Runner consumes much more resources and is slower than Dataflow Runner. It's probably OK for small datastets. But also with a Direct Runner you would miss Dataflow UI graph representation and built-in logging. Kind of another approach could be to modify pipeline to run into streaming mode. This way you can run it on a single dedicated VM with Dataflow and avoid startup time. I think the complexity of pipeline modification to streaming mode highly depends on data source you are using.
– Oleksandr Bushkovskyi
Nov 15 at 6:27
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
From what I've read from Apache Beam's documentation, the Direct Runner should be used as a runner to test/debug your pipeline code.
Thing is, my use-case consists of big and (sometimes) small datasets that should be processed in batches. I want to reuse the same pipeline code for both types of datasets.
I don't think paralellism would be beneficial for writing small datasets (2000/3000 records) in a SQL database most of the times. If I were to use Cloud Dataflow with small datasets, I would have an overhead related to VM startup time, since it isn't possible to use a single dedicated VM in Dataflow.
In this context, I thought of using the Direct Runner to deal with the small datasets. Would this be a bad decision on a production environment? Or is there a runner more suitable for this that I don't know about?
Also, does Direct Runner creates threads for ParDo transforms automatically? If yes, is there an argument/option to specify the maximum number of threads the runner should work with?
google-cloud-dataflow apache-beam asked Nov 12 at 0:40
d4nielfr4nco
8017
From what I've read from Apache Beam's documentation, the Direct Runner should be used as a runner to test/debug your pipeline code.
Thing is, my use-case consists of big and (sometimes) small datasets that should be processed in batches. I want to reuse the same pipeline code for both types of datasets.
I don't think paralellism would be beneficial for writing small datasets (2000/3000 records) in a SQL database most of the times. If I were to use Cloud Dataflow with small datasets, I would have an overhead related to VM startup time, since it isn't possible to use a single dedicated VM in Dataflow.
In this context, I thought of using the Direct Runner to deal with the small datasets. Would this be a bad decision on a production environment? Or is there a runner more suitable for this that I don't know about?
Also, does Direct Runner creates threads for ParDo transforms automatically? If yes, is there an argument/option to specify the maximum number of threads the runner should work with?
google-cloud-dataflow apache-beam google-cloud-dataflow apache-beam asked Nov 12 at 0:40
d4nielfr4nco
8017
asked Nov 12 at 0:40
d4nielfr4nco
8017
asked Nov 12 at 0:40
d4nielfr4nco
8017
asked Nov 12 at 0:40
d4nielfr4nco
8017
asked Nov 12 at 0:40
d4nielfr4nco
8017
8017
From my experience Direct Runner consumes much more resources and is slower than Dataflow Runner. It's probably OK for small datastets. But also with a Direct Runner you would miss Dataflow UI graph representation and built-in logging. Kind of another approach could be to modify pipeline to run into streaming mode. This way you can run it on a single dedicated VM with Dataflow and avoid startup time. I think the complexity of pipeline modification to streaming mode highly depends on data source you are using.
– Oleksandr Bushkovskyi
Nov 15 at 6:27
add a comment |
From my experience Direct Runner consumes much more resources and is slower than Dataflow Runner. It's probably OK for small datastets. But also with a Direct Runner you would miss Dataflow UI graph representation and built-in logging. Kind of another approach could be to modify pipeline to run into streaming mode. This way you can run it on a single dedicated VM with Dataflow and avoid startup time. I think the complexity of pipeline modification to streaming mode highly depends on data source you are using.
– Oleksandr Bushkovskyi
Nov 15 at 6:27
From my experience Direct Runner consumes much more resources and is slower than Dataflow Runner. It's probably OK for small datastets. But also with a Direct Runner you would miss Dataflow UI graph representation and built-in logging. Kind of another approach could be to modify pipeline to run into streaming mode. This way you can run it on a single dedicated VM with Dataflow and avoid startup time. I think the complexity of pipeline modification to streaming mode highly depends on data source you are using.
– Oleksandr Bushkovskyi
Nov 15 at 6:27
From my experience Direct Runner consumes much more resources and is slower than Dataflow Runner. It's probably OK for small datastets. But also with a Direct Runner you would miss Dataflow UI graph representation and built-in logging. Kind of another approach could be to modify pipeline to run into streaming mode. This way you can run it on a single dedicated VM with Dataflow and avoid startup time. I think the complexity of pipeline modification to streaming mode highly depends on data source you are using.
– Oleksandr Bushkovskyi
Nov 15 at 6:27
add a comment |
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53254681%2fapache-beam-runner-for-a-single-machine-in-production%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53254681%2fapache-beam-runner-for-a-single-machine-in-production%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
From my experience Direct Runner consumes much more resources and is slower than Dataflow Runner. It's probably OK for small datastets. But also with a Direct Runner you would miss Dataflow UI graph representation and built-in logging. Kind of another approach could be to modify pipeline to run into streaming mode. This way you can run it on a single dedicated VM with Dataflow and avoid startup time. I think the complexity of pipeline modification to streaming mode highly depends on data source you are using.
– Oleksandr Bushkovskyi
Nov 15 at 6:27