Sync severals spouts and bolt in Heron
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;
I am using Heropy. My topology looks the next way:
Spout1 -> Bolt1 - > Bolt2
Spout2 ---------> Bolt2
So, Bolt2 needs info from both Spout2 and Bolt1 in order to emit the result, however the info from Spout2 comes faster than the info from Bolt1 and Bolt2 has to wait until if has both Spout2 and Bolt1 info. How can I sync information flow using Heron API so that Bolt2 emits the result only after all the info available?
python heron
asked Nov 16 '18 at 14:40
Alexandr ProskurinAlexandr Proskurin
123
add a comment |
I am using Heropy. My topology looks the next way:
Spout1 -> Bolt1 - > Bolt2
Spout2 ---------> Bolt2
So, Bolt2 needs info from both Spout2 and Bolt1 in order to emit the result, however the info from Spout2 comes faster than the info from Bolt1 and Bolt2 has to wait until if has both Spout2 and Bolt1 info. How can I sync information flow using Heron API so that Bolt2 emits the result only after all the info available?
python heron
asked Nov 16 '18 at 14:40
Alexandr ProskurinAlexandr Proskurin
123
add a comment |
I am using Heropy. My topology looks the next way:
Spout1 -> Bolt1 - > Bolt2
Spout2 ---------> Bolt2
So, Bolt2 needs info from both Spout2 and Bolt1 in order to emit the result, however the info from Spout2 comes faster than the info from Bolt1 and Bolt2 has to wait until if has both Spout2 and Bolt1 info. How can I sync information flow using Heron API so that Bolt2 emits the result only after all the info available?
python heron
asked Nov 16 '18 at 14:40
Alexandr ProskurinAlexandr Proskurin
123
I am using Heropy. My topology looks the next way:
Spout1 -> Bolt1 - > Bolt2
Spout2 ---------> Bolt2
So, Bolt2 needs info from both Spout2 and Bolt1 in order to emit the result, however the info from Spout2 comes faster than the info from Bolt1 and Bolt2 has to wait until if has both Spout2 and Bolt1 info. How can I sync information flow using Heron API so that Bolt2 emits the result only after all the info available?
python heron
python heron
asked Nov 16 '18 at 14:40
Alexandr ProskurinAlexandr Proskurin
123
asked Nov 16 '18 at 14:40
Alexandr ProskurinAlexandr Proskurin
123
asked Nov 16 '18 at 14:40
Alexandr ProskurinAlexandr Proskurin
123
asked Nov 16 '18 at 14:40
Alexandr ProskurinAlexandr Proskurin
123
asked Nov 16 '18 at 14:40
Alexandr ProskurinAlexandr Proskurin
123
123
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
In general the order/synchronization is not guaranteed in streaming. It is hard for spout1 and spout2 themselves to be 100% synchronized I feel.
I am thinking there is one options you might consider: hooking bolt1 to spout2 and emiting the tuples from spout2 directly, disconnecting bolt2 from spout2. So that bolt1 becomes the source of truth of the ordering.
Windowing might be another option, but it requires more considerations and works.
answered Nov 16 '18 at 18:20
Ning WangNing Wang
862
add a comment |
Heron doesn't synchronize tuples from different components automatically. So you will need to buffer the tuples from Spout2 and wait until the corresponding tuples from Bolt1 arrive, then do the computation.
answered Nov 16 '18 at 21:37
NengNeng
1462
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53339982%2fsync-severals-spouts-and-bolt-in-heron%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
In general the order/synchronization is not guaranteed in streaming. It is hard for spout1 and spout2 themselves to be 100% synchronized I feel.
I am thinking there is one options you might consider: hooking bolt1 to spout2 and emiting the tuples from spout2 directly, disconnecting bolt2 from spout2. So that bolt1 becomes the source of truth of the ordering.
Windowing might be another option, but it requires more considerations and works.
answered Nov 16 '18 at 18:20
Ning WangNing Wang
862
add a comment |
In general the order/synchronization is not guaranteed in streaming. It is hard for spout1 and spout2 themselves to be 100% synchronized I feel.
I am thinking there is one options you might consider: hooking bolt1 to spout2 and emiting the tuples from spout2 directly, disconnecting bolt2 from spout2. So that bolt1 becomes the source of truth of the ordering.
Windowing might be another option, but it requires more considerations and works.
answered Nov 16 '18 at 18:20
Ning WangNing Wang
862
add a comment |
In general the order/synchronization is not guaranteed in streaming. It is hard for spout1 and spout2 themselves to be 100% synchronized I feel.
I am thinking there is one options you might consider: hooking bolt1 to spout2 and emiting the tuples from spout2 directly, disconnecting bolt2 from spout2. So that bolt1 becomes the source of truth of the ordering.
Windowing might be another option, but it requires more considerations and works.
answered Nov 16 '18 at 18:20
Ning WangNing Wang
862
In general the order/synchronization is not guaranteed in streaming. It is hard for spout1 and spout2 themselves to be 100% synchronized I feel.
I am thinking there is one options you might consider: hooking bolt1 to spout2 and emiting the tuples from spout2 directly, disconnecting bolt2 from spout2. So that bolt1 becomes the source of truth of the ordering.
Windowing might be another option, but it requires more considerations and works.
answered Nov 16 '18 at 18:20
Ning WangNing Wang
862
answered Nov 16 '18 at 18:20
Ning WangNing Wang
862
answered Nov 16 '18 at 18:20
Ning WangNing Wang
862
answered Nov 16 '18 at 18:20
Ning WangNing Wang
862
862
add a comment |
add a comment |
Heron doesn't synchronize tuples from different components automatically. So you will need to buffer the tuples from Spout2 and wait until the corresponding tuples from Bolt1 arrive, then do the computation.
answered Nov 16 '18 at 21:37
NengNeng
1462
add a comment |
Heron doesn't synchronize tuples from different components automatically. So you will need to buffer the tuples from Spout2 and wait until the corresponding tuples from Bolt1 arrive, then do the computation.
answered Nov 16 '18 at 21:37
NengNeng
1462
add a comment |
Heron doesn't synchronize tuples from different components automatically. So you will need to buffer the tuples from Spout2 and wait until the corresponding tuples from Bolt1 arrive, then do the computation.
answered Nov 16 '18 at 21:37
NengNeng
1462
Heron doesn't synchronize tuples from different components automatically. So you will need to buffer the tuples from Spout2 and wait until the corresponding tuples from Bolt1 arrive, then do the computation.
answered Nov 16 '18 at 21:37
NengNeng
1462
answered Nov 16 '18 at 21:37
NengNeng
1462
answered Nov 16 '18 at 21:37
NengNeng
1462
answered Nov 16 '18 at 21:37
NengNeng
1462
1462
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53339982%2fsync-severals-spouts-and-bolt-in-heron%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
