Akka Streams + Akka Cluster
up vote
2
down vote
favorite
I have a question about the use of Akka Streams and Akka Cluster. I'm trying to make a version of distributed word count using Akka Streams and Akka Cluster.
I would like to build an Akka Streams client that reads a text file as streaming I/O and sends the stream of words to a remote cluster. This is the code of the client:
final Path file = Paths.get("example.txt");
final Source<ByteString, CompletionStage<IOResult>> read = FileIO.fromPath(file);
final Source<Pair<String, Integer>, CompletionStage<IOResult>> counts =
read
.via(Framing.delimiter(ByteString.fromString(" "), 256, FramingTruncation.ALLOW))
.map(i -> i.utf8String())
.runWith(/* send to Akka cluster */);
I don't understand what I have to use to send streaming data to an Akka cluster without losing the bases of Akka Streams (backpressure, etc.).
I know of the existence of Stream refs and Cluster Client but I don't understand which of them to use.
java akka akka-stream akka-cluster
edited Nov 10 at 19:43
Jeffrey Chung
13.4k62139
asked Nov 10 at 15:43
Alexh2o
112
New contributor
Alexh2o is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
up vote
2
down vote
favorite
I have a question about the use of Akka Streams and Akka Cluster. I'm trying to make a version of distributed word count using Akka Streams and Akka Cluster.
I would like to build an Akka Streams client that reads a text file as streaming I/O and sends the stream of words to a remote cluster. This is the code of the client:
final Path file = Paths.get("example.txt");
final Source<ByteString, CompletionStage<IOResult>> read = FileIO.fromPath(file);
final Source<Pair<String, Integer>, CompletionStage<IOResult>> counts =
read
.via(Framing.delimiter(ByteString.fromString(" "), 256, FramingTruncation.ALLOW))
.map(i -> i.utf8String())
.runWith(/* send to Akka cluster */);
I don't understand what I have to use to send streaming data to an Akka cluster without losing the bases of Akka Streams (backpressure, etc.).
I know of the existence of Stream refs and Cluster Client but I don't understand which of them to use.
java akka akka-stream akka-cluster
edited Nov 10 at 19:43
Jeffrey Chung
13.4k62139
asked Nov 10 at 15:43
Alexh2o
112
New contributor
Alexh2o is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
There is a way to integrate actors with akka streams: doc.akka.io/docs/akka/2.5.3/scala/stream/…, though I don't think it's a good idea since, in order to preserve the the back pressure, you have to add a lot of boiler plate across your actors...
– RoberMP
Nov 13 at 9:52
add a comment |
up vote
2
down vote
favorite
up vote
2
down vote
favorite
I have a question about the use of Akka Streams and Akka Cluster. I'm trying to make a version of distributed word count using Akka Streams and Akka Cluster.
I would like to build an Akka Streams client that reads a text file as streaming I/O and sends the stream of words to a remote cluster. This is the code of the client:
final Path file = Paths.get("example.txt");
final Source<ByteString, CompletionStage<IOResult>> read = FileIO.fromPath(file);
final Source<Pair<String, Integer>, CompletionStage<IOResult>> counts =
read
.via(Framing.delimiter(ByteString.fromString(" "), 256, FramingTruncation.ALLOW))
.map(i -> i.utf8String())
.runWith(/* send to Akka cluster */);
I don't understand what I have to use to send streaming data to an Akka cluster without losing the bases of Akka Streams (backpressure, etc.).
I know of the existence of Stream refs and Cluster Client but I don't understand which of them to use.
java akka akka-stream akka-cluster
edited Nov 10 at 19:43
Jeffrey Chung
13.4k62139
asked Nov 10 at 15:43
Alexh2o
112
New contributor
Alexh2o is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
I have a question about the use of Akka Streams and Akka Cluster. I'm trying to make a version of distributed word count using Akka Streams and Akka Cluster.
I would like to build an Akka Streams client that reads a text file as streaming I/O and sends the stream of words to a remote cluster. This is the code of the client:
final Path file = Paths.get("example.txt");
final Source<ByteString, CompletionStage<IOResult>> read = FileIO.fromPath(file);
final Source<Pair<String, Integer>, CompletionStage<IOResult>> counts =
read
.via(Framing.delimiter(ByteString.fromString(" "), 256, FramingTruncation.ALLOW))
.map(i -> i.utf8String())
.runWith(/* send to Akka cluster */);
I don't understand what I have to use to send streaming data to an Akka cluster without losing the bases of Akka Streams (backpressure, etc.).
I know of the existence of Stream refs and Cluster Client but I don't understand which of them to use.
java akka akka-stream akka-cluster
java akka akka-stream akka-cluster
edited Nov 10 at 19:43
Jeffrey Chung
13.4k62139
asked Nov 10 at 15:43
Alexh2o
112
New contributor
Alexh2o is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
edited Nov 10 at 19:43
Jeffrey Chung
13.4k62139
asked Nov 10 at 15:43
Alexh2o
112
New contributor
Alexh2o is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
edited Nov 10 at 19:43
Jeffrey Chung
13.4k62139
edited Nov 10 at 19:43
Jeffrey Chung
13.4k62139
edited Nov 10 at 19:43
Jeffrey Chung
13.4k62139
13.4k62139
asked Nov 10 at 15:43
Alexh2o
112
New contributor
Alexh2o is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked Nov 10 at 15:43
Alexh2o
112
asked Nov 10 at 15:43
Alexh2o
112
112
New contributor
Alexh2o is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
Alexh2o is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
Alexh2o is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
There is a way to integrate actors with akka streams: doc.akka.io/docs/akka/2.5.3/scala/stream/…, though I don't think it's a good idea since, in order to preserve the the back pressure, you have to add a lot of boiler plate across your actors...
– RoberMP
Nov 13 at 9:52
add a comment |
There is a way to integrate actors with akka streams: doc.akka.io/docs/akka/2.5.3/scala/stream/…, though I don't think it's a good idea since, in order to preserve the the back pressure, you have to add a lot of boiler plate across your actors...
– RoberMP
Nov 13 at 9:52
There is a way to integrate actors with akka streams: doc.akka.io/docs/akka/2.5.3/scala/stream/…, though I don't think it's a good idea since, in order to preserve the the back pressure, you have to add a lot of boiler plate across your actors...
– RoberMP
Nov 13 at 9:52
There is a way to integrate actors with akka streams: doc.akka.io/docs/akka/2.5.3/scala/stream/…, though I don't think it's a good idea since, in order to preserve the the back pressure, you have to add a lot of boiler plate across your actors...
– RoberMP
Nov 13 at 9:52
add a comment |
2 Answers
2
active
oldest
votes
up vote
0
down vote
Direct Answer
I don't think the functionality that you are looking for is available as of version 2.5.18. The inventory of cluster functionality doesn't list anything related to streams.
Indirect Answer
The computational requirements for your use case would have to be rather extreme to justify an akka-stream spanning multiple servers. The amount of parallelism available in a single server is rather large given the explosion of core count on modern processors. Therefore, each step of computation in your stream would have to require a huge amount of processor resources to justify spanning the stream across a network.
If you are truly working on such a large project then a tool like apache spark may better suite your needs.
answered Nov 11 at 13:31
Ramon J Romero y Vigil
11.5k23970
I know that this use case doesn't justify the use of akka streams but i worked It as initial parte of my thesis. i use this use case only to start to use akka toolkit. Than I hope to use a more realistic and extreme use case.
– Alexh2o
Nov 12 at 14:41
add a comment |
up vote
0
down vote
I don't think what you are looking for currently exists. However, there is this similar thing called streamRefs which allow you to have reactive streams over the network. Take a look here: https://doc.akka.io/docs/akka/2.5/stream/stream-refs.html
answered Nov 12 at 0:11
hveiga
4,67763259
I have read about StreamRef but seams to be something to tie two stream over network. Instead i want to tie a single stream to multiple stream on multiple node in a akka cluster. Now i'm using the first stream as publisher and the streams in cluster as consumer with akka Kafka streams but I don't know if It is the best solution
– Alexh2o
Nov 12 at 14:45
add a comment |
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
Direct Answer
I don't think the functionality that you are looking for is available as of version 2.5.18. The inventory of cluster functionality doesn't list anything related to streams.
Indirect Answer
The computational requirements for your use case would have to be rather extreme to justify an akka-stream spanning multiple servers. The amount of parallelism available in a single server is rather large given the explosion of core count on modern processors. Therefore, each step of computation in your stream would have to require a huge amount of processor resources to justify spanning the stream across a network.
If you are truly working on such a large project then a tool like apache spark may better suite your needs.
answered Nov 11 at 13:31
Ramon J Romero y Vigil
11.5k23970
I know that this use case doesn't justify the use of akka streams but i worked It as initial parte of my thesis. i use this use case only to start to use akka toolkit. Than I hope to use a more realistic and extreme use case.
– Alexh2o
Nov 12 at 14:41
add a comment |
up vote
0
down vote
Direct Answer
I don't think the functionality that you are looking for is available as of version 2.5.18. The inventory of cluster functionality doesn't list anything related to streams.
Indirect Answer
The computational requirements for your use case would have to be rather extreme to justify an akka-stream spanning multiple servers. The amount of parallelism available in a single server is rather large given the explosion of core count on modern processors. Therefore, each step of computation in your stream would have to require a huge amount of processor resources to justify spanning the stream across a network.
If you are truly working on such a large project then a tool like apache spark may better suite your needs.
answered Nov 11 at 13:31
Ramon J Romero y Vigil
11.5k23970
I know that this use case doesn't justify the use of akka streams but i worked It as initial parte of my thesis. i use this use case only to start to use akka toolkit. Than I hope to use a more realistic and extreme use case.
– Alexh2o
Nov 12 at 14:41
add a comment |
up vote
0
down vote
up vote
0
down vote
Direct Answer
I don't think the functionality that you are looking for is available as of version 2.5.18. The inventory of cluster functionality doesn't list anything related to streams.
Indirect Answer
The computational requirements for your use case would have to be rather extreme to justify an akka-stream spanning multiple servers. The amount of parallelism available in a single server is rather large given the explosion of core count on modern processors. Therefore, each step of computation in your stream would have to require a huge amount of processor resources to justify spanning the stream across a network.
If you are truly working on such a large project then a tool like apache spark may better suite your needs.
answered Nov 11 at 13:31
Ramon J Romero y Vigil
11.5k23970
Direct Answer
I don't think the functionality that you are looking for is available as of version 2.5.18. The inventory of cluster functionality doesn't list anything related to streams.
Indirect Answer
The computational requirements for your use case would have to be rather extreme to justify an akka-stream spanning multiple servers. The amount of parallelism available in a single server is rather large given the explosion of core count on modern processors. Therefore, each step of computation in your stream would have to require a huge amount of processor resources to justify spanning the stream across a network.
If you are truly working on such a large project then a tool like apache spark may better suite your needs.
answered Nov 11 at 13:31
Ramon J Romero y Vigil
11.5k23970
answered Nov 11 at 13:31
Ramon J Romero y Vigil
11.5k23970
answered Nov 11 at 13:31
Ramon J Romero y Vigil
11.5k23970
answered Nov 11 at 13:31
Ramon J Romero y Vigil
11.5k23970
11.5k23970
I know that this use case doesn't justify the use of akka streams but i worked It as initial parte of my thesis. i use this use case only to start to use akka toolkit. Than I hope to use a more realistic and extreme use case.
– Alexh2o
Nov 12 at 14:41
add a comment |
I know that this use case doesn't justify the use of akka streams but i worked It as initial parte of my thesis. i use this use case only to start to use akka toolkit. Than I hope to use a more realistic and extreme use case.
– Alexh2o
Nov 12 at 14:41
I know that this use case doesn't justify the use of akka streams but i worked It as initial parte of my thesis. i use this use case only to start to use akka toolkit. Than I hope to use a more realistic and extreme use case.
– Alexh2o
Nov 12 at 14:41
I know that this use case doesn't justify the use of akka streams but i worked It as initial parte of my thesis. i use this use case only to start to use akka toolkit. Than I hope to use a more realistic and extreme use case.
– Alexh2o
Nov 12 at 14:41
add a comment |
up vote
0
down vote
I don't think what you are looking for currently exists. However, there is this similar thing called streamRefs which allow you to have reactive streams over the network. Take a look here: https://doc.akka.io/docs/akka/2.5/stream/stream-refs.html
answered Nov 12 at 0:11
hveiga
4,67763259
I have read about StreamRef but seams to be something to tie two stream over network. Instead i want to tie a single stream to multiple stream on multiple node in a akka cluster. Now i'm using the first stream as publisher and the streams in cluster as consumer with akka Kafka streams but I don't know if It is the best solution
– Alexh2o
Nov 12 at 14:45
add a comment |
up vote
0
down vote
I don't think what you are looking for currently exists. However, there is this similar thing called streamRefs which allow you to have reactive streams over the network. Take a look here: https://doc.akka.io/docs/akka/2.5/stream/stream-refs.html
answered Nov 12 at 0:11
hveiga
4,67763259
I have read about StreamRef but seams to be something to tie two stream over network. Instead i want to tie a single stream to multiple stream on multiple node in a akka cluster. Now i'm using the first stream as publisher and the streams in cluster as consumer with akka Kafka streams but I don't know if It is the best solution
– Alexh2o
Nov 12 at 14:45
add a comment |
up vote
0
down vote
up vote
0
down vote
I don't think what you are looking for currently exists. However, there is this similar thing called streamRefs which allow you to have reactive streams over the network. Take a look here: https://doc.akka.io/docs/akka/2.5/stream/stream-refs.html
answered Nov 12 at 0:11
hveiga
4,67763259
I don't think what you are looking for currently exists. However, there is this similar thing called streamRefs which allow you to have reactive streams over the network. Take a look here: https://doc.akka.io/docs/akka/2.5/stream/stream-refs.html
answered Nov 12 at 0:11
hveiga
4,67763259
answered Nov 12 at 0:11
hveiga
4,67763259
answered Nov 12 at 0:11
hveiga
4,67763259
answered Nov 12 at 0:11
hveiga
4,67763259
4,67763259
I have read about StreamRef but seams to be something to tie two stream over network. Instead i want to tie a single stream to multiple stream on multiple node in a akka cluster. Now i'm using the first stream as publisher and the streams in cluster as consumer with akka Kafka streams but I don't know if It is the best solution
– Alexh2o
Nov 12 at 14:45
add a comment |
I have read about StreamRef but seams to be something to tie two stream over network. Instead i want to tie a single stream to multiple stream on multiple node in a akka cluster. Now i'm using the first stream as publisher and the streams in cluster as consumer with akka Kafka streams but I don't know if It is the best solution
– Alexh2o
Nov 12 at 14:45
I have read about StreamRef but seams to be something to tie two stream over network. Instead i want to tie a single stream to multiple stream on multiple node in a akka cluster. Now i'm using the first stream as publisher and the streams in cluster as consumer with akka Kafka streams but I don't know if It is the best solution
– Alexh2o
Nov 12 at 14:45
I have read about StreamRef but seams to be something to tie two stream over network. Instead i want to tie a single stream to multiple stream on multiple node in a akka cluster. Now i'm using the first stream as publisher and the streams in cluster as consumer with akka Kafka streams but I don't know if It is the best solution
– Alexh2o
Nov 12 at 14:45
add a comment |
Alexh2o is a new contributor. Be nice, and check out our Code of Conduct.
Alexh2o is a new contributor. Be nice, and check out our Code of Conduct.
Alexh2o is a new contributor. Be nice, and check out our Code of Conduct.
Alexh2o is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53240558%2fakka-streams-akka-cluster%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
There is a way to integrate actors with akka streams: doc.akka.io/docs/akka/2.5.3/scala/stream/…, though I don't think it's a good idea since, in order to preserve the the back pressure, you have to add a lot of boiler plate across your actors...
– RoberMP
Nov 13 at 9:52