Writing a chunk stream to a file asynchronously using hyper
I am trying to create a simple function that downloads a remote file to a local filepath using hyper. I need the file write to be asynchronous as well (in my case I am using tokio_fs
for that). Here is the code:
View in the playground
// Parts of the code were omitted, see the playground for full source code
pub fn download_file(
uri: Uri,
file_location: &Path,
) -> Box<Future<Item = (), Error = DownloadFileError>> DownloadFileError::CreateFile(err));
Box::new(
response_future
.join(create_file_future)
.and_then(move
However, I get the following error:
error[E0507]: cannot move out of captured outer variable in an `FnMut` closure
--> src/lib.rs:79:39
|
75 | .and_then(move |(res, file)| {
| ---- captured outer variable
...
79 | io::write_all(file, chunk)
| ^^^^ cannot move out of captured outer variable in an `FnMut` closure
Conceptually, I understand what the error means: Since a FnMut
captures variables by mutable reference, I cannot move a captured variable. However, I do not understand how can I work around this problem in the example given, since I need to write the stream to the file returned by the Join
future.
The write_all
method from the Write
trait would work here since it takes the file as a mutable reference, but the problem is that it does the writing on the same thread.
rust future borrow-checker rust-tokio hyper
add a comment |
I am trying to create a simple function that downloads a remote file to a local filepath using hyper. I need the file write to be asynchronous as well (in my case I am using tokio_fs
for that). Here is the code:
View in the playground
// Parts of the code were omitted, see the playground for full source code
pub fn download_file(
uri: Uri,
file_location: &Path,
) -> Box<Future<Item = (), Error = DownloadFileError>> DownloadFileError::CreateFile(err));
Box::new(
response_future
.join(create_file_future)
.and_then(move
However, I get the following error:
error[E0507]: cannot move out of captured outer variable in an `FnMut` closure
--> src/lib.rs:79:39
|
75 | .and_then(move |(res, file)| {
| ---- captured outer variable
...
79 | io::write_all(file, chunk)
| ^^^^ cannot move out of captured outer variable in an `FnMut` closure
Conceptually, I understand what the error means: Since a FnMut
captures variables by mutable reference, I cannot move a captured variable. However, I do not understand how can I work around this problem in the example given, since I need to write the stream to the file returned by the Join
future.
The write_all
method from the Write
trait would work here since it takes the file as a mutable reference, but the problem is that it does the writing on the same thread.
rust future borrow-checker rust-tokio hyper
I think there is no point to usefor_each()
,into_body()
return an option so just usemap()
instead. But maybe I totally wrong I don't understand a thing in hyper/futures
– Stargateur
Nov 16 '18 at 2:45
As far as I understand from the answers both of them write synchronously to the file usingwrite_all
. In my caseio::write_all
is done asynchronously as well.
– lhahn
Nov 16 '18 at 3:13
add a comment |
I am trying to create a simple function that downloads a remote file to a local filepath using hyper. I need the file write to be asynchronous as well (in my case I am using tokio_fs
for that). Here is the code:
View in the playground
// Parts of the code were omitted, see the playground for full source code
pub fn download_file(
uri: Uri,
file_location: &Path,
) -> Box<Future<Item = (), Error = DownloadFileError>> DownloadFileError::CreateFile(err));
Box::new(
response_future
.join(create_file_future)
.and_then(move
However, I get the following error:
error[E0507]: cannot move out of captured outer variable in an `FnMut` closure
--> src/lib.rs:79:39
|
75 | .and_then(move |(res, file)| {
| ---- captured outer variable
...
79 | io::write_all(file, chunk)
| ^^^^ cannot move out of captured outer variable in an `FnMut` closure
Conceptually, I understand what the error means: Since a FnMut
captures variables by mutable reference, I cannot move a captured variable. However, I do not understand how can I work around this problem in the example given, since I need to write the stream to the file returned by the Join
future.
The write_all
method from the Write
trait would work here since it takes the file as a mutable reference, but the problem is that it does the writing on the same thread.
rust future borrow-checker rust-tokio hyper
I am trying to create a simple function that downloads a remote file to a local filepath using hyper. I need the file write to be asynchronous as well (in my case I am using tokio_fs
for that). Here is the code:
View in the playground
// Parts of the code were omitted, see the playground for full source code
pub fn download_file(
uri: Uri,
file_location: &Path,
) -> Box<Future<Item = (), Error = DownloadFileError>> DownloadFileError::CreateFile(err));
Box::new(
response_future
.join(create_file_future)
.and_then(move
However, I get the following error:
error[E0507]: cannot move out of captured outer variable in an `FnMut` closure
--> src/lib.rs:79:39
|
75 | .and_then(move |(res, file)| {
| ---- captured outer variable
...
79 | io::write_all(file, chunk)
| ^^^^ cannot move out of captured outer variable in an `FnMut` closure
Conceptually, I understand what the error means: Since a FnMut
captures variables by mutable reference, I cannot move a captured variable. However, I do not understand how can I work around this problem in the example given, since I need to write the stream to the file returned by the Join
future.
The write_all
method from the Write
trait would work here since it takes the file as a mutable reference, but the problem is that it does the writing on the same thread.
rust future borrow-checker rust-tokio hyper
rust future borrow-checker rust-tokio hyper
edited Nov 16 '18 at 3:17
lhahn
asked Nov 16 '18 at 2:22
lhahnlhahn
6711626
6711626
I think there is no point to usefor_each()
,into_body()
return an option so just usemap()
instead. But maybe I totally wrong I don't understand a thing in hyper/futures
– Stargateur
Nov 16 '18 at 2:45
As far as I understand from the answers both of them write synchronously to the file usingwrite_all
. In my caseio::write_all
is done asynchronously as well.
– lhahn
Nov 16 '18 at 3:13
add a comment |
I think there is no point to usefor_each()
,into_body()
return an option so just usemap()
instead. But maybe I totally wrong I don't understand a thing in hyper/futures
– Stargateur
Nov 16 '18 at 2:45
As far as I understand from the answers both of them write synchronously to the file usingwrite_all
. In my caseio::write_all
is done asynchronously as well.
– lhahn
Nov 16 '18 at 3:13
I think there is no point to use
for_each()
, into_body()
return an option so just use map()
instead. But maybe I totally wrong I don't understand a thing in hyper/futures– Stargateur
Nov 16 '18 at 2:45
I think there is no point to use
for_each()
, into_body()
return an option so just use map()
instead. But maybe I totally wrong I don't understand a thing in hyper/futures– Stargateur
Nov 16 '18 at 2:45
As far as I understand from the answers both of them write synchronously to the file using
write_all
. In my case io::write_all
is done asynchronously as well.– lhahn
Nov 16 '18 at 3:13
As far as I understand from the answers both of them write synchronously to the file using
write_all
. In my case io::write_all
is done asynchronously as well.– lhahn
Nov 16 '18 at 3:13
add a comment |
1 Answer
1
active
oldest
votes
You do not want to use for_each
. io::write_all
consumes the target and the buffer in exchange for a future that will return the target and the buffer when it is done. You can combine this with Stream::fold
to reuse the file:
.fold(file, |file, chunk|
io::write_all(file, chunk)
.map()
.map(drop)
See also:
- How do I write a futures::Stream to disk without storing it entirely in memory first?
- How to save a file downloaded from S3 with Rusoto to my hard drive?
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53330556%2fwriting-a-chunk-stream-to-a-file-asynchronously-using-hyper%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
You do not want to use for_each
. io::write_all
consumes the target and the buffer in exchange for a future that will return the target and the buffer when it is done. You can combine this with Stream::fold
to reuse the file:
.fold(file, |file, chunk|
io::write_all(file, chunk)
.map()
.map(drop)
See also:
- How do I write a futures::Stream to disk without storing it entirely in memory first?
- How to save a file downloaded from S3 with Rusoto to my hard drive?
add a comment |
You do not want to use for_each
. io::write_all
consumes the target and the buffer in exchange for a future that will return the target and the buffer when it is done. You can combine this with Stream::fold
to reuse the file:
.fold(file, |file, chunk|
io::write_all(file, chunk)
.map()
.map(drop)
See also:
- How do I write a futures::Stream to disk without storing it entirely in memory first?
- How to save a file downloaded from S3 with Rusoto to my hard drive?
add a comment |
You do not want to use for_each
. io::write_all
consumes the target and the buffer in exchange for a future that will return the target and the buffer when it is done. You can combine this with Stream::fold
to reuse the file:
.fold(file, |file, chunk|
io::write_all(file, chunk)
.map()
.map(drop)
See also:
- How do I write a futures::Stream to disk without storing it entirely in memory first?
- How to save a file downloaded from S3 with Rusoto to my hard drive?
You do not want to use for_each
. io::write_all
consumes the target and the buffer in exchange for a future that will return the target and the buffer when it is done. You can combine this with Stream::fold
to reuse the file:
.fold(file, |file, chunk|
io::write_all(file, chunk)
.map()
.map(drop)
See also:
- How do I write a futures::Stream to disk without storing it entirely in memory first?
- How to save a file downloaded from S3 with Rusoto to my hard drive?
edited Nov 17 '18 at 3:41
answered Nov 16 '18 at 16:09
ShepmasterShepmaster
159k15325468
159k15325468
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53330556%2fwriting-a-chunk-stream-to-a-file-asynchronously-using-hyper%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
I think there is no point to use
for_each()
,into_body()
return an option so just usemap()
instead. But maybe I totally wrong I don't understand a thing in hyper/futures– Stargateur
Nov 16 '18 at 2:45
As far as I understand from the answers both of them write synchronously to the file using
write_all
. In my caseio::write_all
is done asynchronously as well.– lhahn
Nov 16 '18 at 3:13