Writing a chunk stream to a file asynchronously using hyper










3















I am trying to create a simple function that downloads a remote file to a local filepath using hyper. I need the file write to be asynchronous as well (in my case I am using tokio_fs for that). Here is the code:



View in the playground



// Parts of the code were omitted, see the playground for full source code
pub fn download_file(
uri: Uri,
file_location: &Path,
) -> Box<Future<Item = (), Error = DownloadFileError>> DownloadFileError::CreateFile(err));

Box::new(
response_future
.join(create_file_future)
.and_then(move


However, I get the following error:



error[E0507]: cannot move out of captured outer variable in an `FnMut` closure
--> src/lib.rs:79:39
|
75 | .and_then(move |(res, file)| {
| ---- captured outer variable
...
79 | io::write_all(file, chunk)
| ^^^^ cannot move out of captured outer variable in an `FnMut` closure


Conceptually, I understand what the error means: Since a FnMut captures variables by mutable reference, I cannot move a captured variable. However, I do not understand how can I work around this problem in the example given, since I need to write the stream to the file returned by the Join future.



The write_all method from the Write trait would work here since it takes the file as a mutable reference, but the problem is that it does the writing on the same thread.










share|improve this question
























  • I think there is no point to use for_each(), into_body() return an option so just use map() instead. But maybe I totally wrong I don't understand a thing in hyper/futures

    – Stargateur
    Nov 16 '18 at 2:45












  • As far as I understand from the answers both of them write synchronously to the file using write_all. In my case io::write_all is done asynchronously as well.

    – lhahn
    Nov 16 '18 at 3:13















3















I am trying to create a simple function that downloads a remote file to a local filepath using hyper. I need the file write to be asynchronous as well (in my case I am using tokio_fs for that). Here is the code:



View in the playground



// Parts of the code were omitted, see the playground for full source code
pub fn download_file(
uri: Uri,
file_location: &Path,
) -> Box<Future<Item = (), Error = DownloadFileError>> DownloadFileError::CreateFile(err));

Box::new(
response_future
.join(create_file_future)
.and_then(move


However, I get the following error:



error[E0507]: cannot move out of captured outer variable in an `FnMut` closure
--> src/lib.rs:79:39
|
75 | .and_then(move |(res, file)| {
| ---- captured outer variable
...
79 | io::write_all(file, chunk)
| ^^^^ cannot move out of captured outer variable in an `FnMut` closure


Conceptually, I understand what the error means: Since a FnMut captures variables by mutable reference, I cannot move a captured variable. However, I do not understand how can I work around this problem in the example given, since I need to write the stream to the file returned by the Join future.



The write_all method from the Write trait would work here since it takes the file as a mutable reference, but the problem is that it does the writing on the same thread.










share|improve this question
























  • I think there is no point to use for_each(), into_body() return an option so just use map() instead. But maybe I totally wrong I don't understand a thing in hyper/futures

    – Stargateur
    Nov 16 '18 at 2:45












  • As far as I understand from the answers both of them write synchronously to the file using write_all. In my case io::write_all is done asynchronously as well.

    – lhahn
    Nov 16 '18 at 3:13













3












3








3








I am trying to create a simple function that downloads a remote file to a local filepath using hyper. I need the file write to be asynchronous as well (in my case I am using tokio_fs for that). Here is the code:



View in the playground



// Parts of the code were omitted, see the playground for full source code
pub fn download_file(
uri: Uri,
file_location: &Path,
) -> Box<Future<Item = (), Error = DownloadFileError>> DownloadFileError::CreateFile(err));

Box::new(
response_future
.join(create_file_future)
.and_then(move


However, I get the following error:



error[E0507]: cannot move out of captured outer variable in an `FnMut` closure
--> src/lib.rs:79:39
|
75 | .and_then(move |(res, file)| {
| ---- captured outer variable
...
79 | io::write_all(file, chunk)
| ^^^^ cannot move out of captured outer variable in an `FnMut` closure


Conceptually, I understand what the error means: Since a FnMut captures variables by mutable reference, I cannot move a captured variable. However, I do not understand how can I work around this problem in the example given, since I need to write the stream to the file returned by the Join future.



The write_all method from the Write trait would work here since it takes the file as a mutable reference, but the problem is that it does the writing on the same thread.










share|improve this question
















I am trying to create a simple function that downloads a remote file to a local filepath using hyper. I need the file write to be asynchronous as well (in my case I am using tokio_fs for that). Here is the code:



View in the playground



// Parts of the code were omitted, see the playground for full source code
pub fn download_file(
uri: Uri,
file_location: &Path,
) -> Box<Future<Item = (), Error = DownloadFileError>> DownloadFileError::CreateFile(err));

Box::new(
response_future
.join(create_file_future)
.and_then(move


However, I get the following error:



error[E0507]: cannot move out of captured outer variable in an `FnMut` closure
--> src/lib.rs:79:39
|
75 | .and_then(move |(res, file)| {
| ---- captured outer variable
...
79 | io::write_all(file, chunk)
| ^^^^ cannot move out of captured outer variable in an `FnMut` closure


Conceptually, I understand what the error means: Since a FnMut captures variables by mutable reference, I cannot move a captured variable. However, I do not understand how can I work around this problem in the example given, since I need to write the stream to the file returned by the Join future.



The write_all method from the Write trait would work here since it takes the file as a mutable reference, but the problem is that it does the writing on the same thread.







rust future borrow-checker rust-tokio hyper






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 16 '18 at 3:17







lhahn

















asked Nov 16 '18 at 2:22









lhahnlhahn

6711626




6711626












  • I think there is no point to use for_each(), into_body() return an option so just use map() instead. But maybe I totally wrong I don't understand a thing in hyper/futures

    – Stargateur
    Nov 16 '18 at 2:45












  • As far as I understand from the answers both of them write synchronously to the file using write_all. In my case io::write_all is done asynchronously as well.

    – lhahn
    Nov 16 '18 at 3:13

















  • I think there is no point to use for_each(), into_body() return an option so just use map() instead. But maybe I totally wrong I don't understand a thing in hyper/futures

    – Stargateur
    Nov 16 '18 at 2:45












  • As far as I understand from the answers both of them write synchronously to the file using write_all. In my case io::write_all is done asynchronously as well.

    – lhahn
    Nov 16 '18 at 3:13
















I think there is no point to use for_each(), into_body() return an option so just use map() instead. But maybe I totally wrong I don't understand a thing in hyper/futures

– Stargateur
Nov 16 '18 at 2:45






I think there is no point to use for_each(), into_body() return an option so just use map() instead. But maybe I totally wrong I don't understand a thing in hyper/futures

– Stargateur
Nov 16 '18 at 2:45














As far as I understand from the answers both of them write synchronously to the file using write_all. In my case io::write_all is done asynchronously as well.

– lhahn
Nov 16 '18 at 3:13





As far as I understand from the answers both of them write synchronously to the file using write_all. In my case io::write_all is done asynchronously as well.

– lhahn
Nov 16 '18 at 3:13












1 Answer
1






active

oldest

votes


















3














You do not want to use for_each. io::write_all consumes the target and the buffer in exchange for a future that will return the target and the buffer when it is done. You can combine this with Stream::fold to reuse the file:



.fold(file, |file, chunk| 
io::write_all(file, chunk)
.map()
.map(drop)


See also:



  • How do I write a futures::Stream to disk without storing it entirely in memory first?

  • How to save a file downloaded from S3 with Rusoto to my hard drive?





share|improve this answer
























    Your Answer






    StackExchange.ifUsing("editor", function ()
    StackExchange.using("externalEditor", function ()
    StackExchange.using("snippets", function ()
    StackExchange.snippets.init();
    );
    );
    , "code-snippets");

    StackExchange.ready(function()
    var channelOptions =
    tags: "".split(" "),
    id: "1"
    ;
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function()
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled)
    StackExchange.using("snippets", function()
    createEditor();
    );

    else
    createEditor();

    );

    function createEditor()
    StackExchange.prepareEditor(
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: true,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: 10,
    bindNavPrevention: true,
    postfix: "",
    imageUploader:
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    ,
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    );



    );













    draft saved

    draft discarded


















    StackExchange.ready(
    function ()
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53330556%2fwriting-a-chunk-stream-to-a-file-asynchronously-using-hyper%23new-answer', 'question_page');

    );

    Post as a guest















    Required, but never shown

























    1 Answer
    1






    active

    oldest

    votes








    1 Answer
    1






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    3














    You do not want to use for_each. io::write_all consumes the target and the buffer in exchange for a future that will return the target and the buffer when it is done. You can combine this with Stream::fold to reuse the file:



    .fold(file, |file, chunk| 
    io::write_all(file, chunk)
    .map()
    .map(drop)


    See also:



    • How do I write a futures::Stream to disk without storing it entirely in memory first?

    • How to save a file downloaded from S3 with Rusoto to my hard drive?





    share|improve this answer





























      3














      You do not want to use for_each. io::write_all consumes the target and the buffer in exchange for a future that will return the target and the buffer when it is done. You can combine this with Stream::fold to reuse the file:



      .fold(file, |file, chunk| 
      io::write_all(file, chunk)
      .map()
      .map(drop)


      See also:



      • How do I write a futures::Stream to disk without storing it entirely in memory first?

      • How to save a file downloaded from S3 with Rusoto to my hard drive?





      share|improve this answer



























        3












        3








        3







        You do not want to use for_each. io::write_all consumes the target and the buffer in exchange for a future that will return the target and the buffer when it is done. You can combine this with Stream::fold to reuse the file:



        .fold(file, |file, chunk| 
        io::write_all(file, chunk)
        .map()
        .map(drop)


        See also:



        • How do I write a futures::Stream to disk without storing it entirely in memory first?

        • How to save a file downloaded from S3 with Rusoto to my hard drive?





        share|improve this answer















        You do not want to use for_each. io::write_all consumes the target and the buffer in exchange for a future that will return the target and the buffer when it is done. You can combine this with Stream::fold to reuse the file:



        .fold(file, |file, chunk| 
        io::write_all(file, chunk)
        .map()
        .map(drop)


        See also:



        • How do I write a futures::Stream to disk without storing it entirely in memory first?

        • How to save a file downloaded from S3 with Rusoto to my hard drive?






        share|improve this answer














        share|improve this answer



        share|improve this answer








        edited Nov 17 '18 at 3:41

























        answered Nov 16 '18 at 16:09









        ShepmasterShepmaster

        159k15325468




        159k15325468





























            draft saved

            draft discarded
















































            Thanks for contributing an answer to Stack Overflow!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid


            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.

            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function ()
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53330556%2fwriting-a-chunk-stream-to-a-file-asynchronously-using-hyper%23new-answer', 'question_page');

            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            Popular posts from this blog

            Top Tejano songwriter Luis Silva dead of heart attack at 64

            ReactJS Fetched API data displays live - need Data displayed static

            政党