Skip to content
This repository has been archived by the owner on Oct 6, 2020. It is now read-only.

Implement streaming operations on blobs #42

Open
MindFlavor opened this issue May 30, 2018 · 3 comments
Open

Implement streaming operations on blobs #42

MindFlavor opened this issue May 30, 2018 · 3 comments
Assignees

Comments

@MindFlavor
Copy link
Owner

Reading and writing blobs in a single operation is cumbersome and really not efficient.
Since we are aync anyway it would make sense to enable data processing as soon as possible without waiting for the complete blob. Also, there might be no need to allocate a big buffer for a file.

The future crate exposes the Stream trait so we should implement it.

@MindFlavor MindFlavor self-assigned this May 30, 2018
@MindFlavor
Copy link
Owner Author

MindFlavor commented May 30, 2018

The read access is completed. It's included in release 0.7.1 merged by #43 .

For example:

let stream = Blob::stream(
        &client,
        &container_name,
        file_name,
        None,
        &Range::new(0, 4096),
        None,
        1024,
);

let fut = stream.for_each(move |mut value| {
        println!("received {:?} bytes", value.len());
        ok(())
});

This will ask the SDK to get the first 4KB of the file in chunks of 1KB. At each chunk you get the chance to do something (power of the future crate!).

@thedodd
Copy link

thedodd commented Jun 4, 2018

Happy to see this! I will be using it soon. Stoked for the streaming upload as well, as I have already started using the fully buffered upload. Being able to stream the upload will be that much more awesome.

@MindFlavor
Copy link
Owner Author

For the streaming upload I need to complete the Put Block REST API first (it was half-baked anyway). This way various chunks of data can be uploaded independently and the finalized with a single Put block list call. Hope to be able to work on this soon!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

2 participants