Shawn Weisfeld

I find when I talk to myself nobody listens. - Shawn Weisfeld
posts - 365, comments - 176, trackbacks - 34

My Links

News


Shawn Weisfeld's Facebook profile

The views expressed in this blog are mine and mine alone, not that of my employer, Microsoft, or anyone else’s. No warrantee is given for the quality of any material on this site.

Archives

Post Categories

Upload files to Azure Blobs FAST

CAUTION: This worked for me...Use it at your own risk.

I was given a task to upload billions of small (10k) files to Azure, yes that is billions with a B. Like everyone else I started by reading the MSDN docs and having it show me the toBlob.UploadFile and toBlob.UploadFromStream. Both proved to be a bit slow in my case. I can only assume the issue is the overhead of opening and closing the connections.

My first iteration was to just wrap everything with a Parallel.ForEach, and while this drastically improved things it still was not good enough.

My second iteration was to bump up the "Default Connection Limit" (see Line one in the example blow), again things got better but not good enough.

My third iteration was to use the Task.Factory.FromAsync to wrap the async calls that the API exposes in a Task, finally things are much better now. (see complete code sample below)

Print | posted on Monday, March 11, 2013 9:18 AM |

Feedback

No comments posted yet.

Post Comment

Title  
Name  
Email
Url
Comment   
Please add 1 and 2 and type the answer here:

Powered by: