Shawn Weisfeld

I find when I talk to myself nobody listens. - Shawn Weisfeld
posts - 356, comments - 173, trackbacks - 34

My Links

News


Shawn Weisfeld's Facebook profile

The views expressed in this blog are mine and mine alone, not that of my employer, Microsoft, or anyone else’s. No warrantee is given for the quality of any material on this site.

Archives

Post Categories

Monday, March 11, 2013

Upload files to Azure Blobs FAST

CAUTION: This worked for me...Use it at your own risk.

I was given a task to upload billions of small (10k) files to Azure, yes that is billions with a B. Like everyone else I started by reading the MSDN docs and having it show me the toBlob.UploadFile and toBlob.UploadFromStream. Both proved to be a bit slow in my case. I can only assume the issue is the overhead of opening and closing the connections.

My first iteration was to just wrap everything with a Parallel.ForEach, and while this drastically improved things it still was not good enough.

My second iteration was to bump up the "Default Connection Limit" (see Line one in the example blow), again things got better but not good enough.

My third iteration was to use the Task.Factory.FromAsync to wrap the async calls that the API exposes in a Task, finally things are much better now. (see complete code sample below)

posted @ Monday, March 11, 2013 9:18 AM | Feedback (0) |

Wednesday, January 23, 2013

Zebra Strip a jqGrid

 

Zebra striping, or greenbar, or odd/even row highlighting whatever you call it, your boss asked you for it and now you are here and you are trying to figure out how to implement it. When I was attempting to do this I saw many examples using javascript to do this, however why do in javascript what you can do in CSS.

If you look for a solution for a standard html table (of which there are millions on the web) they will tell you to do something like this:
    #grdResults tr:nth-child(even) {
            background-color:rgb(226, 228, 240);
        }

While this works great for normal tables for a jqGrid it will not work. Why because the cells themselves are overriding your command. Easy enough to solve, just color the background of each cell. (note the td at the end of the first line)
        #grdResults tr:nth-child(even) td {
            background-color:rgb(226, 228, 240);
        }

Wammo! A zebra striped jqGrid with no additional javascript overhead.

posted @ Wednesday, January 23, 2013 9:05 PM | Feedback (0) |

Tuesday, January 22, 2013

Use ASP.NET MVC as a template engine

Get the code here: https://github.com/shawnweisfeld/MvcRenderAndSaveFile

Got a question today from a colleague. He had a database full of records, lets say customers, and needed to generate static html document for each customer. I did not ask him why he needed the static html, but let’s assume that he was going to use it as the body of an email message. After doing a bunch of thinking I was disappointed that I could not use the power of the MVC template-ing engine, razor. Or could I. What if I wrote a view that pulled a customer from the database and created a view in the normal manner (See the Home\CustomerInfo Action & View). That was easy, boy to I love razor. Now to the hard part, rendering the html for all 100 of our customers. If can use WebClient to download any page off of the internet, why cannot we use it to download a page from our own website (see the Home\ProcessCustomers Action). So with a few lines of code we can now render any html we want. YEA!

While I have not played with it, you could avoid the network traffic with something like the RazorEngine (available from Nuget.org @ http://nuget.org/packages/RazorEngine/)

posted @ Tuesday, January 22, 2013 8:34 PM | Feedback (0) |

Thursday, January 17, 2013

Entity Framework insert performance

While I am big believer in Entity Framework (EF) there are some cases where dropping down to raw ADO.NET make sense. In an application I am working on I need to insert thousands of records into the database, and while the change tracking mechanism and other features in EF provides a great service sometimes the overhead of it doesn’t outweigh the benefits it offers.

I threw together a quick little sample app (you can download it from GitHub here) that shows the problem. The app simply creates a bunch of random customer objects and saves them to sql server. Here are the run times for a set of 1,000 and 10,000 customers. As you can see using raw ADO.NET is about 35% faster with record sets of 1,000 and with the 10,000 size record set I get a 70% performance increase. Additionally it appears that the performance with raw ADO.NET is linear as you scale up while EF appears to perform worse as the record set grows.

 

(times in milliseconds)

Run 1,000 Records     10,000 Records  
  EF DbHelper     EF DbHelper  
1 4,359 2,281     74,010 22,088  
2 3,389 2,233     70,250 20,621  
3 3,370 2,266     71,181 22,470  
4 3,383 2,302     74,096 22,495  
5 3,392 2,288     76,465 20,759  
6 3,367 2,245     70,030 19,778  
7 3,450 2,249     69,176 20,485  
8 3,369 2,252     69,833 19,680  
9 3,368 2,268     71,728 20,751  
10 3,390 2,298     74,084 22,821  
Average 3,484 2,268 34.9%   72,085 21,195 70.6%

 

BTW: SqlBulkCopy is about 100x faster then either of the above if raw speed is important to you. (I have included an example of that in the code on GitHub)

Please note this is all anecdotal information that I have got from my observations of the system, and I could be woefully wrong, please let me know if I am. . .

posted @ Thursday, January 17, 2013 12:28 PM | Feedback (0) | Filed Under [ SQL ]

Thursday, March 22, 2012

Display a list of data as a set of columns not rows

Typically when asked to display a list data, lets say customers, we create a first name column, last name column and then iterate over the rows to produce a table. Something like this:

clip_image001

But recently I was asked to produce a pivot of this data, putting each customer in their own column instead of an individual row. After doing the obligatory internet searching I found no solution that I really liked. So I wrote my own…

Lets start by creating our customer object:

public class Customer
{
    public int Id { get; set; }
    public string FirstName { get; set; }
    public string LastName { get; set; }
    public int Age { get; set; }
}
 

Now that we got him, lets mock up some data in our controller:

public ActionResult Index()
{
    var customers = new List<Customer>();

    for (int i = 0; i < 100; i++)
    {
        customers.Add(new Customer()
                            {
                                Id = i,
                                FirstName = string.Format("First {0}", i),
                                LastName = string.Format("Last {0}", i),
                                Age = i
                            });
    }

    ViewBag.Customers = customers;

    return View();
}
 

Finally, the meat, using a bit of reflection magic lets get a “list” of all the properties on our customer, iterate over each of them to generate our rows, then iterate over each customer and get the “value” for that row.

<div style="width: 600px; overflow: auto">
    <table style="white-space: nowrap;">
        <thead>
            <tr>
                <th></th>
                @foreach (var customer in ViewBag.Customers)
                {
                    <th style="">
                        Customer @customer.Id
                    </th>
                }
            </tr>
        </thead>
        <tbody>
            @foreach (var prop in typeof(Customer).GetProperties())
            {
                <tr>
                    <td>
                        @prop.Name
                    </td>
                    @foreach (var customer in ViewBag.Customers)
                    {
                        <td>
                            @prop.GetValue(customer, null)
                        </td>   
                    }
                </tr>
            }
        </tbody>
    </table>
</div>

 

And ta-da we have a table with dynamically generated columns based on the number of “customers” in our collection.

clip_image002

posted @ Thursday, March 22, 2012 9:14 AM | Feedback (0) |

Saturday, March 17, 2012

Convert flattened data in Excel to a hierarchal XML File

Ever had an excel file that looked like this?

image

and need an xml file that looks like this?

image

 

Converting takes 2 steps:

Step 1: Read the excel file into objects in memory

    var products = new List<Product>();

    using (var con = new OleDbConnection("Provider=Microsoft.ACE.OLEDB.12.0;Data Source=data.xlsx;Extended Properties=\"Excel 8.0;HDR=YES;\""))
    using (var cmd = con.CreateCommand())
    {
        con.Open();
        cmd.CommandText = "SELECT * FROM [Sheet1$]";
        var dr = cmd.ExecuteReader();

        while (dr.Read())
        {
            products.Add(new Product()
            {
                ProductGroup = dr[0].ToString(),
                ProductSubGroup = dr[1].ToString(),
                ProductNumber = dr[2].ToString()
            });
        }
    }

 

Step 2: Use LINQ to group the objects into a hierarchy and convert them into xml

    var xml = new XElement("ProductGroups",
        products.GroupBy(x => x.ProductGroup).Select(x => new XElement("ProductGroup",
            new XElement("name", x.Key),
            new XElement("ProductSubGroups",
                x.GroupBy(y => y.ProductSubGroup).Select(y => new XElement("ProductSubGroup",
                    new XElement("name", y.Key),
                    new XElement("Product", y.Select(z => new XElement("number", z.ProductNumber)))))))));

    xml.Save("test.xml");

posted @ Saturday, March 17, 2012 3:01 PM | Feedback (0) |

Sunday, August 28, 2011

Create a template for Orchard

Orchard the open source CMS from Microsoft (http://www.orchardproject.net/) has a great tempting engine. The out of the box base template “The Theme Machine” is very powerful and flexible, however if you are as design challenged as I, creating something with it that looks decent is out of the question. To that end I use a very cool template builder, Artisteer (http://www.artisteer.com/) if that is not your speed there are about a million websites where you can download free/low cost templates, however most of them generate a standard HTML page with some javascript and css. I wanted to take a minute to document how I converted my HTML template into an Orchard theme.

(Disclaimer this worked for my site, your template might require some different html changes, especially when it comes time to edit the HTML and CSS)

Step 1: Get the html template. (Did I really need to tell you that?)

Step 2: The goal is to make a theme based on the base template “The Theme Machine”. You will need to follow the instructions on the Orchard site to “Generate a New Theme” http://www.orchardproject.net/docs/Writing-a-new-theme.ashx, however their instructions have you creating an empty theme. We need to create one based on “The Theme Machine”. To do that just alter the codegen step to use the following command:

>> codegen theme MyTheme /BasedOn:TheThemeMachine /CreateProject:true /IncludeInSolution:true

Step 3: Copy the CSS & related images into the “Styles” folder of your new theme

Step 4: Copy the javascript into the “Scripts” folder of your new theme

Step 5: Copy the template html file into the Views folder. I do this so that I have it for safe keeping and it stays with the project. Orchard will not be using it directly.

Step 6: Copy the “Layout.cshtml” file from the “Views” folder from the “TheThemeMachine” template into the “Views” folder of our new theme.

Step 7: This is the hard part, time to merge the HTML template that you found/created in Step 1 with the Layout.cshtml file you copied into your “Views” folder in Step 6.

There are 2 things that you will have to worry about here:

First you need to add references for the css and js files. My template requires 3 css files and 1 js file, so I needed to add the following at the top of the layout.cshtml file. I put mine just below the includes “Style.Include” statements that were already in the file.

My template looks like this:

clip_image002

Converted to the layout.cshtml file it looks like:

image

Now that we have the head tag filled with all its goodness, it is time to do the body part. Copy over the html from the template into your Layout.cshtml file. You want to copy the html just below the “@tag.StartElement” line of code. With the markup in place, start moving the placeholders from the Layout.cshtml file into the proper places in your new template.

Once you have moved the placeholders around, you will probably need to add a css file tweak everything just right.

Step 8: The menu. My template and my client required a drop down menu, however out of the box Orchard only supports a single level menu. Time to add a module, the “Advanced Menu” module from Piotr Szmyd to be specific. After installing this module you will now have the ability to create multi level menus. However it comes with its own look and feel. If this works for you great, if not, you will have to modify it. Since my template provides for a menu that I like I am going to use it. To do so I need to copy 2 more cshtml files into my "Views” folder. After adding the module you should see a folder for it listed in your “Modules” folder called “Szmyd.Orchard.Modules.Menu”. In there you will find a “Views” folder and in it you will see Menu.cshtml, and MenuItem.cshtml. Copy those two files into the "Views” folder of your theme. Now alter them to match your template, just like you did the layout.cshtml file.

posted @ Sunday, August 28, 2011 2:08 PM | Feedback (1) |

Tuesday, August 2, 2011

IIS 7 401.2 error

 

I was deploying a website and was seeing the following behavior:

  1. site configured to use Windows Authentication & Impersonation
  2. everything in KB 942043 was set correctly (http://support.microsoft.com/kb/942043)
  3. site was working from IE on the WebServer
  4. site was working from FireFox on my desktop
  5. site was NOT working in Chrome on my desktop “Error 338 (net::ERR_INVALID_AUTH_CREDENTIALS): Unknown error.”
  6. site was NOT working in IE on my desktop, 401 error message after asking for UserName and Password 3 times

The fix was to force the web server to use NTLM. To do this in IIS 7 is just a setting in system.web/system.webServer/security/authentication/windowsAuthentication/providers section of the web.config:

Here is a screen print of my web.config

image

posted @ Tuesday, August 2, 2011 11:43 AM | Feedback (3) |

Tuesday, July 26, 2011

Auditing with Entity Framework

Also take a look at the new logging stuff added to EF 6 http://msdn.microsoft.com/en-us/data/dn469464.aspx 

I was investigating techniques to do auditing and think this is going to fit my needs, thought I would share. . . .

Let me start by saying there are about 1000 examples similar to this on the web. However none of the examples I found meet my specific requirements.

First lets discuss the requirements:

  1. I need to be able to apply auditing to only some of the types in my system
  2. I need to be able to ignore some of the properties on a type (i.e. not audit them)
  3. I need something flexible enough that I can apply it to multiple DbContexts
  4. I needed to have audit columns on each record (CreatedBy, CreatedOn, UpdatedOn, UpdatedBy).
  5. I needed a log table that would record when we got new records, when a field got changed, and when a record was deleted (including who and when it was done)

IAuditable Interface. I am using this guy to allow each type to “opt-in” to the auditing mechanism.

image

 

AuditIgnoreAttribute. I am using this to “opt-out” on a field by field basis of the auditing mechanism.

image

 

So now when I create my business object, I can mark him up appropriately depending on the requirements for that type. As you can see I want to audit my customer object, but not the Bio property.

image

 

Great now for the magic, the AuditableContext. This guy inherits from a normal DbContext, and overrides the SaveChanges method to add the auditing implementation. The auditing implementation is done in 5 steps:

  1. Figure out what the user changed and hang on to those changes for later
  2. Update the CreatedBy, CreatedOn, UpdatedOn, UpdatedBy columns as appropriate
  3. Perform a SaveChanges to write the users changes to the DB
  4. Update our list of user changes with the proper id’s (we need to do this because the data base is assigning the primary keys for our new entities)
  5. Save the list of user changes to the log table

image

image

 

That code should be straight forward. I did implement a helper method that figures out what fields we are interested by using a bit of reflection. NOTE: we are doing the reflection once per type and caching the results in a dictionary.

image

 

We only need to implement the AuditableContext once, each time we create a context in our app, we just inherit from it and we are off and to the races!

image

 

You can download a full working example from here

 

Happy Coding!

posted @ Tuesday, July 26, 2011 10:29 AM | Feedback (2) |

Saturday, July 23, 2011

.NET Micro comes to Dallas

 

I had a great time at the event today and wanted to thank a few folks that helped make this event:

What is next? How about the Kinect? http://research.microsoft.com/en-us/um/redmond/projects/kinectsdk/

 

posted @ Saturday, July 23, 2011 3:12 PM | Feedback (0) | Filed Under [ NETMF ]

Powered by: