.NET 4.0 FAQ Part 1 -- The DLR

Introduction
Where do I get .Net 4.0 from?
What are the important new features in .NET 4.0?
What’s the most important new feature of .NET 4.0?
What is DLR in .NET 4.0 framework?
Can you give more details about DLR subsystem?
How can we consume an object from dynamic language and expose a class to dynamic languages?
Can we see a sample of ‘Dynamic’ objects?
What’s the difference between ‘Dynamic’, ‘Object’ and reflection?
What are the advantages and disadvantage of dynamic keyword?
What are expando objects?
Can we implement our own ‘Expando’ object?
What is the advantage of using custom ‘Expando’ class?
What are IDynamicMetaObjectProvider and DynamicMetaObject?
Can we see performance difference between reflection and dynamic object execution?
Thanks, Thanks and Thanks
.NET 4.0 detail list of new features

Introduction

In this article we will discuss about what new features are provided by .NET framework 4.0. We will then take up the DLR feature and discuss about ‘dynamic’ and ‘expando’ objects. We will also create a custom ‘expando’ class and see what benefits we can get from the same. Many developers mistake ‘dynamic’ objects where made to replace ‘reflection’ and ‘object’ type, we will try to remove this misconception also.

Sorry for the repost. I have deleted the old article due to image upload issues and uploaded again.

Please feel free to download my free 500 question and answer eBook which covers .NET 4.0 , ASP.NET , design patterns, silver light, LINQ, SQL Server , WCF , WPF, WWF@
http://www.questpond.com/

Where do I get .Net 4.0 from?

You can download .NET 4.0 beta from
http://www.microsoft.com/downloads/details.aspx?FamilyID=ee2118cc-51cd-46ad-ab17-af6fff7538c9&displaylang=en

What are the important new features in .NET 4.0?

Rather than walking through the 100 new features list, let’s concentrate on the top 3 features which we think are important. If you are interested to see the detail list of new features click here.
• Windows work flow and WCF 4.0:- This is a major change in 4.0. In WCF they have introduced simplified configuration, discovery, routing service, REST improvements and workflow services. In WWF they have made changes to the core programming model of workflow. Programming model has been made more simple and robust. The biggest thing is the integration between WCF and WWF.
• Dynamic Language runtime: - DLR adds dynamic programming capability to .NET 4.0 CLR. We will talk more about it as this FAQ moves ahead.
• Parallel extensions: - This will help to support parallel computing for multi-core systems. .NET 4.0 has PLINQ in the LINQ engine to support parallel execution. TPL (Task parallel library) is introduced which exposes parallel constructs like parallel ‘For’ and ‘ForEach’ loops, using regular method calls and delegates.

We will be talking in more details of the above features in the coming sections.

What’s the most important new feature of .NET 4.0?

WCF and WWF new features are one of the most interesting features of all. Especially the new programming model of WWF and its integration with WCF will be an interesting thing to watch.

DLR, parallel programming and other new features somehow just seem to be brownie points rather than compelling features. Kathleen Dollard's talks about it in more details
http://msmvps.com/blogs/kathleen/archive/2009/01/07/the-most-important-feature-of-net-4-0.aspx

We will be seeing more of these features as we continue with the FAQ.

What is DLR in .NET 4.0 framework?

DLR (Dynamic language runtime) is set of services which add dynamic programming capability to CLR. DLR makes dynamic languages like LISP, Javascript, PHP,Ruby to run on .NET framework.

There are two types of languages statically typed languages and dynamically typed languages. In statically typed languages you need to specify the object during design time / compile time. Dynamically typed languages can identify the object during runtime. .NET . DLR helps you to host code written in dynamic languages on top of your CLR.

Due to DLR runtime, dynamic languages like ruby, python, JavaScript etc can integrate and run seamlessly with CLR. DLR thus helps to build the best experience for your favorite dynamic language. Your code becomes much cleaner and seamless while integrating with the dynamic languages.

Integration with DLR is not limited to dynamic languages. You can also call MS office components in a much cleaner way by using COM interop binder.

One of the important advantages of DLR is that it provides one central and unified subsystem for dynamic language integration.

Can you give more details about DLR subsystem?

DLR has 3 basic subsystems:-

• Expression trees: - By this we can express language semantics in form of AST (Abstract syntax tree). DLR dynamically generates code using the AST which can be executed by the CLR runtime. An expression tree is a main player which helps to run various dynamic languages javascript, ruby with CLR.
• Call site caching: - When you make method calls to dynamic objects DLR caches information about those method calls. For the other subsequent calls to the method DLR uses the cache history information for fast dispatch.
• Dynamic object interoperability (DOI):- DOI has set of classes which can be used to create dynamic objects. These classes can be used by developers to create classes which can be used in dynamic and static languages.

We will be covering all the above features in more detail in the coming FAQ sections.

How can we consume an object from dynamic language and expose a class to dynamic languages?

To consume a class created in DLR supported dynamic languages we can use the ‘Dynamic’ keyword. For exposing our classes to DLR aware languages we can use the ‘Expando’ class.

So when you want to consume a class constructed in Python , Ruby , Javascript , COM languages etc we need to use the dynamic object to reference the object. If you want your classes to be consumed by dynamic languages you need to create your class by inheriting the ‘Expando’ class. These classes can then be consumed by the dynamic languages. We will be seeing both these classes in the coming section.

Do not forget to download help document for library authors regarding how to enable the dynamic language across platform using DLR
http://dlr.codeplex.com/Wiki/View.aspx?title=Docs

Can we see a sample of ‘Dynamic’ objects?

We had already discussed that ‘Dynamic’ objects helps to consume objects which are created in dynamic languages which support DLR.The dynamic keyword is a part of dynamic object interoperability subsystem.

If you assign an object to a dynamic type variable (dynamic x=new SomeClass()), all method calls, property invocations, and operator invocations on ‘x’ will be delayed till runtime, and the compiler won't perform any type checks for ‘x’ at compile time.

Consider the below code snippet where we are trying to do method calls to excel application using interop services.

// Get the running object of the excel application
object objApp = System.Runtime.InteropServices.Marshal.GetActiveObject("Excel.Application");
// Invoke the member dynamically
object x = objApp.GetType().InvokeMember("Name", System.Reflection.BindingFlags.GetProperty, null, objApp, null);
// Finally get the value by type casting
MessageBox.Show(x.ToString());

The same code we now write using ‘dynamic’ keyword.

// Get the object using
dynamic objApp1 = System.Runtime.InteropServices.Marshal.GetActiveObject("Excel.Application");
// Call the
MessageBox.Show(objApp1.Name);

You can clearly notice the simplification of property invocation syntax. The ‘invokemember’ is pretty cryptic and prone to errors. Using ‘dynamic’ keyword we can see how the code is simplified.

If you try to view the properties in VS IDE you will see a message stating that you can only evaluate during runtime.

What’s the difference between ‘Dynamic’, ‘Object’ and reflection?

Many developers think that ‘Dynamic’ objects where introduced to replace to ‘Reflection’ or the ‘Object’ data type. The main goal of ‘Dynamic’ object is to consume objects created in dynamic languages seamlessly in statically typed languages. But due to this feature some of its goals got overlapped with reflection and object data type.

Eventually it will replace reflection and object data types due to simplified code and caching advantages. The main goal of dynamic object was never introduced in the first place to replace ‘reflection’ or object data type, but due to overlapping features it did.

What are the advantages and disadvantage of dynamic keyword?

We all still remember how we talked bad about VB6 (Well I loved the language) variant keyword and we all appreciated how .NET brought in the compile time check feature, well so why are we changing now.

Well, bad developers will write bad code with the best programming language and good developers will fly with the worst programming language. Dynamic keyword is a good tool to reduce complexity and it’s a curse when not used properly.

So advantages of Dynamic keyword:-

• Helps you interop between dynamic languages.
• Eliminates bad reflection code and simplifies code complexity.
• Improves performance with method call caching.

Disadvantages:-

• Will hit performance if used with strongly typed language.

What are expando objects?

‘Expando’ objects serve the other side of interoperability i.e. enabling your custom classes to be consumed in dynamic languages. So you can create an object of ‘Expando’ class and pass to dynamic languages like ruby, javascript, python etc. ‘Expando’ objects helps to add properties on fly. It’s an efficient implementation of dynamic property bag. In order to use ‘ExpandoObject’ we first import ‘System.Dynamic’ namespace.

using System.Dynamic;

We then create the object of ‘ExpandoObject’ and assign it to an object which created from ‘Dynamic’ class type. Please note if we have used ‘Dynamic’ objects and not expand objects as we still do not know what properties are going to be created during runtime.

dynamic obj = new ExpandoObject();

For creating dynamic property we just need to write the property name and set the value.

obj.Customername = "Some Customer Name";

Finally we display the value.

MessageBox.Show(obj.Customername);

Can we implement our own ‘Expando’ object?

‘Expando’ object internally is nothing but properties added to a collection. So you can create your own version of ‘Expando’ object.

The first thing we need to do is inherit from ‘DynamicObject’ class.

public class clsMyExpando : DynamicObject
{
}

As said previously we need to define a collection where we can save properties. In the second step we have created a dictionary object to maintain the properties in the collection.

public class clsMyExpando : DynamicObject
{
Dictionary items= new Dictionary();
}

We can now use the ‘TryGetMember’ and ‘SetGetMember’ to define our get and set properties.

public class clsMyExpando : DynamicObject
{
Dictionary items = new Dictionary();
public override bool TryGetMember(GetMemberBinder binder, out object result)
{
return items.TryGetValue(binder.Name, out result);
}
public override bool TrySetMember(SetMemberBinder binder, object value)
{
items[binder.Name] = value;
return true;
}}

We can now create object of our customer ‘expando’ class and assign the same to the dynamic class reference. In the below code snippet we have assigned a dynamic property called as ‘Name’.

dynamic obj = new clsMyExpando();
obj.Name = "Dynamic Property";

What is the advantage of using custom ‘Expando’ class?

Custom ‘Expando’ object can be used to gain performance. In case your class has some static properties and some dynamic properties, then you can create static properties in the custom expand class itself as shown in the below code. So when the static properties of the object are called it will not make calls to the dictionary collection thus increasing performance. DLR engine first tries to call the property names rather than calling the ‘TryGetMember’ or ‘TrySetMember’.

First thing avoid ‘expando’ custom classes if you do not have dynamic property requirement and you do not want to communicate with dynamic languages. In case you have dynamic property requirement ensure that properties which you are very sure are added to the class and dynamic properties are implemented by inheriting the dynamicobject class.

public class clsMyExpando : DynamicObject
    {
Dictionary items
        = new Dictionary();
    
        private string _Name;
        public string Name
        {
            get
            {
                return _Name;
            }
            set
            {
                _Name = value;
            }
        }
        public override bool TryGetMember(
            GetMemberBinder binder, out object result)
        {
            return items.TryGetValue(binder.Name, out result);
        }

        public override bool TrySetMember(
            SetMemberBinder binder, object value)
        {
            items[binder.Name] = value;
            return true;
        }

    }

What are IDynamicMetaObjectProvider and DynamicMetaObject?

‘Dynamic’ objects implement ‘IDynamicMetaObjectProvider’ and return ‘DynamicMetaObject’. Both these interfaces are core classes which implement interoperability between dynamic languages. So when should we use it:-

If we want to implement our custom logic for fast dynamic property retrieval we can implement ‘IDynamicMetaObjectProvider’ and ‘DynamicMetaObject’. For instance you would like to provide a fast algorithm to retrieve the most used properties. So you can plug-in the algorithm using ‘IDynamicMetaObjectProvider’.

Can we see performance difference between reflection and dynamic object execution?

Coming soon….

Thanks, Thanks and Thanks

http://tomlev2.wordpress.com/category/code-sample/ has sample code which
shows how to implement dynamic objects.

http://dlr.codeplex.com/Wiki/View.aspx?title=Docs%20and%20specs :- Codeplex
link for the DLR project.

http://msmvps.com/blogs/kathleen/archive/2009/01/07/the-most-important-feature-of-net-4-0.aspx
:- Discusses about the most important feature of .NET 4.0

http://blogs.msdn.com/brada/archive/2008/10/29/net-framework-4-poster.aspx
:- A nice 4.0 complete posture , ensure you have silver light plug-in to take
advantage of the zoom in feature.

http://www.hanselman.com/blog/C4AndTheDynamicKeywordWhirlwindTourAroundNET4AndVisualStudio2010Beta1.aspx
:- Sir Hansel talks about the dynamic keyword.
http://www.codeproject.com/KB/cs/dynamicfun.aspx :- Dynamic example
implementation.

http://en.wikipedia.org/wiki/Dynamic_Language_Runtime :- Wiki link for DLR.
http://msdn.microsoft.com/en-us/library/dd233052(VS.100).aspx :- Download
link for 4.0 and VS 2010 .

http://www.developerfusion.com/article/9576/the-future-of-net-languages/2/
:- Discusses about new language features of 4.0 , I have not seen anything
simpler than this on the web.

http://www.gotnet.biz/Blog/file.axd?file=2009%2F6%2FHow+I+Learned+to+Love+Metaprogramming+-+CodeStock+2009.pdf
:- Nice PDF by Kevin MVP for meta programming.

.NET 4.0 detail list of new features

Thanks to
http://blogs.msdn.com/brada/archive/2008/10/29/net-framework-4-poster.aspx publish a detail poster of .NET 4.0 features.

Best Practices No 5: - Detecting .NET application memory leaks

• Download WindowsAppMemoryLeak - 34.04 KB

Introduction

Memory leaks in .NET application have always being programmer’s nightmare. Memory leaks are biggest problems when it comes to production servers. Productions servers normally need to run with least down time. Memory leaks grow slowly and after sometime they bring down the server by consuming huge chunks of memory. Maximum time people reboot the system, make it work temporarily and send a sorry note to the customer for the downtime.

Please feel free to download my free 500 question and answer eBook which covers .NET , ASP.NET , SQL Server , WCF , WPF , WWF@ http://www.questpond.com .

Avoid task manager to detect memory leak
Using private bytes performance counters to detect memory leak
3 step process to investigate memory leak
What is the type of memory leak? Total Memory = Managed memory + unmanaged memory
How is the memory leak happening?
Where is the memory leak?
Source code
Thanks, Thanks and Thanks

Avoid task manager to detect memory leak

The first and foremost task is to confirm that there is memory leak. Many developers use windows task manager to confirm, is there a memory leak in the application?. Using task manager is not only misleading but it also does not give much information about where the memory leak is.

First let’s try to understand how the task manager memory information is misleading. Task manager shows working set memory and not the actual memory used, ok so what does that mean. This memory is the allocated memory and not the used memory. Adding further some memory from the working set can be shared by other processes / application.

So the working set memory can big in amount than actual memory used.

Using private bytes performance counters to detect memory leak

In order to get right amount of memory consumed by the application we need to track the private bytes consumed by the application. Private bytes are those memory areas which are not shared by other application. In order to detect private bytes consumed by an application we need to use performance counters.

Below are the steps we need to follow to track private bytes in an application using performance counters:-

• Start you application which has memory leak and keep it running.
• Click start à Goto run and type ‘perfmon’.
• Delete all the current performance counters by selecting the counter and deleting the same by hitting the delete button.
• Right click à select ‘Add counters’ à select ‘process’ from performance object.
• From the counter list select ‘Private bytes’.
• From the instance list select the application which you want to test memory leak for.

If you application shows a steady increase in private bytes value that means we have a memory leak issue here. You can see in the below figure how private bytes value is increasing steadily thus confirming that application has memory leak.

The above graph shows a linear increase but in live implementation it can take hours to show the uptrend sign. In order to check memory leak you need to run the performance counter for hours or probably days together on production server to check if really there is a memory leak.

3 step process to investigate memory leak

Once we have confirmed that there is a memory leak, it’s time to investigate the root problem of the memory leak. We will divide our journey to the solution in 3 phases what, how and where.

• What: - We will first try to investigate what is the type of memory leak, is it a managed memory leak or an unmanaged memory leak.
• How: - What is really causing the memory leak. Is it the connection object, some kind of file who handle is not closed etc?
• Where: - Which function / routine or logic is causing the memory leak.
  
  

What is the type of memory leak? Total Memory = Managed memory + unmanaged memory

Before we try to understand what the type of leak is, let’s try to understand how memory is allocated in .Net applications. .NET application have two types of memory managed memory and unmanaged memory. Managed memory is controlled by garbage collection while unmanaged memory is outside of garbage collectors boundary.

So the first thing we need to ensure what is the type of memory leak is it managed leak or unmanaged leak. In order to detect if it’s a managed leak or unmanaged leak we need to measure two performance counters.

The first one is the private bytes counter for the application which we have already seen in the previous session.

The second counter which we need to add is ‘Bytes in all heaps’. So select ‘.NET CLR memory’ in the performance object, from the counter list select ‘Bytes in all heaps’ and the select the application which has the memory leak.

Private bytes are the total memory consumed by the application. Bytes in all heaps are the memory consumed by the managed code. So the equation becomes something as shown in the below figure.

Un Managed memory + Bytes in all helps = private bytes, so if we want to find out unmanaged memory we can always subtract the bytes in all heaps from the private bytes.

Now we will make two statements:-

• If the private bytes increase and bytes in all heaps remain constant that means it’s an unmanaged memory leak.
• If the bytes in all heaps increase linearly that means it’s a managed memory leak.

Below is a typical screenshot of unmanaged leak. You can see private bytes are increasing while bytes in heaps remain constant

Below is a typical screen shot of a managed leak. Bytes in all heaps are increasing.

How is the memory leak happening?

Now that we have answered what type of memory is leaking it’s time to see how is the memory leaking. In other words who is causing the memory leak ?.

So let’s inject an unmanaged memory leak by calling ‘Marshal.AllocHGlobal’ function. This function allocates unmanaged memory and thus injecting unmanaged memory leak in the application. This command is run within the timer number of times to cause huge unmanaged leak.

private void timerUnManaged_Tick(object sender, EventArgs e)
{
 Marshal.AllocHGlobal(7000);
}

It’s very difficult to inject a managed leak as GC ensures that the memory is reclaimed. In order to keep things simple we simulate a managed memory leak by creating lot of brush objects and adding them to a list which is a class level variable. It’s a simulation and not a managed leak. Once the application is closed this memory will be reclaimed.

private void timerManaged_Tick(object sender, EventArgs e)
 {
            for (int i = 0; i < 10000; i++)
            {
                Brush obj = new SolidBrush(Color.Blue);
                objBrushes.Add(obj);
            }
        }

In case you are interested to know how leaks can happen in managed memory you can refer to weak handler for more information
http://msdn.microsoft.com/en-us/library/aa970850.aspx .

The next step is to download ‘debugdiag’ tool from
http://www.microsoft.com/DOWNLOADS/details.aspx?FamilyID=28bd5941-c458-46f1-b24d-f60151d875a3&displaylang=en

Start the debug diagnostic tool and select ‘Memory and handle leak’ and click next.

Select the process in which you want to detect memory leak.


Finally select ‘Activate the rule now’.

Now let the application run and ‘Debugdiag’ tool will run at the backend monitoring memory issues.

Once done click on start analysis and let the tool the analysis.

You should get a detail HTML report which shows how unmanaged memory was allocated. In our code we had allocated huge unmanaged memory using ‘AllochGlobal’ which is shown in the report below.

Managed memory leak of brushes are shown using ‘GdiPlus.dll’ in the below HTML report.

Where is the memory leak?

Once you know the source of memory leak is, it’s time to find out which logic is causing the memory leak. There is no automated tool to detect logic which caused memory leaks. You need to manually go in your code and take the pointers provided by ‘debugdiag’ to conclude in which places the issues are.

For instance from the report it’s clear that ‘AllocHGlobal’ is causing the unmanaged leak while one of the objects of GDI is causing the managed leak. Using these details we need to them go in the code to see where exactly the issue lies.

Source code

You can download the source code from the top of this article which can help you inject memory leak.

Thanks, Thanks and Thanks

It would be unfair on my part to say that the above article is completely my knowledge. Thanks for all the lovely people down who have written articles so that one day someone like me can be benefit.

• http://blogs.msdn.com/tess/ A great blog by a lovely lady Tess on debuggers. There are some great labs on memory leak detection using windbg , do not miss it. Tess god bless you, your blog rocks like anything.
• http://msdn.microsoft.com/en-us/magazine/cc163491.aspx This is a great article by James kovacs on managed and unmanaged memory leaks, must to read.
• http://davybrion.com/blog/2009/08/finding-memory-leaks-in-silverlight-with-windbg/ Great article on finding memory leaks using windbg.
• http://www.itwriting.com/dotnetmem.php :- Great link which explains in detail the difference between working set memory and private bytes.
• http://www.microsoft.com/DOWNLOADS/details.aspx?FamilyID=28bd5941-c458-46f1-b24d-f60151d875a3&displaylang=en Download for the debugdiag tool.
• http://blogs.msdn.com/davidklinems/archive/2005/11/16/493580.aspx David Kline explains the 3 common causes of memory leak in managed application.
  
  

Best Practice No 4:- Improve bandwidth performance of ASP.NET sites using IIS compression

Introduction

Bandwidth performance is one of the critical requirements for every website. In
today’s time major cost of the website is not hard disk space but its bandwidth.
So transferring maximum amount of data over the available bandwidth becomes very
critical. In this article we will see how we can use IIS compression to increase
bandwidth performance.
Please feel free to download my free 500 question and answer videos which covers
Design Pattern, UML, Function Points, Enterprise Application Blocks,OOP'S, SDLC,
.NET, ASP.NET, SQL Server, WCF, WPF, WWF, SharePoint, LINQ, SilverLight, .NET
Best Practices @ these videos http://www.questpond.com/

Best Practice No 4:- Improve bandwidth performance of ASP.NET sites using IIS compression

How does IIS compression work?
Compression fundamentals: - Gzip and deflate
Enabling IIS compression0, 1,2,3,4…10 IIS compression levels
3 point consideration for IIS compression optimization
Static data compression
Dynamic data compression
Compressed file and compression
CPU usage, dynamic compression and load testing
TTFB and Compression levels
IIS 7.0 and CPU roll off
Thanks, Thanks and Thanks
Conclusion
Some known issues on IIS compression
Links for further reading

How does IIS compression work?

Note :- All examples shown in this article is using IIS 6.0. The only reason we have used IIS 6 is because 7.0 is still not that common.

Before we move ahead and talk about how IIS compression works, let’s try to
understand how normally IIS will work. Let’s say the user requests for a
‘Home.html’ page which is 100 KB size. IIS serves this request by passing the
100 KB HTML page over the wire to the end user browser.

When compression is enabled on IIS the sequence of events changes as follows:-

• User requests for a page from the IIS server. While requesting for page the
browser also sends what kind of compression types it supports. Below is a simple
request sent to the browser which says its supports ‘gzip’ and ‘deflate’. We had
used fiddler (http://www.fiddler2.com/fiddler2/version.asp ) to get the request
data.

• Depending on the compression type support sent by the browser IIS compresses data and sends the same over the wire to the end browser.

• Browser then decompresses the data and displays the same on the browser.


Compression fundamentals: - Gzip and deflate

IIS supports to kind of compressions Gzip and deflate. Both are more or less same where Gzip is an extension over deflate. Deflate is a compression algorithm which combines LZ77 and Huffman coding. In case you are interested to read more about LZ77 and Huffman you can read at
http://www.zlib.net/feldspar.html .

Gzip is based on deflate algorithm with extra headers added to the deflate payload.

Below are the headers details which is added to the deflate payload data. It starts with a 10 byte header which has version number and time stamp followed by optional headers for file name. At the end it has the actual deflate compressed payload and 8 byte check sum to ensure data is
not lost in transmission.

Google, Yahoo and Amazon use gzip, so we can safely assume that it’s supported by most of the browsers.

Enabling IIS compression

Till now we have done enough of theory to understand IIS compression. Let’s get our hands dirty to see how we can actually enable IIS compression.

Step 1:- Enable compression

The first step is to enable compression on IIS. So right click on websites  properties and click on the service tab. To enable compression we need to check the below two text boxes from the service tab of IIS website properties. Below figure shows the location of both the checkboxes.

Step 2:- Enable metabase.xml edit

Metadata for IIS comes from ‘Metabase.xml’ which is located at “%windir%\system32\inetsrv\”. In order that compression works properly we need to make some changes to this XML file. In order to make changes to this XML file we need to direct IIS to gives us edit rights. So right click on your IIS server root  go to properties and check ‘enable direct metabase edit’ check box as
shown in the below figure.

Step 3:- Set the compression level and extension types

Next step is to set the compression levels and extension types. Compression level can be defined between 0 to 10, where 0 specifies a mild compression and 10 specifies the highest level of compression. This value is specified using ‘HcDynamicCompressionLevel’ property. There are two types of compression algorithms ‘deflate’ and ‘gzip’. This property needs to be specified in both the
algorithm as shown in the below figures.

We need to also specify which file types need to be compressed. ‘HcScriptFileExtensions’ help us to specify the same. For the current scenario we specified that we need to compress ASPX outputs before they are sent to the end browser.


Step 4:- Does it really work?

Once you are done with the above 4 steps, it’s time to see if the compression
really works. So we will create a simple C# asp.net page which will loop “10000”
times and send some kind of output to the browser.

protected void Page_Load(object sender, EventArgs e)
{
for (int i; i < 10000; i++)
{

Response.Write("Sending huge data" + "<br>");
}
}

In order to see the difference before compression and after compression we will run the fiddler tool as we run our
ASP.NET loop page. You can download fiddler from http://www.fiddler2.com/fiddler2/version.asp .
Below screen shows data captured by fiddler without compression and with compression. Without compression data is “80501 bytes” and with compression it comes to “629 bytes”. I am sure that’s a great performance increase from bandwidth point of view.

0, 1,2,3,4…10 IIS compression levels

In our previous section we have set ‘HcDynamicCompressionLevel’ to value ‘4’. More the compression level value, more the data size will be less. As we increase the compression level the downside is more CPU utilization. One of the big challenges is to figure out what is the optimum compression level. This depends on lot of things type of data, load etc.
In our further coming section we will try to derive which is the best compression level for different scenarios.

3 point consideration for IIS compression optimization

Many developers just enable IIS compression with below default values. But default values do not hold good for every environment. It depends on many other factors like what content type is your site serving. If you site has only static HTML pages then compression levels
will be different as compared to site who are serving mostly dynamic pages.

The above table is taken from

http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/25d2170b-09c0-45fd-8da4-898cf9a7d568.mspx?mfr=true

If your site is only serving compressed data like ‘JPEG’ and ‘PDF’, it’s probably not advisable to enable compression at all as your CPU utilization increases considerably for small compression gains. On the other side we also need to balance compression with CPU utilization. The more we increase the
compression levels the more CPU resources will be utilized.

Different data types needs to be set to different IIS compression levels for optimization. In the further coming section we will take different data types, analyze the same with different compression levels and see how CPU utilization is affected. Below figure shows different data types with some examples of file types.

Static data compression

Let’s start with the easiest one static content type like HTML and HTM. If a user requests for static page from IIS who has compression enabled, IIS compresses the file and puts the same in ‘%windir%\IIS Temporary Compressed Files’ directory.
Below is a simple screen which shows the compressed folder snapshot. Compression
happens only for the first time. On subsequent calls for the same compressed
data is picked from the compressed files directory.

Below are some sample readings we have taken for HTML files of size range from 100 KB to 2048 KB. We have set the compression level to ‘0’.


You can easily see with the least compression level enabled the compression is almost 5 times.


As the compression happens only for the first time, we can happily set the compression level to ‘10’. The first time we will see a huge CPU utilization but on subsequent calls the CPU usage will be small
as compared to the compression gains.

Dynamic data compression

Dynamic data compression is bit different than static compression. Dynamic compression happens every time a page is requested. We need to balance between CPU utilization and compression levels.
In order find the optimized compression level, we did a small experiment as shown below. We took 5 files in a range of 100 KB to 2 MB. We then changed compression levels from 0 to 10 for every file size to check how much was the data was compressed. Below are compressed data readings in Bytes.

The above readings do not show anything specific, its bit messy. So what we did is plotted the below graph using the above data and we hit the sweet spot. You can see even after increasing the compression level from 4 to 10 the compressed size has no effect. We experimented this on 2 to 3 different environments and it always hit the value ‘4’ , the sweet spot.

So the conclusion we draw from this is, setting value ‘4’ compression level for dynamic data pages will be an optimized setting.

Compressed file and compression

Compressed files are file which are already compressed. For example files like JPEG and PDF are already compressed. So we did a small test by taking JPEG compressed files and below are our readings. The compressed files after applying IIS compression did not change much in size.

When we plot a graph you see that the compression benefits are very small. We may end up utilizing more CPU processor resource and gain nothing in terms of compression.


So the conclusion we can draw for compressed files is that we can disable compression for already compressed file types like JPEG and PDF.
CPU usage, dynamic compression and load testing
One of the important points to remember for dynamic data is to optimize between CPU utilization, compression levels and load on the server.


We used WCAT to do stress with 100 concurrent users. For every file size range from 100 KB to 2 MB we recorded CPU utilization for every compression level. We recorded processor time for W3WP exe using performance counter. To add this performance counter you can go to process à select processor time à select w3wp.exe from instances.


If we plot a graph using the above data we hit the sweet spot of 6. Till the IIS compression was 6 CPU utilization was not really affected.

TTFB and Compression levels

TTFB also termed as time to first byte gets the number of milliseconds that have passed before the first byte of the response was received. We also performed a small experiment on 1MB and 2 MB dynamic pages with different compression levels. We then measured the TTFB for every
combination of compression levels and file size. WCAT was used to measure TTFB.

When we plot the above data we get value ‘5’ as a sweet spot. Until the value reaches ‘5’ TTFB remain constant.

Print shot of WCAT output for TTFB measurement.

IIS 7.0 and CPU roll off

All the above experiments and conclusion are done on IIS 6.0. IIS 7.0 has a very important property i.e. CPU roll-off. CPU roll-off acts like cut off gateway so that CPU resources are not consumed
unlimitedly.

When CPU gets beyond a certain level, IIS will stop compressing pages, and when it drops below a different level, it will start up again. This is controlled using ‘staticCompressionEnableCpuUsage’ and ‘dynamicCompressionDisableCpuUsage’ attributes. It’s like a safety valve so that your CPU usage does not come by surprise.

Thanks, Thanks and Thanks

Every bit of inspiration for this article has come from Scott Forsyth's article on IIS compression. You can say I have just created a new version with more details.
http://weblogs.asp.net/owscott/archive/2009/02/22/iis-7-compression-good-bad-how-much.aspx

Thanks to Jimmie to suggest the performance counters and other details of IIS compression.
http://blogs.msdn.com/jimmiet/archive/2009/06/07/iis-6-0-compression.aspx

I also did picked up some bits from this link of Microsoft
http://technet.microsoft.com/hi-in/library/bb742379(en-us).aspx .

Conclusion

• If the files are already compressed do not enable compression on those files.
We can safely disable compression on EXE , JPEG , PDF etc.

• For static pages compression level can be set to 10 as the compression happens
only once.

• Compression level range can be from ‘4’ to ‘6’ for dynamic pages depending on
the server environment and configuration.The best way to judge which compression
level suits best is to perform TTFB, CPU utilization and compression test as
explained in this article.

In case you want to do a sanity check please refer this article
http://weblogs.asp.net/owscott/archive/2009/02/22/iis-7-compression-good-bad-how-much.aspx
, i agree my results do not match exactly with scott but I think we are very much on the same page.

Some known issues on IIS compression

Below are some known issues of IIS compression
http://support.microsoft.com/kb/837251
http://support.microsoft.com/kb/823386

Links for further reading

http://www.zlib.net/feldspar.html
http://www.15seconds.com/Issue/020314.htm
http://blogs.msdn.com/jimmiet/archive/2009/06/07/iis-6-0-compression.aspx
http://www.fiddler2.com/fiddler2/version.asp
http://technet.microsoft.com/hi-in/library/bb742379(en-us).aspx
http://weblogs.asp.net/owscott/archive/2009/02/22/iis-7-compression-good-bad-how-much.aspx
http://www.codinghorror.com/blog/archives/000059.html
http://blogs.microsoft.co.il/blogs/yevgenif/archive/2009/02/08/web-performance-enable-data-compression-on-iis-6-0-server.aspx

.NET Best Practice No: 3:- Using performance counters to gather performance data

Is this Article worth reading ahead?

This article discusses how we can use performance counter to gather data from an application. So we will first understand the fundamentals and then we will see a simple example from which we
will collect some performance data.

Introduction: - My application performance is the best like a rocket 

Let us start this article by a small chat between customer and developer.

Scenario 1

Customer: - How’s your application performance?
Subjective developer: - Well it’s speedy, it’s the best …huuh aaa ooh it’s a like rocket.
Scenario 2
Customer: - How’s your application performance?
Quantitative developer: - With 2 GB RAM , xyz processor and 20000 customer records the customer screen load in 20 secs.I am sure the second developer looks more promising than the first developer. In this article we will explore how we can use performance counters to measure performance of an application. So let’s start counting 1,2,3,4….

Please feel free to download my free 500 question and answer videos which covers Design Pattern, UML, Function Points, Enterprise Application Blocks,OOP'S, SDLC, .NET, ASP.NET, SQL Server, WCF, WPF, WWF, SharePoint, LINQ, SilverLight, .NET Best Practices @ these videos http://www.questpond.com/


Courtesy :- http://scoutbase.org.uk

Thanks Javier and Michael

I really do not have the intellectual to write something on performance counters. But reading the below articles I was able to manage something. So first let me thank these guys and then we can move ahead in the article.
Thanks a bunch Javier Canillas for creating the performance counter helper , it really eases of lot of code http://perfmoncounterhelper.codeplex.com/ Thanks Michael Groeger for the wonderful article, I took the code of counter creation from your article http://www.codeproject.com/KB/dotnet/perfcounter.aspx
I also picked up lot of pieces from
http://msdn.microsoft.com/en-us/library/system.diagnostics.performancecounter.aspx

At the end of the day its count, calculate and display

Any performance evaluation works on count, calculate and display. For instance if you want to count how many pages in memory where processed per second we first need to count number of pages and also how many seconds where elapsed. Once we are finished with counting we then need to calculate i.e. divide the number of pages by seconds elapsed. Finally we need to display the data of our performance.

Now that we know it’s a 3 step process i.e. count, calculate and display. The counting part is done by the application. So the application needs to feed in the data during the counting phase. Please note the data is not automatically detected by the performance counters , some help needs to be provided by the application. The calculation and display is done by the performance counter and monitor.

Performance counters are not magicians

If application does not provide counter data performance counters cannot measure by himself. Performance counter cannot measure applications which do not feed performance data. In other words the application needs to feed in counter data by creating performance counter objects.

Types of measures in application

Almost all application performance measurements fall in to one of the below 6 categories.

Instantaneous values: - Many times we just want to measure the most recent value. For instance we just want to measure how many customer records where processed? , how much RAM memory has been used etc. These types of measures are termed as instantaneous or absolute values. Performance counter supports these measurement types by using instantaneous counters.

Average values: - Sometimes instant / recent values do not really show the real picture. For instance just saying that application consumed 1 GB space is not enough. But if we can get some kind of average data consumption like 10 MB data was consumed per 1000 records probably you can get more insight of what is happening inside the application. Performance counter supports these kinds of measurement types by using average performanance counters like AverageBase, AverageTimer32, AverageCount64 etc.

Rate values: - There are situations when you want to know the rate of events with respect to time. For example you would like to how many records where processed per second. Rate counters help us to calculate these kinds of performance metrics.

Percentage values: - Many times we would like to see values as percentages for comparison purposes. For example you want to compare performance data between 2 computers. Comparing direct values will not be a fair comparison. So if we can have % values from both computers then the comparison can make more sense. If we want to compare values between different performance counters, percentage is much better option rather than using absolute values. For example if you want to compare how much RAM is utilized as compared to hard disk space. Comparing 1 GB ram usage with 50 GB hard disk usage is like comparing apples with oranges. If you can express these values as percentages then comparison will be fair and justifiable. Percentage performance counters
can help us to express absolute values as percentages.

Difference values: - Many times we would like to get difference performance data , for instance how much time was elapsed from the time application started, how much hard disk consumption was done by the application from the time it started etc. In order to collect these kinds of performance
data we need to record the original value and the recent value. To get final performance data we need to subtract the original value from the recent value. Performance counter provides difference counters to calculate such kind of performance data. So summarizing there are 5 types of performance counters which can satisfy all the above counting needs. Below figure shows the same in a pictorial format.


Example on which performance counter will be tested

In this complete article we will be considering a simple counter example as explained below. In this example we will have a timer which generates random number every 100 milliseconds. These random numbers are later checked to see if it’s less than 2. Incase its less than 2 then function ‘MyFunction’ is invoked.

Below is the code where the timer runs every 100 milliseconds and calculates random number. If the random number is smaller than 2 we invoke the function ‘MyFunction’.

private void timer1_Tick(object sender, EventArgs e)
{
// Generate random number between 1 to 5.
Random objRnd = new Random();
int y = objRnd.Next(1, 5);

// If random number is less than 2 call my Function
if (y > 2)
{
MyFunction();
}
}

Below is the code for ‘MyFunction’ which is invoked when the value of random number is less than 2. The method does not do
anything as such.

private void MyFunction()
{

}

All our performance counters example in this article will use the above defined sample.

Adding our first instantaneous performance counter in 4 steps

Before we go in to in depth of how to add performance counters, let’s first understand the structure of performance counters. When we create performance counters it needs to belong to some group.
So we need to create a category and all performance counters will lie under that category.

We will like to just count how many times ‘MyFunction’ was called. So let’s create an instantaneous counter called as 'NumberOfTimeFunctionCalled'. Before we move ahead let’s see how many different types of instantaneous counters are provided by performance counters:-

Below definitions are taken from http://msdn.microsoft.com/enus/library/system.diagnostics.performancecountertype.aspx.

NumberOfItems32:- An instantaneous counter that shows the most recently observed value.

NumberOfItems64:- An instantaneous counter that shows the most recently observed value. Used, for example, to maintain a simple count of a very large number of items or operations. It is the same as NumberOfItems32 except that it uses larger fields to accommodate larger values.

NumberOfItemsHEX32:- An instantaneous counter that shows the most recently observed value in hexadecimal format. Used, for example, to maintain a simple count of items or operations.

NumberOfItemsHEX64:- An instantaneous counter that shows the most recently observed value. Used, for example, to maintain a simple count of a very large number of items or operations. It is the same as NumberOfItemsHEX32 except that it uses larger fields to accommodate larger values.

Step 1 Create the counter: - For our current scenario ‘NumberOfItems32’ will suffice. So let’s first create ‘NumberOfItems32’ instantaneous counter. There are two ways to create counters one is through the code and the other is using the server explorer of VS 2008. The code approach we will see later. For the time we will use server explorer to create our counter. So open your visual
studio  click on view  server explorer and you should see the performance counters section as shown in the below figure. Right click on the performance counters section and select create new category.

When we create a new category you can specify the name of the category and add counters in to this category. For the current example we have given category name as ‘MyApplication’ and added a counter type of ‘NumberOfItem32’ with name ‘NumberOfTimeFunctionCalled’.

Step 2 Add the counter to your visual studio
application: - Once you have added the counter on the server explorer, you can drag and drop the counter on the ASPX page as shown below.

You need to mark ‘ReadOnly’ value as false so that you can modify the counter
value from the code.

Step 3 Add the code to count the counter: -
Finally we need to increment the counter. We have first cleared any old values in the counter during the form load. Please note that counter values are stored globally so they do not do reset by themselves we need to do it explicitly. So in the form load we have cleared the raw value to zero.

private void Form1_Load(object sender, EventArgs e)
{
perfNumberOfTimeFunctionCalled.RawValue = 0;
}

Whenever the function is called we are incrementing the value by using ‘Increment’ method. Every call to the increment
function increases the number by 1.

private void MyFunction()
{
perfNumberOfTimeFunctionCalled.Increment();
}

Step 4 View the counter data: - Now that we have specified the counter in the application which increments every time
‘MyFunction’ function is called. It’s time to use performance monitor to display the performance counter. So go to start  run and type ‘perfmon’. You will see there are lots of by default performance counters. For clarity sake we will remove all the counters for now and add our performance counter i.e. ‘NumberofTimeFunctionCalled’.

You can now view the graphical display as shown in the below figure. Ensure that your application is running because application emits data which is then interpreted by the performance monitor.

Above view is a graphical view of the same. To view the same in textual format you use the view report tab provided by performance monitor. You can see the report shows that ‘MyFunction’ was called 9696 times from the time application started.

Creating more sensible counter

In the previous section we have measured how many times ‘MyFunction’ was called. But this performance count does not really show any kind of measure. It would be great if we can also see the count of how many times the timer was called. Then later we can compare between the numbers of time timer was called and ‘MyFunction’ was called.So create an instantaneous counter and increment this counter when the timer fires as shown in the below code.

private void timer1_Tick(object sender, EventArgs e)
{
perfNumberOfTimeTimerCalled.Increment();
Random objRnd = new Random();
int y = objRnd.Next(1, 5);
if (y > 2)
{
MyFunction();
}
}

You can see both the counters in t he below graph the blue line showing the number of times ‘MyFunction’ was called and the
black one showing number of times timer called.

If we look in to the report view we can see for how many times the timer fired and how many times was ‘MyFunction’ called.

Average performance counters

In the previous section we had counted two counters one which says how many times did the timer fire and the other says how many times ‘MyFunction’ was called. If we can have some kind of average data which says how many times was ‘MyFunctionCalled’ for the number of times timer
called it will be make more sense.In order to get these kinds of metrics Average performance counters can be used. So for our scenario we need to count the number of time function was called and number of time the timer fired. Then we need to divide them to find on a average how many times was the function for the timer fired.

We need to add two counters one for the numerator and the other for the denominator. For the numerator counter we need to add ‘AverageCount64’ type of counter while for the denominator we need to add ‘AverageBase’ type of counter.

You need to add the ‘AverageBase’ counter after the ‘AverageCount64’ type counter or else you will get an error as shown below.

For every timer tick we increment the number of time timer called counter.

private void timer1_Tick(object sender, EventArgs e)
{
perfAvgNumberofTimeTimerCalled.Increment();
Random objRnd = new Random();
int y = objRnd.Next(1, 5);
if (y > 2)
{
MyFunction();
}
}

For every function call we increment the number of time function called counter.

private void MyFunction()
{

perfNumberOfTimeFunctionCalled.Increment();

}

If you run the application in the view report mode you should see something as shown below. You can see on a average
‘MyFunction’ is called 0.5 times.

If you do the calculation you will get the same figure which is been calculated by the performance monitor.

Rate performance counters

From our sample we would now like to find out the rate of ‘MyFunction’ calls with respect to time. So we would like know how many calls are made every second. So browse to the server explorer and add ‘rateofCountsPerSecond32’ counter as shown in the below figure. Increase this counter every time ‘MyFunction’ is called.

If you run the application you should be able to see the ‘RateofMyFunctionCalledPerSecond’ value. Below is a simple report which shows the rate of counter data which was ran for 15 seconds. The total call made in this 15 second was 72. So the average call is 5 ‘MyFunction’ calls per second.

Performance counters left

We have left percentage counters and difference counters as they are pretty simple and straightforward. In order to maintain this article to the point and specific I have excused both these counter types.

Adding counters by C# code

Till now we have added the performance counter using server explorer. You can also add the counter by code. The first thing is we need to import System.Diagnostics namespace.We then need to create object of ‘CounterCreationDataCollection’ object.

CounterCreationDataCollection Mycounters = new CounterCreationDataCollection();

Create our actual counter and specify the counter type.

CounterCreationData totalOps = new CounterCreationData();
totalOps.CounterName = "Numberofoperations";
totalOps.CounterHelp = "Total number of operations executed";
totalOps.CounterType = PerformanceCounterType.NumberOfItems32;
Mycounters.Add(totalOps);

Finally create the counter inside a category. Below code snippet is creating the counter in ‘MyCategory’ category.

PerformanceCounterCategory.Create("MyCategory","Sample category for Codeproject", Mycounters);

Let’s ease some pain using Performance counter helper

Its quiet a pain to write the counter creation code. You can use performance counter helper to ease and make your code smaller. You can find the performance counter helper at
http://perfmoncounterhelper.codeplex.com/ ,

Do not use it in production

Oh yes, use it only when you are doing development. If you are using in production ensure that there is an enabling and disabling mechanism or else it will affect your application performance.

Conclusion

• Use performance counters to measure application data.
• Performance counters comes in various categories like instantaneous, average , rate etc.
• Performance counters should not be used in production. In case it’s used should have a disabling mechanism.
• Performance counter cannot measure by itself application needs to provide data so that performance monitors can calculate and display the data.

Source code

You can find the sample source code for the above performance counters discussed at you can download the source code from here

.NET Best Practice No: 2:- Improve garbage collector performance using finalize/dispose pattern

Introduction and Goal

Ask any developer which is the best place in a .NET class to clean unmanaged resources?, 70% of them will say the destructor. Although it looks the most promising place for cleanup it has a huge impact on performance and memory consumption. Writing clean up code in the destructor
leads to double GC visits and thus affecting the performance multifold times.

In order to validate the same we will first start with bit of theory and then we will actually see how GC algorithm performance is impacted using destructor. So we will first understand the concept of generations and then we will see the finalize dispose pattern.
I am sure this article will change your thought process regarding destructor, dispose and finalize.

Please feel free to download my free 500 question and answer videos which covers Design Pattern, UML, Function Points, Enterprise Application Blocks,OOP'S, SDLC, .NET, ASP.NET, SQL Server, WCF, WPF, WWF, SharePoint, LINQ, SilverLight, .NET Best Practices @ these videos http://www.questpond.com/

.NET Best Practice No: 2:- Improve garbage collector performance using finalize/dispose pattern

Is this Article worth reading ahead?

With this article you will understand how performance of GC algorithm can be improved using finalize dispose pattern. Below figure shows the comparison of what we will be achieving after this article. This article uses CLR profiler to profile how GC works. In case you are new to CLR profiler please do read before you go ahead with this one.

Assumptions

Thanks Mr. Jeffrey Richter and Peter Sollich Let’s start this article by first thanking Mr. Jeffery Richter for explaining in depth how garbage collection algorithm works. He has written two legendary articles about the way garbage collector work. I actually wanted to point to the MSDN magazine article written by Jeffery Richter but for some reason it’s not showing up in MSDN. So I am pointing to a different unofficial location, you can download both the articles from http://www.cs.inf.ethz.ch/ssw/files/GC_in_NET.pdf in a PDF format.

Also thanks to Mr. Peter Sollich who is the CLR Performance Architect to write such a detail help on CLR profiler. When you install the CLR profiler please do not forget to read the detail help document written by Peter Sollich. In this article we will use the CLR profiler to check how the garbage collector performance is affected using finalize.

Thanks a lot to you guys. There was no way I would have completed this article without reading articles written by you. Any time you guys pass by article please do comment on the same would love to hear from you guys.

Garbage collector – The unsung Hero

As said in the introduction writing clean up code in constructor leads to double GC visits. Many developers would like to just shrug off and say ‘Should we really worry about GC and what it does behind scenes?”. Yes, actually we should not worry about GC if you write your code properly. GC has the best algorithm to ensure that your application is notimpacted. But many times the way you have written your code and assigned/cleanedmemory resources in your code affects GC algorithm a lot. Sometimes this impactleads to bad performance of GC and hence bad performance for your application. So let’s first understand what different tasks are performed by the garbage collector to allocate and clean up memory in an application. Let’s say we have 3 classes where in class ‘A’ uses class ‘B’ and class ‘B’ uses class ‘C’.

When the first time your application starts predefined memory is allocated to the application. When the application creates these 3 objects they are assigned in the memory stack with a memory address. You can see in the below figure how the memory looks before the object creation and how it looks after object creation. In case there was an object D created it will be allocated from the address where Object C ends.

Internally GC maintains an object graph to know which objects are reachable. All objects belong to the main application root object. The root object also maintains which object is allocated on which memory address. In case an object is using other objects then that object also holds amemory address of the used object. For example in our case object A uses Object B. So object A stores the memory address of Object B.

Now let’s say Object ‘A’ is removed from memory. So the Object ‘A’ memory is assigned to Object ‘B’ and Object ‘B’ memory is assigned to object ‘C’. So the memory allocation internally looks something as shown below.

As the address pointers are updated GC also needs to ensure that his internal graph is updated with new memory addresses. So the object graph becomes something as shown below. Now that’s a bit of work for GC it needs to ensure that the object is removed from graph and the new
addresses for the existing objects is updated throughout the object tree.

An application other than his own custom objects also has .NET objects which also form the part of the graph. The addresses to those objects also need to be updated. The number of objects of
.NET runtime is very high. For instance below is the number of objects created for a simple console based hello world application. The numbers of objects are approximately in 1000’s. Updating pointers for each of these objects is a huge task.

Generation algorithm – Today, yesterday and day before yesterday

GC uses the concept of generations to improve performance. Concept of generation is based on the way human psychology handles tasks. Below are some points related to how tasks are handled by humans and how garbage collector algorithm works on the same lines:-
• If you decide some task today there is a high possibility of completion of
those tasks.
• If some task is pending from yesterday then probably that task has gained a
low priority and it can be delayed further.
• If some task is pending from day before yesterday then there is a huge
probability that the task can be pending forever.

GC thinks in the same lines and has the below assumptions:-

• If the object is new then the life time of the object can be short.
• If an object is old then it can have a long life time.

So said and done GC supports three generations (Generation 0, Generation 1 and Generation 2).

Generation 0 has all the newly created objects. When the application creates objects they first come and fall in the Generation 0 bucket. A time comes when Generation 0 fills up so GC needs to run to free memory resources. So GC starts building the graph and eliminating any objects which are not used in application any more. In case GC is not able to eliminate an object from generation 0 it promotes it to generation 1. If in the coming iterations it’s not able to remove objects from generation 1 it’s promoted to generation 2. The maximum generation supported by .NET runtime is 2.

Below is a sample display of how generation objects are seen when you run CLR profiler. In case you are new to CLR profiler you can catch up the basics from Best Practices - Part 1

Ok, so how does generation help in optimization

As the objects are now contained in generations, GC can make a choice which generation objects he wants to clean. If you remember in the previous section we talked about the assumptions made by GC regarding object ages. GC assumes that all new objects have shorter life time. So in other words GC will mainly go through generation 0 objects more rather than going through all objects in all generations. If clean up from generation 0 does not provide enough memory it will then move
towards cleaning from generation 1 and so on. This algorithm improves GC performance to a huge extent.

Conclusion about generations

• Huge number of object in Gen 1 and 2 means
memory utilization is not optimized.
• Larger the Gen 1 and Gen 2 regions GC algorithm will perform more worst.

Using finalize/destructor leads to more objects in Gen 1 and Gen 2

The C# compiler translates (renames) the destructor into Finalize. If you see the IL code using IDASM you can see that the destructor is renamed to finalize. So let’s try to understand why
implementing destructor leads to more objects in gen 1 and gen 2 regions. Here’s how the process actually works:-
• When new objects are created they are moved to gen 0.
• When gen 0 fills out GC runs and tries to clear memory.
• If the objects do not have a destructor then it just cleans them up if they are not used.
• If the object has a finalize method it moves those objects to the finalization queue.
• If the objects are reachable it’s moved to the ‘Freachable’ queue. If the objects are unreachable the memory is reclaimed.
• GC work is finished for this iteration.
• Next time when GC again starts its goes to Freachable queue to check if the objects are not reachable. If the objects are not reachable from Freachable memory is claimed back.

In other words objects which have destructor can stay more time in memory.
Let’s try to see the same practically. Below is a simple class which has destructor.

class clsMyClass

{

public clsMyClass()

{

}

~clsMyClass()

{

}

}

We will create 100 * 10000 objects and monitor the same using CLR profiler.

for (int i = 0; i < 100 * 10000; i++)

{

clsMyClass obj = new clsMyClass();

}

If you see the CLR profiler memory by address report you will see lot of objects in gen 1.

Now let’s remove the destructor and do the same exercise.

class clsMyClass

{

public clsMyClass()

{

}

}

You can see the gen 0 has increased considerably while gen 1 and 2 are less in number.

If we see a one to one comparison it’s something as shown in the below figure.


Get rid of the destructor by using Dispose

We can get rid of the destructor by implementing our clean up code in the dispose method For that we need to implement the ‘IDisposable’ method , write our clean up code in this and call suppress finalize method as shown in the below code snippet. ‘SuppressFinalize’
dictates the GC to not call the finalize method. So the double GC call does not happen

class clsMyClass : IDisposable

{

public clsMyClass()

{

}

~clsMyClass()

{

}



public void Dispose()

{

GC.SuppressFinalize(this);

}

}

The client now needs to ensure that it calls the dispose method as shown below.

for (int i = 0; i < 100 ; i++)

{

clsMyClass obj = new clsMyClass();

obj.Dispose();

}

Below is the comparison of how Gen 0 and 1 distribution looks with constructor and with dispose. You can see there is
marked improvement in gen 0 allocation which signifies good memory allocation.


What if developers forget to call Dispose?

It’s not a perfect world. We cannot ensure that the dispose method is always called from the client. So that’s where we can use Finalize / Dispose pattern as explained in the coming section.

There is a detailed implementation of this pattern at
http://msdn.microsoft.com/en-us/library/b1yfkh5e(VS.71).aspx.

Below is how the implementation of finalize / dispose pattern looks like.

class clsMyClass : IDisposable

{

public clsMyClass()

{



}



~clsMyClass()

{

// In case the client forgets to call

// Dispose , destructor will be invoked for

Dispose(false);

}

protected virtual void Dispose(bool disposing)

{

if (disposing)

{

// Free managed objects.

}

// Free unmanaged objects



}



public void Dispose()

{

Dispose(true);

// Ensure that the destructor is not called

GC.SuppressFinalize(this);

}

}

Explanation of the code:-
• We have defined a method called as Dispose which takes a Boolean flag. This flag says is this method called from Dispose or from the destructor. If this is called from the ‘Dispose’ method then we can free both managed and unmanaged resources.
• If this method is called from the destructor then we will just free the unmanaged resources.
• In the dispose method we have suppressed the finalize and called the dispose with true.
• In the destructor we have called the dispose function with false value. In other words we assume that the GC will take care of managed resources and let’s take the destructor call to clean unmanaged resources.
In other words if the client does not call the dispose function the destructor will take care of cleaning the unmanaged resources.

Conclusion

• Do not have empty constructors in your classes.
• In case you need to clean up use finalize dispose pattern with ‘SupressFinalize’ method called.
• If there is a dispose method exposed by a class , ensure to call the same from your client code.
• Application should have more objects allocated in Gen 0 than Gen 1 and Gen 2. More objects in Gen 1 and 2 is sign of bad GC algorithm execution.

Source code

You can find the sample source code for the dispose pattern at from here.

.NET Best Practice No: 1:- Detecting High Memory consuming functions in .NET code

Introduction and Goal

One of the important factors for performance degradation in .NET code is memory consumption. Many developers just concentrate on execution time to determine performance bottle necks in a .NET application. Only measuring execution time does not clearly give idea of where the performance issue resides. Ok, said and done one of the biggest task is to understand which function, assembly or class has consumed how much memory. In this tutorial we will see how we can find which functions consume how much memory. This article discusses the best practices involved using CLR profiler
for studying memory allocation.

Please feel free to download my free 500 question and answer videos which covers Design Pattern, UML, Function Points, Enterprise Application Blocks,OOP'S, SDLC, .NET, ASP.NET, SQL Server, WCF, WPF, WWF, SharePoint, LINQ, SilverLight, .NET Best Practices @ these videos http://www.questpond.com/
Thanks a lot Mr. Peter Sollich

Let’s start this article by first thanking Peter Sollich who is the CLR Performance Architect to write such a detail help on CLR profiler. When you install CLR profiler do not forget to read the detail
help document written by Peter Sollich.Thanks a lot sir, if ever you visit my article let me know your inputs.

CLR Profiler to rescue

CLR profiler is a tool which helps to detect how memory allocation happens in .NET code. CLR profiler tool is provided by Microsoft, you can download the same from

http://www.microsoft.com/downloads/details.aspx?familyid=A362781C-3870-43BE-8926-862B40AA0CD0&displaylang=en
.
Note :- There are two version of CLR profiler one for .NET 1.1 and the other 2.0 framework. For 2.0 CLR profiler you can visit
http://www.microsoft.com/downloads/details.aspx?familyid=A362781C-3870-43BE-8926-862B40AA0CD0&displaylang=en
and to download 1.1 you can use
http://www.microsoft.com/downloads/details.aspx?familyid=86ce6052-d7f4-4aeb-9b7a-94635beebdda&displaylang=en#Overview
Once you download CLR profiler you can unzip and run ‘CLRProfiler.exe’ from the
bin folder.If you have downloaded 2.0 CLR profiler it provides executables for ‘X86’ and ‘X64’ environment. So please ensure to run the appropriate versions.

Features of CLR profiler

CLR profiler is the best tool when it comes to understanding how memory allocations are done in .NET application code. It does two prime importance functions:-
• Gives a complete report of how memory is allocated in a .NET application. So
you can see full report how memory is allocated as per data types, functions,
methods etc
• It also provides how much time the method was called.



Do not user CLR on production and as a starting tool for performance evaluation

CLR is an intrusive tool. In other words it runs its own logic of dumping memory values for every function / class / modules inside the application. In other words it interferes with the application logic. Let’ say you have a normal application which calls function 1 and function 2. When you profile your application with CLR profiler it injects dumping of memory heap data after every function call as shown below.

In other words do not use CLR profiler to find execution time of your application. It actually slows down your application 10 to 100 times. You will end up with wrong results.As said because it’s an intrusion tool you should never use the same in production environment.First thing you should never first start analyzing your performance issues by CLR profiler tool. It’s more of a second step activity where you have zeroed on a function or a class which is having memory issues. So probably you can use performance counters to first find which methods and functions take long execution time and then use CLR profiler to see how the memory allocations are done.

How can we run CLR profiler?

One you have downloaded the CLR profiler from Microsoft site, unzip the files in a folder. Go to the unzipped folder in Binaries  choose your processor and run ‘CLRProfiler.exe’. You will be shown
the CLR profiler as shown in the below figure. The first step we need to decide what do we want to profile. There are two things one is the memory allocation and the other is number of calls to a
method. So select what data you want to profile and then click start application.


Once you are done you can see the complete summary of profiling as shown below. It’s a very complicated report we will see a simplistic approach when we profile the sample application.

Issues faced by CLR profiler

Some issues we faced while running CLR
profiler. If you are getting the below screen and it does not stop, there can be
two reasons:-
• You have .NET 2.0 and you are running CLR profiler 1.1.
• You have not registered ProfilerOBJ.dll in GAC.


The sample application we will profile

The application which we will profile is pretty simple. It has a simple button which calls two functions ‘UseSimpleStrings’ and ‘UseStringBuilders’. Both these functions concatenate strings. One uses ‘+’ for concatenations while the other uses the ‘StringBuilder’ class. We will be 1000 times concatenating the strings.

private void UsingSimpleStrings()

{

string strSimpleStrings="";

for (int i = 0; i < 1000; i++)

{

strSimpleStrings = strSimpleStrings + "Test";

}}

The function which uses ‘StringBuilder’ class to do concatenation.

private void UsingStringBuilders()

{



StringBuilder strBuilder = new StringBuilder();

for (int i = 0; i < 1000; i++)

{

strBuilder.Append("Test");

}

}

Both these functions are called through a button click.

private void btnDoProfiling_Click(object sender, EventArgs e)

{

UsingSimpleStrings();

UsingStringBuilders();



}

Using CLR profiler to profile our sample

Now that we know our application we will try to use profiler to see which function uses how much memory. So click on start application  browse to the application exe and click on check memory allocation button and close the application. You will be popped up with a complete summary dialog box. If you click on histogram button you can see memory allocations as per data type. I understand it’s very confusing. So leave this currently.

If you are interested to see how much memory is allocated as per functions you can click on ‘Allocation Graph’. This gives as per every function how much memory is consumed. Even this report is very confusing because so many function, so many methods and we are not able to locate our two functions of string ‘UsingStringBuilders’ and ‘UsingSimpleStrings’.

To simplify the above graph right click and you will be popped with lot of filtering options. Let’s use the ‘Find Routine’ search to filter unnecessary data. We have entered to get the button click event. From this button click those two functions are called.

The search now zooms on the method as shown in the below figure. Now double click on the ‘btnDoProfiling_Click’ box as highlighted in the below figure.

After double click you will see the below details. It’s now better. But where did the second function go off. It’s only showing ‘UseSimpleStrings’ function. This is because the report is coarse. So click on everything and you should see all the functions.

You can now see the other function also. What’s the 26 bytes ?. It’s just an extra string manipulation needed to when the functions are called, so we can excuse it. Let’s concentrate on our two functions ‘UseSimpleStrings’ and ‘UseStringBuilders’ can now be seen. You can see string concatenation takes 3.8 MB while string builder object consumes only 16 KB. So string builder object is consuming less memory than simple string concatenation.

That was a tough way any easy way

The above shown way was really tough. Let’s say you have 1000 of functions and you want to analyze which function consumes what memory. It’s practically just not possible to go through every call graph, do a find and get your functions.The best way is to export a detail report in to excel and then analyze the data. So click on view  call tree.

Once you click on call tree you will be shown something as shown below. Click on view All functions  you will be shown all functions  click on file and save it as CSV.

Once you have exported the complete data to CSV, you can easily locate your methods and functions and see how much data has been allocated.

Simplifying results using comments

In case you know which methods to profile and you can enable the profiler at that moment when the method is called. In other words we can enable the CLR profiler from the application. In order to enable the profiler from the C# code we need to first reference ‘CLRProfilerControl.dll’. You can get this DLL from the same folder where we have the profiler exe.You can then directly call the profiler control from your code as shown in the below code snippet. We have enabled the profiler before we call both the string manipulation functions. Once the function calls are done we have disabled the profiler.

private void btnDoProfiling_Click(object sender, EventArgs e)

{

CLRProfilerControl.LogWriteLine("Entering loop");

CLRProfilerControl.AllocationLoggingActive = true;

CLRProfilerControl.CallLoggingActive = true;



UsingSimpleStrings();

UsingStringBuilders();



CLRProfilerControl.AllocationLoggingActive = false;

CLRProfilerControl.CallLoggingActive = false;

CLRProfilerControl.LogWriteLine("Exiting loop");

CLRProfilerControl.DumpHeap();

}

So now let’s run the CLR profiler and start our
application. Please ensure that you disable the profile active check box because
we are going to enable profiling from the code itself.

Now if you see the histogram you will see limited data. You can see it has only recorded memory allocation consumed from ‘System.String’ and ‘System.Text.StringBuilder’ data type.


If you see the allocation graph you can see it’s pretty neat now. The cluttered view has completely gone off and we have pretty simple and focused view. Please note to click ‘Everything’ to see the
remaining functions.


As said before do not get carried away with execution time

On the summary page you can see the comments button. If you click on the comments button it shows start time and end time. Do not get carried away with the time recorded. As we have previously cleared that it’s an intrusive tool the results are not proper.

Entering loop (1.987 secs)

Exiting loop (2.022 secs)

Conclusion

• CLR profiler can be used to find memory allocated to functions, classes and assemblies to evaluate performance.
• It should not be used on production.
• It should not be used as a starting point for performance evaluation. We can first run perf counter, get the method having high execution time and then use CLR profiler to see the actual cause.
• You can use histogram to see memory allocation and call graph to see method wise memory allocation.
• If you know which methods you want to profile you can enable profiler from the application itself.
Concluding everything in to one basket as a best practice when you want to see memory allocation nothing can beat CLR profiler.

Source code
You can find the sample source code for profiling
from here

DI using Unity Application Blocks

Introduction

In the previous article we discussed about the fundamentals of IOC and DI design patterns.

In the same article we also discussed about how Windsor can be used to solve this problem. In this article we will take up a simple example and try to implement DI using unity application blocks thus resulting in loosely coupled architecture.

Please feel free to download my free 500 question and answer videos which covers Design Pattern, UML, Function Points, Enterprise Application Blocks,
OOP'S, SDLC, .NET, ASP.NET, SQL Server, WCF, WPF, WWF, SharePoint, LINQ, SilverLight, .NET Best Practices @ these videos http://www.questpond.com/

DI using Unity Application Blocks

• The problem
• Creational patterns
• Dependency Injection
• Solving problem using UNITY block
• Download and installing Unity
• Defining the common interface
• Providing the injection gateway
• Defining the config file
• The client code
• The actual internal working
• Enter the real world of loosely coupling
  

The problem

Any good architecture always stands on the base that components are loosely coupled… let me stress the point truly loosely coupled. Let’s try to understand this with a scenario and how we normally approach this problem using pure object oriented way.

Consider a scenario where we have a customer class which needs to perform insert operation on database. The customer class should be customizable to save in either Sql server or oracle database. In future we should also be able to add new databases with out disturbing the customer class.

Below is what we will do. Define two classes one for SQL server and other for Oracle. Both these classes inherit from some kind of same interface ‘Idatabase’. The customer class points to this interface. So the customer class is theoretically shielded from the concrete implementation of SQL Server or oracle. But because the object creational activity needs to be done by the customer class it still needs to be aware of the concrete classes.

Figure: - General loosely coupled thinking

This is what will happen in actual implementation as shown in figure ‘Concrete classes’. The customer class will be creating objects of either oracle or SQL server depending on condition. That means the customer class is exposed to the concrete classes which defeats the purpose of interface. Any change in the database classes will lead to compiling of the customer class.

Figure :- Concrete classes

Creational patterns

The next thing comes in mind is creation patterns. If we can introduce a factory who takes the creational aspect of the concrete classes thus isolating the concrete classes from the customer class.

Here are the issues with factory which makes us force to think about some other solutions:-

• Everything is hardcoded: - The biggest issues with factory are it can not be reused across applications. All the options are hardcoded in the factory itself which makes the factory stringent to particular implementation.
• Interface dependent: - The base on which factories stands are common interfaces. Interfaces decouple the implementation and the object creation procedure. But then all the classes should implement a common interface. This is a limitation by itself again.
• Factories are custom: - They are very much custom to a particular implementation.
• Everything is compile time: - All dependent objects for an object in factory have to be known at compile time.

I will not be explaining factory pattern in case you are not aware of you can read the same in my previous article SoftArchInter1.aspx

Ok, I will show you a magic…Rather than writing those huge line of code for factory…lets go the DI way (Dependency injection).

Dependency Injection

Rather than writing factory pattern, how about injecting the object directly in to the customer class. So let the customer class references the interface and we should be able to inject the concrete type in to the customer class. With this the customer class does not need to use the new keyword and is complete decoupled from the concrete classes.

Figure: - Dependency injection

Injection can be done using containers. So basically you need to put both the classes in to the container and the container will create object and inject the same in to the other class.

Figure: - Containers

With this your customer class does not need to worry about creating objects and knowing the concrete objects.

Solving problem using UNITY block

Now that we know the benefit of containers we will explore unity application block which helps us to achieve dependency injection.

Download and installing Unity

The first step is to download the unity application block from http://msdn.microsoft.com/en-us/library/cc468366.aspx and install the same. Depending on whether you have 2008 or 2005 the documentation will vary. For this tutorial I will be using VS 2005 because majority professionals are still using 2005.

Figure: - unity block installed

Once you install the block you should be able to see the same in Microsoft patterns and practices.

The first thing you need to do is to get the references of at least two components Unity and Unity configuration.

Figure: - Add referenced to Unity components

Once you have got reference to the components import the “Microsoft.Practices.Unity” namespace in to your code as shown in figure ‘Import unity namespace’.

Figure: - Import unity namespace

Defining the common interface

As said previously we should be able to inject SQL Server database object or oracle database object in the customer class. So as a good practice we will inherit the SQL server implementation and oracle implementation from a common interface ‘Idatabase’.

Figure: - Database interfaces

Providing the injection gateway

In the customer class we will reference the interface and expose the public property using [Dependency] attribute. This exposure is essential for the unity container. Unity container needs some kind of gateway by which it can inject the object. So what we have done is we have exposed the interface ‘iDatabase’ publicly. There are lot of other ways by which we can achieve the same you can read my previous article on the different ways of providing injection gateways for containers
<<here>>.

Figure: - Providing injection gateway

Defining the config file

To achieve high customization unity application allows us to specify how the injection will work in the config files ( web.config and app.config ). As this example is a windows command application we will be defining an app.config file.

You can see below a simple template of App.config file from unity perspective. The Configsection is compulsory and keep it as it is. The important section to be noted is the types section which comes under container section. In this we specify which interfaces map to which concrete class. The unity container creates object from this section and inserts in to the customer object through the dependency attribute.

Figure: - App.config file

The client code

The client code is pretty simple. The first step us we create the unity container object. In the second step we read the unity section. In step 3 we configure the container using the unity section data.

In step 4 we tell the container to create the customer object. This is the most important step of all. In this the container using the config data and the dependency attribute injects either the SQL server database object or oracle database object in to the customer class. This object creation depends on what is defined in the app.config file.

Finally in step 5 we call the save method

Figure: - Client code

The actual internal working

So what happens internally in the container is that the unity container creates the object of either oracle or SQL server data objects and injects in to the customer class through the dependency defined attribute.

Enter the real world of loosely coupling

So if you define the ‘clsSqlServer’ class it will create the object of ‘clsSqlServer’ and inject it in to the customer class. If you define the ‘ClsOracle’ class it will inject the ‘clsOracle’ object. You can see in figure ‘The real fun’ how easy it to change implementation by just modifying the config file type sections

Figure: - The real fun