With my work, I regularly deal with obscene amounts. Of course, our users demand that this information loads instantly and without running out of memory. So optimizing our data storage objects for time and memory constraints is incredibly important. So far, I think I've done a pretty good job, because I can load 125 million data points in a shade over 3 seconds (using a database to store some of this data flew out the window a while ago). But getting to this point has been an interesting journey. Doing quick prototyping of prospective storage mechanisms has been incredibly important and I thought I'd share some of the utility methods/classes I've created to speed up the process.

The first piece of code is incredibly similar to my StopwatchWriter Class. Instead of starting and stopping a Stopwatch, we're asking the garbage collector how much total memory is being used (in bytes), before and after you instantiate an object (or a whole set of objects)

public class MemoryWriter : IDisposable
{
    long _startMem;
    string _text;

    public MemoryWriter(string text)
    {
        _text = text + " - ";
        _startMem = GC.GetTotalMemory(true);
    }

    public void Dispose()
    {
        Console.WriteLine("mem: " + _text + (GC.GetTotalMemory(true) - 
             _startMem).ToString());
    }
}

Usage looks like:

using (new MemoryWriter("CrazyBigObject"))
{
    CrazyBigObject myCrazyBigObject = LoadCrazyBigObject(42);
}

This is the most accurate way of figuring out how much memory a specific object is taking up in memory. But there is a limitation to this method if you're using it in a multi-threaded application, such as a Win Form because another running thread could dereference objects on the heap after you've instantiated your MemoryWriter , but before its been disposed. I prefer to only use this class in small throw away console applications, to make my results as accurate as possible.

Sometimes though, you'll find yourself looking at an object in someone else's code and you want to easily find out how much memory its taking up, but the object in question is "built" over several methods along with several other objects that you don't care about. Which makes it impossible to use theMemoryWriter Class. You can use a memory profiling application to do this, but I've found that they are notoriously hard to pick up and use. And due to the nature of how they work, they take ages to work because they have to take a snapshot before and after the code you care about (i.e. copy your 700MB object heap twice and then "diff" the two heaps). So I've wrote two small methods that serialize the object to a stream and then return the length of the stream. Note: This is approximately how much data the object is holding in memory. It may besignificantly less than how much space it takes up on the heap. Case in point: Dictionary<K, V> takes up much more space in memory than it does when serialized.That's because it only serialized the key value pairs and it re-hydrates the dictionary on deserialization. Nor will it reflect the size of any properties on your object that are marked with the NonSerialized attribute. So it is far from perfect, but it helps give you an idea with out the pain of using a memory profiler . You can use these methods by setting a break point and calling them in your watch window (I prefer doing that over the immediate window).

/// <summary>
/// Returns a rough approximation of the size of an object 
/// (including ALL objects in/directly referenced by the object)
/// </summary>
public static long ApproxSize(object obj)
{
    BinaryFormatter formatter = new BinaryFormatter();
    long length;
    using (MemoryStream stream = new MemoryStream())
    {
        formatter.Serialize(stream, obj);
        stream.Seek(0, SeekOrigin.Begin);
        length = stream.Length;
    }
   
    return length;
}

Sometimes though, your object is very large and serializing the object in memory will cause an OutOfMemory Exception. So use the below method instead, it serializes the object to disk and gives you the size of the file (and deletes the file after its done).

/// <summary>
/// Returns the approximate size of very large objects 
/// (time intensive) in bytes.
/// </summary>
public static long ApproxSizeLarge(object obj)
{
    FileInfo info = new FileInfo(@"c:\approxSizeTemp");
   
    BinaryFormatter formatter = new BinaryFormatter();
    using (StreamWriter writer = new StreamWriter(@"c:\approxSizeTemp"))
    {
        formatter.Serialize(writer.BaseStream, obj);
    }

    long length = info.Length;
    info.Delete();

    return length;
}