Tracking program memory usage.

Since a vector stores information in a implimentaiton defined way, there is really no way to be sure of how much space the vector uses to store its data, but it is probably the number of items in its capacity times the sizeof() those items.  

Now note I said the "capacity" not "size".  ("size" would be okay, but capacity woudl be more accurate.)  A vector may provide itself with room for expansion, so it has a "capacity" which is the number of items it currently has space for.  it can expand to store more items than this capacity, but that will require that the vector resize its storage space.  The capacity is always greater than or equal to the size.  The capacity is returned by the capacity() member function.  

So you could modify your code like

cout<<"sizeof(*xp) is:"<<sizeof(*xp) + x_set.capacity()*sizeof(int) <<"\n

continues

Now that fixes the problem in your code (approximately--actually probably exactly, but there is no guaranttee of that..)  However, you stated

>> I want to track the objects created and
>> the byte required for loading each object
Is that for every object?  That can be a little difficult?  Is it just for dunamcially allocated objects (ones created with new)  That is difficult too, but easier.  It is it just for a simple case like this?  There are some things I can suggest to make it easier, if this really is what you need, but I'm sort of hoping--for your sake--that it isn't necessary.   Is there a reason you want to do this?

Adjusted points to 500

Sorry, maybe I didn't describe clearly.
Here is the refined one:

I'm trying to track the memory space required for loading every object and the implementation detail of the object may be hidden (such as an instance of library class).  Most of these objects are dunamcially allocated on the heap using "new". But It would be better that somebody could provide more general solution?  
Thank you.

>> But It would be better that somebody
>> could provide more general solution?
There really isn't a C++ solution to this.   i.e something you can do in generic C++ to provide the information.   Well, there is one thing you can do that might be of help, but probably not.  You can overload new and delete and in the overloaded version you can keep a running count of the memory usage.  This actually is a good solution, if you have the option of using the overloaded new and delete everywhere.  But you said "mplementation detail of the object may be hidden" and that suggests to me that part of the code might not be under your control and therefore not  able to use the overloaded new and delete.  Is that the case?

If that will be a problem, another approach would be to use compiler/OS specific features to obtain the information.  For example VC provides functions for this type of thing in a debug version of the program.    What compiler are you using?  What OS?

Lastly.  Why?  Maybe you don't really need this information, what are you hoping to do with it.

Can you provide some sample of overloading "new", "delete"?

In fact, the compiler is just G++ running under Linux. So I suppose the debuger for it is GDB, although I didn't use it. And I doubt it has feasures of  tracking the OBJECT memory usage.

>> I doubt it has feasures of  tracking
>> the OBJECT memory usage
What do you mean by "OBJECT memory usage"?  (I'm just trying to make sure I understand what you want.)

You would need to create a global (or static) varaible for storing the memory usage, like

int MemUse = 0;

You need something to help you differentiate the overloaded new from the regular, so I use the following enum (your could also use an "empty" class definition)

enum MemTrkTyp
{
   MemTrk;
};

You would create an oveloaded new like

void *
operator new(size_t  Siz,   // Size of block to allocate.          
             MemTrkTyp     MT)  // Differentiates from regular new.
{
   // Note I save the value from new, then add, then return value
   // to make it work right when new throws an exception.  If you
   // don't care about that, do the add first, then just return what
   // new returns.
   void *MemPtr = new char[Siz];
 
   MemUse += Siz;
   return MemPtr;
};

continues

>What do you mean by "OBJECT memory usage"?  

Tracking memory usage for the object (rather than for the process).

void
operator delete(void *MemPtr,  // -> memory to free.
            MemTrkTyp     MT)  // Differentiates from regular delete
{
   delete [] (char *) MemPtr;
   MemUse -= Siz;  // opps no size.
}

I'll have to to start over with a different new.

But the library class such as Vector and etc  uses default "new" to allocate memory. How can I overload it?

But the library class such as Vector and etc  uses default "new" to allocate memory. How can I overload it?

void *
operator new(size_t  Siz,   // Size of block to allocate.          
               MemTrkTyp     MT)  // Differentiates from regular new.
{
   Siz += sizeof(size_t);  Adjust size for room for storing size.

   void *MemPtr = new char[Siz];
    MemUse += Siz;
    *(size_t *) MemPtr = Siz; // Store the size at the start of the memory.
    return MemPtr + sizeof(size_t); // return pointer past the stored size.
};

void
operator delete(void *MemPtr,  // -> memory to free.
                 MemTrkTyp     MT)  // Differentiates from regular delete
{
   char *BytPtr = (char *) MemPtr

    BytPtr -= sizeof(size_t); // Get -> to real start of block.
    size_t Siz = *(size_t *)BytPtr; // Get the block size.
   delete [] BytPtr;
    MemUse -= Siz; // decrement memory usage.
}

continues.

>> Vector and etc  uses default "new" to allocate memory

The 2nd template parameter is an allocator object that is used to allocate and free memory.  You can supply your own allocator object that calls your overloads.  Note all of STL works this way.  It never forces you to use the default new/delete, all allocations/deletions go through an allocator.  If you don't supply one there is usually a default allocator which ends up using the default new and delete.

Is this going to work for you?  Should I continue?

I missed this

>> >What do you mean by "OBJECT memory usage"?  
>> Tracking memory usage for the object (rather
>> than for the process).

Now that sounds a little different than what you said before.  Are you saying you have some class X and you want to know the space that just it uses, or are you saying you want to know the space used by every object of every class.  If it is just for one class (a few isolated classes), you can use the new and delete member operators.  Then you don't even have to worry about defining an allocator.

Yes, I think I can use it for the purpose. Please continue!

>Now that sounds a little different than what you said before.  >Are you saying you have some class X and you want to know >the space that just it uses, or are you saying you want to know >the space used by every object of every class.  If it is just for >one class (a few isolated classes),
No, I am trying to  tracking space used by every class instance(s).

But what "MemUse" contains is still  total memory allocated at the time. The question is that how can I track which object is calling "new". For example,  a Vector instance can grow mang time, how can I track all the space allcated for it at the last.

For instance:

vector<int>* p1=new vector();

for(int i=1;i<5;i++)
  p1->push_back(i); //grows

//other stuff including call "new" for other objects

p1-> push_back(23); //grows again

Now at this time,  I would like to know how much space the object pointed by p1 occupies?

>> The question is that how can I track which
>> object is calling "new".
Why do you want that?  Didn't you just say you wanted to know the total space beig used.  i.e. you just need a number like "1024 total bytes are being used by all objects"?

>> For example,  a Vector
>> instance can grow mang time, how can I track
>> all the space allcated for it at the last.
I don't understand.

>> I would like to know how much space the
>> object pointed by p1 occupies
Just the space for p1, or for all objects?  

If you need the space on an object-by-object bases, it is goignt o be very hard.  And you will have to either--know the implimentation of each class, or at least be able to use the implimentation of each class to help you.   (There is a function you coudl add to each class that would help you to write this, but it will still be hard.  (Not conceptually hard, jast laborious, you will have to "manually" mantain it to some degree.  i.e. when you add/remove members from a class you may have to make changes tot eh code so the calculations remain correct.

Is there a reason you need this?  

listen...

Are we done?  

What else can you add?

Well I'm still not sure of what you want.  (If I knew why you needed it might help.)  And I'm not sure if you've got all the information you need.)

Hi nietod,

How can I track total momory size required for running a program. You already provide me a way to track the total memory size needed for loading dynamic objects. But how about statically allocated objects? Is there a grneral way to do it? I mean no system call is used. Thank you!

Should I set up this as anther question?

Total memory size fo a program is not the same thing as the total memory size used for storing data (objects).  A program will always require more memory than just the memory used to store its data.  it needs memory for its code, for the standard libraries (data and code), for the stack, for the heap and so on...

What exacty do you want to learn and why?

>What exacty do you want to learn
> and why?

Cann't tracking and analyzing the memory usage of a running progrm be reason?

There are just soooo many interpretations of "memory usage" it would be nice to know why you want it, that would tend to focus on 1 or two interpreations.

Like if you need to know if the program will fit in memory in a particular machine, you do need to know TOTAL memory usage.  That includes the code space, statics, stack etc....

If you need to know how much heap space you are using, then only need to look at dynamic allocation.  

If you want tlo know ho much data you ate storinging, you need to look at the heap, locals, and stack

If you want to know how big to mak your stack, you might want to just know the local sizes.

and so on...

>If you want tlo know ho much data you >ate storinging, you need to look at
> the heap, locals, and stack

Hold on. What's the locals. I thought the memory usage of a program only consists of code, data(for initialized data), heap and stack sections.

Ok, can this be specific?
 
how to measure (or estimate if measure is not possible) the total memory usage of a progrme? without using assembly language or some profiling tools e.g. "purify"?

>>  thought the memory usage of a program only
>> consists of code, data(for initialized
>> data), heap and stack sections.
That is more or less true, but it depends on the OS and implimentation to some extent.  Like on some OSs the code may be shared among different instances of the program, does it still count in the size?  I guess it depends on what you want to do with it....   There also may be data sections used by the OS for the program.  like tables that store handles for the program.  That sort of thing.

>> how to measure (or estimate if measure
>> is not possible) the total memory usage
>> of a progrme?
Well in that case do you want include code size?  There is no mechanism in C++ for doing this, you would have to relly on the OS for this--if possible?  Do you include unused data space?  If the stack is 100K but you are using only 10k of stack space, what is the memory usage due to the stack?  The same with the heap?  Hopefully both the heap and stack have a decent amount of unused space.  What about space used for padding?  like if you decide to add up the sizes of your global varaibles to figure out your static space usage, do you care that there may be padding bytes between some of these globals that would increase the actual memory used for static?  When considering local (stack) usage do you care about the space used for non-object storage.  Like space used to store return addresses and stack frame pointers?  Do you care about the fact that the OS may have to allocate segments in large blocks, like if it has to allocate 64K at a time, and it needs 2 K for the statics, there is 62K wasted?  Do you count it?

The answers to these questions probably depend on what you want to do with this information.  And the answers to these questions really realy effect the approach you will use--if any can be used.

Most likely, you will need to do some OS-specific processing in this, what OS are you using?

Did we make it complicated or is C++  not capable of handling this stuff? I was told in Java, just use following at some point,

static public long totalMemory() {
                      return Runtime.getRuntime().totalMemory();
                    }

then everything is done. Of course the value returned is OS depedant. But the solution is quite is general.

C/C++ does not provide such a function.  (or even a technique for reliably estimating it.)  In fact, probably no languages other than Java have such a function.

It sounds like you are going to need a OS-specific solution.

Hei nietod,
How can I use the overloaded operator delete? Thanks.


---------------------------------
class MemTrkTyp;
void* operator new(size_t Siz MemTrkTyp MT);
void operator delete(void *, MemTrkTyp);

MemTrkTyp r;

//this use is fine. It first calls AClass's constructor,
//then call overloaded new
AClass *p=new (r) AClass;
                  

//But how to delete p??

//this directly calls overloaded "delete"
// without first destroying the object pointed by p.
::operator delete (p,r);

//this first destroys object pointed by p.
//then it calls default "::operator new(size_t)"
//since new and delete for p is not paired, the program crashes.
delete (r,p);

Apparently you can't.  I haven't done anything quite like this before.  You can call the overloaded operator new by passing the additional parameters in parenthesis after the "new".  But for detete you have to go with the default global delete or a class's member delete function.

Well why have the overloaded delete functions?  Those are called if there is a exception when constructing an object that was allocated with overloaded new.  i.e if you allocate an object with an overloaded new and the object's constructor throws an exception, the compiler will delete the memory using the corresponding operator delete.  That appears to be the only time those overloads are called, you can't call them at other times.

However, it doesn't matter for your case.  You just need to define a default global new and delete.  You are allowed to define a single global new and delete that don't take the "extra" parameters.  That is all you need (to track dynamic memory usage, it still doesn't give you all usage...)  Like

void * operator new(size_t  Siz)
{
   return malloc(Siz);
};

void operator delete(void *MemPt)
{
   free(MemPtr);
};

You can then adjust the record of memory usage in these functions.  You also may want to handle calling the new handler and throwing excceptions if malloc() fails.