Solved

Tracking program memory usage.

Posted on 2000-03-05
33
436 Views
Last Modified: 2010-04-02
Hello,

I want to track the objects created and the byte required for loading each object while a program is running and then dump the results to a file or standard output. I tried using "sizeof()" , but it doesn't work. Can you tell me how to do it? Thanks a lot.


p.s. I want to track memory usage of each object created  during the program is running. Not only the total memory usage of the program.



-----------This does not work-----------------


#include <iostream.h>
#include <vector>
using std::vector;

class A1 {
public:
  vector<int> x_set; // could be other user define type

public:
  A1(){ cout<<"A1 is created\n";} //track it when A1 is instantiated
 
  void setVector(int size, int value){
   
    for(int i=1;i<size;i++)
      x_set.push_back(value);
  }

  int getVectorSize(){
    return x_set.size();
  }
 
};

int main(){


cout<<"sizeof(A1) is:"<<sizeof(A1)<<"\n"; //size of class A1?

A1* xp=new A1();

xp->setVector(120,25);

cout<<"sizeof(*xp) is:"<<sizeof(*xp)<<"\n"; //the byte required to load
                                                                           // the object pointed by xp??

cout<<"set: "<<sizeof(xp->x_set)<<"\n";
                              
cout<<"vector size:"<<xp->getVectorSize()<<"\n";
return 0;
}

/*******Running result


sizeof(A1) is:12
A1 is created
sizeof(*xp) is:12
set: 12
vector size:119

***********/
0
Comment
Question by:rongz
  • 17
  • 15
33 Comments
 
LVL 22

Accepted Solution

by:
nietod earned 500 total points
ID: 2585864
The problem is that in

>> out<<"sizeof(*xp) is:"<<sizeof(*xp)<<"\n";

it is returning sizeof(A1).  which seems to be 12.  That is correct.  But not what you want.  That is the size of an A1 object, but an A1 object contains a vector which in turn contains a pointer to other data.  The size of this data is not returned by sizeof()  (It can't be.  For one reason  sizeof() is computed as a constant at compile time, and the number of items in the vecotr is not know at compile time.  (there are other reasons too.))

continues
0
 
LVL 22

Expert Comment

by:nietod
ID: 2585878
Since a vector stores information in a implimentaiton defined way, there is really no way to be sure of how much space the vector uses to store its data, but it is probably the number of items in its capacity times the sizeof() those items.  

Now note I said the "capacity" not "size".  ("size" would be okay, but capacity woudl be more accurate.)  A vector may provide itself with room for expansion, so it has a "capacity" which is the number of items it currently has space for.  it can expand to store more items than this capacity, but that will require that the vector resize its storage space.  The capacity is always greater than or equal to the size.  The capacity is returned by the capacity() member function.  

So you could modify your code like

cout<<"sizeof(*xp) is:"<<sizeof(*xp) + x_set.capacity()*sizeof(int) <<"\n

continues
0
 
LVL 22

Expert Comment

by:nietod
ID: 2585885
Now that fixes the problem in your code (approximately--actually probably exactly, but there is no guaranttee of that..)  However, you stated

>> I want to track the objects created and
>> the byte required for loading each object
Is that for every object?  That can be a little difficult?  Is it just for dunamcially allocated objects (ones created with new)  That is difficult too, but easier.  It is it just for a simple case like this?  There are some things I can suggest to make it easier, if this really is what you need, but I'm sort of hoping--for your sake--that it isn't necessary.   Is there a reason you want to do this?
0
 

Author Comment

by:rongz
ID: 2587047
Adjusted points to 500
0
 

Author Comment

by:rongz
ID: 2587048
Sorry, maybe I didn't describe clearly.
Here is the refined one:

I'm trying to track the memory space required for loading every object and the implementation detail of the object may be hidden (such as an instance of library class).  Most of these objects are dunamcially allocated on the heap using "new". But It would be better that somebody could provide more general solution?  
Thank you.
0
 
LVL 22

Expert Comment

by:nietod
ID: 2587298
>> But It would be better that somebody
>> could provide more general solution?
There really isn't a C++ solution to this.   i.e something you can do in generic C++ to provide the information.   Well, there is one thing you can do that might be of help, but probably not.  You can overload new and delete and in the overloaded version you can keep a running count of the memory usage.  This actually is a good solution, if you have the option of using the overloaded new and delete everywhere.  But you said "mplementation detail of the object may be hidden" and that suggests to me that part of the code might not be under your control and therefore not  able to use the overloaded new and delete.  Is that the case?

If that will be a problem, another approach would be to use compiler/OS specific features to obtain the information.  For example VC provides functions for this type of thing in a debug version of the program.    What compiler are you using?  What OS?

Lastly.  Why?  Maybe you don't really need this information, what are you hoping to do with it.
0
 

Author Comment

by:rongz
ID: 2587434
Can you provide some sample of overloading "new", "delete"?

In fact, the compiler is just G++ running under Linux. So I suppose the debuger for it is GDB, although I didn't use it. And I doubt it has feasures of  tracking the OBJECT memory usage.
0
 
LVL 22

Expert Comment

by:nietod
ID: 2587487
>> I doubt it has feasures of  tracking
>> the OBJECT memory usage
What do you mean by "OBJECT memory usage"?  (I'm just trying to make sure I understand what you want.)

You would need to create a global (or static) varaible for storing the memory usage, like

int MemUse = 0;

You need something to help you differentiate the overloaded new from the regular, so I use the following enum (your could also use an "empty" class definition)

enum MemTrkTyp
{
   MemTrk;
};

You would create an oveloaded new like

void *
operator new(size_t  Siz,   // Size of block to allocate.          
             MemTrkTyp     MT)  // Differentiates from regular new.
{
   // Note I save the value from new, then add, then return value
   // to make it work right when new throws an exception.  If you
   // don't care about that, do the add first, then just return what
   // new returns.
   void *MemPtr = new char[Siz];
 
   MemUse += Siz;
   return MemPtr;
};

continues
0
 

Author Comment

by:rongz
ID: 2587518

>What do you mean by "OBJECT memory usage"?  

Tracking memory usage for the object (rather than for the process).
0
 
LVL 22

Expert Comment

by:nietod
ID: 2587531
void
operator delete(void *MemPtr,  // -> memory to free.
            MemTrkTyp     MT)  // Differentiates from regular delete
{
   delete [] (char *) MemPtr;
   MemUse -= Siz;  // opps no size.
}

I'll have to to start over with a different new.
0
 

Author Comment

by:rongz
ID: 2587574

But the library class such as Vector and etc  uses default "new" to allocate memory. How can I overload it?




0
 

Author Comment

by:rongz
ID: 2587583

But the library class such as Vector and etc  uses default "new" to allocate memory. How can I overload it?




0
 
LVL 22

Expert Comment

by:nietod
ID: 2587599
void *
operator new(size_t  Siz,   // Size of block to allocate.          
               MemTrkTyp     MT)  // Differentiates from regular new.
{
   Siz += sizeof(size_t);  Adjust size for room for storing size.

   void *MemPtr = new char[Siz];
    MemUse += Siz;
    *(size_t *) MemPtr = Siz; // Store the size at the start of the memory.
    return MemPtr + sizeof(size_t); // return pointer past the stored size.
};

void
operator delete(void *MemPtr,  // -> memory to free.
                 MemTrkTyp     MT)  // Differentiates from regular delete
{
   char *BytPtr = (char *) MemPtr

    BytPtr -= sizeof(size_t); // Get -> to real start of block.
    size_t Siz = *(size_t *)BytPtr; // Get the block size.
   delete [] BytPtr;
    MemUse -= Siz; // decrement memory usage.
}

continues.
0
 
LVL 22

Expert Comment

by:nietod
ID: 2587612
>> Vector and etc  uses default "new" to allocate memory

The 2nd template parameter is an allocator object that is used to allocate and free memory.  You can supply your own allocator object that calls your overloads.  Note all of STL works this way.  It never forces you to use the default new/delete, all allocations/deletions go through an allocator.  If you don't supply one there is usually a default allocator which ends up using the default new and delete.

Is this going to work for you?  Should I continue?
0
 
LVL 22

Expert Comment

by:nietod
ID: 2587657
I missed this

>> >What do you mean by "OBJECT memory usage"?  
>> Tracking memory usage for the object (rather
>> than for the process).

Now that sounds a little different than what you said before.  Are you saying you have some class X and you want to know the space that just it uses, or are you saying you want to know the space used by every object of every class.  If it is just for one class (a few isolated classes), you can use the new and delete member operators.  Then you don't even have to worry about defining an allocator.
0
 

Author Comment

by:rongz
ID: 2587672
Yes, I think I can use it for the purpose. Please continue!
0
Enabling OSINT in Activity Based Intelligence

Activity based intelligence (ABI) requires access to all available sources of data. Recorded Future allows analysts to observe structured data on the open, deep, and dark web.

 

Author Comment

by:rongz
ID: 2587690
>Now that sounds a little different than what you said before.  >Are you saying you have some class X and you want to know >the space that just it uses, or are you saying you want to know >the space used by every object of every class.  If it is just for >one class (a few isolated classes),
No, I am trying to  tracking space used by every class instance(s).

0
 

Author Comment

by:rongz
ID: 2587877
But what "MemUse" contains is still  total memory allocated at the time. The question is that how can I track which object is calling "new". For example,  a Vector instance can grow mang time, how can I track all the space allcated for it at the last.

For instance:

vector<int>* p1=new vector();

for(int i=1;i<5;i++)
  p1->push_back(i); //grows

//other stuff including call "new" for other objects

p1-> push_back(23); //grows again

Now at this time,  I would like to know how much space the object pointed by p1 occupies?

0
 
LVL 22

Expert Comment

by:nietod
ID: 2588004
>> The question is that how can I track which
>> object is calling "new".
Why do you want that?  Didn't you just say you wanted to know the total space beig used.  i.e. you just need a number like "1024 total bytes are being used by all objects"?

>> For example,  a Vector
>> instance can grow mang time, how can I track
>> all the space allcated for it at the last.
I don't understand.

>> I would like to know how much space the
>> object pointed by p1 occupies
Just the space for p1, or for all objects?  

If you need the space on an object-by-object bases, it is goignt o be very hard.  And you will have to either--know the implimentation of each class, or at least be able to use the implimentation of each class to help you.   (There is a function you coudl add to each class that would help you to write this, but it will still be hard.  (Not conceptually hard, jast laborious, you will have to "manually" mantain it to some degree.  i.e. when you add/remove members from a class you may have to make changes tot eh code so the calculations remain correct.

Is there a reason you need this?  
0
 
LVL 5

Expert Comment

by:Wyn
ID: 2590077
listen...
0
 
LVL 22

Expert Comment

by:nietod
ID: 2592103
Are we done?  
0
 

Author Comment

by:rongz
ID: 2592260
What else can you add?
0
 
LVL 22

Expert Comment

by:nietod
ID: 2592689
Well I'm still not sure of what you want.  (If I knew why you needed it might help.)  And I'm not sure if you've got all the information you need.)
0
 

Author Comment

by:rongz
ID: 2604325
Hi nietod,

How can I track total momory size required for running a program. You already provide me a way to track the total memory size needed for loading dynamic objects. But how about statically allocated objects? Is there a grneral way to do it? I mean no system call is used. Thank you!

Should I set up this as anther question?
0
 
LVL 22

Expert Comment

by:nietod
ID: 2604379
Total memory size fo a program is not the same thing as the total memory size used for storing data (objects).  A program will always require more memory than just the memory used to store its data.  it needs memory for its code, for the standard libraries (data and code), for the stack, for the heap and so on...

What exacty do you want to learn and why?
0
 

Author Comment

by:rongz
ID: 2604514
>What exacty do you want to learn
> and why?

Cann't tracking and analyzing the memory usage of a running progrm be reason?
0
 
LVL 22

Expert Comment

by:nietod
ID: 2604543
There are just soooo many interpretations of "memory usage" it would be nice to know why you want it, that would tend to focus on 1 or two interpreations.

Like if you need to know if the program will fit in memory in a particular machine, you do need to know TOTAL memory usage.  That includes the code space, statics, stack etc....

If you need to know how much heap space you are using, then only need to look at dynamic allocation.  

If you want tlo know ho much data you ate storinging, you need to look at the heap, locals, and stack

If you want to know how big to mak your stack, you might want to just know the local sizes.

and so on...
0
 

Author Comment

by:rongz
ID: 2604625
>If you want tlo know ho much data you >ate storinging, you need to look at
> the heap, locals, and stack

Hold on. What's the locals. I thought the memory usage of a program only consists of code, data(for initialized data), heap and stack sections.

Ok, can this be specific?
 
how to measure (or estimate if measure is not possible) the total memory usage of a progrme? without using assembly language or some profiling tools e.g. "purify"?
0
 
LVL 22

Expert Comment

by:nietod
ID: 2604701
>>  thought the memory usage of a program only
>> consists of code, data(for initialized
>> data), heap and stack sections.
That is more or less true, but it depends on the OS and implimentation to some extent.  Like on some OSs the code may be shared among different instances of the program, does it still count in the size?  I guess it depends on what you want to do with it....   There also may be data sections used by the OS for the program.  like tables that store handles for the program.  That sort of thing.

>> how to measure (or estimate if measure
>> is not possible) the total memory usage
>> of a progrme?
Well in that case do you want include code size?  There is no mechanism in C++ for doing this, you would have to relly on the OS for this--if possible?  Do you include unused data space?  If the stack is 100K but you are using only 10k of stack space, what is the memory usage due to the stack?  The same with the heap?  Hopefully both the heap and stack have a decent amount of unused space.  What about space used for padding?  like if you decide to add up the sizes of your global varaibles to figure out your static space usage, do you care that there may be padding bytes between some of these globals that would increase the actual memory used for static?  When considering local (stack) usage do you care about the space used for non-object storage.  Like space used to store return addresses and stack frame pointers?  Do you care about the fact that the OS may have to allocate segments in large blocks, like if it has to allocate 64K at a time, and it needs 2 K for the statics, there is 62K wasted?  Do you count it?

The answers to these questions probably depend on what you want to do with this information.  And the answers to these questions really realy effect the approach you will use--if any can be used.

Most likely, you will need to do some OS-specific processing in this, what OS are you using?
0
 

Author Comment

by:rongz
ID: 2604825
Did we make it complicated or is C++  not capable of handling this stuff? I was told in Java, just use following at some point,

static public long totalMemory() {
                      return Runtime.getRuntime().totalMemory();
                    }

then everything is done. Of course the value returned is OS depedant. But the solution is quite is general.
0
 
LVL 22

Expert Comment

by:nietod
ID: 2604883
C/C++ does not provide such a function.  (or even a technique for reliably estimating it.)  In fact, probably no languages other than Java have such a function.

It sounds like you are going to need a OS-specific solution.
0
 

Author Comment

by:rongz
ID: 2616639
Hei nietod,
How can I use the overloaded operator delete? Thanks.


---------------------------------
class MemTrkTyp;
void* operator new(size_t Siz MemTrkTyp MT);
void operator delete(void *, MemTrkTyp);

MemTrkTyp r;

//this use is fine. It first calls AClass's constructor,
//then call overloaded new
AClass *p=new (r) AClass;
                  

//But how to delete p??

//this directly calls overloaded "delete"
// without first destroying the object pointed by p.
::operator delete (p,r);

//this first destroys object pointed by p.
//then it calls default "::operator new(size_t)"
//since new and delete for p is not paired, the program crashes.
delete (r,p);
0
 
LVL 22

Expert Comment

by:nietod
ID: 2617404
Apparently you can't.  I haven't done anything quite like this before.  You can call the overloaded operator new by passing the additional parameters in parenthesis after the "new".  But for detete you have to go with the default global delete or a class's member delete function.

Well why have the overloaded delete functions?  Those are called if there is a exception when constructing an object that was allocated with overloaded new.  i.e if you allocate an object with an overloaded new and the object's constructor throws an exception, the compiler will delete the memory using the corresponding operator delete.  That appears to be the only time those overloads are called, you can't call them at other times.

However, it doesn't matter for your case.  You just need to define a default global new and delete.  You are allowed to define a single global new and delete that don't take the "extra" parameters.  That is all you need (to track dynamic memory usage, it still doesn't give you all usage...)  Like

void * operator new(size_t  Siz)
{
   return malloc(Siz);
};

void operator delete(void *MemPt)
{
   free(MemPtr);
};

You can then adjust the record of memory usage in these functions.  You also may want to handle calling the new handler and throwing excceptions if malloc() fails.
0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

Unlike C#, C++ doesn't have native support for sealing classes (so they cannot be sub-classed). At the cost of a virtual base class pointer it is possible to implement a pseudo sealing mechanism The trick is to virtually inherit from a base class…
C++ Properties One feature missing from standard C++ that you will find in many other Object Oriented Programming languages is something called a Property (http://www.experts-exchange.com/Programming/Languages/CPP/A_3912-Object-Properties-in-C.ht…
The goal of the tutorial is to teach the user how to use functions in C++. The video will cover how to define functions, how to call functions and how to create functions prototypes. Microsoft Visual C++ 2010 Express will be used as a text editor an…
The viewer will learn how to pass data into a function in C++. This is one step further in using functions. Instead of only printing text onto the console, the function will be able to perform calculations with argumentents given by the user.

708 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

17 Experts available now in Live!

Get 1:1 Help Now