Solved

ADO Memory Leak

Posted on 2003-11-06
17
3,577 Views
Last Modified: 2013-12-14
VC++ 6.0 SP5, MDAC 2.8, W2K Pro SP3
I am developing an NT service that uploads data to a database. It will run 24/7. Because of this, memory leaks are of particular concern to me. I wrote a simple application that executes a select from a database continuously until you press a key. After starting the exe, I took a look at the task manager and saw the "mem usage" grow at a linear rate. There doesn't appear to be an upper bound to this growth.

01 #include <stdio.h>
02 #include <conio.h>
03 #import "c:\program files\common files\system\ado\msado15.dll" rename("EOF","adoEOF")
04
05 struct InitOle {
06 InitOle()  { ::CoInitialize(NULL); }
07 ~InitOle() { ::CoUninitialize();   }
08 } _init_InitOle_;
09
10 void hitdb();
11
12 int main(int argc, char* argv[])
13 {
14      bool done = false;
17      printf("\npress any key to cancel the test...");
18      do
19      {
20            if(kbhit()) done = true;
21            hitdb();
22      } while (!done);
23      getch();
24      return 0;
25 }
26
27 void hitdb()
28 {
29      try
30      {
31            ADODB::_RecordsetPtr   Rs1 = NULL;
32            Rs1.CreateInstance( __uuidof( ADODB::Recordset ) );
33            Rs1->Open( L"SELECT count(*) FROM counts",
34                  L"DSN=linestatus",
35                  ADODB::adOpenForwardOnly,
36                  ADODB::adLockReadOnly, -1 );
37            Rs1->Close();
38            Rs1 = NULL;
39      }
40      catch (_com_error &e) { printf("%s\n",e.ErrorMessage()); }
41      catch (...) { printf("Unspecified Error\n"); }
42 }

If I comment out lines 33 - 37 then there is no memory leak, leading me to believe the problem lies with ADO. Adding Rs1.Release(); to line 37.5 doesn't help. Explicitly creating a connection object doesn't help either, in fact, the leak occurs if I only open a connection to the db and nothing else. Additionally, I have tried to use a DSNless connection to avoid ODBC, but without luck. It seems that the memory used *should* return to a constant level after each call to hitdb(), but this doesn't appear to be the case.
0
Comment
Question by:celestian
  • 7
  • 4
  • 2
  • +3
17 Comments
 
LVL 19

Expert Comment

by:Dexstar
ID: 9695151
celestian:

> After starting the exe, I took a look at the task manager and saw the "mem
> usage" grow at a linear rate. There doesn't appear to be an upper bound to this growth.

I've run into something similiar with ADO, but the memory usage always would drop back down after I closed the recordset.  But you're clearly closing the recordset, so that should cover it.

But the task manager isn't really sufficient for diagnosing a memory leak.  Instead, use the Performance Monitor, and add the following items:
     Memory Object -> Available Bytes
     Process -> YourProcess -> Handle Count
     Process -> YourProcess -> Pool Nonpaged Bytes
     Process -> YourProcess -> Pool Pages Bytes
     Process -> YourProcess -> Private Bytes

If the "Private Bytes" of your application increases consistantly, without decreasing, then you have a memory leak.

Hope that helps,
Dex*
0
 
LVL 19

Expert Comment

by:Dexstar
ID: 9695218
celestian:

> If I comment out lines 33 - 37 then there is no memory leak, leading me to
> believe the problem lies with ADO.

You aren't the only one with this issue.  Read this:
     http://dbforums.com/arch/210/2003/1/663190

That's for an Access database, but one of the replies says that there is similiar behavior with other providers.

Dex*
0
 
LVL 9

Expert Comment

by:_ys_
ID: 9700438
Have you tried dumping the connection when you're done.

Rs1->putref_ActiveConnection(NULL); // Line 36.5
0
 

Author Comment

by:celestian
ID: 9701395
Dex,
  I had considered perfmon a more fine grained tool, and if taskman showed an increasing usage of memory, so would perfmon.

_ys_,
  I added the line you specified, but it did not aleviate the problem.

0
 
LVL 19

Expert Comment

by:Dexstar
ID: 9702412
celestian:

My point is that you might not have a memory leak at all.  The Windows memory manager may be doing some things that look odd to you, because it thinks thats the most efficient way to do it.  Maybe you have TONS of RAM on the system, maybe there is nothing else competing for the resources... Who knows?  My point is that "Mem Usage" doesn't indicate a memory leak.

The "Mem Usage" field of the task manager corresponds to the "Working Set" property of the Performance counter, which isn't even one of the ones they tell you to check for a memory leak.  The important measurement is "Private Bytes".  If you want to use the Task Manager, then do this:

1) View -> Select Columns
2) Check "Virtual Memory Size"

If that value keeps going up and up, then you might have a memory leak... But until you know for sure, you could be chasing a ghost.

Dex*
0
 
LVL 3

Expert Comment

by:RJSoft
ID: 9705502
0
 

Author Comment

by:celestian
ID: 9732253
I ran an extended test to see how the memory would be affected:

time                peak mem usage     vm size
12:19              10,948                   8,516
1:19                12,536                   19,408
3:05                13,260                   40,120
4:01                13,260                   51,264
~48 hrs later    17,344                   83,096

Since the program is doing the same thing over and over, I guess I would expect these numbers to remain constant. Since they are not, I am exploring alternatives to ADO.
0
 
LVL 19

Expert Comment

by:Dexstar
ID: 9734165
celestian:

Okay, that is definitely, a memory leak.

I was going to run the code that you posted, but I noticed something that may be your issue:
     38          Rs1 = NULL;

Rs1 is a smart pointer that will clean up after itself.  When you set it to NULL, you're clearing out the pointer that it is supposed to clean up, so when the variable falls out of scope, there is no way to clean it up.  Try removing line 38 and see if you get the same results.

Yes, I know lots of sample VB code does things like:
     Set Rs1 = Nothing

But, when you do that, you are releasing the smart pointer.

If you still are looking for an alternative, then have you looked into OLEDB?  (ADO is basically a higher level wrapper around OLEDB)  Now, OLEDB is much more detailed (and therefore, more to learn before using it), but it gives you access to things on a really low level.  This page gives you some idea of the differences:
     http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnoledb/html/choosingcomponents.asp

ATL even provides you with some really nice templates for creating OLEDB classes, and VS will give you a wizard to generate classes based on a specific table, view, or stored procedure.

Dex*
0
What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

 

Author Comment

by:celestian
ID: 9750712
I have run some more detailed tests in an effort to try and understand what is going on here. I used the following code:

#include <stdio.h>
#include <conio.h>
#import <msado15.dll> rename("EOF","adoEOF")

struct InitOle {
  InitOle()  { ::CoInitialize(NULL); }
  ~InitOle() { ::CoUninitialize();   }
} _init_InitOle_;

void hitdb();

int main(int argc, char* argv[])
{
  bool done = false;
  char c;
  long l = 0;

  printf("\npress any key to start the test...");
  do { Sleep(300);} while (!kbhit()); getch();
  printf("\n'p' to pause 'x' to exit");
  do
  {
    if(kbhit()) {
      c = getch();
      if (c == 'p') {
        printf("\n%d iterations completed",l);
        printf("\npress any key to continue the test...");
        do { Sleep(300); } while (!kbhit()); getch();
      } else if (c == 'x') {
        done = true;
      }
      if (!done) printf("\n'p' to pause 'x' to exit");
    }
    l++;
    hitdb();
    Sleep(300);
  } while (!done);

  return 0;
}
void hitdb() {
  try
  {
    ADODB::_ConnectionPtr  Cn1;
    Cn1.CreateInstance(__uuidof(ADODB::Connection));
    Cn1->CursorLocation = ADODB::adUseClient;
    Cn1->Open(L"dsn=linestatus",L"",L"",-1);
    Cn1->Close();
  }
  catch (_com_error &e) { printf("%s\n",e.ErrorMessage()); }
  catch (...) { printf("Unspecified Error\n"); }
}

I stepped through the code and noticed that as calls were made to hitdb(), threads were created and destroyed, no surprise there. Also, each time a thread was created, 3 handles were also created, but when the thread was destroyed, only 2 handles were returned. This happened each time as I allowed 1 call to hitdb(), and the handle counter grew by one after each call. When I let the loop continue without interruption, I noticed that this didn't always happen. After 671 calls to hitdb(), I saw 184 handles, when after 1 loop there had only been 137. I ran the test continuously 2 more times until I saw 184 handles on 4 threads. This happend after around 480 and 356 iterations. It seems that whatever is going on in this multi-threaded mess, there are handles to something being created, and they are not always destroyed. This might very well be the cause of the memory leak.

Looking for information on the internet related to C++ and ADO is difficult at best, perhaps because others have noticed problems as well and just don't use the two together. I also noticed that when I created my first ADO Connection instance, 50 dlls were loaded. And I am not doing any kind of data manipulation, I am just openning a connection and then closing it right away. I will look into OLEDB and <sqlext.h> and see if I can get something that works a little better.

I will let this question stay open for a little while in case there is someone who knows of a database c++ solution that doesn't leak memory.
0
 
LVL 19

Expert Comment

by:Dexstar
ID: 9750817
What is the backend data source provider that you are using?  SQL Server?  Access?  Oracle?

Dex*
0
 

Author Comment

by:celestian
ID: 9750990
I have tried Oracle on a remote server and a local copy of MySQL server. For Oracle I used a DSNless connection with OraOLEDB.Oracle.1 as the provider, and for MySQL I set up a system dsn with the MySQL ODBC 3.51 Driver. As it so happens both the above code samples are using the DSN with MySQL, but for the memory tests I posted that showed the leak, I was using Oracle. Either way I notice the leak.
0
 
LVL 19

Accepted Solution

by:
Dexstar earned 175 total points
ID: 9766879
I ran the code that you did, but with the SQL Server provider, and I wasn't able to reproduce the leaks.  Maybe the leaks are in the provider layer, not in ADO itself...  Which doesn't help you out, but it would explain why others haven't noticed it yet.

Dex*
0
 
LVL 1

Assisted Solution

by:Paladin_VB
Paladin_VB earned 175 total points
ID: 9804532
By writing code that doesn't have any. Clearly, if your code has new operations, delete operations, and pointer arithmetic all over the place, you are going to mess up somewhere and get leaks, stray pointers, etc. This is true independently of how conscientious you are with your allocations: eventually the complexity of the code will overcome the time and effort you can afford. It follows that successful techniques rely on hiding allocation and deallocation inside more manageable types. Good examples are the standard containers. They manage memory for their elements better than you could without disproportionate effort. Consider writing this without the help of string and vector:
      #include<vector>
      #include<string>
      #include<iostream>
      #include<algorithm>
      using namespace std;

      int main()      // small program messing around with strings
      {
            cout << "enter some whitespace-separated words:\n";
            vector<string> v;
            string s;
            while (cin>>s) v.push_back(s);

            sort(v.begin(),v.end());

            string cat;
            typedef vector<string>::const_iterator Iter;
            for (Iter p = v.begin(); p!=v.end(); ++p) cat += *p+"+";
            cout << cat << '\n';
      }

What would be your chance of getting it right the first time? And how would you know you didn't have a leak?
Note the absence of explicit memory management, macros, casts, overflow checks, explicit size limits, and pointers. By using a function object and a standard algorithm, I could have eliminated the pointer-like use of the iterator, but that seemed overkill for such a tiny program.

These techniques are not perfect and it is not always easy to use them systematically. However, they apply surprisingly widely and by reducing the number of explicit allocations and deallocations you make the remaining examples much easier to keep track of. As early as 1981, I pointed out that by reducing the number of objects that I had to keep track of explicitly from many tens of thousands to a few dozens, I had reduced the intellectual effort needed to get the program right from a Herculean task to something manageable, or even easy.

If your application area doesn't have libraries that make programming that minimizes explicit memory management easy, then the fastest way of getting your program complete and correct might be to first build such a library.

Templates and the standard libraries make this use of containers, resource handles, etc., much easier than it was even a few years ago. The use of exceptions makes it close to essential.

If you cannot handle allocation/deallocation implicitly as part of an object you need in your application anyway, you can use a resource handle to minimize the chance of a leak. Here is an example where I need to return an object allocated on the free store from a function. This is an opportunity to forget to delete that object. After all, we cannot tell just looking at pointer whether it needs to be deallocated and if so who is responsible for that. Using a resource handle, here the standard library auto_ptr, makes it clear where the responsibility lies:

      #include<memory>
      #include<iostream>
      using namespace std;

      struct S {
            S() { cout << "make an S\n"; }
            ~S() { cout << "destroy an S\n"; }
            S(const S&) { cout << "copy initialize an S\n"; }
            S& operator=(const S&) { cout << "copy assign an S\n"; }
      };

      S* f()
      {
            return new S;      // who is responsible for deleting this S?
      };

      auto_ptr<S> g()
      {
            return auto_ptr<S>(new S);      // explicitly transfer responsibility for deleting this S
      }

      int main()
      {
            cout << "start main\n";
            S* p = f();
            cout << "after f() before g()\n";
      //      S* q = g();      // caught by compiler
            auto_ptr<S> q = g();
            cout << "exit main\n";
            // leaks *p
            // implicitly deletes *q
      }

Think about resources in general, rather than simply about memory.

If systematic application of these techniques is not possible in your environment (you have to use code from elsewhere, part of your program was written by Neanderthals, etc.), be sure to use a memory leak detector as part of your standard development procedure, or plug in a garbage collector.

0
 
LVL 19

Expert Comment

by:Dexstar
ID: 9819161
Did that answer apply to this question?
0
 
LVL 9

Expert Comment

by:tinchos
ID: 10242581
No comment has been added lately, so it's time to clean up this TA.
I will leave the following recommendation for this question in the Cleanup topic area:

Split: Dexstar {http:#9766879} & Paladin_VB {http:#9804532}

Please leave any comments here within the next seven days.
PLEASE DO NOT ACCEPT THIS COMMENT AS AN ANSWER!

Tinchos
EE Cleanup Volunteer
0
 
LVL 3

Expert Comment

by:RJSoft
ID: 10245073
Good comment Paladin_VB

RJ
0

Featured Post

Top 6 Sources for Identifying Threat Actor TTPs

Understanding your enemy is essential. These six sources will help you identify the most popular threat actor tactics, techniques, and procedures (TTPs).

Join & Write a Comment

Programmer's Notepad is, one of the best free text editing tools available, simply because the developers appear to have second-guessed every weird problem or issue a programmer is likely to run into. One of these problems is selecting and deleti…
C++ Properties One feature missing from standard C++ that you will find in many other Object Oriented Programming languages is something called a Property (http://www.experts-exchange.com/Programming/Languages/CPP/A_3912-Object-Properties-in-C.ht…
The goal of the video will be to teach the user the concept of local variables and scope. An example of a locally defined variable will be given as well as an explanation of what scope is in C++. The local variable and concept of scope will be relat…
The viewer will learn how to user default arguments when defining functions. This method of defining functions will be contrasted with the non-default-argument of defining functions.

707 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

16 Experts available now in Live!

Get 1:1 Help Now