?
Solved

C++ memory use explosion

Posted on 2004-11-10
6
Medium Priority
?
389 Views
Last Modified: 2008-01-16
Wrote a C++ wrapper class to the MySQL/C API that gets a row of DB data and turns it into vector of floats.

Unfortunately, I'm doing about 1x10^6 all-against-all row comparisons, and the memory use of the program as it runs absolutely explodes, eventually crashing due to lack of memory. The whole point of interfacing the code to a db was, of course, to avoid this sort of excessive memory use. Below is my DB.h class. I don't see any obvious extravagant uses of memory, but nonetheless, upon invoking the getPriceVector() method a few thousands times, my system monitor indicates spiraling memory usage. The compiler is  gnu 3.2.2, the system is RH 9, and it's mysql 4.1.



#include <mysql/mysql.h>
#include <stdio.h>
#include <string.h>
#include <DataSeries.h>
using namespace std;
class DB{
 public:
  string name;
  MYSQL mysql;

  DB(char *dbname){
   
    mysql_init(&mysql);
    if (!mysql_real_connect(&mysql,"localhost.localdomain","agm","chacal", dbname,0,NULL,0))
      {
      printf( "Error connecting to database: %s\n",mysql_error(&mysql));
      }
    //    else printf("Connected...\n");
   
  }

  vector<string> getTickers(){
    char query[255];
    sprintf(query,"show tables");
    char **row;
    vector<string> result;
    MYSQL_RES *res;
    if(mysql_real_query(&mysql, query, strlen(query))){
      printf(mysql_error(&mysql));
      exit(1);
    }
    res=mysql_store_result(&mysql);
    row=mysql_fetch_row(res);
    char *ticker;
    while(row!=NULL){
      string s(row[0]);
      result.push_back(s);
      row=mysql_fetch_row(res);

    }
    return result;
   
   
   
  }
 
  vector<float> getPriceVector(string &t, string &d1, string &d2){
   
    char *startQuery=(char*) malloc(255);
    char *endQuery=(char*) malloc(255);
    char *dataQuery=(char*) malloc(1023);
    //dataQuery=malloc(255);
    //endQuery=malloc(255);
    //startQuery(255);
    sprintf(startQuery, "select orderIndex from `%s` where date=%s", t.c_str(), d1.c_str());
    sprintf(endQuery, "select orderIndex from `%s` where date=%s", t.c_str(), d2.c_str());
    //    cout<<startQuery<<endl;
    //cout<<endQuery<<endl;
    char **row;
    MYSQL_RES *resStart;
    MYSQL_RES *resEnd;
    MYSQL_RES *resData;
    bool startSuccess=false;
    bool endSuccess=false;
    char *startIndex;
    char *endIndex;
    if(mysql_real_query(&mysql, startQuery, strlen(startQuery))){
      //cout<<"error\n";
      printf(mysql_error(&mysql));
      //DataSeries ds();
      //return ds;
      //exit(1);
    }else{
      startSuccess=true;
     
      resStart=mysql_store_result(&mysql);
      row=mysql_fetch_row(resStart);
     
      if(row!=NULL){

      startIndex=row[0];
      //      cout<<"start "<<startIndex<<endl;
      
      }else{
      //      cout<<"couldn't find entry for "<<d1<<endl;
      }
    }

    if(mysql_real_query(&mysql, endQuery, strlen(endQuery))){
      printf(mysql_error(&mysql));
      //      DataSeries ds();
      //return ds;
      //exit(1);
    }else{
      endSuccess=true;
      resEnd=mysql_store_result(&mysql);
      row=mysql_fetch_row(resEnd);

      if(row!=NULL){
      
      endIndex=row[0];
      //      cout<<"end "<<endIndex<<endl;
      }else{
      //      cout<<"couldn't find entry for "<<d2<<endl;
      }    
    }
    //vector<string> dates;
    vector<float> prices;
    if(startSuccess && endSuccess){
      sprintf(dataQuery, "select date, close from `%s` where orderIndex>=%s and orderIndex<=%s order by orderIndex",
            t.c_str(), startIndex, endIndex);
      //      cout<<"start: "<<startIndex<<" stop "<<endIndex<<endl;
   
      if(mysql_real_query(&mysql, dataQuery, strlen(dataQuery))){
      //cout<<"error\n";
      printf(mysql_error(&mysql));
      //      DataSeries ds();
      //return ds;
      //exit(1);
      }else{
      
      resData=mysql_store_result(&mysql);
      row=mysql_fetch_row(resData);
     
      while(row!=NULL){
        string s(row[0]);
        //dates.push_back(s);
        prices.push_back((float) strtod(row[1], NULL));
        row=mysql_fetch_row(resData);
      
      }
      }
    }else{
                 cout<<"couldn't find either start or end\n";
    }
    /*for(int i=0; i<dates.size(); i++){
      cout<<dates[i]<<"\t"<<prices[i]<<endl;
      }*/
    //cout<<"prices "<<prices.size()<<endl;
    //cout<<"dates "<<dates.size()<<endl;
    free(startQuery);
    free(endQuery);
    free(dataQuery);
    return prices;
   
   
   
    //startQ<<"select orderIndex from "<<t.c_str()<<" where date="<<d1.c_str();

  }


};
0
Comment
Question by:antoniogm
  • 3
  • 2
6 Comments
 
LVL 86

Expert Comment

by:jkr
ID: 12548601
Correect me if I got that wrong, but you're never freeing the data that is returned as 'row'. That for sure will lead to memory leaks.
0
 
LVL 22

Expert Comment

by:grg99
ID: 12548731
Are you returning the "prices" vector you return once you're done using the values?

You have several places (commented out) where the function could prematurely return before freeing what it has malloced.
That's not good.

Either fix up the code so it all exits out the bottom, or perhaps better, don't malloc those strings, just declare them as static char arrays, then you don't have to allocate or free them or worry about leaks.

Also it wouldnt hurt to use snprintf just in case one of the strings gets too long and threatens to overflow the char arrays.



0
 

Author Comment

by:antoniogm
ID: 12548767
Tried changing the declaration to the following:

char **row=(char **)malloc(2055);

followed by:

free(row).

Is this even the correct way to malloc a char**?

In any case, it doesn't help the memory problem....memory use is still off the charts. In fact, it throw a seg fault at a seemingly random point in the running (i.e. not immediately...but halfway through).
0
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 86

Expert Comment

by:jkr
ID: 12548841
>>Is this even the correct way to malloc a char**?

Usually, yes. But, you should not use a char** here. According to the docs (http://dev.mysql.com/doc/mysql/en/mysql_fetch_row.html), it is a 'MYSQL_ROW', e.g.

MYSQL_ROW row;
unsigned int num_fields;
unsigned int i;

num_fields = mysql_num_fields(result);
while ((row = mysql_fetch_row(result)))
{
   unsigned long *lengths;
   lengths = mysql_fetch_lengths(result);
   for(i = 0; i < num_fields; i++)
   {
       printf("[%.*s] ", (int) lengths[i], row[i] ? row[i] : "NULL");
   }
   printf("\n");
}

But, what I am also missing is a call to 'mysql_free_result()' (http://dev.mysql.com/doc/mysql/en/mysql_free_result.html), which is necessary to free the memory allocated by a query. When you are done with a result set, you must free the memory it uses by calling 'mysql_free_result()'

0
 

Author Comment

by:antoniogm
ID: 12548890
I changed it to it's a static declartion:

char startQuery[255];
char endQuery[255];
char dataQuery[1023];
char **row;

Same memory problem. Could it be the "row"?
0
 
LVL 86

Accepted Solution

by:
jkr earned 2000 total points
ID: 12548921
>>Could it be the "row"?

Try adding the necessary calls to 'mysql_free_result()' first. And use

MYSQL_ROW row;

instead of 'char**'.
0

Featured Post

Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Written by John Humphreys C++ Threading and the POSIX Library This article will cover the basic information that you need to know in order to make use of the POSIX threading library available for C and C++ on UNIX and most Linux systems.   [s…
Article by: evilrix
Looking for a way to avoid searching through large data sets for data that doesn't exist? A Bloom Filter might be what you need. This data structure is a probabilistic filter that allows you to avoid unnecessary searches when you know the data defin…
The viewer will learn additional member functions of the vector class. Specifically, the capacity and swap member functions will be introduced.
The viewer will learn how to clear a vector as well as how to detect empty vectors in C++.
Suggested Courses

850 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question