Solved

Memory problem

Posted on 2014-11-22
54
177 Views
Last Modified: 2014-11-27
Hi,
when running the codes below
https://dl.dropboxusercontent.com/u/40211031/SaveBinaryFile.cpp

to generate the list. I get this error
https://dl.dropboxusercontent.com/u/40211031/t863.png

how to resolve it?
0
Comment
Question by:HuaMinChen
  • 24
  • 13
  • 7
  • +1
54 Comments
 
LVL 37

Expert Comment

by:Neil Russell
Comment Utility
Please place your code and images on EE where we can ALL see them. A lot of corporate networks will BLOCK Dropbox, and for very good reason :D
0
 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
Here are the codes
//

#include "stdafx.h"
#include <stdio.h>
#include <string.h>
#include <fstream>
#include <string>
#include <ctype.h>
#include <time.h>
#include <process.h>
#include <vector>
#include <iostream>
#include <algorithm>
#include <iomanip>
using namespace std;

struct nameval
{
	char fld_nm[100];
	wchar_t fld_nm_t[100];
	int fld_len;
	int fld_val;
};
nameval binrec;

bool LessComp(const nameval& a1, const nameval& a2)
{
  if(strcmp(a1.fld_nm, a2.fld_nm) < 0) return true;
  if(strcmp(a1.fld_nm, a2.fld_nm) > 0) return false;
  if(a1.fld_val < a2.fld_val) return true;
  return false;
}

int main()
{
    std::vector<nameval> records;
    int cnt;
	for (cnt=0;cnt<3000000;cnt++)
	{
		nameval val={0};
		int j;
		for (j=0;j<20;j++)
		{
			val.fld_nm[j] += (char)(rand () % 58 + 64);
		}

		wchar_t wbuf[100]={0};
		mbstowcs(wbuf,val.fld_nm,_countof(val.fld_nm));
		wmemcpy(val.fld_nm_t,wbuf,100);

		val.fld_val=cnt;
		records.push_back(val);
	}
	std::sort(records.begin(), records.end(), LessComp);

	std::ofstream ostrm("c:\\dp4\\flout.bin", std::ios::binary | std::ios::out );
	if (ostrm.is_open())
	{
		for (size_t i = 0; i < records.size(); i++)
		{
			ostrm.write((char *)&records[i], sizeof(nameval));
		}
	}

	std::wofstream ostrm2("c:\\dp4\\flout.ord", std::ios::out );
	if (ostrm2.is_open())
	{
		for (size_t i = 0; i < records.size(); i++)
		{
			ostrm2 << "\"" << records[i].fld_nm << "\" " << records[i].fld_len << ' '  << records[i].fld_nm_t << ' '<< records[i].fld_val << '\n';
		}
	}

	return 0;
}

Open in new window

and here is the image
t863.png
0
 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
Thanks
0
 
LVL 32

Expert Comment

by:phoffric
Comment Utility
You got a std::bad_alloc exception which means that you ran out of memory as you were allocating it.
   http://www.cplusplus.com/reference/new/bad_alloc/

When you do you vector push_back, you are often allocating memory. Doing that 3 million times apparently is more than your platform can handle. If you drop the number down, to say, 1 million, you might have better luck.
0
 
LVL 37

Expert Comment

by:Neil Russell
Comment Utility
Well im not at a machine where I can compile and test but a quick look says WOW!

You declare binrec but it is never used that I can see.
You are constructing 3 million nameval structures in code and never freeing any of them at all!

I'll have a proper look in a bit if no body answers you in detail.
0
 
LVL 32

Assisted Solution

by:phoffric
phoffric earned 100 total points
Comment Utility
Your vector push back is frequently having to resize the vector. This is probably causing fragmentation, so that as you try for the larger sizes, you cannot find a contiguous memory region. You can reserve the required memory in advance. The following worked on my laptop:
   records.reserve(3000000);
   for (cnt=0;cnt<3000000;cnt++)

Open in new window

http://www.cplusplus.com/reference/vector/vector/reserve/?kw=vector%3A%3Areserve
0
 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
Thanks all.
Phoffric,
It is OK for your way. But if I've increased that to this
	records.reserve(8000000);
	for (cnt=0;cnt<8000000;cnt++)
	...

Open in new window


instead of the original count. I still get this error.
t864.png
0
 
LVL 37

Expert Comment

by:Neil Russell
Comment Utility
Computers have limits just like us humans.  You are now asking for 8 million chunks of contiguous memory the size of your structure. Thats a LOT of memory.
0
 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
How to resolve the problem now? Thanks
0
 
LVL 37

Expert Comment

by:Neil Russell
Comment Utility
ok, back to basics. Why do you want to create and handle IN MEMORY over 8 Million randomly generated strings?

Is this homework? An exercise? for a reason?

Whats wrong with databases? They are pretty good at handling millions of pieces of information.
0
 
LVL 37

Expert Comment

by:Neil Russell
Comment Utility
if you run...

#include "stdafx.h"
#include <stdio.h>
#include <string.h>
#include <fstream>
#include <string>
#include <ctype.h>
#include <time.h>
#include <process.h>
#include <vector>
#include <iostream>
#include <algorithm>
#include <iomanip>
using namespace std;

struct nameval
{
	char fld_nm[100];
	wchar_t fld_nm_t[100];
	int fld_len;
	int fld_val;
};
nameval binrec;

bool LessComp(const nameval& a1, const nameval& a2)
{
  if(strcmp(a1.fld_nm, a2.fld_nm) < 0) return true;
  if(strcmp(a1.fld_nm, a2.fld_nm) > 0) return false;
  if(a1.fld_val < a2.fld_val) return true;
  return false;
}

int main()
{
    std::vector<nameval> records;

  std::cout << "max_size: " << records.max_size() << "\n";
  return 0;
}

Open in new window


What do you get as an answer?
0
 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
Thanks all.
Neilsr,
I get 13944699.
0
 
LVL 37

Expert Comment

by:Neil Russell
Comment Utility
ok that is the ABSOLUTE THEORETICAL maximum size that you could allocate to your vector Records. In reality it will never be anywhere near that and thats only if you dont reorder or use it for anything meaningful.
0
 
LVL 32

Expert Comment

by:phoffric
Comment Utility
Do a sizeof(nameval) and multiply that by the number you wrote, 13944699, and it will probably be close to 4GB, which apparently on your platform is the largest chunk on the heap that you can allocate.

32-bit can hold 2-bit (4) * 10-bit (1K) * 20-bit (1M) = 4*1K*1M = 4GB.

If you get a 64-bit platform and a 64-bit compiler, and more physical memory, then you may be able to reach you larger count.

Reason to keep data in memory is for speed. Databases are extremely much slower. Fastest way to deal with your problem is to memory map your data to a file and let the OS handle the swapping so that you don't actually have read/write to disk in your code. But this approach is outside the scope of this question.
0
 
LVL 32

Expert Comment

by:phoffric
Comment Utility
Since we resolved the original 3 million question, this question is resolved. If you change the question to 8 million count, probably some hardware software changes are in order. If you change it to 80 billion which is a reasonable question in some situations, and still need high speed (eliminating a relational database), the questions are different and potentially much more complex.

You didn't mention where the bad alloc exception occurred. Run the program in the debugger and see where it stops. You may not have been able to reserve the memory dynamically. You could try using a plain array outside of main (i.e., not on the stack) to see if that helps.

You should mention the OS bit size, and the compiler bit-size.
0
 
LVL 32

Expert Comment

by:phoffric
Comment Utility
Neilsr wrote:
Why do you want to create and handle IN MEMORY over 8 Million randomly generated strings?
 Is this homework? An exercise? for a reason?
You addressed other questions from him but did not address this one. The question does appear to be either homework or self-study, so I have limited my answers.
0
 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
Hi,
Thanks all.
Phoffric,
How to resolve the real problem, to handle 8 millions records by the program?
0
 
LVL 32

Expert Comment

by:phoffric
Comment Utility
What OS (including bit size), compiler (including bit size capability), and how much memory do you have?

From the answer you gave earlier, I believe you have a 32-bit platform or possibly a 32-bit compiler. Even if you have enough memory to handle 8 million records, you may not be able to find a contiguous virtual address region for a vector.
0
 
LVL 32

Expert Comment

by:phoffric
Comment Utility
IMO, the real problem is the one you posted in your OP.

Also, please address this question:
Neilsr wrote:
Why do you want to create and handle IN MEMORY over 8 Million randomly generated strings?
  Is this homework? An exercise? for a reason?
You addressed other questions from him but did not address this one.
0
 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
It is in Win 7 machine having 8 GB Ram and I am using VS 2010 to handle the program.
0
 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
This is an exercise, to check the way to handle that amount of records, through VS 2010.
0
 
LVL 32

Expert Comment

by:sarabande
Comment Utility
How to resolve the real problem, to handle 8 millions records by the program?
there are a few options:

1. try to minimize the record size. in your code you were using only 20 bytes for the random string and an integer value.

    that would give a size of 32 bytes while your current size is 308.  conversions from wide to narrow
    characters can be made on the fly and length of text can be determined by checking the text.

2. use a container that doesn't need contiguous memory.

    for your purpose a std::set could be fine which automatically would sort your records, if you provide
    an operator< member function (the implementation would be nearly identical to your LessComp function)
   
3. if you nevertheless run into memory issues, you would need to use a database or a keyed-access file.
   
4. memory-mapped files rarely are an option as they would require contiguous memory as well
    for the view you were looking at.  so actually you would have to do swapping yourself, what is
    much work with a small chance of getting a better over-all speed.

5. a probably better idea would be to use a self-developed b-tree  where you would use a cache for the most
    frequently used b-tree records.

Sara
0
 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
Thanks. Can I see the example using b-tree?
0
 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
Sorry, other than b-tree, I still want to see there is any other way to resolve the same memory problem, which I originally mentioned. Thanks
0
 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
Sorry, as I had been stuck with the memory problem, to the vector in the project. How to resolve the memory problem?
0
IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

 
LVL 32

Expert Comment

by:sarabande
Comment Utility
the b-tree solution is the most difficult suggestion I made. it is nothing where I could give you a quick sample of (even if the academic reservations phoffric has told you wouldn't forbid that anyway), especially not as you needed a "customized" b-tree solution which was using a cache rather than keeping the whole tree in memory. a tree is a container with nodes and leafs. nodes were pointing to other nodes and leafs. because of the pointers you could store the tree as a file and load and unload parts of it dynamically. if you want to go this way, you would need at least two steps. first step is to implement a b-tree container in memory. second step is to make this tree persistent and add a cache. you would need to ask new questions for this and always show your own code such that we are able to assist you rather than doing the work for you.

did you check my other suggestions? what about making the structure more compact? that would allow you to store nearly 10 times more in memory. when using a std::set as basic container it allows to use all heap memory and relieve you of requesting more and more contiguous memory. if you apply both you should be able to store 8 million strings in memory because vs2010 would provide sufficient virtual memory for your program.

Sara
0
 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
Thanks. Can I see the exact way to use std::set instead, to the above codes?
0
 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
I mean if there is one code example to std::set?
0
 
LVL 32

Expert Comment

by:sarabande
Comment Utility
see http://msdn.microsoft.com/en-us/library/e8wh7665.aspx for details.

you now have std::vector<nameval>. you would replace it by std::set<nameval>. instead of push_back you would use std::set::insert function.

then add operator< function to structure nameval:

struct nameval
{
    ....
    bool operator< (const nameval & a2) const;
};

Open in new window


the implementation of the function is identical to your LessComp function beside that you omit 'a1.'  (because a1 is the 'this' object).

after you stored the keys the set already is sorted.

you would use a std::set<nameval>::iterator to iterate the container.

Sara

Sara
0
 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
Sorry, please see these errors

1>------ Rebuild All started: Project: SaveBinaryFile, Configuration: Debug Win32 ------
1>  stdafx.cpp
1>  SaveBinaryFile.cpp
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(44): error C2039: 'set' : is not a member of 'std'
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(44): error C2065: 'set' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(44): error C2275: 'nameval' : illegal use of this type as an expression
1>          c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(21) : see declaration of 'nameval'
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(44): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(46): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(46): error C2228: left of '.reserve' must have class/struct/union
1>          type is ''unknown-type''
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(72): error C3083: 'set': the symbol to the left of a '::' must be a type
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(72): error C2039: 'insert' : is not a member of 'std'
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(72): error C3861: 'insert': identifier not found
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(74): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(74): error C2228: left of '.begin' must have class/struct/union
1>          type is ''unknown-type''
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(74): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(74): error C2228: left of '.end' must have class/struct/union
1>          type is ''unknown-type''
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(74): error C2780: 'void std::sort(_RanIt,_RanIt)' : expects 2 arguments - 3 provided
1>          c:\program files (x86)\microsoft visual studio 10.0\vc\include\algorithm(3639) : see declaration of 'std::sort'
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(79): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(79): error C2228: left of '.size' must have class/struct/union
1>          type is ''unknown-type''
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(84): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(94): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(94): error C2228: left of '.size' must have class/struct/union
1>          type is ''unknown-type''
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(97): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(97): error C2228: left of '.fld_nm' must have class/struct/union
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(97): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(97): error C2228: left of '.fld_len' must have class/struct/union
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(97): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(97): error C2228: left of '.fld_nm_t' must have class/struct/union
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(97): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(97): error C2228: left of '.fld_val' must have class/struct/union
========== Rebuild All: 0 succeeded, 1 failed, 0 skipped ==========

Open in new window


from these codes

//

#include "stdafx.h"
#include <stdio.h>
#include <string.h>
#include <fstream>
#include <string>
#include <ctype.h>
#include <time.h>
#include <process.h>
#include <vector>
#include <iostream>
#include <algorithm>
#include <iomanip>
using namespace std;

struct nameval
{
	bool operator< (const nameval & a2) const;
	char fld_nm[100];
	wchar_t fld_nm_t[100]; 
	int fld_len;
	int fld_val;
};
nameval binrec;

bool LessComp(const nameval& a1, const nameval& a2)
{
  if(strcmp(a1.fld_nm, a2.fld_nm) < 0) return true;
  if(strcmp(a1.fld_nm, a2.fld_nm) > 0) return false;
  if(a1.fld_val < a2.fld_val) return true;
  return false;
}

int main()
{
    std::set<nameval> records;
    int cnt;
	records.reserve(8000000);
	for (cnt=0;cnt<8000000;cnt++)
	{
		nameval val={0};
		int j;
		for (j=0;j<20;j++)
		{
			val.fld_nm[j] += (char)(rand () % 58 + 64);
		}
		
		wchar_t wbuf[100]={0};
		mbstowcs(wbuf,val.fld_nm,_countof(val.fld_nm));
		wmemcpy(val.fld_nm_t,wbuf,100);

		val.fld_val=cnt;
		std::set::insert(val);
	}
	std::sort(records.begin(), records.end(), LessComp);

	std::ofstream ostrm("c:\\dp4\\flout.bin", std::ios::binary | std::ios::out );
	if (ostrm.is_open())
	{
		for (size_t i = 0; i < records.size(); i++)
		{
			ostrm.write((char *)&records[i], sizeof(nameval));
		}
	}

	std::wofstream ostrm2("c:\\dp4\\flout.ord", std::ios::out );
	if (ostrm2.is_open())
	{
		for (size_t i = 0; i < records.size(); i++)
		{
			ostrm2 << "\"" << records[i].fld_nm << "\" " << records[i].fld_len << ' '  << records[i].fld_nm_t << ' '<< records[i].fld_val << '\0' << '\n';
		}
	}

	return 0;
}

Open in new window

0
 
LVL 32

Expert Comment

by:sarabande
Comment Utility
1. add

#include <set> 

Open in new window


to the include statements.

2. the struct nameval needs the operator< member function.

you may implement it by

//forward declaration
bool LessComp(const nameval & a1, const nameval & a2);
struct nameval
{
     ....
     bool operator< (const nameval & a2) const
     {
           return LessComp(*this, a2);
     }

Open in new window


3. the insert function is a member function of std::set, same as push_back is a member function of std::vector.

    so the code to store is:

     
records.insert(val);

Open in new window


4. std::sort is not possible for std::set because the items are already sorted in the set.

5. the for loop to write to the file now must use an iterator:

   
for (std::set<namval>::iterator it = records.begin(); it != records.end(); ++it)

Open in new window


    in the loop you would not use records[j] but (*it) instead.

look in the reference for std::set I posted for sample code and details.

Sara
0
 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
Sorry, I still get these errors
1>------ Rebuild All started: Project: SaveBinaryFile, Configuration: Debug Win32 ------
1>  stdafx.cpp
1>  SaveBinaryFile.cpp
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(3): warning C4627: '#include <set>': skipped when looking for precompiled header use
1>          Add directive to 'StdAfx.h' or rebuild precompiled header
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(29): error C3861: 'LessComp': identifier not found
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(48): error C2039: 'set' : is not a member of 'std'
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(48): error C2065: 'set' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(48): error C2275: 'nameval' : illegal use of this type as an expression
1>          c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(22) : see declaration of 'nameval'
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(48): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(76): error C3083: 'set': the symbol to the left of a '::' must be a type
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(76): error C2039: 'insert' : is not a member of 'std'
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(76): error C3861: 'insert': identifier not found
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(78): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(78): error C2228: left of '.begin' must have class/struct/union
1>          type is ''unknown-type''
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(78): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(78): error C2228: left of '.end' must have class/struct/union
1>          type is ''unknown-type''
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(78): error C2780: 'void std::sort(_RanIt,_RanIt)' : expects 2 arguments - 3 provided
1>          c:\program files (x86)\microsoft visual studio 10.0\vc\include\algorithm(3639) : see declaration of 'std::sort'
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(83): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(83): error C2228: left of '.size' must have class/struct/union
1>          type is ''unknown-type''
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(88): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(98): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(98): error C2228: left of '.size' must have class/struct/union
1>          type is ''unknown-type''
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(101): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(101): error C2228: left of '.fld_nm' must have class/struct/union
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(101): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(101): error C2228: left of '.fld_len' must have class/struct/union
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(101): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(101): error C2228: left of '.fld_nm_t' must have class/struct/union
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(101): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(101): error C2228: left of '.fld_val' must have class/struct/union
========== Rebuild All: 0 succeeded, 1 failed, 0 skipped ==========

Open in new window


due to these?

//

#include <set>
#include "stdafx.h"
#include <stdio.h>
#include <string.h>
#include <fstream>
#include <string>
#include <ctype.h>
#include <time.h>
#include <process.h>
#include <vector>
#include <iostream>
#include <algorithm>
#include <iomanip>
using namespace std;

bool LessComp(const nameval& a1, const nameval& a2)
{
  if(strcmp(a1.fld_nm, a2.fld_nm) < 0) return true;
  if(strcmp(a1.fld_nm, a2.fld_nm) > 0) return false;
  if(a1.fld_val < a2.fld_val) return true;
  return false;
}

struct nameval
{
	char fld_nm[100];
	wchar_t fld_nm_t[100]; 
     bool operator< (const nameval & a2) const
     {
           return LessComp(*this, a2);
     }
	int fld_len;
	int fld_val;
};
nameval binrec;

int main()
{
    std::set<nameval> records;
    int cnt;
	for (cnt=0;cnt<8000000;cnt++)
	{
		nameval val={0};
		int j;
		for (j=0;j<20;j++)
		{
			val.fld_nm[j] += (char)(rand () % 58 + 64);
		}
		
		wchar_t wbuf[100]={0};
		mbstowcs(wbuf,val.fld_nm,_countof(val.fld_nm));
		wmemcpy(val.fld_nm_t,wbuf,100);

		val.fld_val=cnt;
		std::set::insert(val);
	}
	std::sort(records.begin(), records.end(), LessComp);

	std::ofstream ostrm("c:\\dp4\\flout.bin", std::ios::binary | std::ios::out );
	if (ostrm.is_open())
	{
		for (size_t i = 0; i < records.size(); i++)
		{
			ostrm.write((char *)&records[i], sizeof(nameval));
		}
	}

	std::wofstream ostrm2("c:\\dp4\\flout.ord", std::ios::out );
	if (ostrm2.is_open())
	{
		for (size_t i = 0; i < records.size(); i++)
		{
			ostrm2 << "\"" << records[i].fld_nm << "\" " << records[i].fld_len << ' '  << records[i].fld_nm_t << ' '<< records[i].fld_val << '\0' << '\n';
		}
	}

	return 0;
}

Open in new window

0
 
LVL 32

Expert Comment

by:sarabande
Comment Utility
#include <set>': skipped when looking for precompiled header use
you need to add the #include <set> below #include stdafx.h.

all statements above stdafx.h will not be seen by the precompiler cause it uses a precompiled header for them.

std::set::insert(val);
the statement is wrong. see my previous comment.

std::sort(records.begin(), records.end(), LessComp);
as told, the statement must be removed.

also the for loop and access to records is still wrong.


Sara
0
 
LVL 32

Expert Comment

by:sarabande
Comment Utility
HuaMinChen, I've got the impression that the current program is one of your first c++ programs. if that is true and given that you were visiting a course that is suitable for the level of your experience, I wonder why a beginner's exercise should have such "heavy" requirements such as to handle 8 millions of strings in memory.

can you post the full requirement of the exercise such that we could give advice which is appropriate to the given task and your current knowledge. thank you.

Sara
0
 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
Sara,
As you gave the advice to use "set", can you please advise how to correct this line?

		std::set::insert(val);
		...

Open in new window

Phoffric,
It is not good to close the thread while there is error got due to the advice given, right?
0
 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
Sara,
I do expect to resolve the current memory problem to the project. Thanks.
0
 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
And I did need to check such way in C++ project to handle extremely big vector list and that's all.
0
 
LVL 32

Expert Comment

by:sarabande
Comment Utility
how to correct this line?

I already told you the statement in one of my previous comments:

records.insert(val);

Open in new window


insert is a member function of std::set. you defined a container 'std::set<nameval> records;'  and can now call member functions by using operator.

Sara
0
 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
Hi Sara,
how can I now refer to each record instead of "records(i)" (that is the one I previously used)?
0
 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
Please note that I cannot use [] above so I have used () instead.
0
 
LVL 32

Assisted Solution

by:sarabande
sarabande earned 400 total points
Comment Utility
how can I now refer to each record instead of "records(i)" (that is the one I previously used)?
you should check my previous posts if you have questions because I already posted code for all your recent questions.

a std::set internally is organized as a tree. so it cannot accessed by index. however, a tree can iterated by using an iterator so going from one item to the next item.

for (std::set<namval>::iterator it = records.begin(); it != records.end(); ++it)
{
        ....

Open in new window

all stl containers can be accessed by iterator as in the above statement. the iterator is pointing to the current item and you can get a reference to the current set item by using the dereference operator* :

....
nameval & record = *it;
ostrm.write((char *)&record, sizeof(nameval));

Open in new window

or alternatively:

....
ostrm.write((char *)&(*it), sizeof(nameval));

Open in new window


Sara
0
 
LVL 32

Expert Comment

by:sarabande
Comment Utility
also note, that std::set already is an ordered container.

Sara
0
 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
Sorry, I get these errors
1>c:\savebinaryfile tst\savebinaryfile\savebinaryfile.cpp(84): error C2065: 'namval' : undeclared identifier
1>c:\savebinaryfile tst\savebinaryfile\savebinaryfile.cpp(84): error C2955: 'std::set' : use of class template requires template argument list

Open in new window


due to this line
		for (std::set<namval>::iterator it = records.begin(); it != records.end(); ++it)
		...

Open in new window

0
 
LVL 32

Expert Comment

by:sarabande
Comment Utility
it should be 'nameval' and not 'namval'.

Sara
0
 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
Thanks a lot.
how to correct the last line below?
		for (std::set<nameval>::iterator it = records.begin(); it != records.end(); ++it)
		{
			ostrm2 << "\"" << records[i].fld_nm << "\" " << records[i].fld_len << ' '  << records[i].fld_nm_uni << ' '<< records[i].fld_val << '\0' << '\n';
			...

Open in new window

0
 
LVL 32

Assisted Solution

by:sarabande
sarabande earned 400 total points
Comment Utility
replace the records[ i ]  by (*it) for all occurrences.

Sara
0
 
LVL 32

Assisted Solution

by:sarabande
sarabande earned 400 total points
Comment Utility
alternatively you could use 'it->' instead of 'records[ i].' for example

ostrm2 << "\"" << it->fld_nm 

Open in new window


the iterator is similar to a pointer to the current item.

Sara
0
 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
Sorry, most probably one last thing is, how to correct this below?
bool LessComp(const nameval& a1, const nameval& a2)
{
  if(strcmp(a1.fld_nm, a2.fld_nm) < 0) return true;
  if(strcmp(a1.fld_nm, a2.fld_nm) > 0) return false;
  if(a1.fld_val < a2.fld_val) return true;
  return false;
}

Open in new window

0
 
LVL 32

Accepted Solution

by:
sarabande earned 400 total points
Comment Utility
you can leave it as it is and call it from operator< function.

I posted already code for it above:

//forward declarations
struct nameval;
bool LessComp(const nameval & a1, const nameval & a2);

struct nameval
{
     ....
     bool operator< (const nameval & a2) const
     {
           return LessComp(*this, a2);
     }

Open in new window


note, the forward declarations are necessary or you get errors.

alternatively you could remove the LessComp function and implement operator< like

struct nameval
{
     ....
     bool operator< (const nameval & a2) const
     {
           if(strcmp(fld_nm, a2.fld_nm) < 0) return true;
           if(strcmp(fld_nm, a2.fld_nm) > 0) return false;
           if (fld_val < a2.fld_val) return true;
           return false;
     }

Open in new window


note, the code is from the LessComp function and the only difference is that the a1 is not passed but is the (*this) object which can be omitted.

Sara
0
 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
0

Featured Post

Threat Intelligence Starter Resources

Integrating threat intelligence can be challenging, and not all companies are ready. These resources can help you build awareness and prepare for defense.

Join & Write a Comment

Introduction: Dialogs (1) modal - maintaining the database. Continuing from the ninth article about sudoku.   You might have heard of modal and modeless dialogs.  Here with this Sudoku application will we use one of each type: a modal dialog …
This article shows you how to optimize memory allocations in C++ using placement new. Applicable especially to usecases dealing with creation of large number of objects. A brief on problem: Lets take example problem for simplicity: - I have a G…
Video by: Grant
The goal of this video is to provide viewers with basic examples to understand and use for-loops in the C programming language.
This video will show you how to get GIT to work in Eclipse.   It will walk you through how to install the EGit plugin in eclipse and how to checkout an existing repository.

728 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now