Memory problem

Hi,
when running the codes below
https://dl.dropboxusercontent.com/u/40211031/SaveBinaryFile.cpp

to generate the list. I get this error
https://dl.dropboxusercontent.com/u/40211031/t863.png

how to resolve it?
LVL 12
HuaMin ChenProblem resolverAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Neil RussellTechnical Development LeadCommented:
Please place your code and images on EE where we can ALL see them. A lot of corporate networks will BLOCK Dropbox, and for very good reason :D
0
HuaMin ChenProblem resolverAuthor Commented:
Here are the codes
//

#include "stdafx.h"
#include <stdio.h>
#include <string.h>
#include <fstream>
#include <string>
#include <ctype.h>
#include <time.h>
#include <process.h>
#include <vector>
#include <iostream>
#include <algorithm>
#include <iomanip>
using namespace std;

struct nameval
{
	char fld_nm[100];
	wchar_t fld_nm_t[100];
	int fld_len;
	int fld_val;
};
nameval binrec;

bool LessComp(const nameval& a1, const nameval& a2)
{
  if(strcmp(a1.fld_nm, a2.fld_nm) < 0) return true;
  if(strcmp(a1.fld_nm, a2.fld_nm) > 0) return false;
  if(a1.fld_val < a2.fld_val) return true;
  return false;
}

int main()
{
    std::vector<nameval> records;
    int cnt;
	for (cnt=0;cnt<3000000;cnt++)
	{
		nameval val={0};
		int j;
		for (j=0;j<20;j++)
		{
			val.fld_nm[j] += (char)(rand () % 58 + 64);
		}

		wchar_t wbuf[100]={0};
		mbstowcs(wbuf,val.fld_nm,_countof(val.fld_nm));
		wmemcpy(val.fld_nm_t,wbuf,100);

		val.fld_val=cnt;
		records.push_back(val);
	}
	std::sort(records.begin(), records.end(), LessComp);

	std::ofstream ostrm("c:\\dp4\\flout.bin", std::ios::binary | std::ios::out );
	if (ostrm.is_open())
	{
		for (size_t i = 0; i < records.size(); i++)
		{
			ostrm.write((char *)&records[i], sizeof(nameval));
		}
	}

	std::wofstream ostrm2("c:\\dp4\\flout.ord", std::ios::out );
	if (ostrm2.is_open())
	{
		for (size_t i = 0; i < records.size(); i++)
		{
			ostrm2 << "\"" << records[i].fld_nm << "\" " << records[i].fld_len << ' '  << records[i].fld_nm_t << ' '<< records[i].fld_val << '\n';
		}
	}

	return 0;
}

Open in new window

and here is the image
t863.png
0
HuaMin ChenProblem resolverAuthor Commented:
Thanks
0
Learn SQL Server Core 2016

This course will introduce you to SQL Server Core 2016, as well as teach you about SSMS, data tools, installation, server configuration, using Management Studio, and writing and executing queries.

phoffricCommented:
You got a std::bad_alloc exception which means that you ran out of memory as you were allocating it.
   http://www.cplusplus.com/reference/new/bad_alloc/

When you do you vector push_back, you are often allocating memory. Doing that 3 million times apparently is more than your platform can handle. If you drop the number down, to say, 1 million, you might have better luck.
0
Neil RussellTechnical Development LeadCommented:
Well im not at a machine where I can compile and test but a quick look says WOW!

You declare binrec but it is never used that I can see.
You are constructing 3 million nameval structures in code and never freeing any of them at all!

I'll have a proper look in a bit if no body answers you in detail.
0
phoffricCommented:
Your vector push back is frequently having to resize the vector. This is probably causing fragmentation, so that as you try for the larger sizes, you cannot find a contiguous memory region. You can reserve the required memory in advance. The following worked on my laptop:
   records.reserve(3000000);
   for (cnt=0;cnt<3000000;cnt++)

Open in new window

http://www.cplusplus.com/reference/vector/vector/reserve/?kw=vector%3A%3Areserve
0
HuaMin ChenProblem resolverAuthor Commented:
Thanks all.
Phoffric,
It is OK for your way. But if I've increased that to this
	records.reserve(8000000);
	for (cnt=0;cnt<8000000;cnt++)
	...

Open in new window


instead of the original count. I still get this error.
t864.png
0
Neil RussellTechnical Development LeadCommented:
Computers have limits just like us humans.  You are now asking for 8 million chunks of contiguous memory the size of your structure. Thats a LOT of memory.
0
HuaMin ChenProblem resolverAuthor Commented:
How to resolve the problem now? Thanks
0
Neil RussellTechnical Development LeadCommented:
ok, back to basics. Why do you want to create and handle IN MEMORY over 8 Million randomly generated strings?

Is this homework? An exercise? for a reason?

Whats wrong with databases? They are pretty good at handling millions of pieces of information.
0
Neil RussellTechnical Development LeadCommented:
if you run...

#include "stdafx.h"
#include <stdio.h>
#include <string.h>
#include <fstream>
#include <string>
#include <ctype.h>
#include <time.h>
#include <process.h>
#include <vector>
#include <iostream>
#include <algorithm>
#include <iomanip>
using namespace std;

struct nameval
{
	char fld_nm[100];
	wchar_t fld_nm_t[100];
	int fld_len;
	int fld_val;
};
nameval binrec;

bool LessComp(const nameval& a1, const nameval& a2)
{
  if(strcmp(a1.fld_nm, a2.fld_nm) < 0) return true;
  if(strcmp(a1.fld_nm, a2.fld_nm) > 0) return false;
  if(a1.fld_val < a2.fld_val) return true;
  return false;
}

int main()
{
    std::vector<nameval> records;

  std::cout << "max_size: " << records.max_size() << "\n";
  return 0;
}

Open in new window


What do you get as an answer?
0
HuaMin ChenProblem resolverAuthor Commented:
Thanks all.
Neilsr,
I get 13944699.
0
Neil RussellTechnical Development LeadCommented:
ok that is the ABSOLUTE THEORETICAL maximum size that you could allocate to your vector Records. In reality it will never be anywhere near that and thats only if you dont reorder or use it for anything meaningful.
0
phoffricCommented:
Do a sizeof(nameval) and multiply that by the number you wrote, 13944699, and it will probably be close to 4GB, which apparently on your platform is the largest chunk on the heap that you can allocate.

32-bit can hold 2-bit (4) * 10-bit (1K) * 20-bit (1M) = 4*1K*1M = 4GB.

If you get a 64-bit platform and a 64-bit compiler, and more physical memory, then you may be able to reach you larger count.

Reason to keep data in memory is for speed. Databases are extremely much slower. Fastest way to deal with your problem is to memory map your data to a file and let the OS handle the swapping so that you don't actually have read/write to disk in your code. But this approach is outside the scope of this question.
0
phoffricCommented:
Since we resolved the original 3 million question, this question is resolved. If you change the question to 8 million count, probably some hardware software changes are in order. If you change it to 80 billion which is a reasonable question in some situations, and still need high speed (eliminating a relational database), the questions are different and potentially much more complex.

You didn't mention where the bad alloc exception occurred. Run the program in the debugger and see where it stops. You may not have been able to reserve the memory dynamically. You could try using a plain array outside of main (i.e., not on the stack) to see if that helps.

You should mention the OS bit size, and the compiler bit-size.
0
phoffricCommented:
Neilsr wrote:
Why do you want to create and handle IN MEMORY over 8 Million randomly generated strings?
 Is this homework? An exercise? for a reason?
You addressed other questions from him but did not address this one. The question does appear to be either homework or self-study, so I have limited my answers.
0
HuaMin ChenProblem resolverAuthor Commented:
Hi,
Thanks all.
Phoffric,
How to resolve the real problem, to handle 8 millions records by the program?
0
phoffricCommented:
What OS (including bit size), compiler (including bit size capability), and how much memory do you have?

From the answer you gave earlier, I believe you have a 32-bit platform or possibly a 32-bit compiler. Even if you have enough memory to handle 8 million records, you may not be able to find a contiguous virtual address region for a vector.
0
phoffricCommented:
IMO, the real problem is the one you posted in your OP.

Also, please address this question:
Neilsr wrote:
Why do you want to create and handle IN MEMORY over 8 Million randomly generated strings?
  Is this homework? An exercise? for a reason?
You addressed other questions from him but did not address this one.
0
HuaMin ChenProblem resolverAuthor Commented:
It is in Win 7 machine having 8 GB Ram and I am using VS 2010 to handle the program.
0
HuaMin ChenProblem resolverAuthor Commented:
This is an exercise, to check the way to handle that amount of records, through VS 2010.
0
sarabandeCommented:
How to resolve the real problem, to handle 8 millions records by the program?
there are a few options:

1. try to minimize the record size. in your code you were using only 20 bytes for the random string and an integer value.

    that would give a size of 32 bytes while your current size is 308.  conversions from wide to narrow
    characters can be made on the fly and length of text can be determined by checking the text.

2. use a container that doesn't need contiguous memory.

    for your purpose a std::set could be fine which automatically would sort your records, if you provide
    an operator< member function (the implementation would be nearly identical to your LessComp function)
   
3. if you nevertheless run into memory issues, you would need to use a database or a keyed-access file.
   
4. memory-mapped files rarely are an option as they would require contiguous memory as well
    for the view you were looking at.  so actually you would have to do swapping yourself, what is
    much work with a small chance of getting a better over-all speed.

5. a probably better idea would be to use a self-developed b-tree  where you would use a cache for the most
    frequently used b-tree records.

Sara
0
HuaMin ChenProblem resolverAuthor Commented:
Thanks. Can I see the example using b-tree?
0
HuaMin ChenProblem resolverAuthor Commented:
Sorry, other than b-tree, I still want to see there is any other way to resolve the same memory problem, which I originally mentioned. Thanks
0
HuaMin ChenProblem resolverAuthor Commented:
Sorry, as I had been stuck with the memory problem, to the vector in the project. How to resolve the memory problem?
0
sarabandeCommented:
the b-tree solution is the most difficult suggestion I made. it is nothing where I could give you a quick sample of (even if the academic reservations phoffric has told you wouldn't forbid that anyway), especially not as you needed a "customized" b-tree solution which was using a cache rather than keeping the whole tree in memory. a tree is a container with nodes and leafs. nodes were pointing to other nodes and leafs. because of the pointers you could store the tree as a file and load and unload parts of it dynamically. if you want to go this way, you would need at least two steps. first step is to implement a b-tree container in memory. second step is to make this tree persistent and add a cache. you would need to ask new questions for this and always show your own code such that we are able to assist you rather than doing the work for you.

did you check my other suggestions? what about making the structure more compact? that would allow you to store nearly 10 times more in memory. when using a std::set as basic container it allows to use all heap memory and relieve you of requesting more and more contiguous memory. if you apply both you should be able to store 8 million strings in memory because vs2010 would provide sufficient virtual memory for your program.

Sara
0
HuaMin ChenProblem resolverAuthor Commented:
Thanks. Can I see the exact way to use std::set instead, to the above codes?
0
HuaMin ChenProblem resolverAuthor Commented:
I mean if there is one code example to std::set?
0
sarabandeCommented:
see http://msdn.microsoft.com/en-us/library/e8wh7665.aspx for details.

you now have std::vector<nameval>. you would replace it by std::set<nameval>. instead of push_back you would use std::set::insert function.

then add operator< function to structure nameval:

struct nameval
{
    ....
    bool operator< (const nameval & a2) const;
};

Open in new window


the implementation of the function is identical to your LessComp function beside that you omit 'a1.'  (because a1 is the 'this' object).

after you stored the keys the set already is sorted.

you would use a std::set<nameval>::iterator to iterate the container.

Sara

Sara
0
HuaMin ChenProblem resolverAuthor Commented:
Sorry, please see these errors

1>------ Rebuild All started: Project: SaveBinaryFile, Configuration: Debug Win32 ------
1>  stdafx.cpp
1>  SaveBinaryFile.cpp
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(44): error C2039: 'set' : is not a member of 'std'
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(44): error C2065: 'set' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(44): error C2275: 'nameval' : illegal use of this type as an expression
1>          c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(21) : see declaration of 'nameval'
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(44): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(46): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(46): error C2228: left of '.reserve' must have class/struct/union
1>          type is ''unknown-type''
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(72): error C3083: 'set': the symbol to the left of a '::' must be a type
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(72): error C2039: 'insert' : is not a member of 'std'
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(72): error C3861: 'insert': identifier not found
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(74): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(74): error C2228: left of '.begin' must have class/struct/union
1>          type is ''unknown-type''
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(74): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(74): error C2228: left of '.end' must have class/struct/union
1>          type is ''unknown-type''
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(74): error C2780: 'void std::sort(_RanIt,_RanIt)' : expects 2 arguments - 3 provided
1>          c:\program files (x86)\microsoft visual studio 10.0\vc\include\algorithm(3639) : see declaration of 'std::sort'
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(79): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(79): error C2228: left of '.size' must have class/struct/union
1>          type is ''unknown-type''
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(84): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(94): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(94): error C2228: left of '.size' must have class/struct/union
1>          type is ''unknown-type''
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(97): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(97): error C2228: left of '.fld_nm' must have class/struct/union
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(97): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(97): error C2228: left of '.fld_len' must have class/struct/union
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(97): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(97): error C2228: left of '.fld_nm_t' must have class/struct/union
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(97): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(97): error C2228: left of '.fld_val' must have class/struct/union
========== Rebuild All: 0 succeeded, 1 failed, 0 skipped ==========

Open in new window


from these codes

//

#include "stdafx.h"
#include <stdio.h>
#include <string.h>
#include <fstream>
#include <string>
#include <ctype.h>
#include <time.h>
#include <process.h>
#include <vector>
#include <iostream>
#include <algorithm>
#include <iomanip>
using namespace std;

struct nameval
{
	bool operator< (const nameval & a2) const;
	char fld_nm[100];
	wchar_t fld_nm_t[100]; 
	int fld_len;
	int fld_val;
};
nameval binrec;

bool LessComp(const nameval& a1, const nameval& a2)
{
  if(strcmp(a1.fld_nm, a2.fld_nm) < 0) return true;
  if(strcmp(a1.fld_nm, a2.fld_nm) > 0) return false;
  if(a1.fld_val < a2.fld_val) return true;
  return false;
}

int main()
{
    std::set<nameval> records;
    int cnt;
	records.reserve(8000000);
	for (cnt=0;cnt<8000000;cnt++)
	{
		nameval val={0};
		int j;
		for (j=0;j<20;j++)
		{
			val.fld_nm[j] += (char)(rand () % 58 + 64);
		}
		
		wchar_t wbuf[100]={0};
		mbstowcs(wbuf,val.fld_nm,_countof(val.fld_nm));
		wmemcpy(val.fld_nm_t,wbuf,100);

		val.fld_val=cnt;
		std::set::insert(val);
	}
	std::sort(records.begin(), records.end(), LessComp);

	std::ofstream ostrm("c:\\dp4\\flout.bin", std::ios::binary | std::ios::out );
	if (ostrm.is_open())
	{
		for (size_t i = 0; i < records.size(); i++)
		{
			ostrm.write((char *)&records[i], sizeof(nameval));
		}
	}

	std::wofstream ostrm2("c:\\dp4\\flout.ord", std::ios::out );
	if (ostrm2.is_open())
	{
		for (size_t i = 0; i < records.size(); i++)
		{
			ostrm2 << "\"" << records[i].fld_nm << "\" " << records[i].fld_len << ' '  << records[i].fld_nm_t << ' '<< records[i].fld_val << '\0' << '\n';
		}
	}

	return 0;
}

Open in new window

0
sarabandeCommented:
1. add

#include <set> 

Open in new window


to the include statements.

2. the struct nameval needs the operator< member function.

you may implement it by

//forward declaration
bool LessComp(const nameval & a1, const nameval & a2);
struct nameval
{
     ....
     bool operator< (const nameval & a2) const
     {
           return LessComp(*this, a2);
     }

Open in new window


3. the insert function is a member function of std::set, same as push_back is a member function of std::vector.

    so the code to store is:

     
records.insert(val);

Open in new window


4. std::sort is not possible for std::set because the items are already sorted in the set.

5. the for loop to write to the file now must use an iterator:

   
for (std::set<namval>::iterator it = records.begin(); it != records.end(); ++it)

Open in new window


    in the loop you would not use records[j] but (*it) instead.

look in the reference for std::set I posted for sample code and details.

Sara
0
HuaMin ChenProblem resolverAuthor Commented:
Sorry, I still get these errors
1>------ Rebuild All started: Project: SaveBinaryFile, Configuration: Debug Win32 ------
1>  stdafx.cpp
1>  SaveBinaryFile.cpp
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(3): warning C4627: '#include <set>': skipped when looking for precompiled header use
1>          Add directive to 'StdAfx.h' or rebuild precompiled header
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(29): error C3861: 'LessComp': identifier not found
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(48): error C2039: 'set' : is not a member of 'std'
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(48): error C2065: 'set' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(48): error C2275: 'nameval' : illegal use of this type as an expression
1>          c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(22) : see declaration of 'nameval'
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(48): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(76): error C3083: 'set': the symbol to the left of a '::' must be a type
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(76): error C2039: 'insert' : is not a member of 'std'
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(76): error C3861: 'insert': identifier not found
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(78): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(78): error C2228: left of '.begin' must have class/struct/union
1>          type is ''unknown-type''
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(78): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(78): error C2228: left of '.end' must have class/struct/union
1>          type is ''unknown-type''
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(78): error C2780: 'void std::sort(_RanIt,_RanIt)' : expects 2 arguments - 3 provided
1>          c:\program files (x86)\microsoft visual studio 10.0\vc\include\algorithm(3639) : see declaration of 'std::sort'
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(83): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(83): error C2228: left of '.size' must have class/struct/union
1>          type is ''unknown-type''
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(88): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(98): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(98): error C2228: left of '.size' must have class/struct/union
1>          type is ''unknown-type''
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(101): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(101): error C2228: left of '.fld_nm' must have class/struct/union
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(101): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(101): error C2228: left of '.fld_len' must have class/struct/union
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(101): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(101): error C2228: left of '.fld_nm_t' must have class/struct/union
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(101): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(101): error C2228: left of '.fld_val' must have class/struct/union
========== Rebuild All: 0 succeeded, 1 failed, 0 skipped ==========

Open in new window


due to these?

//

#include <set>
#include "stdafx.h"
#include <stdio.h>
#include <string.h>
#include <fstream>
#include <string>
#include <ctype.h>
#include <time.h>
#include <process.h>
#include <vector>
#include <iostream>
#include <algorithm>
#include <iomanip>
using namespace std;

bool LessComp(const nameval& a1, const nameval& a2)
{
  if(strcmp(a1.fld_nm, a2.fld_nm) < 0) return true;
  if(strcmp(a1.fld_nm, a2.fld_nm) > 0) return false;
  if(a1.fld_val < a2.fld_val) return true;
  return false;
}

struct nameval
{
	char fld_nm[100];
	wchar_t fld_nm_t[100]; 
     bool operator< (const nameval & a2) const
     {
           return LessComp(*this, a2);
     }
	int fld_len;
	int fld_val;
};
nameval binrec;

int main()
{
    std::set<nameval> records;
    int cnt;
	for (cnt=0;cnt<8000000;cnt++)
	{
		nameval val={0};
		int j;
		for (j=0;j<20;j++)
		{
			val.fld_nm[j] += (char)(rand () % 58 + 64);
		}
		
		wchar_t wbuf[100]={0};
		mbstowcs(wbuf,val.fld_nm,_countof(val.fld_nm));
		wmemcpy(val.fld_nm_t,wbuf,100);

		val.fld_val=cnt;
		std::set::insert(val);
	}
	std::sort(records.begin(), records.end(), LessComp);

	std::ofstream ostrm("c:\\dp4\\flout.bin", std::ios::binary | std::ios::out );
	if (ostrm.is_open())
	{
		for (size_t i = 0; i < records.size(); i++)
		{
			ostrm.write((char *)&records[i], sizeof(nameval));
		}
	}

	std::wofstream ostrm2("c:\\dp4\\flout.ord", std::ios::out );
	if (ostrm2.is_open())
	{
		for (size_t i = 0; i < records.size(); i++)
		{
			ostrm2 << "\"" << records[i].fld_nm << "\" " << records[i].fld_len << ' '  << records[i].fld_nm_t << ' '<< records[i].fld_val << '\0' << '\n';
		}
	}

	return 0;
}

Open in new window

0
sarabandeCommented:
#include <set>': skipped when looking for precompiled header use
you need to add the #include <set> below #include stdafx.h.

all statements above stdafx.h will not be seen by the precompiler cause it uses a precompiled header for them.

std::set::insert(val);
the statement is wrong. see my previous comment.

std::sort(records.begin(), records.end(), LessComp);
as told, the statement must be removed.

also the for loop and access to records is still wrong.


Sara
0
sarabandeCommented:
HuaMinChen, I've got the impression that the current program is one of your first c++ programs. if that is true and given that you were visiting a course that is suitable for the level of your experience, I wonder why a beginner's exercise should have such "heavy" requirements such as to handle 8 millions of strings in memory.

can you post the full requirement of the exercise such that we could give advice which is appropriate to the given task and your current knowledge. thank you.

Sara
0
HuaMin ChenProblem resolverAuthor Commented:
Sara,
As you gave the advice to use "set", can you please advise how to correct this line?

		std::set::insert(val);
		...

Open in new window

Phoffric,
It is not good to close the thread while there is error got due to the advice given, right?
0
HuaMin ChenProblem resolverAuthor Commented:
Sara,
I do expect to resolve the current memory problem to the project. Thanks.
0
HuaMin ChenProblem resolverAuthor Commented:
And I did need to check such way in C++ project to handle extremely big vector list and that's all.
0
sarabandeCommented:
how to correct this line?

I already told you the statement in one of my previous comments:

records.insert(val);

Open in new window


insert is a member function of std::set. you defined a container 'std::set<nameval> records;'  and can now call member functions by using operator.

Sara
0
HuaMin ChenProblem resolverAuthor Commented:
Hi Sara,
how can I now refer to each record instead of "records(i)" (that is the one I previously used)?
0
HuaMin ChenProblem resolverAuthor Commented:
Please note that I cannot use [] above so I have used () instead.
0
sarabandeCommented:
how can I now refer to each record instead of "records(i)" (that is the one I previously used)?
you should check my previous posts if you have questions because I already posted code for all your recent questions.

a std::set internally is organized as a tree. so it cannot accessed by index. however, a tree can iterated by using an iterator so going from one item to the next item.

for (std::set<namval>::iterator it = records.begin(); it != records.end(); ++it)
{
        ....

Open in new window

all stl containers can be accessed by iterator as in the above statement. the iterator is pointing to the current item and you can get a reference to the current set item by using the dereference operator* :

....
nameval & record = *it;
ostrm.write((char *)&record, sizeof(nameval));

Open in new window

or alternatively:

....
ostrm.write((char *)&(*it), sizeof(nameval));

Open in new window


Sara
0
sarabandeCommented:
also note, that std::set already is an ordered container.

Sara
0
HuaMin ChenProblem resolverAuthor Commented:
Sorry, I get these errors
1>c:\savebinaryfile tst\savebinaryfile\savebinaryfile.cpp(84): error C2065: 'namval' : undeclared identifier
1>c:\savebinaryfile tst\savebinaryfile\savebinaryfile.cpp(84): error C2955: 'std::set' : use of class template requires template argument list

Open in new window


due to this line
		for (std::set<namval>::iterator it = records.begin(); it != records.end(); ++it)
		...

Open in new window

0
sarabandeCommented:
it should be 'nameval' and not 'namval'.

Sara
0
HuaMin ChenProblem resolverAuthor Commented:
Thanks a lot.
how to correct the last line below?
		for (std::set<nameval>::iterator it = records.begin(); it != records.end(); ++it)
		{
			ostrm2 << "\"" << records[i].fld_nm << "\" " << records[i].fld_len << ' '  << records[i].fld_nm_uni << ' '<< records[i].fld_val << '\0' << '\n';
			...

Open in new window

0
sarabandeCommented:
replace the records[ i ]  by (*it) for all occurrences.

Sara
0
sarabandeCommented:
alternatively you could use 'it->' instead of 'records[ i].' for example

ostrm2 << "\"" << it->fld_nm 

Open in new window


the iterator is similar to a pointer to the current item.

Sara
0
HuaMin ChenProblem resolverAuthor Commented:
Sorry, most probably one last thing is, how to correct this below?
bool LessComp(const nameval& a1, const nameval& a2)
{
  if(strcmp(a1.fld_nm, a2.fld_nm) < 0) return true;
  if(strcmp(a1.fld_nm, a2.fld_nm) > 0) return false;
  if(a1.fld_val < a2.fld_val) return true;
  return false;
}

Open in new window

0
sarabandeCommented:
you can leave it as it is and call it from operator< function.

I posted already code for it above:

//forward declarations
struct nameval;
bool LessComp(const nameval & a1, const nameval & a2);

struct nameval
{
     ....
     bool operator< (const nameval & a2) const
     {
           return LessComp(*this, a2);
     }

Open in new window


note, the forward declarations are necessary or you get errors.

alternatively you could remove the LessComp function and implement operator< like

struct nameval
{
     ....
     bool operator< (const nameval & a2) const
     {
           if(strcmp(fld_nm, a2.fld_nm) < 0) return true;
           if(strcmp(fld_nm, a2.fld_nm) > 0) return false;
           if (fld_val < a2.fld_val) return true;
           return false;
     }

Open in new window


note, the code is from the LessComp function and the only difference is that the a1 is not passed but is the (*this) object which can be omitted.

Sara
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
HuaMin ChenProblem resolverAuthor Commented:
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
C++

From novice to tech pro — start learning today.