Avatar of Peter Chan
Peter Chan
Flag for Hong Kong asked on

Memory problem

Hi,
when running the codes below
https://dl.dropboxusercontent.com/u/40211031/SaveBinaryFile.cpp

to generate the list. I get this error
https://dl.dropboxusercontent.com/u/40211031/t863.png

how to resolve it?
C++CSystem Programming

Avatar of undefined
Last Comment
Peter Chan

8/22/2022 - Mon
Neil Russell

Please place your code and images on EE where we can ALL see them. A lot of corporate networks will BLOCK Dropbox, and for very good reason :D
Peter Chan

ASKER
Here are the codes
//

#include "stdafx.h"
#include <stdio.h>
#include <string.h>
#include <fstream>
#include <string>
#include <ctype.h>
#include <time.h>
#include <process.h>
#include <vector>
#include <iostream>
#include <algorithm>
#include <iomanip>
using namespace std;

struct nameval
{
	char fld_nm[100];
	wchar_t fld_nm_t[100];
	int fld_len;
	int fld_val;
};
nameval binrec;

bool LessComp(const nameval& a1, const nameval& a2)
{
  if(strcmp(a1.fld_nm, a2.fld_nm) < 0) return true;
  if(strcmp(a1.fld_nm, a2.fld_nm) > 0) return false;
  if(a1.fld_val < a2.fld_val) return true;
  return false;
}

int main()
{
    std::vector<nameval> records;
    int cnt;
	for (cnt=0;cnt<3000000;cnt++)
	{
		nameval val={0};
		int j;
		for (j=0;j<20;j++)
		{
			val.fld_nm[j] += (char)(rand () % 58 + 64);
		}

		wchar_t wbuf[100]={0};
		mbstowcs(wbuf,val.fld_nm,_countof(val.fld_nm));
		wmemcpy(val.fld_nm_t,wbuf,100);

		val.fld_val=cnt;
		records.push_back(val);
	}
	std::sort(records.begin(), records.end(), LessComp);

	std::ofstream ostrm("c:\\dp4\\flout.bin", std::ios::binary | std::ios::out );
	if (ostrm.is_open())
	{
		for (size_t i = 0; i < records.size(); i++)
		{
			ostrm.write((char *)&records[i], sizeof(nameval));
		}
	}

	std::wofstream ostrm2("c:\\dp4\\flout.ord", std::ios::out );
	if (ostrm2.is_open())
	{
		for (size_t i = 0; i < records.size(); i++)
		{
			ostrm2 << "\"" << records[i].fld_nm << "\" " << records[i].fld_len << ' '  << records[i].fld_nm_t << ' '<< records[i].fld_val << '\n';
		}
	}

	return 0;
}

Open in new window

and here is the image
t863.png
Peter Chan

ASKER
Thanks
All of life is about relationships, and EE has made a viirtual community a real community. It lifts everyone's boat
William Peck
phoffric

You got a std::bad_alloc exception which means that you ran out of memory as you were allocating it.
   http://www.cplusplus.com/reference/new/bad_alloc/

When you do you vector push_back, you are often allocating memory. Doing that 3 million times apparently is more than your platform can handle. If you drop the number down, to say, 1 million, you might have better luck.
Neil Russell

Well im not at a machine where I can compile and test but a quick look says WOW!

You declare binrec but it is never used that I can see.
You are constructing 3 million nameval structures in code and never freeing any of them at all!

I'll have a proper look in a bit if no body answers you in detail.
SOLUTION
phoffric

THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
GET A PERSONALIZED SOLUTION
Ask your own question & get feedback from real experts
Find out why thousands trust the EE community with their toughest problems.
Peter Chan

ASKER
Thanks all.
Phoffric,
It is OK for your way. But if I've increased that to this
	records.reserve(8000000);
	for (cnt=0;cnt<8000000;cnt++)
	...

Open in new window


instead of the original count. I still get this error.
t864.png
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
Neil Russell

Computers have limits just like us humans.  You are now asking for 8 million chunks of contiguous memory the size of your structure. Thats a LOT of memory.
Peter Chan

ASKER
How to resolve the problem now? Thanks
Neil Russell

ok, back to basics. Why do you want to create and handle IN MEMORY over 8 Million randomly generated strings?

Is this homework? An exercise? for a reason?

Whats wrong with databases? They are pretty good at handling millions of pieces of information.
Experts Exchange has (a) saved my job multiple times, (b) saved me hours, days, and even weeks of work, and often (c) makes me look like a superhero! This place is MAGIC!
Walt Forbes
Neil Russell

if you run...

#include "stdafx.h"
#include <stdio.h>
#include <string.h>
#include <fstream>
#include <string>
#include <ctype.h>
#include <time.h>
#include <process.h>
#include <vector>
#include <iostream>
#include <algorithm>
#include <iomanip>
using namespace std;

struct nameval
{
	char fld_nm[100];
	wchar_t fld_nm_t[100];
	int fld_len;
	int fld_val;
};
nameval binrec;

bool LessComp(const nameval& a1, const nameval& a2)
{
  if(strcmp(a1.fld_nm, a2.fld_nm) < 0) return true;
  if(strcmp(a1.fld_nm, a2.fld_nm) > 0) return false;
  if(a1.fld_val < a2.fld_val) return true;
  return false;
}

int main()
{
    std::vector<nameval> records;

  std::cout << "max_size: " << records.max_size() << "\n";
  return 0;
}

Open in new window


What do you get as an answer?
Peter Chan

ASKER
Thanks all.
Neilsr,
I get 13944699.
Neil Russell

ok that is the ABSOLUTE THEORETICAL maximum size that you could allocate to your vector Records. In reality it will never be anywhere near that and thats only if you dont reorder or use it for anything meaningful.
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
phoffric

Do a sizeof(nameval) and multiply that by the number you wrote, 13944699, and it will probably be close to 4GB, which apparently on your platform is the largest chunk on the heap that you can allocate.

32-bit can hold 2-bit (4) * 10-bit (1K) * 20-bit (1M) = 4*1K*1M = 4GB.

If you get a 64-bit platform and a 64-bit compiler, and more physical memory, then you may be able to reach you larger count.

Reason to keep data in memory is for speed. Databases are extremely much slower. Fastest way to deal with your problem is to memory map your data to a file and let the OS handle the swapping so that you don't actually have read/write to disk in your code. But this approach is outside the scope of this question.
phoffric

Since we resolved the original 3 million question, this question is resolved. If you change the question to 8 million count, probably some hardware software changes are in order. If you change it to 80 billion which is a reasonable question in some situations, and still need high speed (eliminating a relational database), the questions are different and potentially much more complex.

You didn't mention where the bad alloc exception occurred. Run the program in the debugger and see where it stops. You may not have been able to reserve the memory dynamically. You could try using a plain array outside of main (i.e., not on the stack) to see if that helps.

You should mention the OS bit size, and the compiler bit-size.
phoffric

Neilsr wrote:
Why do you want to create and handle IN MEMORY over 8 Million randomly generated strings?
 Is this homework? An exercise? for a reason?
You addressed other questions from him but did not address this one. The question does appear to be either homework or self-study, so I have limited my answers.
I started with Experts Exchange in 2004 and it's been a mainstay of my professional computing life since. It helped me launch a career as a programmer / Oracle data analyst
William Peck
Peter Chan

ASKER
Hi,
Thanks all.
Phoffric,
How to resolve the real problem, to handle 8 millions records by the program?
phoffric

What OS (including bit size), compiler (including bit size capability), and how much memory do you have?

From the answer you gave earlier, I believe you have a 32-bit platform or possibly a 32-bit compiler. Even if you have enough memory to handle 8 million records, you may not be able to find a contiguous virtual address region for a vector.
phoffric

IMO, the real problem is the one you posted in your OP.

Also, please address this question:
Neilsr wrote:
Why do you want to create and handle IN MEMORY over 8 Million randomly generated strings?
  Is this homework? An exercise? for a reason?
You addressed other questions from him but did not address this one.
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
Peter Chan

ASKER
It is in Win 7 machine having 8 GB Ram and I am using VS 2010 to handle the program.
Peter Chan

ASKER
This is an exercise, to check the way to handle that amount of records, through VS 2010.
sarabande

How to resolve the real problem, to handle 8 millions records by the program?
there are a few options:

1. try to minimize the record size. in your code you were using only 20 bytes for the random string and an integer value.

    that would give a size of 32 bytes while your current size is 308.  conversions from wide to narrow
    characters can be made on the fly and length of text can be determined by checking the text.

2. use a container that doesn't need contiguous memory.

    for your purpose a std::set could be fine which automatically would sort your records, if you provide
    an operator< member function (the implementation would be nearly identical to your LessComp function)
   
3. if you nevertheless run into memory issues, you would need to use a database or a keyed-access file.
   
4. memory-mapped files rarely are an option as they would require contiguous memory as well
    for the view you were looking at.  so actually you would have to do swapping yourself, what is
    much work with a small chance of getting a better over-all speed.

5. a probably better idea would be to use a self-developed b-tree  where you would use a cache for the most
    frequently used b-tree records.

Sara
Your help has saved me hundreds of hours of internet surfing.
fblack61
Peter Chan

ASKER
Thanks. Can I see the example using b-tree?
Peter Chan

ASKER
Sorry, other than b-tree, I still want to see there is any other way to resolve the same memory problem, which I originally mentioned. Thanks
Peter Chan

ASKER
Sorry, as I had been stuck with the memory problem, to the vector in the project. How to resolve the memory problem?
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
sarabande

the b-tree solution is the most difficult suggestion I made. it is nothing where I could give you a quick sample of (even if the academic reservations phoffric has told you wouldn't forbid that anyway), especially not as you needed a "customized" b-tree solution which was using a cache rather than keeping the whole tree in memory. a tree is a container with nodes and leafs. nodes were pointing to other nodes and leafs. because of the pointers you could store the tree as a file and load and unload parts of it dynamically. if you want to go this way, you would need at least two steps. first step is to implement a b-tree container in memory. second step is to make this tree persistent and add a cache. you would need to ask new questions for this and always show your own code such that we are able to assist you rather than doing the work for you.

did you check my other suggestions? what about making the structure more compact? that would allow you to store nearly 10 times more in memory. when using a std::set as basic container it allows to use all heap memory and relieve you of requesting more and more contiguous memory. if you apply both you should be able to store 8 million strings in memory because vs2010 would provide sufficient virtual memory for your program.

Sara
Peter Chan

ASKER
Thanks. Can I see the exact way to use std::set instead, to the above codes?
Peter Chan

ASKER
I mean if there is one code example to std::set?
This is the best money I have ever spent. I cannot not tell you how many times these folks have saved my bacon. I learn so much from the contributors.
rwheeler23
sarabande

see http://msdn.microsoft.com/en-us/library/e8wh7665.aspx for details.

you now have std::vector<nameval>. you would replace it by std::set<nameval>. instead of push_back you would use std::set::insert function.

then add operator< function to structure nameval:

struct nameval
{
    ....
    bool operator< (const nameval & a2) const;
};

Open in new window


the implementation of the function is identical to your LessComp function beside that you omit 'a1.'  (because a1 is the 'this' object).

after you stored the keys the set already is sorted.

you would use a std::set<nameval>::iterator to iterate the container.

Sara

Sara
Peter Chan

ASKER
Sorry, please see these errors

1>------ Rebuild All started: Project: SaveBinaryFile, Configuration: Debug Win32 ------
1>  stdafx.cpp
1>  SaveBinaryFile.cpp
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(44): error C2039: 'set' : is not a member of 'std'
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(44): error C2065: 'set' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(44): error C2275: 'nameval' : illegal use of this type as an expression
1>          c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(21) : see declaration of 'nameval'
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(44): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(46): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(46): error C2228: left of '.reserve' must have class/struct/union
1>          type is ''unknown-type''
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(72): error C3083: 'set': the symbol to the left of a '::' must be a type
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(72): error C2039: 'insert' : is not a member of 'std'
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(72): error C3861: 'insert': identifier not found
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(74): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(74): error C2228: left of '.begin' must have class/struct/union
1>          type is ''unknown-type''
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(74): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(74): error C2228: left of '.end' must have class/struct/union
1>          type is ''unknown-type''
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(74): error C2780: 'void std::sort(_RanIt,_RanIt)' : expects 2 arguments - 3 provided
1>          c:\program files (x86)\microsoft visual studio 10.0\vc\include\algorithm(3639) : see declaration of 'std::sort'
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(79): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(79): error C2228: left of '.size' must have class/struct/union
1>          type is ''unknown-type''
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(84): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(94): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(94): error C2228: left of '.size' must have class/struct/union
1>          type is ''unknown-type''
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(97): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(97): error C2228: left of '.fld_nm' must have class/struct/union
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(97): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(97): error C2228: left of '.fld_len' must have class/struct/union
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(97): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(97): error C2228: left of '.fld_nm_t' must have class/struct/union
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(97): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(97): error C2228: left of '.fld_val' must have class/struct/union
========== Rebuild All: 0 succeeded, 1 failed, 0 skipped ==========

Open in new window


from these codes

//

#include "stdafx.h"
#include <stdio.h>
#include <string.h>
#include <fstream>
#include <string>
#include <ctype.h>
#include <time.h>
#include <process.h>
#include <vector>
#include <iostream>
#include <algorithm>
#include <iomanip>
using namespace std;

struct nameval
{
	bool operator< (const nameval & a2) const;
	char fld_nm[100];
	wchar_t fld_nm_t[100]; 
	int fld_len;
	int fld_val;
};
nameval binrec;

bool LessComp(const nameval& a1, const nameval& a2)
{
  if(strcmp(a1.fld_nm, a2.fld_nm) < 0) return true;
  if(strcmp(a1.fld_nm, a2.fld_nm) > 0) return false;
  if(a1.fld_val < a2.fld_val) return true;
  return false;
}

int main()
{
    std::set<nameval> records;
    int cnt;
	records.reserve(8000000);
	for (cnt=0;cnt<8000000;cnt++)
	{
		nameval val={0};
		int j;
		for (j=0;j<20;j++)
		{
			val.fld_nm[j] += (char)(rand () % 58 + 64);
		}
		
		wchar_t wbuf[100]={0};
		mbstowcs(wbuf,val.fld_nm,_countof(val.fld_nm));
		wmemcpy(val.fld_nm_t,wbuf,100);

		val.fld_val=cnt;
		std::set::insert(val);
	}
	std::sort(records.begin(), records.end(), LessComp);

	std::ofstream ostrm("c:\\dp4\\flout.bin", std::ios::binary | std::ios::out );
	if (ostrm.is_open())
	{
		for (size_t i = 0; i < records.size(); i++)
		{
			ostrm.write((char *)&records[i], sizeof(nameval));
		}
	}

	std::wofstream ostrm2("c:\\dp4\\flout.ord", std::ios::out );
	if (ostrm2.is_open())
	{
		for (size_t i = 0; i < records.size(); i++)
		{
			ostrm2 << "\"" << records[i].fld_nm << "\" " << records[i].fld_len << ' '  << records[i].fld_nm_t << ' '<< records[i].fld_val << '\0' << '\n';
		}
	}

	return 0;
}

Open in new window

sarabande

1. add

#include <set> 

Open in new window


to the include statements.

2. the struct nameval needs the operator< member function.

you may implement it by

//forward declaration
bool LessComp(const nameval & a1, const nameval & a2);
struct nameval
{
     ....
     bool operator< (const nameval & a2) const
     {
           return LessComp(*this, a2);
     }

Open in new window


3. the insert function is a member function of std::set, same as push_back is a member function of std::vector.

    so the code to store is:

     
records.insert(val);

Open in new window


4. std::sort is not possible for std::set because the items are already sorted in the set.

5. the for loop to write to the file now must use an iterator:

   
for (std::set<namval>::iterator it = records.begin(); it != records.end(); ++it)

Open in new window


    in the loop you would not use records[j] but (*it) instead.

look in the reference for std::set I posted for sample code and details.

Sara
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
Peter Chan

ASKER
Sorry, I still get these errors
1>------ Rebuild All started: Project: SaveBinaryFile, Configuration: Debug Win32 ------
1>  stdafx.cpp
1>  SaveBinaryFile.cpp
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(3): warning C4627: '#include <set>': skipped when looking for precompiled header use
1>          Add directive to 'StdAfx.h' or rebuild precompiled header
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(29): error C3861: 'LessComp': identifier not found
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(48): error C2039: 'set' : is not a member of 'std'
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(48): error C2065: 'set' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(48): error C2275: 'nameval' : illegal use of this type as an expression
1>          c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(22) : see declaration of 'nameval'
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(48): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(76): error C3083: 'set': the symbol to the left of a '::' must be a type
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(76): error C2039: 'insert' : is not a member of 'std'
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(76): error C3861: 'insert': identifier not found
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(78): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(78): error C2228: left of '.begin' must have class/struct/union
1>          type is ''unknown-type''
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(78): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(78): error C2228: left of '.end' must have class/struct/union
1>          type is ''unknown-type''
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(78): error C2780: 'void std::sort(_RanIt,_RanIt)' : expects 2 arguments - 3 provided
1>          c:\program files (x86)\microsoft visual studio 10.0\vc\include\algorithm(3639) : see declaration of 'std::sort'
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(83): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(83): error C2228: left of '.size' must have class/struct/union
1>          type is ''unknown-type''
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(88): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(98): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(98): error C2228: left of '.size' must have class/struct/union
1>          type is ''unknown-type''
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(101): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(101): error C2228: left of '.fld_nm' must have class/struct/union
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(101): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(101): error C2228: left of '.fld_len' must have class/struct/union
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(101): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(101): error C2228: left of '.fld_nm_t' must have class/struct/union
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(101): error C2065: 'records' : undeclared identifier
1>c:\savebinaryfile\savebinaryfile\savebinaryfile.cpp(101): error C2228: left of '.fld_val' must have class/struct/union
========== Rebuild All: 0 succeeded, 1 failed, 0 skipped ==========

Open in new window


due to these?

//

#include <set>
#include "stdafx.h"
#include <stdio.h>
#include <string.h>
#include <fstream>
#include <string>
#include <ctype.h>
#include <time.h>
#include <process.h>
#include <vector>
#include <iostream>
#include <algorithm>
#include <iomanip>
using namespace std;

bool LessComp(const nameval& a1, const nameval& a2)
{
  if(strcmp(a1.fld_nm, a2.fld_nm) < 0) return true;
  if(strcmp(a1.fld_nm, a2.fld_nm) > 0) return false;
  if(a1.fld_val < a2.fld_val) return true;
  return false;
}

struct nameval
{
	char fld_nm[100];
	wchar_t fld_nm_t[100]; 
     bool operator< (const nameval & a2) const
     {
           return LessComp(*this, a2);
     }
	int fld_len;
	int fld_val;
};
nameval binrec;

int main()
{
    std::set<nameval> records;
    int cnt;
	for (cnt=0;cnt<8000000;cnt++)
	{
		nameval val={0};
		int j;
		for (j=0;j<20;j++)
		{
			val.fld_nm[j] += (char)(rand () % 58 + 64);
		}
		
		wchar_t wbuf[100]={0};
		mbstowcs(wbuf,val.fld_nm,_countof(val.fld_nm));
		wmemcpy(val.fld_nm_t,wbuf,100);

		val.fld_val=cnt;
		std::set::insert(val);
	}
	std::sort(records.begin(), records.end(), LessComp);

	std::ofstream ostrm("c:\\dp4\\flout.bin", std::ios::binary | std::ios::out );
	if (ostrm.is_open())
	{
		for (size_t i = 0; i < records.size(); i++)
		{
			ostrm.write((char *)&records[i], sizeof(nameval));
		}
	}

	std::wofstream ostrm2("c:\\dp4\\flout.ord", std::ios::out );
	if (ostrm2.is_open())
	{
		for (size_t i = 0; i < records.size(); i++)
		{
			ostrm2 << "\"" << records[i].fld_nm << "\" " << records[i].fld_len << ' '  << records[i].fld_nm_t << ' '<< records[i].fld_val << '\0' << '\n';
		}
	}

	return 0;
}

Open in new window

sarabande

#include <set>': skipped when looking for precompiled header use
you need to add the #include <set> below #include stdafx.h.

all statements above stdafx.h will not be seen by the precompiler cause it uses a precompiled header for them.

std::set::insert(val);
the statement is wrong. see my previous comment.

std::sort(records.begin(), records.end(), LessComp);
as told, the statement must be removed.

also the for loop and access to records is still wrong.


Sara
sarabande

HuaMinChen, I've got the impression that the current program is one of your first c++ programs. if that is true and given that you were visiting a course that is suitable for the level of your experience, I wonder why a beginner's exercise should have such "heavy" requirements such as to handle 8 millions of strings in memory.

can you post the full requirement of the exercise such that we could give advice which is appropriate to the given task and your current knowledge. thank you.

Sara
Experts Exchange is like having an extremely knowledgeable team sitting and waiting for your call. Couldn't do my job half as well as I do without it!
James Murphy
Peter Chan

ASKER
Sara,
As you gave the advice to use "set", can you please advise how to correct this line?

		std::set::insert(val);
		...

Open in new window

Phoffric,
It is not good to close the thread while there is error got due to the advice given, right?
Peter Chan

ASKER
Sara,
I do expect to resolve the current memory problem to the project. Thanks.
Peter Chan

ASKER
And I did need to check such way in C++ project to handle extremely big vector list and that's all.
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
sarabande

how to correct this line?

I already told you the statement in one of my previous comments:

records.insert(val);

Open in new window


insert is a member function of std::set. you defined a container 'std::set<nameval> records;'  and can now call member functions by using operator.

Sara
Peter Chan

ASKER
Hi Sara,
how can I now refer to each record instead of "records(i)" (that is the one I previously used)?
Peter Chan

ASKER
Please note that I cannot use [] above so I have used () instead.
All of life is about relationships, and EE has made a viirtual community a real community. It lifts everyone's boat
William Peck
SOLUTION
sarabande

THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
sarabande

also note, that std::set already is an ordered container.

Sara
Peter Chan

ASKER
Sorry, I get these errors
1>c:\savebinaryfile tst\savebinaryfile\savebinaryfile.cpp(84): error C2065: 'namval' : undeclared identifier
1>c:\savebinaryfile tst\savebinaryfile\savebinaryfile.cpp(84): error C2955: 'std::set' : use of class template requires template argument list

Open in new window


due to this line
		for (std::set<namval>::iterator it = records.begin(); it != records.end(); ++it)
		...

Open in new window

sarabande

it should be 'nameval' and not 'namval'.

Sara
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
Peter Chan

ASKER
Thanks a lot.
how to correct the last line below?
		for (std::set<nameval>::iterator it = records.begin(); it != records.end(); ++it)
		{
			ostrm2 << "\"" << records[i].fld_nm << "\" " << records[i].fld_len << ' '  << records[i].fld_nm_uni << ' '<< records[i].fld_val << '\0' << '\n';
			...

Open in new window

SOLUTION
sarabande

THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
SOLUTION
sarabande

THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
Peter Chan

ASKER
Sorry, most probably one last thing is, how to correct this below?
bool LessComp(const nameval& a1, const nameval& a2)
{
  if(strcmp(a1.fld_nm, a2.fld_nm) < 0) return true;
  if(strcmp(a1.fld_nm, a2.fld_nm) > 0) return false;
  if(a1.fld_val < a2.fld_val) return true;
  return false;
}

Open in new window

ASKER CERTIFIED SOLUTION
sarabande

THIS SOLUTION ONLY AVAILABLE TO MEMBERS.
View this solution by signing up for a free trial.
Members can start a 7-Day free trial and enjoy unlimited access to the platform.
See Pricing Options
Start Free Trial
⚡ FREE TRIAL OFFER
Try out a week of full access for free.
Find out why thousands trust the EE community with their toughest problems.
Peter Chan

ASKER
Experts Exchange has (a) saved my job multiple times, (b) saved me hours, days, and even weeks of work, and often (c) makes me look like a superhero! This place is MAGIC!
Walt Forbes