?
Solved

Problem with the memory

Posted on 2014-11-27
21
Medium Priority
?
232 Views
1 Endorsement
Last Modified: 2014-11-28
Hi,
I do get the attached error

using these codes

#include "stdafx.h"
#include <set>
#include <stdio.h>
#include <string.h>
#include <fstream>
#include <string>
#include <ctype.h>
#include <time.h>
#include <process.h>
#include <vector>
#include <iostream>
#include <algorithm>
#include <iomanip>
using namespace std;


struct nameval
{
	char fld_nm[100];
	wchar_t fld_nm_t[100];
     bool operator< (const nameval & a2) const
     {
           if(strcmp(fld_nm, a2.fld_nm) < 0) return true;
           if(strcmp(fld_nm, a2.fld_nm) > 0) return false;
           if (fld_val < a2.fld_val) return true;
           return false;
     }
	int fld_len;
	int fld_val;
};
nameval binrec;

int main()
{
    std::set<nameval> records;
    int cnt;
	for (cnt=0;cnt<8000000;cnt++)
	{
		nameval val={0};
		int j;
		for (j=0;j<20;j++)
		{
			val.fld_nm[j] += (char)(rand () % 58 + 64);
		}
		
		wchar_t wbuf[100]={0};
		mbstowcs(wbuf,val.fld_nm,_countof(val.fld_nm));
		wmemcpy(val.fld_nm_t,wbuf,100);

		val.fld_val=cnt;
		records.insert(val);
	}

	std::ofstream ostrm("c:\\dp4\\flout.bin", std::ios::binary | std::ios::out );
	if (ostrm.is_open())
	{
		for (std::set<nameval>::iterator it = records.begin(); it != records.end(); ++it)
		{
			ostrm.write((char *)&records, sizeof(nameval));
		}
	}

	std::wofstream ostrm2("c:\\dp4\\flout.ord", std::ios::out );
	if (ostrm2.is_open())
	{
		for (std::set<nameval>::iterator it = records.begin(); it != records.end(); ++it)
		{
			ostrm2 << "\"" << it->fld_nm << "\" " << it->fld_len << ' '  << it->fld_nm_t << ' '<< it->fld_val << '\0' << '\n';
		}
	}

 	system("pause>null");

	return 0;
}

Open in new window

t867.png
1
Comment
Question by:HuaMinChen
  • 11
  • 5
  • 3
19 Comments
 
LVL 11

Author Comment

by:HuaMinChen
ID: 40469943
No, I am to confirm if the way is fine to handle very big vector list. Thanks.
0
 
LVL 25

Assisted Solution

by:chaau
chaau earned 1000 total points
ID: 40469967
I think you wanted this on line 59:
ostrm.write((char *)&it, sizeof(nameval));

Open in new window

(basically you need to write iterator)
0
 
LVL 11

Author Comment

by:HuaMinChen
ID: 40469971
Thanks to your advice to do the correction.
I've corrected it above but I still have the memory problem, after having re-built the project.
0
Never miss a deadline with monday.com

The revolutionary project management tool is here!   Plan visually with a single glance and make sure your projects get done.

 
LVL 11

Author Comment

by:HuaMinChen
ID: 40469972
It is still the same problem.
0
 
LVL 25

Assisted Solution

by:chaau
chaau earned 1000 total points
ID: 40469976
try to embed everything inside your for() loop into a try/catch block and handle the exceptions
0
 
LVL 11

Author Comment

by:HuaMinChen
ID: 40469983
I have this part now
		for (std::set<nameval>::iterator it = records.begin(); it != records.end(); ++it)
		{
			try
			{
				ostrm.write((char *)&it, sizeof(nameval));
			}
			catch (exception& e)
			{
				cout << e.what() << '\n';
			}
		}
		...

Open in new window

but I still encounter the same when running the re-built project.
0
 
LVL 25

Expert Comment

by:chaau
ID: 40470068
The problem is most likely when you assign 80 million records to the set. So, pit the try  catch in the first loop
0
 
LVL 31

Expert Comment

by:Zoppo
ID: 40470094
Hi HuaMinChen,

IMO this can't ever work in a 32-bit build (from the screenshot I assume your app is 32-bit) because if you want allocate 8.000.000 elements of size about 300 byte you'll need around 2,4 GB - this is more than a windows 32-bit app usually can allocate (i.e. take a look at http://stackoverflow.com/questions/639540/how-much-memory-can-a-32-bit-process-access-on-a-64-bit-operating-system about this).

I would suggest to change it to build a 64-bit application to avoid this limit.

Hope that helps,

ZOPPO
0
 
LVL 11

Author Comment

by:HuaMinChen
ID: 40470100
Thanks all.
Zoppo,
I am using VS 2010 to the project. Can you advise how to adjust the project to x64?
0
 
LVL 11

Author Comment

by:HuaMinChen
ID: 40470103
Chaau,
I do add try...catch to 1st for loop within that, and do get repeating "bad allocation" there. How to resolve it? Thanks.
0
 
LVL 31

Assisted Solution

by:Zoppo
Zoppo earned 1000 total points
ID: 40470117
Hi HuaMinChen,

to do this you'll have to add a new configuration. To do this open 'Configuration Mananger' (i.e. from Build menu), open drop-down Active solution platform and select New. In the following dialog select x64 as platform type and press OK.

Following this you now should have all configurations you had before (usually 'Release' and 'Debug') twice, one for platform Win32, the other one for x64.

With a bit luck that's all you have to do to build 64 bit. It maybe you'll encounter build- or runtime errors, if so you need to fix them all. Even it's a good idea to check all compiler warnings if any, coz one of the most common cause for problems when porting from 32- to 64-bit is bad type casting or mixing of integer types which size may be different in x64 allthough they're the same in Win32.

If you have problems you can google for something like 'Visual Studio port Win32 x64', you'll find a lot of information.

ZOPPO
0
 
LVL 11

Author Comment

by:HuaMinChen
ID: 40470122
Sorry Zoppo.
I am using VS 2010 express. Can I open Configuration manager from that?
0
 
LVL 31

Expert Comment

by:Zoppo
ID: 40470126
Sorry, I can't say exactly coz I don't use 'Express', but I'm nearly 100% sure it exists in VS 2010 Express too. In VS 2010 there are two possibilities (I know) to open it:

- Menu 'Build' -> 'Configration Manager'
- In properties dialog (i.e. Menu 'Project' -> '<project name> Properties') there's a button 'Configuration Manager' at top-right corner.
0
 
LVL 31

Expert Comment

by:Zoppo
ID: 40470130
Addition: There might be more work needed depending on the history of the project. If it was i.e. created in VS 2010 there shouldn't be to do more, but if it's older it might be there are settings where the same path is set for both b platforms allthough they shouldn't.

If you don't plan to further be able to build as 32 bit you can ignore those settings and I would suggest to remove the Win32 platform from the configuration (even possible with 'Configuration Mananger').

But if you want to keep both platforms you should check if everything is fine.

As a first step you should check if this works:

1. select Win32 platform and do a Rebuild All
2. switch to x64 platform and do a Rebuild All too
3. switch back to Win32 platform and do a normal Build
4. stay in Win32 platform and do a Rebuild All again
5. switch to x64 platform again and do a normal Build

If everything is ok the steps 3 and 5 should finish immediate without recompiling anything.

Otherwise please tell then I can show you which settings probably aren't correct ...

ZOPPO
0
 
LVL 11

Author Comment

by:HuaMinChen
ID: 40470140
Many many thanks Zoppo.
I now open the original project (that is created using VS 2010 express), within VS 2013 Pro version. Can you tell me how to open configuration manager within VS 2013 Pro version?
0
 
LVL 11

Author Comment

by:HuaMinChen
ID: 40470145
Sorry, please disregard my last question.
0
 
LVL 11

Author Comment

by:HuaMinChen
ID: 40470147
Zoppo,
The attached one is the screenshot, within Config manager of VS 2010, can you please advise what to adjust on that? Thanks
t868.png
0
 
LVL 11

Author Comment

by:HuaMinChen
ID: 40470148
Please again disregard my current question. Sorry.
0
 
LVL 31

Accepted Solution

by:
Zoppo earned 1000 total points
ID: 40470152
As told select <New> in the dropdown Active solution platform to open the dialog to create a 64 bit configuration.

Attached you see a screenshot (or better two in one) of the important settings.
screenshot.png
0

Featured Post

The 14th Annual Expert Award Winners

The results are in! Meet the top members of our 2017 Expert Awards. Congratulations to all who qualified!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

The article covers five tools all IT professionals should know about, as they up productivity by a great deal!
Article by: evilrix
Looking for a way to avoid searching through large data sets for data that doesn't exist? A Bloom Filter might be what you need. This data structure is a probabilistic filter that allows you to avoid unnecessary searches when you know the data defin…
The goal of this video is to provide viewers with basic examples to understand and use conditional statements in the C programming language.
The goal of the tutorial is to teach the user how to use functions in C++. The video will cover how to define functions, how to call functions and how to create functions prototypes. Microsoft Visual C++ 2010 Express will be used as a text editor an…

601 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question