Solved

Problem with the memory

Posted on 2014-11-27
21
217 Views
1 Endorsement
Last Modified: 2014-11-28
Hi,
I do get the attached error

using these codes

#include "stdafx.h"
#include <set>
#include <stdio.h>
#include <string.h>
#include <fstream>
#include <string>
#include <ctype.h>
#include <time.h>
#include <process.h>
#include <vector>
#include <iostream>
#include <algorithm>
#include <iomanip>
using namespace std;


struct nameval
{
	char fld_nm[100];
	wchar_t fld_nm_t[100];
     bool operator< (const nameval & a2) const
     {
           if(strcmp(fld_nm, a2.fld_nm) < 0) return true;
           if(strcmp(fld_nm, a2.fld_nm) > 0) return false;
           if (fld_val < a2.fld_val) return true;
           return false;
     }
	int fld_len;
	int fld_val;
};
nameval binrec;

int main()
{
    std::set<nameval> records;
    int cnt;
	for (cnt=0;cnt<8000000;cnt++)
	{
		nameval val={0};
		int j;
		for (j=0;j<20;j++)
		{
			val.fld_nm[j] += (char)(rand () % 58 + 64);
		}
		
		wchar_t wbuf[100]={0};
		mbstowcs(wbuf,val.fld_nm,_countof(val.fld_nm));
		wmemcpy(val.fld_nm_t,wbuf,100);

		val.fld_val=cnt;
		records.insert(val);
	}

	std::ofstream ostrm("c:\\dp4\\flout.bin", std::ios::binary | std::ios::out );
	if (ostrm.is_open())
	{
		for (std::set<nameval>::iterator it = records.begin(); it != records.end(); ++it)
		{
			ostrm.write((char *)&records, sizeof(nameval));
		}
	}

	std::wofstream ostrm2("c:\\dp4\\flout.ord", std::ios::out );
	if (ostrm2.is_open())
	{
		for (std::set<nameval>::iterator it = records.begin(); it != records.end(); ++it)
		{
			ostrm2 << "\"" << it->fld_nm << "\" " << it->fld_len << ' '  << it->fld_nm_t << ' '<< it->fld_val << '\0' << '\n';
		}
	}

 	system("pause>null");

	return 0;
}

Open in new window

t867.png
1
Comment
Question by:HuaMinChen
  • 11
  • 5
  • 3
21 Comments
 
LVL 10

Author Comment

by:HuaMinChen
ID: 40469943
No, I am to confirm if the way is fine to handle very big vector list. Thanks.
0
 
LVL 24

Assisted Solution

by:chaau
chaau earned 250 total points
ID: 40469967
I think you wanted this on line 59:
ostrm.write((char *)&it, sizeof(nameval));

Open in new window

(basically you need to write iterator)
0
 
LVL 10

Author Comment

by:HuaMinChen
ID: 40469971
Thanks to your advice to do the correction.
I've corrected it above but I still have the memory problem, after having re-built the project.
0
Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

 
LVL 10

Author Comment

by:HuaMinChen
ID: 40469972
It is still the same problem.
0
 
LVL 24

Assisted Solution

by:chaau
chaau earned 250 total points
ID: 40469976
try to embed everything inside your for() loop into a try/catch block and handle the exceptions
0
 
LVL 10

Author Comment

by:HuaMinChen
ID: 40469983
I have this part now
		for (std::set<nameval>::iterator it = records.begin(); it != records.end(); ++it)
		{
			try
			{
				ostrm.write((char *)&it, sizeof(nameval));
			}
			catch (exception& e)
			{
				cout << e.what() << '\n';
			}
		}
		...

Open in new window

but I still encounter the same when running the re-built project.
0
 
LVL 24

Expert Comment

by:chaau
ID: 40470068
The problem is most likely when you assign 80 million records to the set. So, pit the try  catch in the first loop
0
 
LVL 31

Expert Comment

by:Zoppo
ID: 40470094
Hi HuaMinChen,

IMO this can't ever work in a 32-bit build (from the screenshot I assume your app is 32-bit) because if you want allocate 8.000.000 elements of size about 300 byte you'll need around 2,4 GB - this is more than a windows 32-bit app usually can allocate (i.e. take a look at http://stackoverflow.com/questions/639540/how-much-memory-can-a-32-bit-process-access-on-a-64-bit-operating-system about this).

I would suggest to change it to build a 64-bit application to avoid this limit.

Hope that helps,

ZOPPO
0
 
LVL 10

Author Comment

by:HuaMinChen
ID: 40470100
Thanks all.
Zoppo,
I am using VS 2010 to the project. Can you advise how to adjust the project to x64?
0
 
LVL 10

Author Comment

by:HuaMinChen
ID: 40470103
Chaau,
I do add try...catch to 1st for loop within that, and do get repeating "bad allocation" there. How to resolve it? Thanks.
0
 
LVL 31

Assisted Solution

by:Zoppo
Zoppo earned 250 total points
ID: 40470117
Hi HuaMinChen,

to do this you'll have to add a new configuration. To do this open 'Configuration Mananger' (i.e. from Build menu), open drop-down Active solution platform and select New. In the following dialog select x64 as platform type and press OK.

Following this you now should have all configurations you had before (usually 'Release' and 'Debug') twice, one for platform Win32, the other one for x64.

With a bit luck that's all you have to do to build 64 bit. It maybe you'll encounter build- or runtime errors, if so you need to fix them all. Even it's a good idea to check all compiler warnings if any, coz one of the most common cause for problems when porting from 32- to 64-bit is bad type casting or mixing of integer types which size may be different in x64 allthough they're the same in Win32.

If you have problems you can google for something like 'Visual Studio port Win32 x64', you'll find a lot of information.

ZOPPO
0
 
LVL 10

Author Comment

by:HuaMinChen
ID: 40470122
Sorry Zoppo.
I am using VS 2010 express. Can I open Configuration manager from that?
0
 
LVL 31

Expert Comment

by:Zoppo
ID: 40470126
Sorry, I can't say exactly coz I don't use 'Express', but I'm nearly 100% sure it exists in VS 2010 Express too. In VS 2010 there are two possibilities (I know) to open it:

- Menu 'Build' -> 'Configration Manager'
- In properties dialog (i.e. Menu 'Project' -> '<project name> Properties') there's a button 'Configuration Manager' at top-right corner.
0
 
LVL 31

Expert Comment

by:Zoppo
ID: 40470130
Addition: There might be more work needed depending on the history of the project. If it was i.e. created in VS 2010 there shouldn't be to do more, but if it's older it might be there are settings where the same path is set for both b platforms allthough they shouldn't.

If you don't plan to further be able to build as 32 bit you can ignore those settings and I would suggest to remove the Win32 platform from the configuration (even possible with 'Configuration Mananger').

But if you want to keep both platforms you should check if everything is fine.

As a first step you should check if this works:

1. select Win32 platform and do a Rebuild All
2. switch to x64 platform and do a Rebuild All too
3. switch back to Win32 platform and do a normal Build
4. stay in Win32 platform and do a Rebuild All again
5. switch to x64 platform again and do a normal Build

If everything is ok the steps 3 and 5 should finish immediate without recompiling anything.

Otherwise please tell then I can show you which settings probably aren't correct ...

ZOPPO
0
 
LVL 10

Author Comment

by:HuaMinChen
ID: 40470140
Many many thanks Zoppo.
I now open the original project (that is created using VS 2010 express), within VS 2013 Pro version. Can you tell me how to open configuration manager within VS 2013 Pro version?
0
 
LVL 10

Author Comment

by:HuaMinChen
ID: 40470145
Sorry, please disregard my last question.
0
 
LVL 10

Author Comment

by:HuaMinChen
ID: 40470147
Zoppo,
The attached one is the screenshot, within Config manager of VS 2010, can you please advise what to adjust on that? Thanks
t868.png
0
 
LVL 10

Author Comment

by:HuaMinChen
ID: 40470148
Please again disregard my current question. Sorry.
0
 
LVL 31

Accepted Solution

by:
Zoppo earned 250 total points
ID: 40470152
As told select <New> in the dropdown Active solution platform to open the dialog to create a 64 bit configuration.

Attached you see a screenshot (or better two in one) of the important settings.
screenshot.png
0

Featured Post

Free Tool: Postgres Monitoring System

A PHP and Perl based system to collect and display usage statistics from PostgreSQL databases.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Active Directory Audit 18 117
Issues with C++ Class 19 101
How to remove Recent Projects from Embarcadero C++ builder XE10. Berlin 2 66
C qsort compare function issue 6 16
This is a short and sweet, but (hopefully) to the point article. There seems to be some fundamental misunderstanding about the function prototype for the "main" function in C and C++, more specifically what type this function should return. I see so…
In a recent article here at Experts Exchange (http://www.experts-exchange.com/articles/18880/PaperPort-14-in-Windows-10-A-First-Look.html), I discussed my nine-month sandbox testing of the Windows 10 Technical Preview, specifically with respect to r…
The viewer will be introduced to the member functions push_back and pop_back of the vector class. The video will teach the difference between the two as well as how to use each one along with its functionality.
This is used to tweak the memory usage for your computer, it is used for servers more so than workstations but just be careful editing registry settings as it may cause irreversible results. I hold no responsibility for anything you do to the regist…

856 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question