Solved

Problem with the memory

Posted on 2014-11-27
21
209 Views
1 Endorsement
Last Modified: 2014-11-28
Hi,
I do get the attached error

using these codes

#include "stdafx.h"
#include <set>
#include <stdio.h>
#include <string.h>
#include <fstream>
#include <string>
#include <ctype.h>
#include <time.h>
#include <process.h>
#include <vector>
#include <iostream>
#include <algorithm>
#include <iomanip>
using namespace std;


struct nameval
{
	char fld_nm[100];
	wchar_t fld_nm_t[100];
     bool operator< (const nameval & a2) const
     {
           if(strcmp(fld_nm, a2.fld_nm) < 0) return true;
           if(strcmp(fld_nm, a2.fld_nm) > 0) return false;
           if (fld_val < a2.fld_val) return true;
           return false;
     }
	int fld_len;
	int fld_val;
};
nameval binrec;

int main()
{
    std::set<nameval> records;
    int cnt;
	for (cnt=0;cnt<8000000;cnt++)
	{
		nameval val={0};
		int j;
		for (j=0;j<20;j++)
		{
			val.fld_nm[j] += (char)(rand () % 58 + 64);
		}
		
		wchar_t wbuf[100]={0};
		mbstowcs(wbuf,val.fld_nm,_countof(val.fld_nm));
		wmemcpy(val.fld_nm_t,wbuf,100);

		val.fld_val=cnt;
		records.insert(val);
	}

	std::ofstream ostrm("c:\\dp4\\flout.bin", std::ios::binary | std::ios::out );
	if (ostrm.is_open())
	{
		for (std::set<nameval>::iterator it = records.begin(); it != records.end(); ++it)
		{
			ostrm.write((char *)&records, sizeof(nameval));
		}
	}

	std::wofstream ostrm2("c:\\dp4\\flout.ord", std::ios::out );
	if (ostrm2.is_open())
	{
		for (std::set<nameval>::iterator it = records.begin(); it != records.end(); ++it)
		{
			ostrm2 << "\"" << it->fld_nm << "\" " << it->fld_len << ' '  << it->fld_nm_t << ' '<< it->fld_val << '\0' << '\n';
		}
	}

 	system("pause>null");

	return 0;
}

Open in new window

t867.png
1
Comment
Question by:HuaMinChen
  • 11
  • 5
  • 3
21 Comments
 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
No, I am to confirm if the way is fine to handle very big vector list. Thanks.
0
 
LVL 24

Assisted Solution

by:chaau
chaau earned 250 total points
Comment Utility
I think you wanted this on line 59:
ostrm.write((char *)&it, sizeof(nameval));

Open in new window

(basically you need to write iterator)
0
 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
Thanks to your advice to do the correction.
I've corrected it above but I still have the memory problem, after having re-built the project.
0
 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
It is still the same problem.
0
 
LVL 24

Assisted Solution

by:chaau
chaau earned 250 total points
Comment Utility
try to embed everything inside your for() loop into a try/catch block and handle the exceptions
0
 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
I have this part now
		for (std::set<nameval>::iterator it = records.begin(); it != records.end(); ++it)
		{
			try
			{
				ostrm.write((char *)&it, sizeof(nameval));
			}
			catch (exception& e)
			{
				cout << e.what() << '\n';
			}
		}
		...

Open in new window

but I still encounter the same when running the re-built project.
0
 
LVL 24

Expert Comment

by:chaau
Comment Utility
The problem is most likely when you assign 80 million records to the set. So, pit the try  catch in the first loop
0
 
LVL 30

Expert Comment

by:Zoppo
Comment Utility
Hi HuaMinChen,

IMO this can't ever work in a 32-bit build (from the screenshot I assume your app is 32-bit) because if you want allocate 8.000.000 elements of size about 300 byte you'll need around 2,4 GB - this is more than a windows 32-bit app usually can allocate (i.e. take a look at http://stackoverflow.com/questions/639540/how-much-memory-can-a-32-bit-process-access-on-a-64-bit-operating-system about this).

I would suggest to change it to build a 64-bit application to avoid this limit.

Hope that helps,

ZOPPO
0
 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
Thanks all.
Zoppo,
I am using VS 2010 to the project. Can you advise how to adjust the project to x64?
0
Better Security Awareness With Threat Intelligence

See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
Chaau,
I do add try...catch to 1st for loop within that, and do get repeating "bad allocation" there. How to resolve it? Thanks.
0
 
LVL 30

Assisted Solution

by:Zoppo
Zoppo earned 250 total points
Comment Utility
Hi HuaMinChen,

to do this you'll have to add a new configuration. To do this open 'Configuration Mananger' (i.e. from Build menu), open drop-down Active solution platform and select New. In the following dialog select x64 as platform type and press OK.

Following this you now should have all configurations you had before (usually 'Release' and 'Debug') twice, one for platform Win32, the other one for x64.

With a bit luck that's all you have to do to build 64 bit. It maybe you'll encounter build- or runtime errors, if so you need to fix them all. Even it's a good idea to check all compiler warnings if any, coz one of the most common cause for problems when porting from 32- to 64-bit is bad type casting or mixing of integer types which size may be different in x64 allthough they're the same in Win32.

If you have problems you can google for something like 'Visual Studio port Win32 x64', you'll find a lot of information.

ZOPPO
0
 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
Sorry Zoppo.
I am using VS 2010 express. Can I open Configuration manager from that?
0
 
LVL 30

Expert Comment

by:Zoppo
Comment Utility
Sorry, I can't say exactly coz I don't use 'Express', but I'm nearly 100% sure it exists in VS 2010 Express too. In VS 2010 there are two possibilities (I know) to open it:

- Menu 'Build' -> 'Configration Manager'
- In properties dialog (i.e. Menu 'Project' -> '<project name> Properties') there's a button 'Configuration Manager' at top-right corner.
0
 
LVL 30

Expert Comment

by:Zoppo
Comment Utility
Addition: There might be more work needed depending on the history of the project. If it was i.e. created in VS 2010 there shouldn't be to do more, but if it's older it might be there are settings where the same path is set for both b platforms allthough they shouldn't.

If you don't plan to further be able to build as 32 bit you can ignore those settings and I would suggest to remove the Win32 platform from the configuration (even possible with 'Configuration Mananger').

But if you want to keep both platforms you should check if everything is fine.

As a first step you should check if this works:

1. select Win32 platform and do a Rebuild All
2. switch to x64 platform and do a Rebuild All too
3. switch back to Win32 platform and do a normal Build
4. stay in Win32 platform and do a Rebuild All again
5. switch to x64 platform again and do a normal Build

If everything is ok the steps 3 and 5 should finish immediate without recompiling anything.

Otherwise please tell then I can show you which settings probably aren't correct ...

ZOPPO
0
 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
Many many thanks Zoppo.
I now open the original project (that is created using VS 2010 express), within VS 2013 Pro version. Can you tell me how to open configuration manager within VS 2013 Pro version?
0
 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
Sorry, please disregard my last question.
0
 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
Zoppo,
The attached one is the screenshot, within Config manager of VS 2010, can you please advise what to adjust on that? Thanks
t868.png
0
 
LVL 10

Author Comment

by:HuaMinChen
Comment Utility
Please again disregard my current question. Sorry.
0
 
LVL 30

Accepted Solution

by:
Zoppo earned 250 total points
Comment Utility
As told select <New> in the dropdown Active solution platform to open the dialog to create a 64 bit configuration.

Attached you see a screenshot (or better two in one) of the important settings.
screenshot.png
0

Featured Post

What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

Join & Write a Comment

Go is an acronym of golang, is a programming language developed Google in 2007. Go is a new language that is mostly in the C family, with significant input from Pascal/Modula/Oberon family. Hence Go arisen as low-level language with fast compilation…
#Citrix #POC #XenDesktop #vCenter #VMware #ESX
Video by: Grant
The goal of this video is to provide viewers with basic examples to understand and use while-loops in the C programming language.
The viewer will learn how to user default arguments when defining functions. This method of defining functions will be contrasted with the non-default-argument of defining functions.

763 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now