?
Solved

Problem with the memory

Posted on 2014-11-27
21
Medium Priority
?
224 Views
1 Endorsement
Last Modified: 2014-11-28
Hi,
I do get the attached error

using these codes

#include "stdafx.h"
#include <set>
#include <stdio.h>
#include <string.h>
#include <fstream>
#include <string>
#include <ctype.h>
#include <time.h>
#include <process.h>
#include <vector>
#include <iostream>
#include <algorithm>
#include <iomanip>
using namespace std;


struct nameval
{
	char fld_nm[100];
	wchar_t fld_nm_t[100];
     bool operator< (const nameval & a2) const
     {
           if(strcmp(fld_nm, a2.fld_nm) < 0) return true;
           if(strcmp(fld_nm, a2.fld_nm) > 0) return false;
           if (fld_val < a2.fld_val) return true;
           return false;
     }
	int fld_len;
	int fld_val;
};
nameval binrec;

int main()
{
    std::set<nameval> records;
    int cnt;
	for (cnt=0;cnt<8000000;cnt++)
	{
		nameval val={0};
		int j;
		for (j=0;j<20;j++)
		{
			val.fld_nm[j] += (char)(rand () % 58 + 64);
		}
		
		wchar_t wbuf[100]={0};
		mbstowcs(wbuf,val.fld_nm,_countof(val.fld_nm));
		wmemcpy(val.fld_nm_t,wbuf,100);

		val.fld_val=cnt;
		records.insert(val);
	}

	std::ofstream ostrm("c:\\dp4\\flout.bin", std::ios::binary | std::ios::out );
	if (ostrm.is_open())
	{
		for (std::set<nameval>::iterator it = records.begin(); it != records.end(); ++it)
		{
			ostrm.write((char *)&records, sizeof(nameval));
		}
	}

	std::wofstream ostrm2("c:\\dp4\\flout.ord", std::ios::out );
	if (ostrm2.is_open())
	{
		for (std::set<nameval>::iterator it = records.begin(); it != records.end(); ++it)
		{
			ostrm2 << "\"" << it->fld_nm << "\" " << it->fld_len << ' '  << it->fld_nm_t << ' '<< it->fld_val << '\0' << '\n';
		}
	}

 	system("pause>null");

	return 0;
}

Open in new window

t867.png
1
Comment
Question by:HuaMinChen
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 11
  • 5
  • 3
21 Comments
 
LVL 11

Author Comment

by:HuaMinChen
ID: 40469943
No, I am to confirm if the way is fine to handle very big vector list. Thanks.
0
 
LVL 25

Assisted Solution

by:chaau
chaau earned 1000 total points
ID: 40469967
I think you wanted this on line 59:
ostrm.write((char *)&it, sizeof(nameval));

Open in new window

(basically you need to write iterator)
0
 
LVL 11

Author Comment

by:HuaMinChen
ID: 40469971
Thanks to your advice to do the correction.
I've corrected it above but I still have the memory problem, after having re-built the project.
0
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 11

Author Comment

by:HuaMinChen
ID: 40469972
It is still the same problem.
0
 
LVL 25

Assisted Solution

by:chaau
chaau earned 1000 total points
ID: 40469976
try to embed everything inside your for() loop into a try/catch block and handle the exceptions
0
 
LVL 11

Author Comment

by:HuaMinChen
ID: 40469983
I have this part now
		for (std::set<nameval>::iterator it = records.begin(); it != records.end(); ++it)
		{
			try
			{
				ostrm.write((char *)&it, sizeof(nameval));
			}
			catch (exception& e)
			{
				cout << e.what() << '\n';
			}
		}
		...

Open in new window

but I still encounter the same when running the re-built project.
0
 
LVL 25

Expert Comment

by:chaau
ID: 40470068
The problem is most likely when you assign 80 million records to the set. So, pit the try  catch in the first loop
0
 
LVL 31

Expert Comment

by:Zoppo
ID: 40470094
Hi HuaMinChen,

IMO this can't ever work in a 32-bit build (from the screenshot I assume your app is 32-bit) because if you want allocate 8.000.000 elements of size about 300 byte you'll need around 2,4 GB - this is more than a windows 32-bit app usually can allocate (i.e. take a look at http://stackoverflow.com/questions/639540/how-much-memory-can-a-32-bit-process-access-on-a-64-bit-operating-system about this).

I would suggest to change it to build a 64-bit application to avoid this limit.

Hope that helps,

ZOPPO
0
 
LVL 11

Author Comment

by:HuaMinChen
ID: 40470100
Thanks all.
Zoppo,
I am using VS 2010 to the project. Can you advise how to adjust the project to x64?
0
 
LVL 11

Author Comment

by:HuaMinChen
ID: 40470103
Chaau,
I do add try...catch to 1st for loop within that, and do get repeating "bad allocation" there. How to resolve it? Thanks.
0
 
LVL 31

Assisted Solution

by:Zoppo
Zoppo earned 1000 total points
ID: 40470117
Hi HuaMinChen,

to do this you'll have to add a new configuration. To do this open 'Configuration Mananger' (i.e. from Build menu), open drop-down Active solution platform and select New. In the following dialog select x64 as platform type and press OK.

Following this you now should have all configurations you had before (usually 'Release' and 'Debug') twice, one for platform Win32, the other one for x64.

With a bit luck that's all you have to do to build 64 bit. It maybe you'll encounter build- or runtime errors, if so you need to fix them all. Even it's a good idea to check all compiler warnings if any, coz one of the most common cause for problems when porting from 32- to 64-bit is bad type casting or mixing of integer types which size may be different in x64 allthough they're the same in Win32.

If you have problems you can google for something like 'Visual Studio port Win32 x64', you'll find a lot of information.

ZOPPO
0
 
LVL 11

Author Comment

by:HuaMinChen
ID: 40470122
Sorry Zoppo.
I am using VS 2010 express. Can I open Configuration manager from that?
0
 
LVL 31

Expert Comment

by:Zoppo
ID: 40470126
Sorry, I can't say exactly coz I don't use 'Express', but I'm nearly 100% sure it exists in VS 2010 Express too. In VS 2010 there are two possibilities (I know) to open it:

- Menu 'Build' -> 'Configration Manager'
- In properties dialog (i.e. Menu 'Project' -> '<project name> Properties') there's a button 'Configuration Manager' at top-right corner.
0
 
LVL 31

Expert Comment

by:Zoppo
ID: 40470130
Addition: There might be more work needed depending on the history of the project. If it was i.e. created in VS 2010 there shouldn't be to do more, but if it's older it might be there are settings where the same path is set for both b platforms allthough they shouldn't.

If you don't plan to further be able to build as 32 bit you can ignore those settings and I would suggest to remove the Win32 platform from the configuration (even possible with 'Configuration Mananger').

But if you want to keep both platforms you should check if everything is fine.

As a first step you should check if this works:

1. select Win32 platform and do a Rebuild All
2. switch to x64 platform and do a Rebuild All too
3. switch back to Win32 platform and do a normal Build
4. stay in Win32 platform and do a Rebuild All again
5. switch to x64 platform again and do a normal Build

If everything is ok the steps 3 and 5 should finish immediate without recompiling anything.

Otherwise please tell then I can show you which settings probably aren't correct ...

ZOPPO
0
 
LVL 11

Author Comment

by:HuaMinChen
ID: 40470140
Many many thanks Zoppo.
I now open the original project (that is created using VS 2010 express), within VS 2013 Pro version. Can you tell me how to open configuration manager within VS 2013 Pro version?
0
 
LVL 11

Author Comment

by:HuaMinChen
ID: 40470145
Sorry, please disregard my last question.
0
 
LVL 11

Author Comment

by:HuaMinChen
ID: 40470147
Zoppo,
The attached one is the screenshot, within Config manager of VS 2010, can you please advise what to adjust on that? Thanks
t868.png
0
 
LVL 11

Author Comment

by:HuaMinChen
ID: 40470148
Please again disregard my current question. Sorry.
0
 
LVL 31

Accepted Solution

by:
Zoppo earned 1000 total points
ID: 40470152
As told select <New> in the dropdown Active solution platform to open the dialog to create a 64 bit configuration.

Attached you see a screenshot (or better two in one) of the important settings.
screenshot.png
0

Featured Post

Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

I don't know if many of you have made the great mistake of using the Cisco Thin Client model with the management software VXC. If you have then you are probably more then familiar with the incredibly clunky interface, the numerous work arounds, and …
In a recent article here at Experts Exchange (http://www.experts-exchange.com/articles/18880/PaperPort-14-in-Windows-10-A-First-Look.html), I discussed my nine-month sandbox testing of the Windows 10 Technical Preview, specifically with respect to r…
The viewer will learn how to clear a vector as well as how to detect empty vectors in C++.
This is used to tweak the memory usage for your computer, it is used for servers more so than workstations but just be careful editing registry settings as it may cause irreversible results. I hold no responsibility for anything you do to the regist…

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question