[2 days left] What’s wrong with your cloud strategy? Learn why multicloud solutions matter with Nimble Storage.Register Now

x
?
Solved

ADO .NET Large Data Set - DataTable Out Of Memory Exception

Posted on 2008-06-09
5
Medium Priority
?
3,462 Views
Last Modified: 2013-11-26
Hi Experts!

Environment:
Visual Basic 2005
Visual Studio 2005 SP1
SQL Server 2005 (on Windows Server 2003 64-bit)
Crystal Reports XI R2 SP3
Host OS - Windows XP SP2 32 bit
.NET FrameWork 1.1 (latest)
.NET FrameWork 2.0 (latest)
.NET FrameWork 3.0 (latest)
.NET FrameWork 3.5 (latest)

I am working with an error that I can't seem to find a resolution to.  I have to populate a Crystal Report with data from the database.  We are using a DataTable object.  The query returns 24 columns from the database.  The application works fine until we have about 1,400,000 rows returned.  At about 1.4 million rows the DataSet/DataTable we are using causes an "System.OutOfMemoryException".  I cannot reduce the number of rows or the number of columns because this report is a drill down report for statistics.  In testing I was watching memory usage and found that while the host computer had 3 GB of RAM and a 3 GB pagefile, the application used about 26 MB of Active Memory and about 20 MB of VM.  Once the Database pull started the amounts added 1 GB of active memory and 1 GB of VM.  At the point it hits about 1 GB of each of VM and Active, the application would exception.

Is there a language barrier of 2GB for DataTables/DataSets?  Is there another approach that can be used to gather the data for display?
0
Comment
Question by:Venoym
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
5 Comments
 
LVL 17

Expert Comment

by:MIKE
ID: 21746031
Can you ADD GROUPING to the report...If it doesn't already have it...and flag it to GROUP ON SERVER..?

Mike
0
 

Author Comment

by:Venoym
ID: 21746610
We tried that first because we thought it was directly a Crystal Reports XI issue and it didn't help because of the number of rows returned even grouped.  I believe this is more of a .NET issue than a Crystal Reports XI issue.  I also think it's directly related to some kind of 2 GB limit in the DataTables/DataSets.
0
 

Author Comment

by:Venoym
ID: 21759668
I found a solution!

From: http://msdn.microsoft.com/en-us/library/bb147385.aspx

"      32-bit applications on 32-bit platforms can address up to 2 GB
"      32-bit applications built with the /LARGEADDRESSAWARE:YES linker flag on 32-bit Windows XP or Windows Server 2003 with the special /3gb boot option can address up to 3 GB. This constrains the kernel to only 1 GB which may cause some drivers and/or services to fail.
"      32-bit applications built with the /LARGEADDRESSAWARE:YES linker flag on 32-bit versions of Windows Vista, and on 32-bit versions of Windows Server Code Name "Longhorn" operating systems, can address memory up to the number specified by the boot configuration data (BCD) element IncreaseUserVa. IncreaseUserVa can have a value ranging from 2048, the default, to 3072 (which matches the amount of memory configured by the /3gb boot option on Windows XP). The remainder of 4 GB is allocated to the kernel and can result in failing driver and service configurations.
For more information about BCD, see Boot Configuration Data on MSDN.
"      32-bit applications on 64-bit platforms can address up to 2 GB, or up to 4 GB with the /LARGEADDRESSAWARE:YES linker flag.
"      64-bit applications use 43 bits for addressing, which provides 8 TB of virtual address for applications and 8 TB reserved for the kernel.

It is a good practice to specify large-address-aware when building 32-bit applications, by using the linker flag /LARGEADDRESSAWARE, even if the application is not intended for a 64-bit platform, because of the advantages that are gained at no cost. As explained earlier, enabling this flag for a build allows a 32-bit program to access more memory with special boot options on a 32-bit OS or on a 64-bit OS. However, developers must be careful that pointer assumptions are not made, such as assuming that the high-bit is never set in a 32-bit pointer. In general, enabling the /LARGEADDRESSAWARE flag is a good practice.


Moderator: How should I close this question?  And should points be awarded?
0
 
LVL 1

Accepted Solution

by:
Computer101 earned 0 total points
ID: 22022155
PAQed with points refunded (250)

Computer101
EE Admin
0

Featured Post

Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In my previous two articles we discussed Binary Serialization (http://www.experts-exchange.com/A_4362.html) and XML Serialization (http://www.experts-exchange.com/A_4425.html). In this article we will try to know more about SOAP (Simple Object Acces…
More often than not, we developers are confronted with a need: a need to make some kind of magic happen via code. Whether it is for a client, for the boss, or for our own personal projects, the need must be satisfied. Most of the time, the Framework…
Sometimes it takes a new vantage point, apart from our everyday security practices, to truly see our Active Directory (AD) vulnerabilities. We get used to implementing the same techniques and checking the same areas for a breach. This pattern can re…
In a question here at Experts Exchange (https://www.experts-exchange.com/questions/29062564/Adobe-acrobat-reader-DC.html), a member asked how to create a signature in Adobe Acrobat Reader DC (the free Reader product, not the paid, full Acrobat produ…

649 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question