ADO .NET Large Data Set - DataTable Out Of Memory Exception

Hi Experts!

Environment:
Visual Basic 2005
Visual Studio 2005 SP1
SQL Server 2005 (on Windows Server 2003 64-bit)
Crystal Reports XI R2 SP3
Host OS - Windows XP SP2 32 bit
.NET FrameWork 1.1 (latest)
.NET FrameWork 2.0 (latest)
.NET FrameWork 3.0 (latest)
.NET FrameWork 3.5 (latest)

I am working with an error that I can't seem to find a resolution to.  I have to populate a Crystal Report with data from the database.  We are using a DataTable object.  The query returns 24 columns from the database.  The application works fine until we have about 1,400,000 rows returned.  At about 1.4 million rows the DataSet/DataTable we are using causes an "System.OutOfMemoryException".  I cannot reduce the number of rows or the number of columns because this report is a drill down report for statistics.  In testing I was watching memory usage and found that while the host computer had 3 GB of RAM and a 3 GB pagefile, the application used about 26 MB of Active Memory and about 20 MB of VM.  Once the Database pull started the amounts added 1 GB of active memory and 1 GB of VM.  At the point it hits about 1 GB of each of VM and Active, the application would exception.

Is there a language barrier of 2GB for DataTables/DataSets?  Is there another approach that can be used to gather the data for display?
VenoymAsked:
Who is Participating?

Improve company productivity with a Business Account.Sign Up

x
 
Computer101Connect With a Mentor Commented:
PAQed with points refunded (250)

Computer101
EE Admin
0
 
MIKESoftware Solutions ConsultantCommented:
Can you ADD GROUPING to the report...If it doesn't already have it...and flag it to GROUP ON SERVER..?

Mike
0
 
VenoymAuthor Commented:
We tried that first because we thought it was directly a Crystal Reports XI issue and it didn't help because of the number of rows returned even grouped.  I believe this is more of a .NET issue than a Crystal Reports XI issue.  I also think it's directly related to some kind of 2 GB limit in the DataTables/DataSets.
0
 
VenoymAuthor Commented:
I found a solution!

From: http://msdn.microsoft.com/en-us/library/bb147385.aspx

"      32-bit applications on 32-bit platforms can address up to 2 GB
"      32-bit applications built with the /LARGEADDRESSAWARE:YES linker flag on 32-bit Windows XP or Windows Server 2003 with the special /3gb boot option can address up to 3 GB. This constrains the kernel to only 1 GB which may cause some drivers and/or services to fail.
"      32-bit applications built with the /LARGEADDRESSAWARE:YES linker flag on 32-bit versions of Windows Vista, and on 32-bit versions of Windows Server Code Name "Longhorn" operating systems, can address memory up to the number specified by the boot configuration data (BCD) element IncreaseUserVa. IncreaseUserVa can have a value ranging from 2048, the default, to 3072 (which matches the amount of memory configured by the /3gb boot option on Windows XP). The remainder of 4 GB is allocated to the kernel and can result in failing driver and service configurations.
For more information about BCD, see Boot Configuration Data on MSDN.
"      32-bit applications on 64-bit platforms can address up to 2 GB, or up to 4 GB with the /LARGEADDRESSAWARE:YES linker flag.
"      64-bit applications use 43 bits for addressing, which provides 8 TB of virtual address for applications and 8 TB reserved for the kernel.

It is a good practice to specify large-address-aware when building 32-bit applications, by using the linker flag /LARGEADDRESSAWARE, even if the application is not intended for a 64-bit platform, because of the advantages that are gained at no cost. As explained earlier, enabling this flag for a build allows a 32-bit program to access more memory with special boot options on a 32-bit OS or on a 64-bit OS. However, developers must be careful that pointer assumptions are not made, such as assuming that the high-bit is never set in a 32-bit pointer. In general, enabling the /LARGEADDRESSAWARE flag is a good practice.


Moderator: How should I close this question?  And should points be awarded?
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.