Solved

Array System.OutofMemoryException

Posted on 2004-03-23
15
825 Views
Last Modified: 2012-06-27
Hi,

I am puzzled and looking for a workaround if possible, and some how too info.

I have an array, 4 dimensions

Per MS documentation at <a href="http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vbcn7/html/vacondeclaringarrays.asp">Declaring Array Variables</a>

No one of my dimensions is greater than 2^64 -1
actually not even close, maximum dimensions size is 256

So I could have Dim MyArray(256,256,256,256) As Integer
as the largest array [actually, do a ReDim based upon input from user, but above is maximum size]

When I have such an array, I get System.OutofMemoryException, error.
By trial and error, so far largest array that does not throw the error is 100,100,100,100
[actually I am somewhere between 100 & <130, on my machine]

But MS documentation, same link also says:
"The total size limit of an array varies, based on your operating system and how much memory is available. Using an array that exceeds the amount of RAM available on your system is slower because the data must be read from and written to disk."

OK, so I have an array larger than RAM can handle . . . I'll take the hit to speed, it can read/write to disk, but it is obviously not doing so.

The way I read MS doc is that with my large array it should be reading/writing to disk, yes it slows down program, but that is better than it throwing OutofMemoryException.

Some questions:
1. Is MS doc wrong, or is there some parameter, class, setting I need to specify to tell it to read/write large arrays? e.g. somewhere in program a line that reads in essence "OK stupid, if an array is too big for memory, read/write it to disk, like MS doc says you should"

2. Ignoring idiocy of doc for a second, is there some other way I can store the data that gets stored in the array and achieve the functionality [I know of a couple of ways myself, but they would involve major rewrite of code - read/write to a file, bust array up into parts - that is there some other object/class that could do it?]

3. I suspect MS doc is wrong. In which case how do I tell before program gets to the error, how big of an array the memory for a given machine can handle?
That is, when user specifies array size, have program do a calculation so as determine at user input time, that the array is either too big for that user's machine, or machine does have enough RAM?
And if not enough RAM for say 256^4, how big the array could be and still run?
Rather than user enters size, runs it, gets error, tries again w different value, gets error, tries again...?
0
Comment
Question by:tlfeet
  • 7
  • 7
15 Comments
 
LVL 26

Expert Comment

by:EDDYKT
Comment Utility
I've no problem to doing this

a(101, 101, 101, 101) = 30
Dim b As Integer = a(101, 101, 101, 101)

the IDE threw an exception when I try to read the data, but I get the value b back
0
 

Author Comment

by:tlfeet
Comment Utility
Hi EDDYKT,

Perhaps I was not clear.
I know I can do an array of 100,100,100,100
I also know 130,130,130,130 cause error.
In between, I do not know.

2 problems:
1. really need to be able to handle up to 256,256,256,256
2. if I can't handle that, need to be able to tell, at time user inputs, if the dim user specified will be too big for their machine, so that:
a. can tell them so, before processing starts
b. aid me in coming up with some sort of work around.

[trial and error to find the limit not best way - either at my end writing the program & at user end]
0
 
LVL 12

Expert Comment

by:dfiala13
Comment Utility
1. Don't think it will.

2a.  You can call a test function and see if their machine will handle what they need...

Private Function TestMaxSize(i As Integer) As Boolean

On Error GoTo ErrHandler
ReDim ia(i, i, i, i) As Integer
    TestMaxSize = True
   
  Exit Function
 
ErrHandler:
    TestMaxSize = False
End Function
0
 

Author Comment

by:tlfeet
Comment Utility
Hi dfiala13

Re: #2 yes that would work.
Still a bit of trial and error in that user enters number, it tells user its too big, try another smaller number, user tries another smaller number & that turns out to be too big too, so try another, etc.

Would be very nice to construct a Procedure/Function that determines how big an array the computer/a computer/any computer could handle and just tell them upfront, no trial and error
"Hey user, the biggest array your 'puter can handle is X^4"

BTW, from trial and error found my computer handles up to 128^4
also figured out that when it gets close to that - 120^4 at least, it does read/write to virtual memory

Also "found out", not solid info, but likely culprit that Windows XP can read/write an "object" only up to 4 gigs in size. Which in practice/real world is probably more like 2 gigs given OS, program overhead.

And, guess what?
256^4 = 4.294 G +- a few bytes ;-)

I might have to do some major rewrite on my program and figure out how to split the larger arrays up.
Concerned that splitting for example 256^4 into say 2 x 128^4 is going to cause problems too (or 4 x 64^4 or 8x32^4, etc.) as the total amount of memory is actually going up given the overhead of the extra arrays.

0
 
LVL 12

Expert Comment

by:dfiala13
Comment Utility
The problem is the stack size:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vbcon98/html/vbcondatalimitations.asp

Right around 150 elements per rank, you start to use around 1 MB of stack and that is all that can be on the stack in any one thread.  

I tried to load as a module level variable, with the same issues, though supposedly VB uses a different data segment in those cases.

here's a function that calls the one I wrote above to find the max size available
Private Function GetMaxSize(iStart As Integer) As Integer
Dim i As Integer

    For i = iStart To 256
        If TestMaxSize(i) = False Then
            GetMaxSize = i - 1
            Exit For
        End If
    Next
End Function
0
 
LVL 12

Expert Comment

by:dfiala13
Comment Utility
I need to do better math that's 1 GB around 150. -- doh!

What are you trying to use the arrays for?  VB might not be the best tool to do what you need.  Could be worth the time to write a C dll that does the heavy lifting.
0
 

Author Comment

by:tlfeet
Comment Utility
Hi dfiala13,

I am doing some image processing.
Certainly VB is not best language, but it was what  I had access to at the time I started and most importantly knew how to get the info (pixel data) out of the image using VB.

Looking at moving it over to Java, to plug in/work with a program called ImageJ
-but beofre I do that, want to make sure Java can handle data too. ;-)
0
Enabling OSINT in Activity Based Intelligence

Activity based intelligence (ABI) requires access to all available sources of data. Recorded Future allows analysts to observe structured data on the open, deep, and dark web.

 
LVL 12

Expert Comment

by:dfiala13
Comment Utility
Good plan.  Java "should" be less encumbered by memory limitations.
0
 

Author Comment

by:tlfeet
Comment Utility
Hi dfiala13,

OK, wierd things are happening.
I just got two notices from EE about this thread and they were for posts before your last one, starting w "I need to do better math that's 1 GB around 150"

Anyway, thanks for the link to the MSDN article.
Had not seen that (even after extensive search ;-) )

I started taking a deeper look at imageJ angle.
It is going to mean a much steeper learning curve than I had hoped.
But a bigger problem, I just realized is that I need to write results to a MS SQL Server database and that if I do it in ImageJ then I have to get an ODBC driver....etc etc.

My little program has one main task:
To process thousands of images, of Fabrics (cloth), figure out what color(s) the images, hence the Fabrics are and write that "ColorResults" data set to the database for each Fabric/image.

The images are jpegs, so VB gives me the A, R, G, B values for each pixel
As an intermediary step, I store the frequency of each color.

Right now, I am trying to figure out what set of parameters returns the best results. e.g. do I average pixel values, over what size of an area?  What shape of area? Do I analyze individual, unique colors or work say in WebSafe colors (or some other space).  If I average, do I do a weighted average? etc. etc.

For example if I use unique indiviudal colors, the full ARGB space, and analyze a Fabric/Image, call it TestImage2, then the most frequent colors are all black, but the Fabric is mutli-colored.

So if I use Websafe colors (0, 51, 102...etc - which I actually use an index in the intermediate stage) then the blacks are all grouped together, as are all the other shades of say red, orange, etc.
And the results is what I need.

In the first case, using/allowing the full RGB range, I dim my array like this:
ReDim FrequencyArray(255,255,255,255)
then each color gets counted like this:
FrequencyArray(currenta, currentr, currentg, currentb)+=1

In second case I convert the actuall values to an index running from 0 to 5, an my array works like this:
ReDim FrequencyArray(5,5,5,5)
and count like this:
FrequencyArray(currenta, currentr, currentg, currentb)+=1

Of course the 255^4 dim statement bombs out.
But in playing around with different parameters, I found WebSafe returns too few colors, too broad of a brush, while of course, the full range 0-255 for RGB returns results that are too fine.

So I need somewhere in between, as I do not know yet, what, I am running into the the question:
"How big can my space be before I hit the upper limit of VB handling the resulting/needed array?"

Since the images I am currently working with are gif/jpg, the A value is always 255 anyway, so I can just ingore it and do this
ReDim frequencyArray(255,255,255) or (0,255,255,255)
or likes this in second case:
ReDim FrequncyArray(5,5,5) or (0,5,5,5)

The problem is:
1. who knows what kind of images I will be using say in a year or 2, and then need that 4th rank?
2. as I said, I am still trying to figure out the best processing parameters to use, so who knows what I will end up needing.
3. If I ignore Alpha all together or handle it "specially" I have all this code to write to say, in essence:
"Do this unless it is the Alpha value, in which case do this..."
Just be nicer/faster/cleaner to be able to treat all the values the same.
4. I am trying to write the program so as to be as generally usefull as possible (there are other uses to which it could be put) and those other uses may need the 4th rank.

So, for my immediate task/goal & image inputs I can just ignore alpha values, or even wirte code to handle them specially/differently

It would just be nice to be able to handle the larger arrays.
Then I would have clean, simple code, that works with all the data in an Image.

Thanks for the little code tidbits - they will be helpful.
Basically, have a bee in my bonnet about the large arrays ;-)
0
 
LVL 12

Accepted Solution

by:
dfiala13 earned 500 total points
Comment Utility
Interesting problem....

My first thought is that while you are dimensioning a huge array, it sounds like it is sparsely populated (ie, you don't have 4 GB of individual colors to count frequencies for).

So instead of creating this huge array, why not create a dictionary of color objects.  I have listed out two classes, one that is the master used to manage a dictionary of color objects.  The color objects are defined in MyColor.cls.

You can create the color counter object for a new fabric and call it's CountColor method for each pixel you process.  it will add unique colors (defined by unique combinations of ARGB) or simply add to the frequency count if the color is already in the dictionary.

The dictionary class is available in the Microsoft Scriptiing Runtime which is likely already on your machine.  Just need to add a reference to it.

'---class file ColorCounter.cls

Option Explicit

Private m_dictColor As Dictionary


Private Sub Class_Initialize()
    Set m_dictColor = New Dictionary
   
End Sub

Public Sub CountColor(ByVal btAlpha As Byte, ByVal btRed As Byte, ByVal btGreen As Byte, ByVal btBlue As Byte)
Dim sKey As String
Dim c As MyColor

    sKey = Hex$(btAlpha) & Hex$(btRed) & Hex$(btGreen) & Hex$(btBlue)
    If m_dictColor.Exists(sKey) Then
        Set c = m_dictColor.Item(sKey)
        c.Add
    Else
         Set c = New MyColor
         c.Init btAlpha, btRed, btGreen, btBlue, sKey
         m_dictColor.Add sKey, c
    End If
   
End Sub

Public Property Get Colors() As Dictionary
    Set Colors = m_dictColor
End Property
Private Sub Class_Terminate()
    Set m_dictColor = Nothing
End Sub


'---class file MyColor.cls

Option Explicit

Private m_btAlpha As Byte
Private m_btRed As Byte
Private m_btGreen As Byte
Private m_btBlue As Byte

Private m_iCount As Integer
Private m_sKey As String
Public Sub Init(ByVal btAlpha As Byte, ByVal btRed As Byte, ByVal btGreen As Byte, ByVal btBlue As Byte, ByVal sKey As String)
    m_btAlpha = btAlpha
    m_btRed = btRed
    m_btGreen = btGreen
    m_btBlue = btBlue
    m_iCount = 1
    m_sKey = sKey
End Sub

Public Sub Add()
    m_iCount = m_iCount + 1
End Sub

Public Property Get Key() As String
    Key = m_sKey
End Property
Public Property Get Description() As String
    Description = m_btAlpha & " : " & m_btRed & " : " & m_btGreen & " : " & m_btBlue
End Property

Public Property Get Frequency() As Integer
    Frequency = m_iCount
End Property
0
 

Author Comment

by:tlfeet
Comment Utility
Hi dfiala13

WOW!!!
That will do the trick.

I was using arrays, cause well did not know what else to use and excepting the outofmemory issue, it does work pretty well and is easy to code and understand what is going on.

Yes you are right, I do not need all 4GB the array offers, it is sparsely populated.

So Ieven thought of doing something similar to dictionary with arrays.
i.e. create an array that holds the unique values and go from there.
But that involved a lot of sorting & going thru the array, whose size needed would very from image to image, etc. etc.

The image I spoke of is 400x400px = 160,000 and has over 100,000 unique colors (some only a pixel or 2) - so while even that one sparsely populates 4GB array, holding 100,000 values in some other array leads to real performance hit.

I just knew there had to be some other method/class/feaute/thingy in VB that would handle what I need.
Your dictionary idea certainly seems like it will do the trick.

Now off to do a little reading up on it ;-)

Thanks again dfiala13

I will come back shortly - going to try a qucky verison of it, after doing some reading.
WOW!
0
 

Author Comment

by:tlfeet
Comment Utility
Hi dfiala13,

Just an update.
Got side-tracked here at work with some other things....geez go figure ;-)
Started importing/implementing your codes snippets for the Dictionary object.

Ran into a couple of snags
1. I have no clue what I am doing - but learning fast!!!!
2. Forgot to mention that I am doing this in VB.NET in Visual Studio.NET
[but please do not rewrite your code - I am well into reworking for VB.NET (e.g "Public Property Get Colors() As Dictionary" is not valid .NET becuase of the "GET", as the "GET" gets put inside the Property . . . no big deal, just to illustrate reworking)]

3. I did want to double check a couple of things:

A.
You said to reference "Microsoft Scriptiing Runtime" for the dictionary
I could not find such a namespace/class
So I am inheriting "System.Collections.DictionaryBase"
Is that the correct/best animal?  Seems to be....but see #1 above ;-)

B.
Have a few minor points.  Not really stuck on them, just have to do more reading, searching...
e.g.
m_dictColor.Exists(sKey) - Exists not used in .NET
c = m_dictColor.Item(sKey) is not a member of MyColor
and similar type Build Errors

I am pretty sure I can fix them (already have fixed about 20 of them) - but I just wanted to make sure I am on the right track.

Does it sound like it?
Or should I go to school? ;-) Or should I fire myself and hire someone else?
0
 
LVL 12

Expert Comment

by:dfiala13
Comment Utility
Well, if you are doing this in .NET, why didn't you say so!?! Much, much easier. Geesh.

I was giving you VB6 code, hence the reference to the scripting runtime and the errors.

Since you are using .NET you yon't have to use the scripting runtime or do any inhertance.  Just use the Hashtable instead of the dictionary and instead of Init on the MyColor class use a constructor.

instead of Exists on the Hashtable use Contains
and you can get the object out of the hashtable like this:

c = m_ht(sKey)

But you are more or less on the right track.

Feel free to hire me.  Have compiler will travel ;)


0
 

Author Comment

by:tlfeet
Comment Utility
Hi dfiala13,

I got the Hashtable/MyColor working.

Not only does it work, but becuase I am not dealing with huge, sparsely populated arrays:
1. code runs faster
2. I can get a little more sophisticated with what/how it does its thing.

e.g. as I mentioned I am trying to figure out the best way to find the colors, average over what size, shape of an area, what size of a colorspace to use, etc.

So I have it loop/batch thru some images, and loop thru various settings, the hashtable gives me a much better way to store the info, so that I get better control over how/what it loops thru and how it spits out the imtermediary/final results - which I then take a look at to see which set of parameters gives the best results.

Much, much better . . .  now off to figure out how to sort it!


Thanks a ton!!!
0
 
LVL 12

Expert Comment

by:dfiala13
Comment Utility
You're welcome.

Glad you got it working.

0

Featured Post

6 Surprising Benefits of Threat Intelligence

All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

Join & Write a Comment

The debugging module of the VB 6 IDE can be accessed by way of the Debug menu item. That menu item can normally be found in the IDE's main menu line as shown in this picture.   There is also a companion Debug Toolbar that looks like the followin…
Enums (shorthand for ‘enumerations’) are not often used by programmers but they can be quite valuable when they are.  What are they? An Enum is just a type of variable like a string or an Integer, but in this case one that you create that contains…
Get people started with the process of using Access VBA to control Outlook using automation, Microsoft Access can control other applications. An example is the ability to programmatically talk to Microsoft Outlook. Using automation, an Access applic…
This lesson covers basic error handling code in Microsoft Excel using VBA. This is the first lesson in a 3-part series that uses code to loop through an Excel spreadsheet in VBA and then fix errors, taking advantage of error handling code. This l…

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now