Link to home
Start Free TrialLog in
Avatar of tlfeet
tlfeet

asked on

Array System.OutofMemoryException

Hi,

I am puzzled and looking for a workaround if possible, and some how too info.

I have an array, 4 dimensions

Per MS documentation at <a href="http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vbcn7/html/vacondeclaringarrays.asp">Declaring Array Variables</a>

No one of my dimensions is greater than 2^64 -1
actually not even close, maximum dimensions size is 256

So I could have Dim MyArray(256,256,256,256) As Integer
as the largest array [actually, do a ReDim based upon input from user, but above is maximum size]

When I have such an array, I get System.OutofMemoryException, error.
By trial and error, so far largest array that does not throw the error is 100,100,100,100
[actually I am somewhere between 100 & <130, on my machine]

But MS documentation, same link also says:
"The total size limit of an array varies, based on your operating system and how much memory is available. Using an array that exceeds the amount of RAM available on your system is slower because the data must be read from and written to disk."

OK, so I have an array larger than RAM can handle . . . I'll take the hit to speed, it can read/write to disk, but it is obviously not doing so.

The way I read MS doc is that with my large array it should be reading/writing to disk, yes it slows down program, but that is better than it throwing OutofMemoryException.

Some questions:
1. Is MS doc wrong, or is there some parameter, class, setting I need to specify to tell it to read/write large arrays? e.g. somewhere in program a line that reads in essence "OK stupid, if an array is too big for memory, read/write it to disk, like MS doc says you should"

2. Ignoring idiocy of doc for a second, is there some other way I can store the data that gets stored in the array and achieve the functionality [I know of a couple of ways myself, but they would involve major rewrite of code - read/write to a file, bust array up into parts - that is there some other object/class that could do it?]

3. I suspect MS doc is wrong. In which case how do I tell before program gets to the error, how big of an array the memory for a given machine can handle?
That is, when user specifies array size, have program do a calculation so as determine at user input time, that the array is either too big for that user's machine, or machine does have enough RAM?
And if not enough RAM for say 256^4, how big the array could be and still run?
Rather than user enters size, runs it, gets error, tries again w different value, gets error, tries again...?
Avatar of EDDYKT
EDDYKT
Flag of Canada image

I've no problem to doing this

a(101, 101, 101, 101) = 30
Dim b As Integer = a(101, 101, 101, 101)

the IDE threw an exception when I try to read the data, but I get the value b back
Avatar of tlfeet
tlfeet

ASKER

Hi EDDYKT,

Perhaps I was not clear.
I know I can do an array of 100,100,100,100
I also know 130,130,130,130 cause error.
In between, I do not know.

2 problems:
1. really need to be able to handle up to 256,256,256,256
2. if I can't handle that, need to be able to tell, at time user inputs, if the dim user specified will be too big for their machine, so that:
a. can tell them so, before processing starts
b. aid me in coming up with some sort of work around.

[trial and error to find the limit not best way - either at my end writing the program & at user end]
1. Don't think it will.

2a.  You can call a test function and see if their machine will handle what they need...

Private Function TestMaxSize(i As Integer) As Boolean

On Error GoTo ErrHandler
ReDim ia(i, i, i, i) As Integer
    TestMaxSize = True
   
  Exit Function
 
ErrHandler:
    TestMaxSize = False
End Function
Avatar of tlfeet

ASKER

Hi dfiala13

Re: #2 yes that would work.
Still a bit of trial and error in that user enters number, it tells user its too big, try another smaller number, user tries another smaller number & that turns out to be too big too, so try another, etc.

Would be very nice to construct a Procedure/Function that determines how big an array the computer/a computer/any computer could handle and just tell them upfront, no trial and error
"Hey user, the biggest array your 'puter can handle is X^4"

BTW, from trial and error found my computer handles up to 128^4
also figured out that when it gets close to that - 120^4 at least, it does read/write to virtual memory

Also "found out", not solid info, but likely culprit that Windows XP can read/write an "object" only up to 4 gigs in size. Which in practice/real world is probably more like 2 gigs given OS, program overhead.

And, guess what?
256^4 = 4.294 G +- a few bytes ;-)

I might have to do some major rewrite on my program and figure out how to split the larger arrays up.
Concerned that splitting for example 256^4 into say 2 x 128^4 is going to cause problems too (or 4 x 64^4 or 8x32^4, etc.) as the total amount of memory is actually going up given the overhead of the extra arrays.

The problem is the stack size:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vbcon98/html/vbcondatalimitations.asp

Right around 150 elements per rank, you start to use around 1 MB of stack and that is all that can be on the stack in any one thread.  

I tried to load as a module level variable, with the same issues, though supposedly VB uses a different data segment in those cases.

here's a function that calls the one I wrote above to find the max size available
Private Function GetMaxSize(iStart As Integer) As Integer
Dim i As Integer

    For i = iStart To 256
        If TestMaxSize(i) = False Then
            GetMaxSize = i - 1
            Exit For
        End If
    Next
End Function
I need to do better math that's 1 GB around 150. -- doh!

What are you trying to use the arrays for?  VB might not be the best tool to do what you need.  Could be worth the time to write a C dll that does the heavy lifting.
Avatar of tlfeet

ASKER

Hi dfiala13,

I am doing some image processing.
Certainly VB is not best language, but it was what  I had access to at the time I started and most importantly knew how to get the info (pixel data) out of the image using VB.

Looking at moving it over to Java, to plug in/work with a program called ImageJ
-but beofre I do that, want to make sure Java can handle data too. ;-)
Good plan.  Java "should" be less encumbered by memory limitations.
Avatar of tlfeet

ASKER

Hi dfiala13,

OK, wierd things are happening.
I just got two notices from EE about this thread and they were for posts before your last one, starting w "I need to do better math that's 1 GB around 150"

Anyway, thanks for the link to the MSDN article.
Had not seen that (even after extensive search ;-) )

I started taking a deeper look at imageJ angle.
It is going to mean a much steeper learning curve than I had hoped.
But a bigger problem, I just realized is that I need to write results to a MS SQL Server database and that if I do it in ImageJ then I have to get an ODBC driver....etc etc.

My little program has one main task:
To process thousands of images, of Fabrics (cloth), figure out what color(s) the images, hence the Fabrics are and write that "ColorResults" data set to the database for each Fabric/image.

The images are jpegs, so VB gives me the A, R, G, B values for each pixel
As an intermediary step, I store the frequency of each color.

Right now, I am trying to figure out what set of parameters returns the best results. e.g. do I average pixel values, over what size of an area?  What shape of area? Do I analyze individual, unique colors or work say in WebSafe colors (or some other space).  If I average, do I do a weighted average? etc. etc.

For example if I use unique indiviudal colors, the full ARGB space, and analyze a Fabric/Image, call it TestImage2, then the most frequent colors are all black, but the Fabric is mutli-colored.

So if I use Websafe colors (0, 51, 102...etc - which I actually use an index in the intermediate stage) then the blacks are all grouped together, as are all the other shades of say red, orange, etc.
And the results is what I need.

In the first case, using/allowing the full RGB range, I dim my array like this:
ReDim FrequencyArray(255,255,255,255)
then each color gets counted like this:
FrequencyArray(currenta, currentr, currentg, currentb)+=1

In second case I convert the actuall values to an index running from 0 to 5, an my array works like this:
ReDim FrequencyArray(5,5,5,5)
and count like this:
FrequencyArray(currenta, currentr, currentg, currentb)+=1

Of course the 255^4 dim statement bombs out.
But in playing around with different parameters, I found WebSafe returns too few colors, too broad of a brush, while of course, the full range 0-255 for RGB returns results that are too fine.

So I need somewhere in between, as I do not know yet, what, I am running into the the question:
"How big can my space be before I hit the upper limit of VB handling the resulting/needed array?"

Since the images I am currently working with are gif/jpg, the A value is always 255 anyway, so I can just ingore it and do this
ReDim frequencyArray(255,255,255) or (0,255,255,255)
or likes this in second case:
ReDim FrequncyArray(5,5,5) or (0,5,5,5)

The problem is:
1. who knows what kind of images I will be using say in a year or 2, and then need that 4th rank?
2. as I said, I am still trying to figure out the best processing parameters to use, so who knows what I will end up needing.
3. If I ignore Alpha all together or handle it "specially" I have all this code to write to say, in essence:
"Do this unless it is the Alpha value, in which case do this..."
Just be nicer/faster/cleaner to be able to treat all the values the same.
4. I am trying to write the program so as to be as generally usefull as possible (there are other uses to which it could be put) and those other uses may need the 4th rank.

So, for my immediate task/goal & image inputs I can just ignore alpha values, or even wirte code to handle them specially/differently

It would just be nice to be able to handle the larger arrays.
Then I would have clean, simple code, that works with all the data in an Image.

Thanks for the little code tidbits - they will be helpful.
Basically, have a bee in my bonnet about the large arrays ;-)
ASKER CERTIFIED SOLUTION
Avatar of dfiala13
dfiala13

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Avatar of tlfeet

ASKER

Hi dfiala13

WOW!!!
That will do the trick.

I was using arrays, cause well did not know what else to use and excepting the outofmemory issue, it does work pretty well and is easy to code and understand what is going on.

Yes you are right, I do not need all 4GB the array offers, it is sparsely populated.

So Ieven thought of doing something similar to dictionary with arrays.
i.e. create an array that holds the unique values and go from there.
But that involved a lot of sorting & going thru the array, whose size needed would very from image to image, etc. etc.

The image I spoke of is 400x400px = 160,000 and has over 100,000 unique colors (some only a pixel or 2) - so while even that one sparsely populates 4GB array, holding 100,000 values in some other array leads to real performance hit.

I just knew there had to be some other method/class/feaute/thingy in VB that would handle what I need.
Your dictionary idea certainly seems like it will do the trick.

Now off to do a little reading up on it ;-)

Thanks again dfiala13

I will come back shortly - going to try a qucky verison of it, after doing some reading.
WOW!
Avatar of tlfeet

ASKER

Hi dfiala13,

Just an update.
Got side-tracked here at work with some other things....geez go figure ;-)
Started importing/implementing your codes snippets for the Dictionary object.

Ran into a couple of snags
1. I have no clue what I am doing - but learning fast!!!!
2. Forgot to mention that I am doing this in VB.NET in Visual Studio.NET
[but please do not rewrite your code - I am well into reworking for VB.NET (e.g "Public Property Get Colors() As Dictionary" is not valid .NET becuase of the "GET", as the "GET" gets put inside the Property . . . no big deal, just to illustrate reworking)]

3. I did want to double check a couple of things:

A.
You said to reference "Microsoft Scriptiing Runtime" for the dictionary
I could not find such a namespace/class
So I am inheriting "System.Collections.DictionaryBase"
Is that the correct/best animal?  Seems to be....but see #1 above ;-)

B.
Have a few minor points.  Not really stuck on them, just have to do more reading, searching...
e.g.
m_dictColor.Exists(sKey) - Exists not used in .NET
c = m_dictColor.Item(sKey) is not a member of MyColor
and similar type Build Errors

I am pretty sure I can fix them (already have fixed about 20 of them) - but I just wanted to make sure I am on the right track.

Does it sound like it?
Or should I go to school? ;-) Or should I fire myself and hire someone else?
Well, if you are doing this in .NET, why didn't you say so!?! Much, much easier. Geesh.

I was giving you VB6 code, hence the reference to the scripting runtime and the errors.

Since you are using .NET you yon't have to use the scripting runtime or do any inhertance.  Just use the Hashtable instead of the dictionary and instead of Init on the MyColor class use a constructor.

instead of Exists on the Hashtable use Contains
and you can get the object out of the hashtable like this:

c = m_ht(sKey)

But you are more or less on the right track.

Feel free to hire me.  Have compiler will travel ;)


Avatar of tlfeet

ASKER

Hi dfiala13,

I got the Hashtable/MyColor working.

Not only does it work, but becuase I am not dealing with huge, sparsely populated arrays:
1. code runs faster
2. I can get a little more sophisticated with what/how it does its thing.

e.g. as I mentioned I am trying to figure out the best way to find the colors, average over what size, shape of an area, what size of a colorspace to use, etc.

So I have it loop/batch thru some images, and loop thru various settings, the hashtable gives me a much better way to store the info, so that I get better control over how/what it loops thru and how it spits out the imtermediary/final results - which I then take a look at to see which set of parameters gives the best results.

Much, much better . . .  now off to figure out how to sort it!


Thanks a ton!!!
You're welcome.

Glad you got it working.