Array sizes in Microsoft Visual C++ 6.0

I am attempting to store a lot of memory into an array of type unsigned char in Microsoft Visual C++, to be exact, I need to set up an array containing 1.6 million characters, at present I can successfully implement an array containing 1028096 values, leaving me around 580,000 characters short of my required total. Increasing the array size at the time of declaration of this array results in an instant program crash at runtime.

I would like to know if it is possible to allocate more memory to where these variables are stored, either at runtime or design time. I am sure that it is possible to do this but I cannot find the correct way to do so. I should also add that the array of 1028096 values behaves exactly as it ought to, and experiences no overflow problems.

Thanks in advance

Liam
LVL 1
lamarshAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

cookreCommented:
Do it at runtime.

MAPI will let you allocate that much and more:

         void * ReadBuff;
         ...
         MapiRC=MAPIInitialize(NULL);
         if (MapiRC!=S_OK)
            {
            // not enough RAM
            }
         MapiRC=MAPIAllocateBuffer(BuffSize,&ReadBuff);
         if (S_OK!=MapiRC)
            {
            // not enough RAM
            MAPIUninitialize();
            }
nthnhungCommented:
You used a static or dynamic array? If you used static array, how about think of a dynamic array as instead (pointers, linked list).
Do you really need to load all of characters at the same time? Maybe you are doing some thing regarding to dictionary. Why don't you use an database to save and query (index files, access - sql database, ...).
jhanceCommented:
The most common problem with large allcations using the C-runtime has its own memory allocator that can be overflowed and/or you are allocating the space on the stack which is 1MB by default.

While you can write your own allocator or increase the stack space, the simplest way in my opinion is to use the Windows API VirtualAlloc()/VirtualFree().  These functions permit you to allocate any size block of memory up to the maximum possible virtual space for the process, which is 4GB on a WIN32 system.

Big Business Goals? Which KPIs Will Help You

The most successful MSPs rely on metrics – known as key performance indicators (KPIs) – for making informed decisions that help their businesses thrive, rather than just survive. This eBook provides an overview of the most important KPIs used by top MSPs.

lamarshAuthor Commented:
Thanks for the prompt responses, I'll start looking into them straight away. Just to clarify things a bit, yes it is essential to have this many characters stored in the array simulatenously as this program is part of a hardware data capture system. I am looking to acquire large amounts of digital data from a circuit board that I have put together. At present I am able to shift my target of 12.8Mb/sec from the board to the PC (which is done as characters) and have implemented a special method of data storage which is suitable for this purpose. In order for this to be the case however, all 1.6 million characters must be piled into either one array, or multiple arrays. My experience with the particular compiler so far is that the memory is in fact treated as a total rather than segregated, and consequently one array consumes the same amount of memory as several smaller ones that total the same storage space. I am aware that there may be several methods for enhancing the amount of memory that I may use for variables, but this is an area of programming that I am quite unsure about. I would be grateful for any further advice that can be offered on this matter whilst I investigate the methods sugested above,. Thanks again
sserafimescuCommented:
Hi Liam,

As jhance pointed out above, you are probably using a static array, something like:
unsigned char x[1000000];
If you are doing this, the memory is allocated on the stack which is not really good. If you need large amounts of memory, allocate it dinamically like this:
unsigned char *x=(char*)malloc(HUGE_AMOUNT_OF_BYTES);
After doing that you may use the array x just as before, but you should remember to free it when you're done with it:
free(x);

Using malloc() is sure to work even if you need hundreds of megabytes of memory. The amount is only limited by the virtual memory of the operating system.

Good luck,
Serban

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
lamarshAuthor Commented:
Thanks Serban for the response above, was wondering if yo ucould clarify exactly how this array behaves compared to the basic C++ array type. The reason for this is that previously I was able to pull data from the array using an integer pointer to an address e.g. datahigh[x], datahigh[x+1], datahigh[x+2] etc, which yielded an otput in the following format:

2923 BE84 E16C etc,

using malloc, and making no changes other than the variable declaration as you described I get the following output:

2923 FFFFFFBEFFFFFF84 FFFFFFE16C etc.

if this was simplay a case of removing the 0xFFFFFF from before each value it would not be a problem, however the blocks of FFFFFF do not appear to occur sequentially. In fact, not all values are affected by these values. I would be greatful for any further help you can offer regarding this matter

Liam
sserafimescuCommented:
Hi again,

First I will clarify a little bit the two array types:

For most purposes I can think of, an array allocated with malloc behaves exactly the same as an array statically allocated of the same size and type. To be more exact, the following two lines of code should yield similar arrays:

Mytype my_array_1[SIZE];

Mytype *my_array_2 = (Mytype*)malloc(SIZE * sizeof(Mytype))

The reason for using SIZE * sizeof(Mytype) with malloc is that malloc accepts the array size in bytes as an argument, whereas the first array declaration accepts the number of elements in the array.

In case you are allocating chars or unsigned chars, sizeof(char) == 1, so there is no need for a multiplication; but in case you are allocating ints or something else, sizeof() is important.

After allocating the arrays as above, you can use my_array_1[i] and my_array_2[i] in the same manner for any i in the range 0 .. SIZE - 1.

Now, regarding your current problem, I suspect you have declared the array of one type (say char) and are trying to access it as if it were an array of integers, or vice-versa.

Please post a relevant snippet of your code and I will surely help you find out the exact problem.

Regards,
Serban
lamarshAuthor Commented:
Serban,
Many thanks for the clarification offered. Sincerest apologies for making you goto the trouble of doing so as the problem was origintaing from the fact that i'd declared my malloc array as a char, and previously it was unsigned. Once i'd hunted that out everything seemed to be working fine. Thanks again for the help offered.

Liam
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Editors IDEs

From novice to tech pro — start learning today.