Solved

Memory allocation and disk thrashing.

Posted on 1998-01-08
4
189 Views
Last Modified: 2010-08-05
Hi, All
I refer to a previous question regarding Global alloc et al.

I've get a program that receives data from host at 2k a time.

It eventually saves the data in paradox table.

The guy who wrote it (a contracter long gone) does many, many, many mallocs as he is converting the data for paradox etc...

So 600 transactions, converting to some 3000-10000 records, and allocation say 25 bytes for each field ...

Could the thrashing I am getting on the Win 95 machine running 24Mb, be attributed to what ever it was you have been talking about?


Zonnald
0
Comment
Question by:Zonnald
4 Comments
 
LVL 32

Accepted Solution

by:
jhance earned 100 total points
ID: 1177924
Yes, I would say that the above situation is very likely to cause thrashing.  It's very poor practice to malloc a lot of small blocks.  The problem is that malloc is not all that efficient and when allocating a small block it ends up wasting some space.  For a relatively small number of mallocs this is not a problem but when you do it thousands of times and put it in a loop with malloc's and free's you end up with a fragmented mess in memory.  In any kind of high volume transaction based program (like you have mentioned above) it's well worth the time and effort to build your own memory allocation utility which has a foreknowledge of the structure of the objects you are allocating and can do it much more eficiently.
0
 
LVL 22

Expert Comment

by:nietod
ID: 1177925
I dissagree.  (and would refer you back to the question that Zonnald is talking about if I could remember/find the QID.)

The total amount of heap space we are talking about here is on the order of a few hundred k.  The total number of allocations is a lot and may make for an innefficient heap, but the total size is not large enough to cause thrashing from virtual memory swapping.  Not these days, 5 years ago maybe. 10 years ago it was out of the question.

I suspect the problem is that the code is accessing the disk and causing the thrashing.  Perhaps it is writting out the information in small chuncks, which can be innefficient.


0
 
LVL 2

Expert Comment

by:anichini
ID: 1177926
If it is writing in small chunks, either buffer up large (i.e. at least 4 K) blocks of data or use memory-mapped files which does the same thing for you automatically (i'm simplifying).


0
 
LVL 1

Author Comment

by:Zonnald
ID: 1177927
It is writing the records to the disk via calls to Paradox.

I am not sure if it is doing several records at a time or if it is writing a record and closing the table.

Should this be the case perhaps we could open and close the database table only at the beginning and end of a particular set of records.

I will leave this open as there is a differing of opionions.  When we have a rewrite of the code which proves a praticular theory then I will sort out the points.

Zonnald
0

Featured Post

Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

In days of old, returning something by value from a function in C++ was necessarily avoided because it would, invariably, involve one or even two copies of the object being created and potentially costly calls to a copy-constructor and destructor. A…
Basic understanding on "OO- Object Orientation" is needed for designing a logical solution to solve a problem. Basic OOAD is a prerequisite for a coder to ensure that they follow the basic design of OO. This would help developers to understand the b…
The viewer will learn how to use the return statement in functions in C++. The video will also teach the user how to pass data to a function and have the function return data back for further processing.
The viewer will learn additional member functions of the vector class. Specifically, the capacity and swap member functions will be introduced.

789 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question