Solved

Memory allocation and disk thrashing.

Posted on 1998-01-08
4
193 Views
Last Modified: 2010-08-05
Hi, All
I refer to a previous question regarding Global alloc et al.

I've get a program that receives data from host at 2k a time.

It eventually saves the data in paradox table.

The guy who wrote it (a contracter long gone) does many, many, many mallocs as he is converting the data for paradox etc...

So 600 transactions, converting to some 3000-10000 records, and allocation say 25 bytes for each field ...

Could the thrashing I am getting on the Win 95 machine running 24Mb, be attributed to what ever it was you have been talking about?


Zonnald
0
Comment
Question by:Zonnald
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
4 Comments
 
LVL 32

Accepted Solution

by:
jhance earned 100 total points
ID: 1177924
Yes, I would say that the above situation is very likely to cause thrashing.  It's very poor practice to malloc a lot of small blocks.  The problem is that malloc is not all that efficient and when allocating a small block it ends up wasting some space.  For a relatively small number of mallocs this is not a problem but when you do it thousands of times and put it in a loop with malloc's and free's you end up with a fragmented mess in memory.  In any kind of high volume transaction based program (like you have mentioned above) it's well worth the time and effort to build your own memory allocation utility which has a foreknowledge of the structure of the objects you are allocating and can do it much more eficiently.
0
 
LVL 22

Expert Comment

by:nietod
ID: 1177925
I dissagree.  (and would refer you back to the question that Zonnald is talking about if I could remember/find the QID.)

The total amount of heap space we are talking about here is on the order of a few hundred k.  The total number of allocations is a lot and may make for an innefficient heap, but the total size is not large enough to cause thrashing from virtual memory swapping.  Not these days, 5 years ago maybe. 10 years ago it was out of the question.

I suspect the problem is that the code is accessing the disk and causing the thrashing.  Perhaps it is writting out the information in small chuncks, which can be innefficient.


0
 
LVL 2

Expert Comment

by:anichini
ID: 1177926
If it is writing in small chunks, either buffer up large (i.e. at least 4 K) blocks of data or use memory-mapped files which does the same thing for you automatically (i'm simplifying).


0
 
LVL 1

Author Comment

by:Zonnald
ID: 1177927
It is writing the records to the disk via calls to Paradox.

I am not sure if it is doing several records at a time or if it is writing a record and closing the table.

Should this be the case perhaps we could open and close the database table only at the beginning and end of a particular set of records.

I will leave this open as there is a differing of opionions.  When we have a rewrite of the code which proves a praticular theory then I will sort out the points.

Zonnald
0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Written by John Humphreys C++ Threading and the POSIX Library This article will cover the basic information that you need to know in order to make use of the POSIX threading library available for C and C++ on UNIX and most Linux systems.   [s…
Many modern programming languages support the concept of a property -- a class member that combines characteristics of both a data member and a method.  These are sometimes called "smart fields" because you can add logic that is applied automaticall…
The goal of the tutorial is to teach the user how to use functions in C++. The video will cover how to define functions, how to call functions and how to create functions prototypes. Microsoft Visual C++ 2010 Express will be used as a text editor an…
The goal of the video will be to teach the user the difference and consequence of passing data by value vs passing data by reference in C++. An example of passing data by value as well as an example of passing data by reference will be be given. Bot…

696 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question