Solved

Memory allocation and disk thrashing.

Posted on 1998-01-08
4
183 Views
Last Modified: 2010-08-05
Hi, All
I refer to a previous question regarding Global alloc et al.

I've get a program that receives data from host at 2k a time.

It eventually saves the data in paradox table.

The guy who wrote it (a contracter long gone) does many, many, many mallocs as he is converting the data for paradox etc...

So 600 transactions, converting to some 3000-10000 records, and allocation say 25 bytes for each field ...

Could the thrashing I am getting on the Win 95 machine running 24Mb, be attributed to what ever it was you have been talking about?


Zonnald
0
Comment
Question by:Zonnald
4 Comments
 
LVL 32

Accepted Solution

by:
jhance earned 100 total points
ID: 1177924
Yes, I would say that the above situation is very likely to cause thrashing.  It's very poor practice to malloc a lot of small blocks.  The problem is that malloc is not all that efficient and when allocating a small block it ends up wasting some space.  For a relatively small number of mallocs this is not a problem but when you do it thousands of times and put it in a loop with malloc's and free's you end up with a fragmented mess in memory.  In any kind of high volume transaction based program (like you have mentioned above) it's well worth the time and effort to build your own memory allocation utility which has a foreknowledge of the structure of the objects you are allocating and can do it much more eficiently.
0
 
LVL 22

Expert Comment

by:nietod
ID: 1177925
I dissagree.  (and would refer you back to the question that Zonnald is talking about if I could remember/find the QID.)

The total amount of heap space we are talking about here is on the order of a few hundred k.  The total number of allocations is a lot and may make for an innefficient heap, but the total size is not large enough to cause thrashing from virtual memory swapping.  Not these days, 5 years ago maybe. 10 years ago it was out of the question.

I suspect the problem is that the code is accessing the disk and causing the thrashing.  Perhaps it is writting out the information in small chuncks, which can be innefficient.


0
 
LVL 2

Expert Comment

by:anichini
ID: 1177926
If it is writing in small chunks, either buffer up large (i.e. at least 4 K) blocks of data or use memory-mapped files which does the same thing for you automatically (i'm simplifying).


0
 
LVL 1

Author Comment

by:Zonnald
ID: 1177927
It is writing the records to the disk via calls to Paradox.

I am not sure if it is doing several records at a time or if it is writing a record and closing the table.

Should this be the case perhaps we could open and close the database table only at the beginning and end of a particular set of records.

I will leave this open as there is a differing of opionions.  When we have a rewrite of the code which proves a praticular theory then I will sort out the points.

Zonnald
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

Article by: SunnyDark
This article's goal is to present you with an easy to use XML wrapper for C++ and also present some interesting techniques that you might use with MS C++. The reason I built this class is to ease the pain of using XML files with C++, since there is…
Basic understanding on "OO- Object Orientation" is needed for designing a logical solution to solve a problem. Basic OOAD is a prerequisite for a coder to ensure that they follow the basic design of OO. This would help developers to understand the b…
The goal of the video will be to teach the user the difference and consequence of passing data by value vs passing data by reference in C++. An example of passing data by value as well as an example of passing data by reference will be be given. Bot…
The viewer will be introduced to the technique of using vectors in C++. The video will cover how to define a vector, store values in the vector and retrieve data from the values stored in the vector.

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now