Hard disk access slows down important thread

I use a Thread descendant to gather data from external hardware. The acquisition rate is given by the hardware. However, every time I write or read files (in the main thread) I get an exception because this slows down my acquisition thread. Any ideas?
LVL 2
__alexAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

aikimarkCommented:
Suggestion:
Use a memory mapped file to share the output buffer between two different threads.  The data acquisition thread writes to the MMF and a different thread outputs blocks of data from the MMF to the hard drive.
0
__alexAuthor Commented:
> to share the output buffer between two different threads
I don't want to share data. The data from the external hardware and the data I want to load and save are not related in any way.
0
aikimarkCommented:
1. set the priority of the hard drive I/O thread lower than that of the data acquisition thread (if this is even possible).

2. spawn a completely different process for the data acquisition code and run it at a higher priority.  Use atoms or named pipes to communicate.

3. Install and run the data acquisition as a Windows service.

4. limit the amount of I/O you do at any one time.  If you have a large file, write it out in several smaller operations, rather than one big operation.

5. 'outsource' the data read/write operations to a different process on a different processor and different hard drive.

6. Reconfigure your hardware to write more efficiently/quickly.  For instance, add a RAID hard drive card, which will buffer your reads/writes and splay them out to the hard drives in independent operations.

============================
Of course, these comments are strictly out-of-the-box thinking with no regard to costs or other restrictions you (your users) might be facing.
0
Keep up with what's happening at Experts Exchange!

Sign up to receive Decoded, a new monthly digest with product updates, feature release info, continuing education opportunities, and more.

robert_marquardtCommented:
The problem may be that the PCI bus is at its limit. Tell more about the amount of data you aquire.
0
aikimarkCommented:
while you're at it, think about these factors for the non-data-acquisition code...

* How much data is read at any one time (min, max, avg, bytes, files)?
* How much data is written at any one time (min, max, avg, bytes, files)?
* Are the read/write targets different between I/O operations?
     - If the same target files, then is there a requirement to open/close between I/O operations?
* What kind of read/write operations are you doing (sequential, direct, record, blocked)?
* What contention is there for the device?
     - other processes
     - operating system
     - network sharing
     - slave vs master
     - channel contention (CD-ROM, USB, etc.)
* What inefficiencies are you facing?
     - physical partition vs. logical partition
     - path is buried deep in the directory structure
     - particion needs defragmenting
     - drive has problems (run scandisk-like utility)
     - size of the drive/partition/sector relative to the size files you are reading/writing?
     - format of the file system (FAT32, NTFS, HPFS, etc.)?
* Is this a single-processor CPU, multi-processor, hyperthreading Intel processor?
0
__alexAuthor Commented:
Thanks for the input!
I acquire data frames with a size of about 200 kByte and a rate of 100 frames each second. That is 20 MByte each second. At any time the user is able to go to the file menu and open or save files with a size up to 100 MByte. Actually I use a TFileStream. The performance of the loading and saving is less important.

> 1.
I'll give it a try
> 2.
Not an option (as far as I understand)
> 3.
I don't know anything about services -> Maybe in a next release
> 4.
That sounds good!
> 5.
Duh, ...
> 6.
Not an option. Application must even run with an Intel 486 (I'm kidding).

> What contention is there for the device?
> What inefficiencies are you facing?
Well, we tell our customers to close all other applications if they want to acquire data but there's no other restriction. They can even try to save to floppy disk. Usually we ship with Win2k or XP and a single CPU.

I will try your suggestions (alter thread prioriy and write in small chunks) and come back later. Thank you!
0
robert_marquardtCommented:
This definitely sounds as if you saturate the PCI bus. Modern Harddisks go > 50 MB/sec.
This is a fat part of the (theoretical) PCI maximum of 133 MB/sec.

Try writing your file in smaller portions and call Sleep(0) or other functions yielding the processor to other threads.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
__alexAuthor Commented:
No success with thread priorities or Sleep() while accessing hard disk. I guess I’ll have to stop data acquisition while loading / saving. Thank you anyway!
0
aikimarkCommented:
Alex,

Thanks for the points.  I think you can still acquire data while your other process performs I/O, just buffer the data you are acquiring until the other I/O process has finished.  To accomplish this, you will need to enable efficient inter-thread communications and/or an efficient locking protocol (mutex is preferred).

========================
200,000 bytes collected every 1/100th of a second -- WOW!!!

W1. I would certainly look into a compression scheme for this data.  Depending on the amount of change between frames, my gut feel is that an inter-frame different scheme may be the best choice here.  Certainly look at the schemes that eliminate repeating bytes/words/sequences.
W2. How many hard drives do you go through in a day/week?  At 20MB/sec, you are consuming a LOT of hard drive space (72GB/hr).
Bytes/second calculation:
 20,000,000       1/second
 100,000,000       5/seconds
 1,000,000,000       50/seconds
 72,000,000,000       3600/seconds

W3. How long does this data acquisition process run?

W4. Maybe it would be worth asking a follow-up question about locking out ALL other I/O (not just your other thread, but all other application process I/O) while you are in data acquisition mode.  If you are just concerned about your two threads, then inter-thread locking, data compression, and data buffering would probably be a reasonable solution.  Of course, you will need programmatically speed test the hard drive and set a 'buffer size' (max read/write bytes in a single operation) that will not take more than 10ms (1/100th of a second).

W5. With latency and rotational delay, it might not be physically possible to share arbitrary I/O operations between two (or more) different processes.
0
__alexAuthor Commented:
> At 20MB/sec, you are consuming a LOT of hard drive space (72GB/hr).
Nope. I wait for a trigger within the data and hold just a tiny piece in memory. The thing I want to save on my hard drive is a very different beast.

:-)
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Delphi

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.