Hard disk access slows down important thread

Posted on 2004-11-10
Last Modified: 2010-04-05
I use a Thread descendant to gather data from external hardware. The acquisition rate is given by the hardware. However, every time I write or read files (in the main thread) I get an exception because this slows down my acquisition thread. Any ideas?
Question by:__alex
    LVL 44

    Expert Comment

    Use a memory mapped file to share the output buffer between two different threads.  The data acquisition thread writes to the MMF and a different thread outputs blocks of data from the MMF to the hard drive.
    LVL 2

    Author Comment

    > to share the output buffer between two different threads
    I don't want to share data. The data from the external hardware and the data I want to load and save are not related in any way.
    LVL 44

    Assisted Solution

    1. set the priority of the hard drive I/O thread lower than that of the data acquisition thread (if this is even possible).

    2. spawn a completely different process for the data acquisition code and run it at a higher priority.  Use atoms or named pipes to communicate.

    3. Install and run the data acquisition as a Windows service.

    4. limit the amount of I/O you do at any one time.  If you have a large file, write it out in several smaller operations, rather than one big operation.

    5. 'outsource' the data read/write operations to a different process on a different processor and different hard drive.

    6. Reconfigure your hardware to write more efficiently/quickly.  For instance, add a RAID hard drive card, which will buffer your reads/writes and splay them out to the hard drives in independent operations.

    Of course, these comments are strictly out-of-the-box thinking with no regard to costs or other restrictions you (your users) might be facing.
    LVL 11

    Expert Comment

    The problem may be that the PCI bus is at its limit. Tell more about the amount of data you aquire.
    LVL 44

    Expert Comment

    while you're at it, think about these factors for the non-data-acquisition code...

    * How much data is read at any one time (min, max, avg, bytes, files)?
    * How much data is written at any one time (min, max, avg, bytes, files)?
    * Are the read/write targets different between I/O operations?
         - If the same target files, then is there a requirement to open/close between I/O operations?
    * What kind of read/write operations are you doing (sequential, direct, record, blocked)?
    * What contention is there for the device?
         - other processes
         - operating system
         - network sharing
         - slave vs master
         - channel contention (CD-ROM, USB, etc.)
    * What inefficiencies are you facing?
         - physical partition vs. logical partition
         - path is buried deep in the directory structure
         - particion needs defragmenting
         - drive has problems (run scandisk-like utility)
         - size of the drive/partition/sector relative to the size files you are reading/writing?
         - format of the file system (FAT32, NTFS, HPFS, etc.)?
    * Is this a single-processor CPU, multi-processor, hyperthreading Intel processor?
    LVL 2

    Author Comment

    Thanks for the input!
    I acquire data frames with a size of about 200 kByte and a rate of 100 frames each second. That is 20 MByte each second. At any time the user is able to go to the file menu and open or save files with a size up to 100 MByte. Actually I use a TFileStream. The performance of the loading and saving is less important.

    > 1.
    I'll give it a try
    > 2.
    Not an option (as far as I understand)
    > 3.
    I don't know anything about services -> Maybe in a next release
    > 4.
    That sounds good!
    > 5.
    Duh, ...
    > 6.
    Not an option. Application must even run with an Intel 486 (I'm kidding).

    > What contention is there for the device?
    > What inefficiencies are you facing?
    Well, we tell our customers to close all other applications if they want to acquire data but there's no other restriction. They can even try to save to floppy disk. Usually we ship with Win2k or XP and a single CPU.

    I will try your suggestions (alter thread prioriy and write in small chunks) and come back later. Thank you!
    LVL 11

    Accepted Solution

    This definitely sounds as if you saturate the PCI bus. Modern Harddisks go > 50 MB/sec.
    This is a fat part of the (theoretical) PCI maximum of 133 MB/sec.

    Try writing your file in smaller portions and call Sleep(0) or other functions yielding the processor to other threads.
    LVL 2

    Author Comment

    No success with thread priorities or Sleep() while accessing hard disk. I guess I’ll have to stop data acquisition while loading / saving. Thank you anyway!
    LVL 44

    Expert Comment


    Thanks for the points.  I think you can still acquire data while your other process performs I/O, just buffer the data you are acquiring until the other I/O process has finished.  To accomplish this, you will need to enable efficient inter-thread communications and/or an efficient locking protocol (mutex is preferred).

    200,000 bytes collected every 1/100th of a second -- WOW!!!

    W1. I would certainly look into a compression scheme for this data.  Depending on the amount of change between frames, my gut feel is that an inter-frame different scheme may be the best choice here.  Certainly look at the schemes that eliminate repeating bytes/words/sequences.
    W2. How many hard drives do you go through in a day/week?  At 20MB/sec, you are consuming a LOT of hard drive space (72GB/hr).
    Bytes/second calculation:
     20,000,000       1/second
     100,000,000       5/seconds
     1,000,000,000       50/seconds
     72,000,000,000       3600/seconds

    W3. How long does this data acquisition process run?

    W4. Maybe it would be worth asking a follow-up question about locking out ALL other I/O (not just your other thread, but all other application process I/O) while you are in data acquisition mode.  If you are just concerned about your two threads, then inter-thread locking, data compression, and data buffering would probably be a reasonable solution.  Of course, you will need programmatically speed test the hard drive and set a 'buffer size' (max read/write bytes in a single operation) that will not take more than 10ms (1/100th of a second).

    W5. With latency and rotational delay, it might not be physically possible to share arbitrary I/O operations between two (or more) different processes.
    LVL 2

    Author Comment

    > At 20MB/sec, you are consuming a LOT of hard drive space (72GB/hr).
    Nope. I wait for a trigger within the data and hold just a tiny piece in memory. The thing I want to save on my hard drive is a very different beast.


    Featured Post

    How your wiki can always stay up-to-date

    Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
    - Increase transparency
    - Onboard new hires faster
    - Access from mobile/offline

    Join & Write a Comment

    A lot of questions regard threads in Delphi.   One of the more specific questions is how to show progress of the thread.   Updating a progressbar from inside a thread is a mistake. A solution to this would be to send a synchronized message to the…
    In my programming career I have only very rarely run into situations where operator overloading would be of any use in my work.  Normally those situations involved math with either overly large numbers (hundreds of thousands of digits or accuracy re…
    Access reports are powerful and flexible. Learn how to create a query and then a grouped report using the wizard. Modify the report design after the wizard is done to make it look better. There will be another video to explain how to put the final p…
    This video gives you a great overview about bandwidth monitoring with SNMP and WMI with our network monitoring solution PRTG Network Monitor ( If you're looking for how to monitor bandwidth using netflow or packet s…

    734 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    21 Experts available now in Live!

    Get 1:1 Help Now