[Last Call] Learn how to a build a cloud-first strategyRegister Now

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 163
  • Last Modified:

Hard disk access slows down important thread

I use a Thread descendant to gather data from external hardware. The acquisition rate is given by the hardware. However, every time I write or read files (in the main thread) I get an exception because this slows down my acquisition thread. Any ideas?
0
__alex
Asked:
__alex
  • 4
  • 4
  • 2
2 Solutions
 
aikimarkCommented:
Suggestion:
Use a memory mapped file to share the output buffer between two different threads.  The data acquisition thread writes to the MMF and a different thread outputs blocks of data from the MMF to the hard drive.
0
 
__alexAuthor Commented:
> to share the output buffer between two different threads
I don't want to share data. The data from the external hardware and the data I want to load and save are not related in any way.
0
 
aikimarkCommented:
1. set the priority of the hard drive I/O thread lower than that of the data acquisition thread (if this is even possible).

2. spawn a completely different process for the data acquisition code and run it at a higher priority.  Use atoms or named pipes to communicate.

3. Install and run the data acquisition as a Windows service.

4. limit the amount of I/O you do at any one time.  If you have a large file, write it out in several smaller operations, rather than one big operation.

5. 'outsource' the data read/write operations to a different process on a different processor and different hard drive.

6. Reconfigure your hardware to write more efficiently/quickly.  For instance, add a RAID hard drive card, which will buffer your reads/writes and splay them out to the hard drives in independent operations.

============================
Of course, these comments are strictly out-of-the-box thinking with no regard to costs or other restrictions you (your users) might be facing.
0
VIDEO: THE CONCERTO CLOUD FOR HEALTHCARE

Modern healthcare requires a modern cloud. View this brief video to understand how the Concerto Cloud for Healthcare can help your organization.

 
robert_marquardtCommented:
The problem may be that the PCI bus is at its limit. Tell more about the amount of data you aquire.
0
 
aikimarkCommented:
while you're at it, think about these factors for the non-data-acquisition code...

* How much data is read at any one time (min, max, avg, bytes, files)?
* How much data is written at any one time (min, max, avg, bytes, files)?
* Are the read/write targets different between I/O operations?
     - If the same target files, then is there a requirement to open/close between I/O operations?
* What kind of read/write operations are you doing (sequential, direct, record, blocked)?
* What contention is there for the device?
     - other processes
     - operating system
     - network sharing
     - slave vs master
     - channel contention (CD-ROM, USB, etc.)
* What inefficiencies are you facing?
     - physical partition vs. logical partition
     - path is buried deep in the directory structure
     - particion needs defragmenting
     - drive has problems (run scandisk-like utility)
     - size of the drive/partition/sector relative to the size files you are reading/writing?
     - format of the file system (FAT32, NTFS, HPFS, etc.)?
* Is this a single-processor CPU, multi-processor, hyperthreading Intel processor?
0
 
__alexAuthor Commented:
Thanks for the input!
I acquire data frames with a size of about 200 kByte and a rate of 100 frames each second. That is 20 MByte each second. At any time the user is able to go to the file menu and open or save files with a size up to 100 MByte. Actually I use a TFileStream. The performance of the loading and saving is less important.

> 1.
I'll give it a try
> 2.
Not an option (as far as I understand)
> 3.
I don't know anything about services -> Maybe in a next release
> 4.
That sounds good!
> 5.
Duh, ...
> 6.
Not an option. Application must even run with an Intel 486 (I'm kidding).

> What contention is there for the device?
> What inefficiencies are you facing?
Well, we tell our customers to close all other applications if they want to acquire data but there's no other restriction. They can even try to save to floppy disk. Usually we ship with Win2k or XP and a single CPU.

I will try your suggestions (alter thread prioriy and write in small chunks) and come back later. Thank you!
0
 
robert_marquardtCommented:
This definitely sounds as if you saturate the PCI bus. Modern Harddisks go > 50 MB/sec.
This is a fat part of the (theoretical) PCI maximum of 133 MB/sec.

Try writing your file in smaller portions and call Sleep(0) or other functions yielding the processor to other threads.
0
 
__alexAuthor Commented:
No success with thread priorities or Sleep() while accessing hard disk. I guess I’ll have to stop data acquisition while loading / saving. Thank you anyway!
0
 
aikimarkCommented:
Alex,

Thanks for the points.  I think you can still acquire data while your other process performs I/O, just buffer the data you are acquiring until the other I/O process has finished.  To accomplish this, you will need to enable efficient inter-thread communications and/or an efficient locking protocol (mutex is preferred).

========================
200,000 bytes collected every 1/100th of a second -- WOW!!!

W1. I would certainly look into a compression scheme for this data.  Depending on the amount of change between frames, my gut feel is that an inter-frame different scheme may be the best choice here.  Certainly look at the schemes that eliminate repeating bytes/words/sequences.
W2. How many hard drives do you go through in a day/week?  At 20MB/sec, you are consuming a LOT of hard drive space (72GB/hr).
Bytes/second calculation:
 20,000,000       1/second
 100,000,000       5/seconds
 1,000,000,000       50/seconds
 72,000,000,000       3600/seconds

W3. How long does this data acquisition process run?

W4. Maybe it would be worth asking a follow-up question about locking out ALL other I/O (not just your other thread, but all other application process I/O) while you are in data acquisition mode.  If you are just concerned about your two threads, then inter-thread locking, data compression, and data buffering would probably be a reasonable solution.  Of course, you will need programmatically speed test the hard drive and set a 'buffer size' (max read/write bytes in a single operation) that will not take more than 10ms (1/100th of a second).

W5. With latency and rotational delay, it might not be physically possible to share arbitrary I/O operations between two (or more) different processes.
0
 
__alexAuthor Commented:
> At 20MB/sec, you are consuming a LOT of hard drive space (72GB/hr).
Nope. I wait for a trigger within the data and hold just a tiny piece in memory. The thing I want to save on my hard drive is a very different beast.

:-)
0

Featured Post

Important Lessons on Recovering from Petya

In their most recent webinar, Skyport Systems explores ways to isolate and protect critical databases to keep the core of your company safe from harm.

  • 4
  • 4
  • 2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now