• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 727
  • Last Modified:

fread fwrite taking a long time

Hi,

I have a program which uses fread, fwrite and when I do a top, it shows that the program uses 1% of the CPU for the read write. This explains the reason the program is slow.

Why is the CPU only using 1% for fread and fwrite ? Thanks a million
0
zizi21
Asked:
zizi21
  • 14
  • 6
  • 3
  • +1
3 Solutions
 
Kent OlsenData Warehouse Architect / DBACommented:
Hi zizi,

How did you determine that the program uses only 1% of the CPU?  If there is no contention for resources, it should be a lot higher.

The only exception that I can think of is if the I/O is unbuffered so that every call to fread/fwrite forces a physical I/O.


Kent
0
 
zizi21Author Commented:
I did a top and it showed the percentage of CPU being used is 1 %.
0
 
Kent OlsenData Warehouse Architect / DBACommented:

How long does the program run?
0
Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

 
phoffricCommented:
Also, make sure that your files are defragged. If large files are not contiguous on the disk, then your I/O's may be taking much time with head seeks.
0
 
zizi21Author Commented:
This is from the unix  time function.
real    318m20.610s
user    14m13.200s
sys     7m20.180s
0
 
zizi21Author Commented:
when you say degragged, do you mean that the names of files are short formed ?
0
 
zizi21Author Commented:
sorry, is this what you mean?

"(defrag) as you add and delete files on your hard drive the individual files end up being stored in a large number of fragments wherever there is free space on the drive. This slows down the process of reading the file. To resolve this we need to defragment the drive."

I need to run a program to deframent the drive..
0
 
zizi21Author Commented:
i recently added a partition of hard disk to my computer..not sure if that is the problem.
0
 
phoffricCommented:
>> sorry, is this what you mean? "(defrag) as you add and delete files.
Yes. From one compile to the next of exactly the same program (i.e., no source code changes), the program speed varied significantly until the disk was defragged allowing a sizeable chunk of free contiguous disk space. Then, the program ran with minor time variance.

If your program is running now on a smaller partition, then it is more likely that the partition will be more fragmented.
0
 
zizi21Author Commented:
The files are on 250 GB partition and the program is running on a 30 GB partition.
0
 
zizi21Author Commented:
How do i defragment it. I was reading and some people say it is not necessary in linux and some people say it is necessary and the e2defrag tool is not included for ext 4. Some say, you can move the files over the folders and copy back..
0
 
phoffricCommented:
How much free space in each partition? If there is much adding and deleting of files, then less free space in a partition, then the more likely that the added files will be fragmented.
0
 
zizi21Author Commented:
I didn't decrease teh size of the linux partition but I decreased the size for the windows so that i can have more space in linux
0
 
zizi21Author Commented:
There is around 220GB in the partition that has lots of reads and writes...And in the 30GB partition there is not much read and writes...Thanks you..
0
 
zizi21Author Commented:
would copying a file from one folder to another and copying it back defrag the file ? thanks a lot for answering my queries..
0
 
phoffricCommented:
If you have not changed the program, and if you boot up as Linux-only, and nothing has changed in the Linux partition, then the speed should remain as before. If you are running Linux on top of Windows, then possibly you have less swap space that is affecting the Linux run (but I'm not sure about this configuration).
0
 
zizi21Author Commented:
thanks very much for the reply..i only boot up as linux only.
0
 
phoffricCommented:
Linux only - good. Then Windows is out of the picture. One less interaction to worry about. Did the slowness appear only after the partition changes? If not, what other factors (e.g., increased number of writes, increased number of files, program changes, etc.) changed since the time your program ran ok?

I see that top has some fields associated with swap. You should check that out to see if that gives a clue.

>> would copying a file from one folder to another and copying it back defrag the file ?
Not if the folders were in the same partition.

My understanding is that Linux writes to center of partition making fragmentation less likely under normal usage. But if the partition has little free space, then still, fragmentation is possible.

At this point, we don't know if fragmentation is the problem; but you can rule it out if you copy all the files from the big partition to another partition wiping the big partition clean; and then copying the files back in one sequential operation - this should reduce the amount of fragmentation. Same concern for the program partition.
I haven't had to work with swap space; but check this out:
    http://www.cyberciti.biz/faq/linux-add-a-swap-file-howto/
0
 
Infinity08Commented:
It's probably better, if we go back to Kdo's original idea of getting more information, before jumping to conclusions. ie. :

1) what does your code do exactly ? (actual code would be ideal)

2) what makes you think it runs slow ?

3) how did you measure that ? (you've mentioned looking at the output from top, but if you just ran top without parameters, and observed the output, that's too generic, and not fine-grained enough to get accurate results)
0
 
phoffricCommented:
In the swap link, notice that partitions are not necessarily contiguous. For high speed disk access, whether read or write, you want the files to be physically contiguous to avoid long physical seeks of logical bytes. If you find that fragmentation is a problem, then I suggest that you try to find out how to get contiguous partitions in Linux.
0
 
zizi21Author Commented:
Thanks for all the replies and links. I am studying them. Pls give me some time.

The code is very very huge to be posted but it does lots of disk read and writes. I would run the program again and post more information.I just used top it showed that the cpu % for the program was 1%. I didn't use any other parameter.
0
 
zizi21Author Commented:
When you said "and then copying the files back in one sequential operation - this should reduce the amount of fragmentation" ?

Do you mean by doing cp * (for one sequential operation) ?
0
 
Kent OlsenData Warehouse Architect / DBACommented:

Yes.  Use *cp* to copy the file(s).

0
 
Infinity08Commented:
>> The code is very very huge to be posted but it does lots of disk read and writes.

Can you then describe what it does ? What files does it read to ? What files does it write to ? How often does it do each of them ? How much does it read/write ? How do you call fread and fwrite ? Etc.


>> I just used top it showed that the cpu % for the program was 1%. I didn't use any other parameter.

Right. That won't give you an accurate idea of what your process is using. The 'time' output you posted earlier gives a slightly better idea, and there it comes down to about 7% average CPU usage for your process (both user and system).

Better tools to check the I/O performance of your process (or the system in general) are iostat and iotop eg. Check out what kind of I/O load your process places on the system, and whether it's not maxing it out.

0
 
zizi21Author Commented:
Thanks a lot for all the replies. I think, I would close the question first and if I have more questions, i will post another one.
0
 
Infinity08Commented:
You should only close a question when it has been resolved. Have you found out the cause ? If so, can you post it here ? If not, it's best to re-open this question, and continue working here until you have a solution.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Keep up with what's happening at Experts Exchange!

Sign up to receive Decoded, a new monthly digest with product updates, feature release info, continuing education opportunities, and more.

  • 14
  • 6
  • 3
  • +1
Tackle projects and never again get stuck behind a technical roadblock.
Join Now