Solved

fread fwrite taking a long time

Posted on 2011-09-10
26
647 Views
Last Modified: 2012-05-12
Hi,

I have a program which uses fread, fwrite and when I do a top, it shows that the program uses 1% of the CPU for the read write. This explains the reason the program is slow.

Why is the CPU only using 1% for fread and fwrite ? Thanks a million
0
Comment
Question by:zizi21
  • 14
  • 6
  • 3
  • +1
26 Comments
 
LVL 45

Assisted Solution

by:Kdo
Kdo earned 100 total points
ID: 36517966
Hi zizi,

How did you determine that the program uses only 1% of the CPU?  If there is no contention for resources, it should be a lot higher.

The only exception that I can think of is if the I/O is unbuffered so that every call to fread/fwrite forces a physical I/O.


Kent
0
 

Author Comment

by:zizi21
ID: 36517967
I did a top and it showed the percentage of CPU being used is 1 %.
0
 
LVL 45

Expert Comment

by:Kdo
ID: 36517984

How long does the program run?
0
 
LVL 32

Expert Comment

by:phoffric
ID: 36518051
Also, make sure that your files are defragged. If large files are not contiguous on the disk, then your I/O's may be taking much time with head seeks.
0
 

Author Comment

by:zizi21
ID: 36518102
This is from the unix  time function.
real    318m20.610s
user    14m13.200s
sys     7m20.180s
0
 

Author Comment

by:zizi21
ID: 36518103
when you say degragged, do you mean that the names of files are short formed ?
0
 

Author Comment

by:zizi21
ID: 36518112
sorry, is this what you mean?

"(defrag) as you add and delete files on your hard drive the individual files end up being stored in a large number of fragments wherever there is free space on the drive. This slows down the process of reading the file. To resolve this we need to defragment the drive."

I need to run a program to deframent the drive..
0
 

Author Comment

by:zizi21
ID: 36518118
i recently added a partition of hard disk to my computer..not sure if that is the problem.
0
 
LVL 32

Expert Comment

by:phoffric
ID: 36518142
>> sorry, is this what you mean? "(defrag) as you add and delete files.
Yes. From one compile to the next of exactly the same program (i.e., no source code changes), the program speed varied significantly until the disk was defragged allowing a sizeable chunk of free contiguous disk space. Then, the program ran with minor time variance.

If your program is running now on a smaller partition, then it is more likely that the partition will be more fragmented.
0
 

Author Comment

by:zizi21
ID: 36518143
The files are on 250 GB partition and the program is running on a 30 GB partition.
0
 

Author Comment

by:zizi21
ID: 36518147
How do i defragment it. I was reading and some people say it is not necessary in linux and some people say it is necessary and the e2defrag tool is not included for ext 4. Some say, you can move the files over the folders and copy back..
0
 
LVL 32

Expert Comment

by:phoffric
ID: 36518151
How much free space in each partition? If there is much adding and deleting of files, then less free space in a partition, then the more likely that the added files will be fragmented.
0
 

Author Comment

by:zizi21
ID: 36518153
I didn't decrease teh size of the linux partition but I decreased the size for the windows so that i can have more space in linux
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 

Author Comment

by:zizi21
ID: 36518155
There is around 220GB in the partition that has lots of reads and writes...And in the 30GB partition there is not much read and writes...Thanks you..
0
 

Author Comment

by:zizi21
ID: 36518156
would copying a file from one folder to another and copying it back defrag the file ? thanks a lot for answering my queries..
0
 
LVL 32

Expert Comment

by:phoffric
ID: 36518160
If you have not changed the program, and if you boot up as Linux-only, and nothing has changed in the Linux partition, then the speed should remain as before. If you are running Linux on top of Windows, then possibly you have less swap space that is affecting the Linux run (but I'm not sure about this configuration).
0
 

Author Comment

by:zizi21
ID: 36518164
thanks very much for the reply..i only boot up as linux only.
0
 
LVL 32

Assisted Solution

by:phoffric
phoffric earned 200 total points
ID: 36518179
Linux only - good. Then Windows is out of the picture. One less interaction to worry about. Did the slowness appear only after the partition changes? If not, what other factors (e.g., increased number of writes, increased number of files, program changes, etc.) changed since the time your program ran ok?

I see that top has some fields associated with swap. You should check that out to see if that gives a clue.

>> would copying a file from one folder to another and copying it back defrag the file ?
Not if the folders were in the same partition.

My understanding is that Linux writes to center of partition making fragmentation less likely under normal usage. But if the partition has little free space, then still, fragmentation is possible.

At this point, we don't know if fragmentation is the problem; but you can rule it out if you copy all the files from the big partition to another partition wiping the big partition clean; and then copying the files back in one sequential operation - this should reduce the amount of fragmentation. Same concern for the program partition.
I haven't had to work with swap space; but check this out:
    http://www.cyberciti.biz/faq/linux-add-a-swap-file-howto/
0
 
LVL 53

Expert Comment

by:Infinity08
ID: 36518241
It's probably better, if we go back to Kdo's original idea of getting more information, before jumping to conclusions. ie. :

1) what does your code do exactly ? (actual code would be ideal)

2) what makes you think it runs slow ?

3) how did you measure that ? (you've mentioned looking at the output from top, but if you just ran top without parameters, and observed the output, that's too generic, and not fine-grained enough to get accurate results)
0
 
LVL 32

Expert Comment

by:phoffric
ID: 36518243
In the swap link, notice that partitions are not necessarily contiguous. For high speed disk access, whether read or write, you want the files to be physically contiguous to avoid long physical seeks of logical bytes. If you find that fragmentation is a problem, then I suggest that you try to find out how to get contiguous partitions in Linux.
0
 

Author Comment

by:zizi21
ID: 36518668
Thanks for all the replies and links. I am studying them. Pls give me some time.

The code is very very huge to be posted but it does lots of disk read and writes. I would run the program again and post more information.I just used top it showed that the cpu % for the program was 1%. I didn't use any other parameter.
0
 

Author Comment

by:zizi21
ID: 36518669
When you said "and then copying the files back in one sequential operation - this should reduce the amount of fragmentation" ?

Do you mean by doing cp * (for one sequential operation) ?
0
 
LVL 45

Expert Comment

by:Kdo
ID: 36518756

Yes.  Use *cp* to copy the file(s).

0
 
LVL 53

Accepted Solution

by:
Infinity08 earned 200 total points
ID: 36519242
>> The code is very very huge to be posted but it does lots of disk read and writes.

Can you then describe what it does ? What files does it read to ? What files does it write to ? How often does it do each of them ? How much does it read/write ? How do you call fread and fwrite ? Etc.


>> I just used top it showed that the cpu % for the program was 1%. I didn't use any other parameter.

Right. That won't give you an accurate idea of what your process is using. The 'time' output you posted earlier gives a slightly better idea, and there it comes down to about 7% average CPU usage for your process (both user and system).

Better tools to check the I/O performance of your process (or the system in general) are iostat and iotop eg. Check out what kind of I/O load your process places on the system, and whether it's not maxing it out.

0
 

Author Comment

by:zizi21
ID: 36520246
Thanks a lot for all the replies. I think, I would close the question first and if I have more questions, i will post another one.
0
 
LVL 53

Expert Comment

by:Infinity08
ID: 36520999
You should only close a question when it has been resolved. Have you found out the cause ? If so, can you post it here ? If not, it's best to re-open this question, and continue working here until you have a solution.
0

Featured Post

Better Security Awareness With Threat Intelligence

See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

Join & Write a Comment

Little introduction about CP: CP is a command on linux that use to copy files and folder from one location to another location. Example usage of CP as follow: cp /myfoder /pathto/destination/folder/ cp abc.tar.gz /pathto/destination/folder/ab…
I. Introduction There's an interesting discussion going on now in an Experts Exchange Group — Attachments with no extension (http://www.experts-exchange.com/discussions/210281/Attachments-with-no-extension.html). This reminded me of questions tha…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
The goal of this video is to provide viewers with basic examples to understand opening and writing to files in the C programming language.

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now