Solved

C-Files related Question

Posted on 2002-04-19
18
280 Views
Last Modified: 2010-05-18
OK.

This is what I want.
I have a C program. It creates a series of binary files on the disk. The record (buffer) is about 1000 bytes per cycle. There can be maximum of 5000 cycles per file. So each file could be maximum of 5000 * 1000 bytes = 5000000 bytes.

Since this writing is done across a Ethernet, after each file is closing, it closes and locks up. I find that this is due to inability of trasfring that huge amount of data over the Ethernet because in my program there is a time-out period. The close() call exceeds the time-out. So it throws me out. That is a requirement I must have in the program.

So, now I try to flush the buffer every time the buffer is ready to be written. So I use the setvbuf() and
fflush() together. So I have something like this.


/*I am creating 10 files */

int i = 0;
for (int i = 0; i < 10; i++)
{
 /*CReate a new file name here*/
 
 fptr =  fopen(filename);

 while (i < 5000)
 {  
  /*Create the buffer here, and fill it*/
  ..............
  /*set the buffer */
  setvbuf( fptr, buf, _IOFBF,  sizeof(buf));

  /*flush it */
  fflush(fptr);
 }

 /*Ok. 500 cycles done. So close it */
 fclose(fptr);
}

I want to see when I do a dir or ls (in unix) the file is growing (changing the size). I do not see this hapenning.  In fact I do not see even the file is created on the disk when I do a dir or ls when the loop is running. Only time I see the file created is when close() is called. However with ZERO file size. So the fflush() really did not work for me. I thought when I use fflush() along with setvbuf(), the data is trasferred immediately to the disk so that I can see the file is created and growing each cycle.

Can anyone out there throw some light into this?. I do not mind using any other method.

Thanks
prain
 

 


0
Comment
Question by:prain
  • 6
  • 3
  • 3
  • +5
18 Comments
 
LVL 3

Expert Comment

by:marcjb
Comment Utility
It sounds like the OS is keeping a lot of data in the cache.  I had a similar problem, and I used 'fsync' to fix it.  According to the man page, fsync "copies all in-core parts of a file to disk".  So, before calling fclose, try:

fsync(fileno(fptr));

The call to fileno is needed here because you have a FILE* and fsync takes a file handle (int).

fsync is POSIX, but fileno is not.  However, you should either have fileno or a similar function available to you.

Hope this helps,

Marc
0
 
LVL 86

Expert Comment

by:jkr
Comment Utility
Hmm, what about turning the buffering off using

setvbuf( fptr, NULL, _IONBF,  0);

instead?
0
 
LVL 30

Expert Comment

by:Axter
Comment Utility
Use ftell.  ftell will give you the current file position.

If your file position is not at the end of the file already, then you can use a combination of ftell

and fseek.
Example:
long GetMySize(FILE *MyStream)
{
long OldPos = ftell(MyStream);
fseek(MyStream, 0, SEEK_END);
long CurLenFile = ftell(MyStream);
fseek(MyStream, OldPos, SEEK_SET);
return CurLenFile;
}
0
 
LVL 30

Expert Comment

by:Axter
Comment Utility
FYI:
IMHO, If all you want to do is find out the current size of a file, ftell is a better method for your requirements.
0
 
LVL 86

Expert Comment

by:jkr
Comment Utility
Axter, 'ftell()' will not help with a 'dir' command...
0
 
LVL 30

Expert Comment

by:Axter
Comment Utility
jkr,
You're right. I misunderstood the question.
0
 
LVL 2

Expert Comment

by:triso
Comment Utility
Prain,
There are a few problems with your code.  If you just typed it in then ignore this message, it's probably just typos.

But if not:

Having i delcared twice like so could be a problem:

int i = 0;
for (int i = 0; i < 10; i++)

This while loop uses i, inisde the for also using i, so it looks suspicious:

 while (i < 5000)
If buf is declared as "char *buf;" and not "char buf [X];"
then the sizeof (buf) is not correct in:

  setvbuf( fptr, buf, _IOFBF,  sizeof(buf));



0
 
LVL 2

Expert Comment

by:CSuvendra
Comment Utility
Alternatives.

1. Create each file in a temp folder in the same m/c where this code is executed. Call a different thread to transfer the file to remote (using ftp).

2. If ftp is not accessible, I guess copying the file in smaller chunks is advisable, each time checking if the connection to remote is active.

Finally remove the temporary file.

N.B. transferring Approx 5 MB data across networks may be cumbersome in slow networks. FTP seems to be a better alternative (you do not need to check for validity of packets each time)
0
 
LVL 3

Expert Comment

by:elfie
Comment Utility
Can you post the complete code. Also the write/fprintf/... calls are important. (and also as indicated the declarations of the buffers)
0
Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

 

Accepted Solution

by:
pariveshj_jain1 earned 200 total points
Comment Utility
parin,
i got the ans.
in your program u give condition
"while(i<5000)"
but the program can't come out of this loop because "i" will always remain less than 10.
now if u are increasing it in while loop then after coming out of while loop the for loop condition will cancel because u give i<10.
i think now ur prob. is solved.
if u can send the complete coding of this program then i can do my best.
my email id is pariveshj_jain1@rediffmail.com
ok bye
hopeing for best
parivesh
0
 
LVL 30

Expert Comment

by:Axter
Comment Utility
Hi (pariveshj_jain1 ), welcome to EE.
All of the experts here, for the most part have learn from other experts as to the proper etiquette for posting answer.

An answer should not be posted as an answer, if other experts have previously posted possible answers as comments, and/or have already made contributions to the question.

There are many experts who never post answers as answer.  Instead, they post their answers as comments.
If you read the following link, you'll see why this is the preferred method for many of our valued experts, including myself.

http://www.experts-exchange.com/jsp/cmtyQuestAnswer.jsp


Hi (prain):
Feel free to click the [Reject Answer] button near (Answer-poster's) response, even if it seems like a good answer.
Doing so will increase your chance of obtaining additional input from other experts.  Later, you can click the [Select Comment as Answer] button on any response.
0
 

Author Comment

by:prain
Comment Utility
Guys...
Thanks all for all inputs. I pretty much followed the CSuvendra's advice. I read some more docs on the net pertaining to this kind of a situation. I found that the problem I have is "Pretty Common" when doing IO on a network. So I created a Queue and enqueued my data into the Queue, while a separate thread dequeues the data and writes it into the files. Boom, problem solved. Of scourse,
it was really not solved in the first run. Since I am writing embeded SW on a real-time environment, there were few others issues too to fix the problem. Anyways..... the problem is solved now. I really appreciate everyone's input, and it is a blessing that we have boards such as these to exchange ideas.

prain.
 
0
 

Author Comment

by:prain
Comment Utility
Guys...
Thanks all for all inputs. I pretty much followed the CSuvendra's advice. I read some more docs on the net pertaining to this kind of a situation. I found that the problem I have is "Pretty Common" when doing IO on a network. So I created a Queue and enqueued my data into the Queue, while a separate thread dequeues the data and writes it into the files. Boom, problem solved. Of scourse,
it was really not solved in the first run. Since I am writing embeded SW on a real-time environment, there were few others issues too to fix the problem. Anyways..... the problem is solved now. I really appreciate everyone's input, and it is a blessing that we have boards such as these to exchange ideas.

prain.
 
0
 
LVL 30

Expert Comment

by:Axter
Comment Utility
prain,
>>I pretty much followed the CSuvendra's advice

If you followed CSuvendra's advice, then why did you award the points to pariveshj_jain1?

Did you use pariveshj_jain1 answer?
0
 
LVL 86

Expert Comment

by:jkr
Comment Utility
>>If you followed CSuvendra's advice, then why did you award the points to pariveshj_jain1?

That was what I was wondering about, too - especially since you accepted an answer of mine in the C++ area regarding the same issue a while ago.
0
 

Author Comment

by:prain
Comment Utility
Ok. Ok wait......

First of all, I apologise for the mistake. I forgot to click the "Reject" button. Just now I sent an e-mail to EE requesting them to remove points I accidently gave to pariveshj_jain1. Obviously lot of people posted their comments and views for this question. So no wonder if someone gets upset If I gave the points to an incorrent answer.

jkr....
Well yes you did. I appreciate that. And in fact I accepted your answer in C++ area. However I was testing "Parts" of my problem, and then I had different thoughts. Hope you would not get upset on this....I have accepted your answer in C++ area. But then the idea for a real solution came from CSuvendra. Thanks anyway. I appreciate everyone's effort.

I have asked EE to award the points to CSuvendra. I do not know at this point what they would on that. Let wait and see.

again.. my apology if anyone got upset on this minor issue.
(Mistake happens- Specially SW guys... :-) :-))


Thanks again guys.
Good Day..
prain.
0
 
LVL 2

Expert Comment

by:CSuvendra
Comment Utility
glad to be of some help ;-)
0
 
LVL 30

Expert Comment

by:Axter
Comment Utility
prain,
I don't think jkr was upset, and I certainly was not upset.

We just wanted to make sure the points went to the intended expert.

Thanks for working to clear it up.
0

Featured Post

Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

Join & Write a Comment

Have you thought about creating an iPhone application (app), but didn't even know where to get started? Here's how: ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ Important pre-programming comments: I’ve never tri…
Summary: This tutorial covers some basics of pointer, pointer arithmetic and function pointer. What is a pointer: A pointer is a variable which holds an address. This address might be address of another variable/address of devices/address of fu…
The goal of this video is to provide viewers with basic examples to understand how to use strings and some functions related to them in the C programming language.
Video by: Grant
The goal of this video is to provide viewers with basic examples to understand and use nested-loops in the C programming language.

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now