Solved

C-Files related Question

Posted on 2002-04-19
18
281 Views
Last Modified: 2010-05-18
OK.

This is what I want.
I have a C program. It creates a series of binary files on the disk. The record (buffer) is about 1000 bytes per cycle. There can be maximum of 5000 cycles per file. So each file could be maximum of 5000 * 1000 bytes = 5000000 bytes.

Since this writing is done across a Ethernet, after each file is closing, it closes and locks up. I find that this is due to inability of trasfring that huge amount of data over the Ethernet because in my program there is a time-out period. The close() call exceeds the time-out. So it throws me out. That is a requirement I must have in the program.

So, now I try to flush the buffer every time the buffer is ready to be written. So I use the setvbuf() and
fflush() together. So I have something like this.


/*I am creating 10 files */

int i = 0;
for (int i = 0; i < 10; i++)
{
 /*CReate a new file name here*/
 
 fptr =  fopen(filename);

 while (i < 5000)
 {  
  /*Create the buffer here, and fill it*/
  ..............
  /*set the buffer */
  setvbuf( fptr, buf, _IOFBF,  sizeof(buf));

  /*flush it */
  fflush(fptr);
 }

 /*Ok. 500 cycles done. So close it */
 fclose(fptr);
}

I want to see when I do a dir or ls (in unix) the file is growing (changing the size). I do not see this hapenning.  In fact I do not see even the file is created on the disk when I do a dir or ls when the loop is running. Only time I see the file created is when close() is called. However with ZERO file size. So the fflush() really did not work for me. I thought when I use fflush() along with setvbuf(), the data is trasferred immediately to the disk so that I can see the file is created and growing each cycle.

Can anyone out there throw some light into this?. I do not mind using any other method.

Thanks
prain
 

 


0
Comment
Question by:prain
  • 6
  • 3
  • 3
  • +5
18 Comments
 
LVL 3

Expert Comment

by:marcjb
ID: 6954883
It sounds like the OS is keeping a lot of data in the cache.  I had a similar problem, and I used 'fsync' to fix it.  According to the man page, fsync "copies all in-core parts of a file to disk".  So, before calling fclose, try:

fsync(fileno(fptr));

The call to fileno is needed here because you have a FILE* and fsync takes a file handle (int).

fsync is POSIX, but fileno is not.  However, you should either have fileno or a similar function available to you.

Hope this helps,

Marc
0
 
LVL 86

Expert Comment

by:jkr
ID: 6956487
Hmm, what about turning the buffering off using

setvbuf( fptr, NULL, _IONBF,  0);

instead?
0
 
LVL 30

Expert Comment

by:Axter
ID: 6960452
Use ftell.  ftell will give you the current file position.

If your file position is not at the end of the file already, then you can use a combination of ftell

and fseek.
Example:
long GetMySize(FILE *MyStream)
{
long OldPos = ftell(MyStream);
fseek(MyStream, 0, SEEK_END);
long CurLenFile = ftell(MyStream);
fseek(MyStream, OldPos, SEEK_SET);
return CurLenFile;
}
0
 
LVL 30

Expert Comment

by:Axter
ID: 6960469
FYI:
IMHO, If all you want to do is find out the current size of a file, ftell is a better method for your requirements.
0
 
LVL 86

Expert Comment

by:jkr
ID: 6960523
Axter, 'ftell()' will not help with a 'dir' command...
0
 
LVL 30

Expert Comment

by:Axter
ID: 6960534
jkr,
You're right. I misunderstood the question.
0
 
LVL 2

Expert Comment

by:triso
ID: 6975996
Prain,
There are a few problems with your code.  If you just typed it in then ignore this message, it's probably just typos.

But if not:

Having i delcared twice like so could be a problem:

int i = 0;
for (int i = 0; i < 10; i++)

This while loop uses i, inisde the for also using i, so it looks suspicious:

 while (i < 5000)
If buf is declared as "char *buf;" and not "char buf [X];"
then the sizeof (buf) is not correct in:

  setvbuf( fptr, buf, _IOFBF,  sizeof(buf));



0
 
LVL 2

Expert Comment

by:CSuvendra
ID: 6976330
Alternatives.

1. Create each file in a temp folder in the same m/c where this code is executed. Call a different thread to transfer the file to remote (using ftp).

2. If ftp is not accessible, I guess copying the file in smaller chunks is advisable, each time checking if the connection to remote is active.

Finally remove the temporary file.

N.B. transferring Approx 5 MB data across networks may be cumbersome in slow networks. FTP seems to be a better alternative (you do not need to check for validity of packets each time)
0
 
LVL 3

Expert Comment

by:elfie
ID: 7010590
Can you post the complete code. Also the write/fprintf/... calls are important. (and also as indicated the declarations of the buffers)
0
ScreenConnect 6.0 Free Trial

Want empowering updates? You're in the right place! Discover new features in ScreenConnect 6.0, based on partner feedback, to keep you business operating smoothly and optimally (the way it should be). Explore all of the extras and enhancements for yourself!

 

Accepted Solution

by:
pariveshj_jain1 earned 200 total points
ID: 7103206
parin,
i got the ans.
in your program u give condition
"while(i<5000)"
but the program can't come out of this loop because "i" will always remain less than 10.
now if u are increasing it in while loop then after coming out of while loop the for loop condition will cancel because u give i<10.
i think now ur prob. is solved.
if u can send the complete coding of this program then i can do my best.
my email id is pariveshj_jain1@rediffmail.com
ok bye
hopeing for best
parivesh
0
 
LVL 30

Expert Comment

by:Axter
ID: 7103261
Hi (pariveshj_jain1 ), welcome to EE.
All of the experts here, for the most part have learn from other experts as to the proper etiquette for posting answer.

An answer should not be posted as an answer, if other experts have previously posted possible answers as comments, and/or have already made contributions to the question.

There are many experts who never post answers as answer.  Instead, they post their answers as comments.
If you read the following link, you'll see why this is the preferred method for many of our valued experts, including myself.

http://www.experts-exchange.com/jsp/cmtyQuestAnswer.jsp


Hi (prain):
Feel free to click the [Reject Answer] button near (Answer-poster's) response, even if it seems like a good answer.
Doing so will increase your chance of obtaining additional input from other experts.  Later, you can click the [Select Comment as Answer] button on any response.
0
 

Author Comment

by:prain
ID: 7103566
Guys...
Thanks all for all inputs. I pretty much followed the CSuvendra's advice. I read some more docs on the net pertaining to this kind of a situation. I found that the problem I have is "Pretty Common" when doing IO on a network. So I created a Queue and enqueued my data into the Queue, while a separate thread dequeues the data and writes it into the files. Boom, problem solved. Of scourse,
it was really not solved in the first run. Since I am writing embeded SW on a real-time environment, there were few others issues too to fix the problem. Anyways..... the problem is solved now. I really appreciate everyone's input, and it is a blessing that we have boards such as these to exchange ideas.

prain.
 
0
 

Author Comment

by:prain
ID: 7103569
Guys...
Thanks all for all inputs. I pretty much followed the CSuvendra's advice. I read some more docs on the net pertaining to this kind of a situation. I found that the problem I have is "Pretty Common" when doing IO on a network. So I created a Queue and enqueued my data into the Queue, while a separate thread dequeues the data and writes it into the files. Boom, problem solved. Of scourse,
it was really not solved in the first run. Since I am writing embeded SW on a real-time environment, there were few others issues too to fix the problem. Anyways..... the problem is solved now. I really appreciate everyone's input, and it is a blessing that we have boards such as these to exchange ideas.

prain.
 
0
 
LVL 30

Expert Comment

by:Axter
ID: 7103585
prain,
>>I pretty much followed the CSuvendra's advice

If you followed CSuvendra's advice, then why did you award the points to pariveshj_jain1?

Did you use pariveshj_jain1 answer?
0
 
LVL 86

Expert Comment

by:jkr
ID: 7103614
>>If you followed CSuvendra's advice, then why did you award the points to pariveshj_jain1?

That was what I was wondering about, too - especially since you accepted an answer of mine in the C++ area regarding the same issue a while ago.
0
 

Author Comment

by:prain
ID: 7103705
Ok. Ok wait......

First of all, I apologise for the mistake. I forgot to click the "Reject" button. Just now I sent an e-mail to EE requesting them to remove points I accidently gave to pariveshj_jain1. Obviously lot of people posted their comments and views for this question. So no wonder if someone gets upset If I gave the points to an incorrent answer.

jkr....
Well yes you did. I appreciate that. And in fact I accepted your answer in C++ area. However I was testing "Parts" of my problem, and then I had different thoughts. Hope you would not get upset on this....I have accepted your answer in C++ area. But then the idea for a real solution came from CSuvendra. Thanks anyway. I appreciate everyone's effort.

I have asked EE to award the points to CSuvendra. I do not know at this point what they would on that. Let wait and see.

again.. my apology if anyone got upset on this minor issue.
(Mistake happens- Specially SW guys... :-) :-))


Thanks again guys.
Good Day..
prain.
0
 
LVL 2

Expert Comment

by:CSuvendra
ID: 7111610
glad to be of some help ;-)
0
 
LVL 30

Expert Comment

by:Axter
ID: 7111631
prain,
I don't think jkr was upset, and I certainly was not upset.

We just wanted to make sure the points went to the intended expert.

Thanks for working to clear it up.
0

Featured Post

Want to promote your upcoming event?

Attending an event? Speaking at a conference? Or exhibiting at a trade show? Easily inform your contacts by using a promotional banner in your email signature. This will ensure your organization’s most important contacts are in the know.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Windows programmers of the C/C++ variety, how many of you realise that since Window 9x Microsoft has been lying to you about what constitutes Unicode (http://en.wikipedia.org/wiki/Unicode)? They will have you believe that Unicode requires you to use…
This is a short and sweet, but (hopefully) to the point article. There seems to be some fundamental misunderstanding about the function prototype for the "main" function in C and C++, more specifically what type this function should return. I see so…
The goal of this video is to provide viewers with basic examples to understand opening and writing to files in the C programming language.
The goal of this video is to provide viewers with basic examples to understand and use conditional statements in the C programming language.

867 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now