C-Files related Question
OK.
This is what I want.
I have a C program. It creates a series of binary files on the disk. The record (buffer) is about 1000 bytes per cycle. There can be maximum of 5000 cycles per file. So each file could be maximum of 5000 * 1000 bytes = 5000000 bytes.
Since this writing is done across a Ethernet, after each file is closing, it closes and locks up. I find that this is due to inability of trasfring that huge amount of data over the Ethernet because in my program there is a time-out period. The close() call exceeds the time-out. So it throws me out. That is a requirement I must have in the program.
So, now I try to flush the buffer every time the buffer is ready to be written. So I use the setvbuf() and
fflush() together. So I have something like this.
/*I am creating 10 files */
int i = 0;
for (int i = 0; i < 10; i++)
{
/*CReate a new file name here*/
fptr = fopen(filename);
while (i < 5000)
{
/*Create the buffer here, and fill it*/
..............
/*set the buffer */
setvbuf( fptr, buf, _IOFBF, sizeof(buf));
/*flush it */
fflush(fptr);
}
/*Ok. 500 cycles done. So close it */
fclose(fptr);
}
I want to see when I do a dir or ls (in unix) the file is growing (changing the size). I do not see this hapenning. In fact I do not see even the file is created on the disk when I do a dir or ls when the loop is running. Only time I see the file created is when close() is called. However with ZERO file size. So the fflush() really did not work for me. I thought when I use fflush() along with setvbuf(), the data is trasferred immediately to the disk so that I can see the file is created and growing each cycle.
Can anyone out there throw some light into this?. I do not mind using any other method.
Thanks
prain
This is what I want.
I have a C program. It creates a series of binary files on the disk. The record (buffer) is about 1000 bytes per cycle. There can be maximum of 5000 cycles per file. So each file could be maximum of 5000 * 1000 bytes = 5000000 bytes.
Since this writing is done across a Ethernet, after each file is closing, it closes and locks up. I find that this is due to inability of trasfring that huge amount of data over the Ethernet because in my program there is a time-out period. The close() call exceeds the time-out. So it throws me out. That is a requirement I must have in the program.
So, now I try to flush the buffer every time the buffer is ready to be written. So I use the setvbuf() and
fflush() together. So I have something like this.
/*I am creating 10 files */
int i = 0;
for (int i = 0; i < 10; i++)
{
/*CReate a new file name here*/
fptr = fopen(filename);
while (i < 5000)
{
/*Create the buffer here, and fill it*/
..............
/*set the buffer */
setvbuf( fptr, buf, _IOFBF, sizeof(buf));
/*flush it */
fflush(fptr);
}
/*Ok. 500 cycles done. So close it */
fclose(fptr);
}
I want to see when I do a dir or ls (in unix) the file is growing (changing the size). I do not see this hapenning. In fact I do not see even the file is created on the disk when I do a dir or ls when the loop is running. Only time I see the file created is when close() is called. However with ZERO file size. So the fflush() really did not work for me. I thought when I use fflush() along with setvbuf(), the data is trasferred immediately to the disk so that I can see the file is created and growing each cycle.
Can anyone out there throw some light into this?. I do not mind using any other method.
Thanks
prain
Hmm, what about turning the buffering off using
setvbuf( fptr, NULL, _IONBF, 0);
instead?
setvbuf( fptr, NULL, _IONBF, 0);
instead?
Use ftell. ftell will give you the current file position.
If your file position is not at the end of the file already, then you can use a combination of ftell
and fseek.
Example:
long GetMySize(FILE *MyStream)
{
long OldPos = ftell(MyStream);
fseek(MyStream, 0, SEEK_END);
long CurLenFile = ftell(MyStream);
fseek(MyStream, OldPos, SEEK_SET);
return CurLenFile;
}
If your file position is not at the end of the file already, then you can use a combination of ftell
and fseek.
Example:
long GetMySize(FILE *MyStream)
{
long OldPos = ftell(MyStream);
fseek(MyStream, 0, SEEK_END);
long CurLenFile = ftell(MyStream);
fseek(MyStream, OldPos, SEEK_SET);
return CurLenFile;
}
FYI:
IMHO, If all you want to do is find out the current size of a file, ftell is a better method for your requirements.
IMHO, If all you want to do is find out the current size of a file, ftell is a better method for your requirements.
Axter, 'ftell()' will not help with a 'dir' command...
jkr,
You're right. I misunderstood the question.
You're right. I misunderstood the question.
Prain,
There are a few problems with your code. If you just typed it in then ignore this message, it's probably just typos.
But if not:
Having i delcared twice like so could be a problem:
int i = 0;
for (int i = 0; i < 10; i++)
This while loop uses i, inisde the for also using i, so it looks suspicious:
while (i < 5000)
If buf is declared as "char *buf;" and not "char buf [X];"
then the sizeof (buf) is not correct in:
setvbuf( fptr, buf, _IOFBF, sizeof(buf));
There are a few problems with your code. If you just typed it in then ignore this message, it's probably just typos.
But if not:
Having i delcared twice like so could be a problem:
int i = 0;
for (int i = 0; i < 10; i++)
This while loop uses i, inisde the for also using i, so it looks suspicious:
while (i < 5000)
If buf is declared as "char *buf;" and not "char buf [X];"
then the sizeof (buf) is not correct in:
setvbuf( fptr, buf, _IOFBF, sizeof(buf));
Alternatives.
1. Create each file in a temp folder in the same m/c where this code is executed. Call a different thread to transfer the file to remote (using ftp).
2. If ftp is not accessible, I guess copying the file in smaller chunks is advisable, each time checking if the connection to remote is active.
Finally remove the temporary file.
N.B. transferring Approx 5 MB data across networks may be cumbersome in slow networks. FTP seems to be a better alternative (you do not need to check for validity of packets each time)
1. Create each file in a temp folder in the same m/c where this code is executed. Call a different thread to transfer the file to remote (using ftp).
2. If ftp is not accessible, I guess copying the file in smaller chunks is advisable, each time checking if the connection to remote is active.
Finally remove the temporary file.
N.B. transferring Approx 5 MB data across networks may be cumbersome in slow networks. FTP seems to be a better alternative (you do not need to check for validity of packets each time)
Can you post the complete code. Also the write/fprintf/... calls are important. (and also as indicated the declarations of the buffers)
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Hi (pariveshj_jain1 ), welcome to EE.
All of the experts here, for the most part have learn from other experts as to the proper etiquette for posting answer.
An answer should not be posted as an answer, if other experts have previously posted possible answers as comments, and/or have already made contributions to the question.
There are many experts who never post answers as answer. Instead, they post their answers as comments.
If you read the following link, you'll see why this is the preferred method for many of our valued experts, including myself.
https://www.experts-exchange.com/jsp/cmtyQuestAnswer.jsp
Hi (prain):
Feel free to click the [Reject Answer] button near (Answer-poster's) response, even if it seems like a good answer.
Doing so will increase your chance of obtaining additional input from other experts. Later, you can click the [Select Comment as Answer] button on any response.
All of the experts here, for the most part have learn from other experts as to the proper etiquette for posting answer.
An answer should not be posted as an answer, if other experts have previously posted possible answers as comments, and/or have already made contributions to the question.
There are many experts who never post answers as answer. Instead, they post their answers as comments.
If you read the following link, you'll see why this is the preferred method for many of our valued experts, including myself.
https://www.experts-exchange.com/jsp/cmtyQuestAnswer.jsp
Hi (prain):
Feel free to click the [Reject Answer] button near (Answer-poster's) response, even if it seems like a good answer.
Doing so will increase your chance of obtaining additional input from other experts. Later, you can click the [Select Comment as Answer] button on any response.
ASKER
Guys...
Thanks all for all inputs. I pretty much followed the CSuvendra's advice. I read some more docs on the net pertaining to this kind of a situation. I found that the problem I have is "Pretty Common" when doing IO on a network. So I created a Queue and enqueued my data into the Queue, while a separate thread dequeues the data and writes it into the files. Boom, problem solved. Of scourse,
it was really not solved in the first run. Since I am writing embeded SW on a real-time environment, there were few others issues too to fix the problem. Anyways..... the problem is solved now. I really appreciate everyone's input, and it is a blessing that we have boards such as these to exchange ideas.
prain.
Thanks all for all inputs. I pretty much followed the CSuvendra's advice. I read some more docs on the net pertaining to this kind of a situation. I found that the problem I have is "Pretty Common" when doing IO on a network. So I created a Queue and enqueued my data into the Queue, while a separate thread dequeues the data and writes it into the files. Boom, problem solved. Of scourse,
it was really not solved in the first run. Since I am writing embeded SW on a real-time environment, there were few others issues too to fix the problem. Anyways..... the problem is solved now. I really appreciate everyone's input, and it is a blessing that we have boards such as these to exchange ideas.
prain.
ASKER
Guys...
Thanks all for all inputs. I pretty much followed the CSuvendra's advice. I read some more docs on the net pertaining to this kind of a situation. I found that the problem I have is "Pretty Common" when doing IO on a network. So I created a Queue and enqueued my data into the Queue, while a separate thread dequeues the data and writes it into the files. Boom, problem solved. Of scourse,
it was really not solved in the first run. Since I am writing embeded SW on a real-time environment, there were few others issues too to fix the problem. Anyways..... the problem is solved now. I really appreciate everyone's input, and it is a blessing that we have boards such as these to exchange ideas.
prain.
Thanks all for all inputs. I pretty much followed the CSuvendra's advice. I read some more docs on the net pertaining to this kind of a situation. I found that the problem I have is "Pretty Common" when doing IO on a network. So I created a Queue and enqueued my data into the Queue, while a separate thread dequeues the data and writes it into the files. Boom, problem solved. Of scourse,
it was really not solved in the first run. Since I am writing embeded SW on a real-time environment, there were few others issues too to fix the problem. Anyways..... the problem is solved now. I really appreciate everyone's input, and it is a blessing that we have boards such as these to exchange ideas.
prain.
prain,
>>I pretty much followed the CSuvendra's advice
If you followed CSuvendra's advice, then why did you award the points to pariveshj_jain1?
Did you use pariveshj_jain1 answer?
>>I pretty much followed the CSuvendra's advice
If you followed CSuvendra's advice, then why did you award the points to pariveshj_jain1?
Did you use pariveshj_jain1 answer?
>>If you followed CSuvendra's advice, then why did you award the points to pariveshj_jain1?
That was what I was wondering about, too - especially since you accepted an answer of mine in the C++ area regarding the same issue a while ago.
That was what I was wondering about, too - especially since you accepted an answer of mine in the C++ area regarding the same issue a while ago.
ASKER
Ok. Ok wait......
First of all, I apologise for the mistake. I forgot to click the "Reject" button. Just now I sent an e-mail to EE requesting them to remove points I accidently gave to pariveshj_jain1. Obviously lot of people posted their comments and views for this question. So no wonder if someone gets upset If I gave the points to an incorrent answer.
jkr....
Well yes you did. I appreciate that. And in fact I accepted your answer in C++ area. However I was testing "Parts" of my problem, and then I had different thoughts. Hope you would not get upset on this....I have accepted your answer in C++ area. But then the idea for a real solution came from CSuvendra. Thanks anyway. I appreciate everyone's effort.
I have asked EE to award the points to CSuvendra. I do not know at this point what they would on that. Let wait and see.
again.. my apology if anyone got upset on this minor issue.
(Mistake happens- Specially SW guys... :-) :-))
Thanks again guys.
Good Day..
prain.
First of all, I apologise for the mistake. I forgot to click the "Reject" button. Just now I sent an e-mail to EE requesting them to remove points I accidently gave to pariveshj_jain1. Obviously lot of people posted their comments and views for this question. So no wonder if someone gets upset If I gave the points to an incorrent answer.
jkr....
Well yes you did. I appreciate that. And in fact I accepted your answer in C++ area. However I was testing "Parts" of my problem, and then I had different thoughts. Hope you would not get upset on this....I have accepted your answer in C++ area. But then the idea for a real solution came from CSuvendra. Thanks anyway. I appreciate everyone's effort.
I have asked EE to award the points to CSuvendra. I do not know at this point what they would on that. Let wait and see.
again.. my apology if anyone got upset on this minor issue.
(Mistake happens- Specially SW guys... :-) :-))
Thanks again guys.
Good Day..
prain.
glad to be of some help ;-)
prain,
I don't think jkr was upset, and I certainly was not upset.
We just wanted to make sure the points went to the intended expert.
Thanks for working to clear it up.
I don't think jkr was upset, and I certainly was not upset.
We just wanted to make sure the points went to the intended expert.
Thanks for working to clear it up.
fsync(fileno(fptr));
The call to fileno is needed here because you have a FILE* and fsync takes a file handle (int).
fsync is POSIX, but fileno is not. However, you should either have fileno or a similar function available to you.
Hope this helps,
Marc