Solved

binary fwrite in C

Posted on 2009-04-06
38
1,473 Views
Last Modified: 2012-05-06
Hi,

I'm trying to write binary code into a file - that means, create a struct, dump the contents to a file and later retrieve it. I'm doing this to make retrieval easy and to save disk space.
I've tried it doing the following:
- use fwrite() several times
- close the file
- use fread()
and it works well, only on some occasions after making several fread()'s the fread() fails and can't read anymore(all subsequent freads fail).
I tried to circumvent it using fseek() but then I saw that the data comes skewed. Looks like some unwanted data was added to the file.

My questions are:
- what is the recommended option for binary write/read?
- how is this problem caused and how can we fix it?
typedef struct _BIN_TO_SEND
{	
	int x;
	int y;
	char SbinDesc[SBIN_DESC];
}BIN_TO_SEND;
BIN_TO_SEND data;
 
 
the following is called repeatedly:	
	if(pBinHandle == NULL)
	{
		pBinHandle = fopen( binFileName , "wb");
		if(pBinHandle == NULL)
		{
			printf("Error\n!!!");
		}
	}
	fwrite(&data , sizeof(BIN_TO_SEND), 1, pBinHandle);
	binWrittenCnt++;
	fflush(pBinHandle);
 
the read is similar

Open in new window

0
Comment
Question by:optimaltest
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 15
  • 13
  • 9
  • +1
38 Comments
 
LVL 11

Expert Comment

by:princeatapi
ID: 24075474
Hi !

 Here comes the issue , >> fwrite(&data , sizeof(BIN_TO_SEND), 1, pBinHandle);
 just alter the second argument of the above function call using a variable !

 int RecSize =  sizeof(BIN_TO_SEND);
.
.
.
fwrite(&data , RecSize , 1 , pBinHandle);

it will help u out i think

0
 
LVL 53

Expert Comment

by:Infinity08
ID: 24075484
>> the read is similar

Are you reading this on the same platform, from an executable created by the same compiler ?
Are you using sizeof(BIN_TO_SEND) just like you did in the fwrite ?


>> only on some occasions after making several fread()'s the fread() fails and can't read anymore

What do ferror and feof report ?

        http://www.cplusplus.com/reference/clibrary/cstdio/fread.html
0
 
LVL 53

Expert Comment

by:Infinity08
ID: 24075487
>>  just alter the second argument of the above function call using a variable !

Why ?
0
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 7

Expert Comment

by:HalfAsleep
ID: 24075857
It would be helpful to see the read code too, and as Infinity says, what error is fread reporting?

And when you say on some occasions, do you mean that you have been running this code on the same machine for a while, and suddenly it stops working?  Or are you trying this on multiple machines, and some machines get the error?
0
 

Author Comment

by:optimaltest
ID: 24076070
I run the write code and wait for it to finish, then run the read code.
I do the same thing several times on the same machine. Sometimes it works, sometimes it doesn't.
ferror and feof return 0, but readBytes = 0
0
 

Author Comment

by:optimaltest
ID: 24076081
Posting the read code
0
 
LVL 53

Assisted Solution

by:Infinity08
Infinity08 earned 125 total points
ID: 24076084
>> ferror and feof return 0, but readBytes = 0

That doesn't make sense. If fread returns a value that is different from its count parameter (third parameter), then either feof or ferror will return a non-zero value.

Can you show the exact code you used (all of it) ?
0
 
LVL 7

Assisted Solution

by:HalfAsleep
HalfAsleep earned 125 total points
ID: 24076087
And can you show us your read code?
0
 

Author Comment

by:optimaltest
ID: 24076089
doing it in the comment area:

do
      {
            seekRes = fseek(pInputFileHandle,numOfParam*sizeof(PARAM_TO_SEND),SEEK_SET);
            readBytes = fread(&currentParam , sizeof(PARAM_TO_SEND) , 1  , pInputFileHandle);
            if(readBytes > 0)
            {
                  fprintf(pOutputFileHandle, "%d, %d, %d, %d, %lf\n",
                                     currentParam.partId,
                                     currentParam.siteId,
                                     currentParam.paramUniqueId,
                                     currentParam.paramPassFail,
                                     currentParam.paramValue);
                  memset(&currentParam , 0 , sizeof(PARAM_TO_SEND));

                  numOfParam++;
            }
            else
            {
                  errorCode = ferror(pOutputFileHandle);
                  printf( "ParseFile_Param - Failed to Write. Error=%d!\n", errorCode);             
            }
      }
0
 
LVL 53

Expert Comment

by:Infinity08
ID: 24076099
Your code doesn't check feof.
How is PARAM_TO_SEND defined ?
Could you post the whole code ?
Can you also confirm that this code is compiled with the same compiler (and the same compiler settings), and run on the same machine ?
0
 
LVL 7

Expert Comment

by:HalfAsleep
ID: 24076125
Also, the fread moves the file read pointer forward, so no need to call fseek inside the do while for multiple reads, unless you are trying to skip reading a struct at the beginning or in the middle of the loop.  I am also curious if the PARAM_TO_SEND matches the size of BIN_TO_SEND
0
 

Author Comment

by:optimaltest
ID: 24076135
Both project are part of the same solution. The write mechanism is one of the threads that receives write commands through a queue.
every write command should add astruct entry.

Here is the whole code.

write:
void HandleBinWrite(SUPPORTED_COMMANDS *pParam)
{      
      if(pParam == NULL)
      {
            return;
      }
      // This bin is already stored in the CSV (resend)
      if( (pParam->data.Bin.managementFlag & IS_RESEND) != 0)
      {
            return;
      }

      if(pBinHandle == NULL)
      {
            sprintf(binFileName , "%s\\bin" , logFolderPath);
            pBinHandle = fopen( binFileName , "wb");
            if(pBinHandle == NULL)
            {
                  printf("Error\n!!!");
            }
      }
      fwrite(&pParam->data.Bin.data , sizeof(BIN_TO_SEND), 1, pBinHandle);
      printf("partId=%d, siteId=%d\n",pParam->data.Bin.data.partId, pParam->data.Bin.data.siteId);
      binWrittenCnt++;
      //fprintf( pBinHandle , "HandleBinWrite => partId= %d,hbin=%d,sbin=%d\n",pParam->data.Bin.data.partId,pParam->data.Bin.data.Hbin , pParam->data.Bin.data.Sbin);
      fflush(pBinHandle);
      DeleteCommandIfNeeded(&pParam);
}

read:
void ParseFile_Param(char *fileToParse)
{
      PARAM_TO_SEND currentParam;
      int readBytes = -1;
      FILE *pInputFileHandle = NULL;
      FILE *pOutputFileHandle = NULL;
      int numOfParam = 0;
      int seekRes=0;
      int errorCode = 0;

      OpenIOFiles(fileToParse, &pInputFileHandle, &pOutputFileHandle);
      
      fprintf(pOutputFileHandle, "partId, siteId, paramUniqueId, paramPassFail, paramValue\n");

      do
      {
            seekRes = fseek(pInputFileHandle,numOfParam*sizeof(PARAM_TO_SEND),SEEK_SET);
            readBytes = fread(&currentParam , sizeof(PARAM_TO_SEND) , 1  , pInputFileHandle);
            if(readBytes > 0)
            {
                  fprintf(pOutputFileHandle, "%d, %d, %d, %d, %lf\n",
                                     currentParam.partId,
                                     currentParam.siteId,
                                     currentParam.paramUniqueId,
                                     currentParam.paramPassFail,
                                     currentParam.paramValue);
                  memset(&currentParam , 0 , sizeof(PARAM_TO_SEND));

                  numOfParam++;
            }
            else
            {
                  errorCode = ferror(pOutputFileHandle);
                  printf( "ParseFile_Param - Failed to Write. Error=%d!\n", errorCode);             
            }
      }
      //while( !feof(pOutputFileHandle));
      while(readBytes > 0);

      fclose(pInputFileHandle);
      fclose(pOutputFileHandle);
}
0
 

Author Comment

by:optimaltest
ID: 24076146
Sorry posted the wrong write code. Here it is:

void HandleParamWrite(SUPPORTED_COMMANDS *pParam)
{      
      if(pParam == NULL)
      {
            return;
      }
      // This param is already stored in the CSV (resend)
      if( (pParam->data.Param.managementFlag & IS_RESEND) != 0)
      {
            return;
      }
      if(pParamHandle == NULL)
      {
            sprintf(paramFileName , "%s\\param" , logFolderPath);
            pParamHandle = fopen( paramFileName , "wb");
            if(pParamHandle == NULL)
            {
                  printf("Error\n!!!");
            }
      }
      fwrite(&pParam->data.Param.data , sizeof(PARAM_TO_SEND), 1, pParamHandle);
      printf("partId=%d, siteId = %d, paramValue=%lf\n",pParam->data.Param.data.partId, pParam->data.Param.data.siteId, pParam->data.Param.data.paramValue);
      paramWrittenCnt++;
      //fprintf( pParamHandle , "HandleParamWrite => ParatId=%d,Name=%s,Val=%lf,paramNumber=%d\n",pParam->data.Param.data.partId,pParam->data.Param.data.paramTestName , pParam->data.Param.data.paramValue , pParam->data.Param.data.paramNumber);
      fflush(pParamHandle);
      DeleteCommandIfNeeded(&pParam);
}
0
 
LVL 53

Expert Comment

by:Infinity08
ID: 24076152
>> Here is the whole code.

That's not the whole code. Those are just two functions.

Can you also answer my other questions ?
0
 
LVL 53

Expert Comment

by:Infinity08
ID: 24076161
Btw, where are you closing the output file in the write code ?
0
 

Author Comment

by:optimaltest
ID: 24076202
I agree, no need for the fseek.
I've added it to try to see if it would solve the case.
The compiler has a lot of options. which options (in VS2005) should I check for?
Also, I haven't confirmed it but I believe the problem exists in VS2005 but doesn't exist on Linux.
typedef struct _PARAM_TO_SEND
{		
	int partId;
	int siteId;
	int paramUniqueId;
	int paramPassFail;
	double paramValue;
}PARAM_TO_SEND;

Open in new window

0
 
LVL 53

Expert Comment

by:Infinity08
ID: 24076238
>> The compiler has a lot of options. which options (in VS2005) should I check for?

Any options that might impact struct padding, and struct member alignment.


Now, BIN_TO_SEND :

>> typedef struct _BIN_TO_SEND
>> {      
>>         int x;
>>         int y;
>>         char SbinDesc[SBIN_DESC];
>> }BIN_TO_SEND;

is not the same as PARAM_TO_SEND :

>> typedef struct _PARAM_TO_SEND
>> {              
>>         int partId;
>>         int siteId;
>>         int paramUniqueId;
>>         int paramPassFail;
>>         double paramValue;
>> }PARAM_TO_SEND;

You're writing the first, and reading the second ... It's not surprising that you see odd behavior.
0
 
LVL 53

Expert Comment

by:Infinity08
ID: 24076260
There are also still several questions that remain unanswered. I'll summarize them here so you know which ones :

>> Your code doesn't check feof.

>> Could you post the whole code ?

>> Can you also confirm that this code is compiled with the same compiler (and the same compiler settings), and run on the same machine ?

>> Btw, where are you closing the output file in the write code ?
0
 
LVL 7

Expert Comment

by:HalfAsleep
ID: 24076288
Compiler options should not be an issue, as long as you have the same options for both read and write, compiled with same compiler, for the same target.

When the code works, have you checked the read with the debugger, and made sure that you have read the exact number of bytes that you would expect, and that the bytes contain the correct data?

The write and read has to use the exact same size.  And it is clear by now, that you are not using the same struct, or data definition on both sides.  On the write side, you are using BIN_TO_SEND, while on the read side you are using PARAM_TO_SEND.  PARAM_TO_SEND is a permanent size, while we cannot determine the size of BIN_TO_SEND, until you provide us the size of SBIN_DESC.
0
 

Author Comment

by:optimaltest
ID: 24076295
There is no close command for it.
I tried closing it and it didn't help
0
 
LVL 53

Expert Comment

by:Infinity08
ID: 24076317
>> Compiler options should not be an issue, as long as you have the same options for both read and write, compiled with same compiler, for the same target.

Which was exactly what I was asking about ;)


>> There is no close command for it.
>> I tried closing it and it didn't help

What do you mean ? Just use fclose like you did in the read code.
0
 
LVL 7

Expert Comment

by:HalfAsleep
ID: 24076352
I think maybe one of the issues here is that you are trying to write one particuar data struct, and the read something else.  Usually, when you write something binary, you have to be absolutely specific about how to read it as well.  The best way to do that, is to read the same struct typedef that you used for writing.

In other words, can you explain why you are using one struct typedef for write and a different for read, instead of using just one struct typedef for both read and write?
0
 

Author Comment

by:optimaltest
ID: 24076499
>> Your code doesn't check feof.
feof returns 1 when readBytes=0

>> Could you post the whole code ?
I can't post the whole project. I'll see which parts of it I can post

>> Can you also confirm that this code is compiled with the same compiler (and the same compiler settings), and run on the same machine ?
same compiler, same machine. Regarding the compiler options, it was defined in a similar way but I can't guarantee they are all the same

>> Btw, where are you closing the output file in the write code ?
It's not closed properly.
I tried to add an fclose() command in the end, but it didn't make any difference

>> In other words, can you explain why you are using one struct typedef for write and a different for read, instead of using just one struct typedef for both read and write?
That was due to an error in my post, not the code. the write and read both use PARAM_TO_SEND
0
 
LVL 53

Expert Comment

by:Infinity08
ID: 24076537
>> >> Your code doesn't check feof.
>> feof returns 1 when readBytes=0

That means that the end of the file has been reached while trying to read the requested data from the file. Are you sure that there's supposed to be more data to read ?


>> same compiler, same machine. Regarding the compiler options, it was defined in a similar way but I can't guarantee they are all the same

Then that should be ok.


>> It's not closed properly.
>> I tried to add an fclose() command in the end, but it didn't make any difference

You'll need it anyway.


>> That was due to an error in my post, not the code. the write and read both use PARAM_TO_SEND

Do they also use the exact same definition of the struct ?
Did you verify that sizeof(PARAM_TO_SEND) returns the same value for both reader and writer ?
0
 
LVL 53

Expert Comment

by:Infinity08
ID: 24076560
Btw, if the code is too big, you could write a representative (compilable) smaller program that exhibits the same behavior.

Doing that (ie. trimming the code down to only the part that causes/shows the problem) will also make it easier for you to track down what the problem is.
0
 
LVL 7

Expert Comment

by:HalfAsleep
ID: 24076712
As Infinity suggests, are you sure that you are not done when you get eof?  Maybe the eof is actually valid, ie, you have written 3 times, but try to read 4 times.
0
 

Author Comment

by:optimaltest
ID: 24076744
I will write a smaller program and see how it goes. Probably not today though.
Thanks for your help, I will let you know how it works out.
0
 
LVL 53

Expert Comment

by:Infinity08
ID: 24076769
For now, did you look into the eof issue ? That's where the problem is : the reader code encounters the end of the file ,while you think there should be more data in the file. Are you sure there should be more data ? Why ?
0
 

Author Comment

by:optimaltest
ID: 24077231
>>For now, did you look into the eof issue ? That's where the problem is : the reader code encounters the end of the file ,while you think there should be more data in the file. Are you sure there should be more data ? Why ?
I'm sure because I can read data written after the EOF from the file using fseek() to the next record .
0
 
LVL 7

Expert Comment

by:HalfAsleep
ID: 24077277
Are you running write and read concurrently in 2 different threads, or is the writer always running before the reader?

And can you show how you use fseek to "get past the eof"?  Are you checking the file position after an eof, and are you able to fseek past this point?
0
 
LVL 53

Expert Comment

by:Infinity08
ID: 24077411
>> I'm sure because I can read data written after the EOF from the file using fseek() to the next record .

That can't be right. If you have reached the end of the file, then there's no more data to read ... you can also not fseek beyond the end of the file.

Maybe you didn't check whether fseek returned an error ? And when you thought that fseek went "past the end of the file", it really didn't, and the get pointer just stayed where it was before the fseek.

In any case, what you're describing is not possible, so it's likely an error on your side :)

Now would be a good time to see the code heh ... Or at least a representative sample of it that has the same problem.
0
 

Author Comment

by:optimaltest
ID: 24077447
>Are you running write and read concurrently in 2 different threads, or is the writer always running before the reader?

I'm waiting for the write to finish and the program to exit then running the read.
I can run the read several times for the same result.
When writing I write on several files concurrently. I've posted 2 functions - HandleParamWrite() and HandleBinWrite() that write concurrently to different files.

>And can you show how you use fseek to "get past the eof"?  Are you checking the file position after an eof, and are you able to fseek past this point?

In the debugger I increment numOfParam and return to the fseek.
0
 
LVL 7

Expert Comment

by:HalfAsleep
ID: 24077634
You cannot incremnet that variable and expect to read more in the file. fseek will never allow you to go beyond the eof position. I think you must be reading from an already read part of the file. do you have unique data in each struct, so that you can see if you read something twice? In any regard, the eof is probably valid, and you cannot fseek past the eof, just by incrementing that variable when debugging. Some more code would help alot I think
0
 
LVL 53

Expert Comment

by:Infinity08
ID: 24084915
Any progress on that code ?
0
 

Author Comment

by:optimaltest
ID: 24129886
I rewrote the code in one read and one write function.
No matter how i write it, it stops reading after 25 records.
If I take the same output file and use only the reader function on Linux, it reads the whole file.

//WriterThread.c
#include <stdio.h>
#include <stdlib.h>
#include <sys/stat.h>
char paramFileName[512];
FILE *pParamHandle = NULL;
 
 
 
////////////////////////////////////////////////////////////////////////////////////////////
// Structures
////////////////////////////////////////////////////////////////////////////////////////////
 
typedef struct _PARAM_TO_SEND
{		
	int partId;
	int siteId;
	int paramUniqueId;
	int paramPassFail;
	double paramValue;
}PARAM_TO_SEND;
 
PARAM_TO_SEND param;
PARAM_TO_SEND currentParam;
 
void temp_writer()
{	
	if(pParamHandle == NULL)
	{
        sprintf(paramFileName , "param_short" );
		pParamHandle = fopen( paramFileName , "wb");
		if(pParamHandle == NULL)
		{
			printf("Error\n!!!");
		}
	}
	fwrite( &param, sizeof(PARAM_TO_SEND), 1, pParamHandle);
	
	//fprintf( pParamHandle , "HandleParamWrite => ParatId=%d,Name=%s,Val=%lf,paramNumber=%d\n",pParam->data.Param.data.partId,pParam->data.Param.data.paramTestName , pParam->data.Param.data.paramValue , pParam->data.Param.data.paramNumber);
	fflush(pParamHandle);
	
 
}
void temp_reader()
{
	int readBytes=0;
	int currentParam_Len;
	int errorCode=0;
 
	currentParam_Len = sizeof(PARAM_TO_SEND);
	readBytes = fread(&currentParam , currentParam_Len , 1  , pParamHandle);
	errorCode = ferror(pParamHandle);
	if(feof(pParamHandle))
		printf("reached end of file\n");
	if(errorCode != 0)
		printf("error=%d\n", errorCode);
 
	if(readBytes > 0)
	{
		printf( "%d, %d, %d, %d, %lf\n",
					 currentParam.partId, 
					 currentParam.siteId, 
					 currentParam.paramUniqueId, 
					 currentParam.paramPassFail, 
					 currentParam.paramValue);
		//memset(&currentParam , 0 , sizeof(PARAM_TO_SEND));
	}
	else
	{
		printf( "ParseFile_Param - Failed to Write\n"); 		
	}
}
 
int main()
{
	int i=0;
 
 
	param.partId = 0;
	param.siteId = 0;
	param.paramPassFail = 0;
	param.paramValue = 100.234;
	for( ; i<100;i++)
	{
		param.paramUniqueId = i;
		temp_writer();
	}
	fclose(pParamHandle);
	
	pParamHandle = fopen( paramFileName , "r");
	for( i=0 ; i<100 ; i++ )
	{
		temp_reader();
	}
	fclose(pParamHandle);
 
}

Open in new window

reader.JPG
0
 

Accepted Solution

by:
optimaltest earned 0 total points
ID: 24130133
Issue is solved.
the fwrite was "wb" and the fread was "r".
when I change the fread to "rb" it worked perfectly.
0
 
LVL 7

Expert Comment

by:HalfAsleep
ID: 24130185
Heh.  Just goes to show that sometimes, it is the small things that throw you completely off!  Good job!
0
 
LVL 53

Expert Comment

by:Infinity08
ID: 24134988
I'm glad you found it :)
0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

An Outlet in Cocoa is a persistent reference to a GUI control; it connects a property (a variable) to a control.  For example, it is common to create an Outlet for the text field GUI control and change the text that appears in this field via that Ou…
This tutorial is posted by Aaron Wojnowski, administrator at SDKExpert.net.  To view more iPhone tutorials, visit www.sdkexpert.net. This is a very simple tutorial on finding the user's current location easily. In this tutorial, you will learn ho…
The goal of this video is to provide viewers with basic examples to understand and use pointers in the C programming language.
Video by: Grant
The goal of this video is to provide viewers with basic examples to understand and use while-loops in the C programming language.

707 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question