Solved

binary fwrite in C

Posted on 2009-04-06
38
1,457 Views
Last Modified: 2012-05-06
Hi,

I'm trying to write binary code into a file - that means, create a struct, dump the contents to a file and later retrieve it. I'm doing this to make retrieval easy and to save disk space.
I've tried it doing the following:
- use fwrite() several times
- close the file
- use fread()
and it works well, only on some occasions after making several fread()'s the fread() fails and can't read anymore(all subsequent freads fail).
I tried to circumvent it using fseek() but then I saw that the data comes skewed. Looks like some unwanted data was added to the file.

My questions are:
- what is the recommended option for binary write/read?
- how is this problem caused and how can we fix it?
typedef struct _BIN_TO_SEND

{	

	int x;

	int y;

	char SbinDesc[SBIN_DESC];

}BIN_TO_SEND;

BIN_TO_SEND data;
 
 

the following is called repeatedly:	

	if(pBinHandle == NULL)

	{

		pBinHandle = fopen( binFileName , "wb");

		if(pBinHandle == NULL)

		{

			printf("Error\n!!!");

		}

	}

	fwrite(&data , sizeof(BIN_TO_SEND), 1, pBinHandle);

	binWrittenCnt++;

	fflush(pBinHandle);
 

the read is similar

Open in new window

0
Comment
Question by:optimaltest
  • 15
  • 13
  • 9
  • +1
38 Comments
 
LVL 11

Expert Comment

by:princeatapi
Comment Utility
Hi !

 Here comes the issue , >> fwrite(&data , sizeof(BIN_TO_SEND), 1, pBinHandle);
 just alter the second argument of the above function call using a variable !

 int RecSize =  sizeof(BIN_TO_SEND);
.
.
.
fwrite(&data , RecSize , 1 , pBinHandle);

it will help u out i think

0
 
LVL 53

Expert Comment

by:Infinity08
Comment Utility
>> the read is similar

Are you reading this on the same platform, from an executable created by the same compiler ?
Are you using sizeof(BIN_TO_SEND) just like you did in the fwrite ?


>> only on some occasions after making several fread()'s the fread() fails and can't read anymore

What do ferror and feof report ?

        http://www.cplusplus.com/reference/clibrary/cstdio/fread.html
0
 
LVL 53

Expert Comment

by:Infinity08
Comment Utility
>>  just alter the second argument of the above function call using a variable !

Why ?
0
 
LVL 7

Expert Comment

by:HalfAsleep
Comment Utility
It would be helpful to see the read code too, and as Infinity says, what error is fread reporting?

And when you say on some occasions, do you mean that you have been running this code on the same machine for a while, and suddenly it stops working?  Or are you trying this on multiple machines, and some machines get the error?
0
 

Author Comment

by:optimaltest
Comment Utility
I run the write code and wait for it to finish, then run the read code.
I do the same thing several times on the same machine. Sometimes it works, sometimes it doesn't.
ferror and feof return 0, but readBytes = 0
0
 

Author Comment

by:optimaltest
Comment Utility
Posting the read code
0
 
LVL 53

Assisted Solution

by:Infinity08
Infinity08 earned 125 total points
Comment Utility
>> ferror and feof return 0, but readBytes = 0

That doesn't make sense. If fread returns a value that is different from its count parameter (third parameter), then either feof or ferror will return a non-zero value.

Can you show the exact code you used (all of it) ?
0
 
LVL 7

Assisted Solution

by:HalfAsleep
HalfAsleep earned 125 total points
Comment Utility
And can you show us your read code?
0
 

Author Comment

by:optimaltest
Comment Utility
doing it in the comment area:

do
      {
            seekRes = fseek(pInputFileHandle,numOfParam*sizeof(PARAM_TO_SEND),SEEK_SET);
            readBytes = fread(&currentParam , sizeof(PARAM_TO_SEND) , 1  , pInputFileHandle);
            if(readBytes > 0)
            {
                  fprintf(pOutputFileHandle, "%d, %d, %d, %d, %lf\n",
                                     currentParam.partId,
                                     currentParam.siteId,
                                     currentParam.paramUniqueId,
                                     currentParam.paramPassFail,
                                     currentParam.paramValue);
                  memset(&currentParam , 0 , sizeof(PARAM_TO_SEND));

                  numOfParam++;
            }
            else
            {
                  errorCode = ferror(pOutputFileHandle);
                  printf( "ParseFile_Param - Failed to Write. Error=%d!\n", errorCode);             
            }
      }
0
 
LVL 53

Expert Comment

by:Infinity08
Comment Utility
Your code doesn't check feof.
How is PARAM_TO_SEND defined ?
Could you post the whole code ?
Can you also confirm that this code is compiled with the same compiler (and the same compiler settings), and run on the same machine ?
0
 
LVL 7

Expert Comment

by:HalfAsleep
Comment Utility
Also, the fread moves the file read pointer forward, so no need to call fseek inside the do while for multiple reads, unless you are trying to skip reading a struct at the beginning or in the middle of the loop.  I am also curious if the PARAM_TO_SEND matches the size of BIN_TO_SEND
0
 

Author Comment

by:optimaltest
Comment Utility
Both project are part of the same solution. The write mechanism is one of the threads that receives write commands through a queue.
every write command should add astruct entry.

Here is the whole code.

write:
void HandleBinWrite(SUPPORTED_COMMANDS *pParam)
{      
      if(pParam == NULL)
      {
            return;
      }
      // This bin is already stored in the CSV (resend)
      if( (pParam->data.Bin.managementFlag & IS_RESEND) != 0)
      {
            return;
      }

      if(pBinHandle == NULL)
      {
            sprintf(binFileName , "%s\\bin" , logFolderPath);
            pBinHandle = fopen( binFileName , "wb");
            if(pBinHandle == NULL)
            {
                  printf("Error\n!!!");
            }
      }
      fwrite(&pParam->data.Bin.data , sizeof(BIN_TO_SEND), 1, pBinHandle);
      printf("partId=%d, siteId=%d\n",pParam->data.Bin.data.partId, pParam->data.Bin.data.siteId);
      binWrittenCnt++;
      //fprintf( pBinHandle , "HandleBinWrite => partId= %d,hbin=%d,sbin=%d\n",pParam->data.Bin.data.partId,pParam->data.Bin.data.Hbin , pParam->data.Bin.data.Sbin);
      fflush(pBinHandle);
      DeleteCommandIfNeeded(&pParam);
}

read:
void ParseFile_Param(char *fileToParse)
{
      PARAM_TO_SEND currentParam;
      int readBytes = -1;
      FILE *pInputFileHandle = NULL;
      FILE *pOutputFileHandle = NULL;
      int numOfParam = 0;
      int seekRes=0;
      int errorCode = 0;

      OpenIOFiles(fileToParse, &pInputFileHandle, &pOutputFileHandle);
      
      fprintf(pOutputFileHandle, "partId, siteId, paramUniqueId, paramPassFail, paramValue\n");

      do
      {
            seekRes = fseek(pInputFileHandle,numOfParam*sizeof(PARAM_TO_SEND),SEEK_SET);
            readBytes = fread(&currentParam , sizeof(PARAM_TO_SEND) , 1  , pInputFileHandle);
            if(readBytes > 0)
            {
                  fprintf(pOutputFileHandle, "%d, %d, %d, %d, %lf\n",
                                     currentParam.partId,
                                     currentParam.siteId,
                                     currentParam.paramUniqueId,
                                     currentParam.paramPassFail,
                                     currentParam.paramValue);
                  memset(&currentParam , 0 , sizeof(PARAM_TO_SEND));

                  numOfParam++;
            }
            else
            {
                  errorCode = ferror(pOutputFileHandle);
                  printf( "ParseFile_Param - Failed to Write. Error=%d!\n", errorCode);             
            }
      }
      //while( !feof(pOutputFileHandle));
      while(readBytes > 0);

      fclose(pInputFileHandle);
      fclose(pOutputFileHandle);
}
0
 

Author Comment

by:optimaltest
Comment Utility
Sorry posted the wrong write code. Here it is:

void HandleParamWrite(SUPPORTED_COMMANDS *pParam)
{      
      if(pParam == NULL)
      {
            return;
      }
      // This param is already stored in the CSV (resend)
      if( (pParam->data.Param.managementFlag & IS_RESEND) != 0)
      {
            return;
      }
      if(pParamHandle == NULL)
      {
            sprintf(paramFileName , "%s\\param" , logFolderPath);
            pParamHandle = fopen( paramFileName , "wb");
            if(pParamHandle == NULL)
            {
                  printf("Error\n!!!");
            }
      }
      fwrite(&pParam->data.Param.data , sizeof(PARAM_TO_SEND), 1, pParamHandle);
      printf("partId=%d, siteId = %d, paramValue=%lf\n",pParam->data.Param.data.partId, pParam->data.Param.data.siteId, pParam->data.Param.data.paramValue);
      paramWrittenCnt++;
      //fprintf( pParamHandle , "HandleParamWrite => ParatId=%d,Name=%s,Val=%lf,paramNumber=%d\n",pParam->data.Param.data.partId,pParam->data.Param.data.paramTestName , pParam->data.Param.data.paramValue , pParam->data.Param.data.paramNumber);
      fflush(pParamHandle);
      DeleteCommandIfNeeded(&pParam);
}
0
 
LVL 53

Expert Comment

by:Infinity08
Comment Utility
>> Here is the whole code.

That's not the whole code. Those are just two functions.

Can you also answer my other questions ?
0
 
LVL 53

Expert Comment

by:Infinity08
Comment Utility
Btw, where are you closing the output file in the write code ?
0
 

Author Comment

by:optimaltest
Comment Utility
I agree, no need for the fseek.
I've added it to try to see if it would solve the case.
The compiler has a lot of options. which options (in VS2005) should I check for?
Also, I haven't confirmed it but I believe the problem exists in VS2005 but doesn't exist on Linux.
typedef struct _PARAM_TO_SEND

{		

	int partId;

	int siteId;

	int paramUniqueId;

	int paramPassFail;

	double paramValue;

}PARAM_TO_SEND;

Open in new window

0
 
LVL 53

Expert Comment

by:Infinity08
Comment Utility
>> The compiler has a lot of options. which options (in VS2005) should I check for?

Any options that might impact struct padding, and struct member alignment.


Now, BIN_TO_SEND :

>> typedef struct _BIN_TO_SEND
>> {      
>>         int x;
>>         int y;
>>         char SbinDesc[SBIN_DESC];
>> }BIN_TO_SEND;

is not the same as PARAM_TO_SEND :

>> typedef struct _PARAM_TO_SEND
>> {              
>>         int partId;
>>         int siteId;
>>         int paramUniqueId;
>>         int paramPassFail;
>>         double paramValue;
>> }PARAM_TO_SEND;

You're writing the first, and reading the second ... It's not surprising that you see odd behavior.
0
 
LVL 53

Expert Comment

by:Infinity08
Comment Utility
There are also still several questions that remain unanswered. I'll summarize them here so you know which ones :

>> Your code doesn't check feof.

>> Could you post the whole code ?

>> Can you also confirm that this code is compiled with the same compiler (and the same compiler settings), and run on the same machine ?

>> Btw, where are you closing the output file in the write code ?
0
 
LVL 7

Expert Comment

by:HalfAsleep
Comment Utility
Compiler options should not be an issue, as long as you have the same options for both read and write, compiled with same compiler, for the same target.

When the code works, have you checked the read with the debugger, and made sure that you have read the exact number of bytes that you would expect, and that the bytes contain the correct data?

The write and read has to use the exact same size.  And it is clear by now, that you are not using the same struct, or data definition on both sides.  On the write side, you are using BIN_TO_SEND, while on the read side you are using PARAM_TO_SEND.  PARAM_TO_SEND is a permanent size, while we cannot determine the size of BIN_TO_SEND, until you provide us the size of SBIN_DESC.
0
How to improve team productivity

Quip adds documents, spreadsheets, and tasklists to your Slack experience
- Elevate ideas to Quip docs
- Share Quip docs in Slack
- Get notified of changes to your docs
- Available on iOS/Android/Desktop/Web
- Online/Offline

 

Author Comment

by:optimaltest
Comment Utility
There is no close command for it.
I tried closing it and it didn't help
0
 
LVL 53

Expert Comment

by:Infinity08
Comment Utility
>> Compiler options should not be an issue, as long as you have the same options for both read and write, compiled with same compiler, for the same target.

Which was exactly what I was asking about ;)


>> There is no close command for it.
>> I tried closing it and it didn't help

What do you mean ? Just use fclose like you did in the read code.
0
 
LVL 7

Expert Comment

by:HalfAsleep
Comment Utility
I think maybe one of the issues here is that you are trying to write one particuar data struct, and the read something else.  Usually, when you write something binary, you have to be absolutely specific about how to read it as well.  The best way to do that, is to read the same struct typedef that you used for writing.

In other words, can you explain why you are using one struct typedef for write and a different for read, instead of using just one struct typedef for both read and write?
0
 

Author Comment

by:optimaltest
Comment Utility
>> Your code doesn't check feof.
feof returns 1 when readBytes=0

>> Could you post the whole code ?
I can't post the whole project. I'll see which parts of it I can post

>> Can you also confirm that this code is compiled with the same compiler (and the same compiler settings), and run on the same machine ?
same compiler, same machine. Regarding the compiler options, it was defined in a similar way but I can't guarantee they are all the same

>> Btw, where are you closing the output file in the write code ?
It's not closed properly.
I tried to add an fclose() command in the end, but it didn't make any difference

>> In other words, can you explain why you are using one struct typedef for write and a different for read, instead of using just one struct typedef for both read and write?
That was due to an error in my post, not the code. the write and read both use PARAM_TO_SEND
0
 
LVL 53

Expert Comment

by:Infinity08
Comment Utility
>> >> Your code doesn't check feof.
>> feof returns 1 when readBytes=0

That means that the end of the file has been reached while trying to read the requested data from the file. Are you sure that there's supposed to be more data to read ?


>> same compiler, same machine. Regarding the compiler options, it was defined in a similar way but I can't guarantee they are all the same

Then that should be ok.


>> It's not closed properly.
>> I tried to add an fclose() command in the end, but it didn't make any difference

You'll need it anyway.


>> That was due to an error in my post, not the code. the write and read both use PARAM_TO_SEND

Do they also use the exact same definition of the struct ?
Did you verify that sizeof(PARAM_TO_SEND) returns the same value for both reader and writer ?
0
 
LVL 53

Expert Comment

by:Infinity08
Comment Utility
Btw, if the code is too big, you could write a representative (compilable) smaller program that exhibits the same behavior.

Doing that (ie. trimming the code down to only the part that causes/shows the problem) will also make it easier for you to track down what the problem is.
0
 
LVL 7

Expert Comment

by:HalfAsleep
Comment Utility
As Infinity suggests, are you sure that you are not done when you get eof?  Maybe the eof is actually valid, ie, you have written 3 times, but try to read 4 times.
0
 

Author Comment

by:optimaltest
Comment Utility
I will write a smaller program and see how it goes. Probably not today though.
Thanks for your help, I will let you know how it works out.
0
 
LVL 53

Expert Comment

by:Infinity08
Comment Utility
For now, did you look into the eof issue ? That's where the problem is : the reader code encounters the end of the file ,while you think there should be more data in the file. Are you sure there should be more data ? Why ?
0
 

Author Comment

by:optimaltest
Comment Utility
>>For now, did you look into the eof issue ? That's where the problem is : the reader code encounters the end of the file ,while you think there should be more data in the file. Are you sure there should be more data ? Why ?
I'm sure because I can read data written after the EOF from the file using fseek() to the next record .
0
 
LVL 7

Expert Comment

by:HalfAsleep
Comment Utility
Are you running write and read concurrently in 2 different threads, or is the writer always running before the reader?

And can you show how you use fseek to "get past the eof"?  Are you checking the file position after an eof, and are you able to fseek past this point?
0
 
LVL 53

Expert Comment

by:Infinity08
Comment Utility
>> I'm sure because I can read data written after the EOF from the file using fseek() to the next record .

That can't be right. If you have reached the end of the file, then there's no more data to read ... you can also not fseek beyond the end of the file.

Maybe you didn't check whether fseek returned an error ? And when you thought that fseek went "past the end of the file", it really didn't, and the get pointer just stayed where it was before the fseek.

In any case, what you're describing is not possible, so it's likely an error on your side :)

Now would be a good time to see the code heh ... Or at least a representative sample of it that has the same problem.
0
 

Author Comment

by:optimaltest
Comment Utility
>Are you running write and read concurrently in 2 different threads, or is the writer always running before the reader?

I'm waiting for the write to finish and the program to exit then running the read.
I can run the read several times for the same result.
When writing I write on several files concurrently. I've posted 2 functions - HandleParamWrite() and HandleBinWrite() that write concurrently to different files.

>And can you show how you use fseek to "get past the eof"?  Are you checking the file position after an eof, and are you able to fseek past this point?

In the debugger I increment numOfParam and return to the fseek.
0
 
LVL 7

Expert Comment

by:HalfAsleep
Comment Utility
You cannot incremnet that variable and expect to read more in the file. fseek will never allow you to go beyond the eof position. I think you must be reading from an already read part of the file. do you have unique data in each struct, so that you can see if you read something twice? In any regard, the eof is probably valid, and you cannot fseek past the eof, just by incrementing that variable when debugging. Some more code would help alot I think
0
 
LVL 53

Expert Comment

by:Infinity08
Comment Utility
Any progress on that code ?
0
 

Author Comment

by:optimaltest
Comment Utility
I rewrote the code in one read and one write function.
No matter how i write it, it stops reading after 25 records.
If I take the same output file and use only the reader function on Linux, it reads the whole file.

//WriterThread.c

#include <stdio.h>

#include <stdlib.h>

#include <sys/stat.h>

char paramFileName[512];

FILE *pParamHandle = NULL;
 
 
 

////////////////////////////////////////////////////////////////////////////////////////////

// Structures

////////////////////////////////////////////////////////////////////////////////////////////
 

typedef struct _PARAM_TO_SEND

{		

	int partId;

	int siteId;

	int paramUniqueId;

	int paramPassFail;

	double paramValue;

}PARAM_TO_SEND;
 

PARAM_TO_SEND param;

PARAM_TO_SEND currentParam;
 

void temp_writer()

{	

	if(pParamHandle == NULL)

	{

        sprintf(paramFileName , "param_short" );

		pParamHandle = fopen( paramFileName , "wb");

		if(pParamHandle == NULL)

		{

			printf("Error\n!!!");

		}

	}

	fwrite( &param, sizeof(PARAM_TO_SEND), 1, pParamHandle);

	

	//fprintf( pParamHandle , "HandleParamWrite => ParatId=%d,Name=%s,Val=%lf,paramNumber=%d\n",pParam->data.Param.data.partId,pParam->data.Param.data.paramTestName , pParam->data.Param.data.paramValue , pParam->data.Param.data.paramNumber);

	fflush(pParamHandle);

	
 

}

void temp_reader()

{

	int readBytes=0;

	int currentParam_Len;

	int errorCode=0;
 

	currentParam_Len = sizeof(PARAM_TO_SEND);

	readBytes = fread(&currentParam , currentParam_Len , 1  , pParamHandle);

	errorCode = ferror(pParamHandle);

	if(feof(pParamHandle))

		printf("reached end of file\n");

	if(errorCode != 0)

		printf("error=%d\n", errorCode);
 

	if(readBytes > 0)

	{

		printf( "%d, %d, %d, %d, %lf\n",

					 currentParam.partId, 

					 currentParam.siteId, 

					 currentParam.paramUniqueId, 

					 currentParam.paramPassFail, 

					 currentParam.paramValue);

		//memset(&currentParam , 0 , sizeof(PARAM_TO_SEND));

	}

	else

	{

		printf( "ParseFile_Param - Failed to Write\n"); 		

	}

}
 

int main()

{

	int i=0;
 
 

	param.partId = 0;

	param.siteId = 0;

	param.paramPassFail = 0;

	param.paramValue = 100.234;

	for( ; i<100;i++)

	{

		param.paramUniqueId = i;

		temp_writer();

	}

	fclose(pParamHandle);

	

	pParamHandle = fopen( paramFileName , "r");

	for( i=0 ; i<100 ; i++ )

	{

		temp_reader();

	}

	fclose(pParamHandle);
 

}

Open in new window

reader.JPG
0
 

Accepted Solution

by:
optimaltest earned 0 total points
Comment Utility
Issue is solved.
the fwrite was "wb" and the fread was "r".
when I change the fread to "rb" it worked perfectly.
0
 
LVL 7

Expert Comment

by:HalfAsleep
Comment Utility
Heh.  Just goes to show that sometimes, it is the small things that throw you completely off!  Good job!
0
 
LVL 53

Expert Comment

by:Infinity08
Comment Utility
I'm glad you found it :)
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

Summary: This tutorial covers some basics of pointer, pointer arithmetic and function pointer. What is a pointer: A pointer is a variable which holds an address. This address might be address of another variable/address of devices/address of fu…
This is a short and sweet, but (hopefully) to the point article. There seems to be some fundamental misunderstanding about the function prototype for the "main" function in C and C++, more specifically what type this function should return. I see so…
Video by: Grant
The goal of this video is to provide viewers with basic examples to understand and use while-loops in the C programming language.
The goal of this video is to provide viewers with basic examples to understand opening and reading files in the C programming language.

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now