Solved

Boost Serialization/Deserialization techniques?

Posted on 2010-09-01
6
1,339 Views
Last Modified: 2012-05-10
I want to use the boost serialization library to serialize data from a network source to a binary archive.  The data is heterogenous and I have wrapper classes that will take care of any special processing needed to serialize and deserialize.  I've created a test case and it seems to write the data correctly but deserialization returns garbage after the second iteration. I have attempted to just open the file stream and recreate the wrapper class object.  The first pass seems to work fine and the data looks good.  After the second iteration I receive garbage back from the boost library.  So I feel like I am not using this library correctly.  Are there any suggestions on how to deserialize a large amount of heterogenous data using boost?  The idea is to serialize a series of these wrapper classes depending on what is received from the network and then recreate this data at another time.  I have provided the code to the wrapper class and the deserialization code.
class CReflectAttributeValuesWrapper{

	friend class boost::serialization::access;

	template<class Archive>
	void save(Archive & ar, const unsigned int version) const
	{
		//save attribute handle value pair set
		RTI::ULong ulLengthBytes = 0;
		AttributeHandle hAttribute;
		char* pszValue = 0;
		stringstream ss;
		RTI::ULong size = m_pAttributes->size();
		static int count = 0;

		ss<<"Object Count: "<<++count<<endl;

		ar<<count;

		ss<<"Object handle: "<<m_hObjectHandle<<endl;

		ar << m_hObjectHandle;

		ss<<"attribute set size is "<<size<<endl;

		ar << size;

		for(unsigned long i = 0; i < size; i++)
		{			
			ulLengthBytes = m_pAttributes->getValueLength(i);
			pszValue = new char[ulLengthBytes];
			m_pAttributes->getValue(i,pszValue,ulLengthBytes);

			hAttribute = m_pAttributes->getHandle ((RTI::ULong)i);

			ss<<"serializing attribute handle whose value is "<<(unsigned long)hAttribute;
			ss<<" whose length is "<<(unsigned long)ulLengthBytes<<endl;
			ss<<"serializing attribute ";

			ss<<endl;

			ar << hAttribute;
			ar << ulLengthBytes;
			ar << boost::serialization::make_array(pszValue,ulLengthBytes);
		}

.....



		ss<<"tag is "<<m_sTheTag<<endl;

		CDiagnosticLogfile::writeMessage(CDiagnosticLogfile::DEBUG,ss.str(),__FILE__,__LINE__);
		
		ar<<m_sTheTag;
	}

	template<class Archive>
	void load(Archive & ar, const unsigned int version)
	{
		RTI::AttributeHandleValuePairSet* pAttributes;
		RTI::ULong ulLengthBytes = 0;
		AttributeHandle hAttribute;
		stringstream ss;		
		int objNum = -1;

		RTI::ULong size;

		ar >> objNum;

		ss<<"Object Number "<<objNum<<endl;

		ar >> m_hObjectHandle;

		ss<<"Object handle: "<<m_hObjectHandle<<endl;

		ar >> size;

		ss<<"attribute set size is "<<size<<endl;

		pAttributes = RTI::AttributeSetFactory::create (size);
		
		cout<<ss.str();

		ss.str("");

		for(unsigned long i = 0; i < size; i++)
		{
			ar >> hAttribute;
			ss<<"deserializing attribute handle whose value is "<<(unsigned long)hAttribute;

			ar >> ulLengthBytes;
			ss<<" whose length is "<<(unsigned long)ulLengthBytes<<endl;

			char* pszValue = new char[ulLengthBytes];

			ar >> boost::serialization::make_array(pszValue,ulLengthBytes);

			ss<<endl;
			pAttributes->add (hAttribute, pszValue,ulLengthBytes);
			cout<<ss.str();

			ss.str("");
		}

		ar>>m_sTheTag;

		ss<<"tag is "<<m_sTheTag<<endl;

        cout<<ss.str();
		
	}
	BOOST_SERIALIZATION_SPLIT_MEMBER()

public:
	CReflectAttributeValuesWrapper(){}
	~CReflectAttributeValuesWrapper(){}

	void setAttributeValues(const RTI::AttributeHandleValuePairSet& attributes)
	{
		m_pAttributes = const_cast <RTI::AttributeHandleValuePairSet*> (&attributes);
	}
	
	void setObjectHandle(RTI::ObjectHandle handle){m_hObjectHandle = handle;}
	void setTag(const char* tag){m_sTheTag = const_cast<char*>(tag);}

	RTI::AttributeHandleValuePairSet* getAttributeValues(){return m_pAttributes;}
	RTI::ObjectHandle getObjectHandle(){return m_hObjectHandle;}
	string getTag(){return m_sTheTag;}

private:

	RTI::AttributeHandleValuePairSet* m_pAttributes;
	RTI::ObjectHandle m_hObjectHandle;
	string m_sTheTag;

};

/********  this is in another executable file *********/
std::ifstream ifs("HLAData.bin",ios::binary);
boost::archive::binary_iarchive ia(ifs);

while(!ifs.eof())
{
	CReflectAttributeValuesWrapper ravw;

	ia>>ravw;	
}

Open in new window

0
Comment
Question by:shootah
  • 3
  • 3
6 Comments
 
LVL 53

Expert Comment

by:Infinity08
ID: 33584352
What code was used to serialize the data ?
0
 

Author Comment

by:shootah
ID: 33586654
I've attached the serialize code.
void reflectAttributeValues (
    RTI::ObjectHandle                 theObject,    
    const RTI::AttributeHandleValuePairSet& theAttributes, 
    const char                             *theTag) 
{    
/* 	m_ofStream is declared as a member variable and intialized in constructor like this:
	m_ofStream.open("HLAData.bin",ios::binary);
*/
	boost::archive::binary_oarchive oa(m_ofStream);

	CReflectAttributeValuesWrapper data;

	data.setAttributeValues(theAttributes);
	data.setObjectHandle(theObject);
	data.setTag(theTag);

	oa<<data;
	
	
	/********/
	//irrelevant code below
}

Open in new window

0
 
LVL 53

Expert Comment

by:Infinity08
ID: 33587312
>> I've attached the serialize code.

That only places one object in the serialized file.

But when you're getting the data from the file again, you try to get more than one.
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 

Author Comment

by:shootah
ID: 33587751
That function (ReflectAttributeValues) is a callback function that is called with a high frequency.  So as far as I can tell, each time it is called, an object is placed in the serialized file.  I have a version of this code that writes to a text file, and just from eyeing the file, it looks like it is placing the data in sequence in the file (The file gets to about 100k after a few minutes).
Another thing is on the second iteration, the first value that is written to the first variable during deserialization is "22", which is the number in what looks like header information for each entry.

Attached is a portion of the txt version of the serialized file. 11534337 is a valid handle, 3 is the number of attributes, 186 is the "attribute handle", 4 is the size of the attribute in the attached file.  All of this works fine on the first pass.  When I call the insertion operator again, I get "22" for the first variable (the first number in the header for the next sequence).
22 serialization::archive 7 0 0 11534337 3 186 4 0 0 0 0 158 8 -60 125 -73 60 88 -43 0 0 162 176 63 -31 -126 -35 34 -90 -110 47 -65 -7 -30 -45 -44 -22 -89 -19 -64 30 -117 -17 -85 96 0 0 64 46 21 -32 103 6 58 79 63 -68 17 60 17 3 43 -19 63 -11 67 113 -7 -118 -82 -90 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 63 -33 65 -123 -66 -76 -104 108 -65 -21 -29 112 -109 -97 76 -45 -65 -103 52 -66 96 -80 -57 -32 -65 -94 -51 -96 0 -6 -32 -95 63 -16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 63 -16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 -60 125 -73 60 88 -43 0 0 13 Tag_Name<00>
22 serialization::archive 7 0 0 11534338 3 186 4 0 0 0 0 158 8 -60 125 -73 60 -116 -117 64 0 162 176 63 -31 -126 -104 24 118 -29 21 -65 -7 -30 -30 -69 -89 16 10 -64 30 -105 85 90 0 0 0 64 45 61 36 -1 30 63 -94 63 -69 50 -119 -91 -27 -81 46 63 -12 -85 126 70 -76 -5 -19 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 63 -33 66 95 -95 104 -100 -31 -65 -21 -29 58 -98 -119 -19 -6 -65 -103 40 98 4 80 76 26 -65 -94 -57 60 71 -90 -46 -125 63 -16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 63 -16 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 -60 125 -73 60 -116 -117 64 0 13 Tag_name<00>

Open in new window

0
 
LVL 53

Accepted Solution

by:
Infinity08 earned 500 total points
ID: 33588473
>> So as far as I can tell, each time it is called, an object is placed in the serialized file

You're still creating a new boost::archive::binary_oarchive for every object, which you don't do when reading the file.

You should probably align both the serializing and de-serializing loops, to both use the same approach (either one archive instance defined outside of the loop, or an archive instance for each iteration).
0
 

Author Comment

by:shootah
ID: 33589020
Wow.  That sure was a lot of typing on my part for such a simple answer.  Thanks for the help.
0

Featured Post

Better Security Awareness With Threat Intelligence

See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

Join & Write a Comment

Suggested Solutions

Written by John Humphreys C++ Threading and the POSIX Library This article will cover the basic information that you need to know in order to make use of the POSIX threading library available for C and C++ on UNIX and most Linux systems.   [s…
  Included as part of the C++ Standard Template Library (STL) is a collection of generic containers. Each of these containers serves a different purpose and has different pros and cons. It is often difficult to decide which container to use and …
The goal of the tutorial is to teach the user how to use functions in C++. The video will cover how to define functions, how to call functions and how to create functions prototypes. Microsoft Visual C++ 2010 Express will be used as a text editor an…
The goal of the video will be to teach the user the concept of local variables and scope. An example of a locally defined variable will be given as well as an explanation of what scope is in C++. The local variable and concept of scope will be relat…

759 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

17 Experts available now in Live!

Get 1:1 Help Now