ostringstream preallocate

I'm currently working on Pandromeda's MojoWorld project.

We're using Microsoft VC++6.0.
Within MojoWorld, we use ostreams to save the state of our planet.  This works fine for saving to ofstreams.
(When saving a planet to a file).
We also have a log, which users can save states of planets while they're changing them.  So to do this we use (binary) ostringstreams, and so just store this log of states as a vector or strings.

This works fine for small files ( < 1meg) but when we have meshes in our planes, file sizes can quickly grow to >4 megs.  As a result the ostringstream grinds to a very (extremly) slow crawl. ( around >~1.5megs ).
I suspect this is because we havn't preallocated memory and the memory allocation scheme in the stings is slow for >1.5megs.  

How can I preallocate memory for a ostringstream??  I've tried preallocating a string and then passing it in the constructor for ostringstream, but it seems to just clear it and start from scratch again!  

I want to avoid having to create my own allocator object as this seems like it should be a simple operation.....

Thanks heaps.

Brian Sharpe.
Who is Participating?
AxterConnect With a Mentor Commented:
>>I looked into them but they seem very tightly
>>bound to SGI's STL.  
I was experimenting with the hash_map class that is also tightly bound to SGI's STL, and I had the same problem.

I wanted the rest of my code to use the MSVC++ STL, so what I did was change the namespace for the SGI's STL.
It took me about 30 minutes of work making all the modifications, but it worked.  I was able to use the hash_map from the SGI's STL, and still able to have the rest of my code use the MSVC++ STL.
>> I suspect this is because we havn't preallocated memory
>> and the memory allocation scheme in the stings
>> is slow for >1.5megs.  
yes, each time the string runs out of room, the string needs to expand and this requires a new allocation and then a copy of all the existing data.  So this gets slow, but tehoretically it shouldn't be that bad.   STL strings are supposed to grown in amortized constant time, which means that on average the large strings should not take longer to grow than the small strings.  This is accomplsihed by making the additional space allocated proportional to the space already used.  i.e. like when the string runs out of room always double the size.  This means that while a long string takes longer to grown, it needs to grow much less often than a small string, son avereage a long string is no slower than a short one.  

So this might be a factor, but it should not be a huge factor.  It think a bigger problem is likely to be memory swapping and the fact that you are allocating such huge chunks of memory.

Can you write your code in a stream-genral way.  i..e make your code output to a ostream of any type, not just a string stream.   The advantage of this is that you can try other strings types and see if you get better results.  For, example a fle stream might be much faster for this case.   Or you might try using a stream class of your own devising.  For example, you might write a stringstream-like class that uses a rope for storage rather than a string.  (Ropes are oftem much more efficient than strings for long text sequences.)
For more information on rope class, check out the following link:
The 14th Annual Expert Award Winners

The results are in! Meet the top members of our 2017 Expert Awards. Congratulations to all who qualified!

briansharpeAuthor Commented:
Yea!!! Ropes!!!
I looked into them but they seem very tightly
bound to SGI's STL.  We're using Microsofts MCV++ STL
implementation.  Its tool late in the project for
us to think of changing that.

Has anybody ever tried implementing a rope in MCV++???
I've been looking but all I can see is the SGI one.  And people that have used it with MCV++ use the whole SGISTL rather than the MVC++ one.

I've looked harder and I also tried calling the
pubsetbuf method to set a huge buffer of about 1meg for the stringstream, but this still had little effect on its speed.  

Thanks heaps you guys.
Brian Sharpe.
>> ts tool late in the project for
>> us to think of changing that.
If that's true, its probably too late for any solution.

But I doubt that it is too later.   How often does your code rely on the fact fact that the stream object is a string stream object.   usually this is relied on only in a small part of the code.  Usually the function that creates the string stream object, because this is usually the only palce that the string is extracted.    Most other portions of the code probalby treat the stream object as a general stream object, that is, they make no use of string-stream-specific behavior.  if so, Then "fixing" these portions is fast and easy.  You just change the decalred type to "ostream" from "ostringstreamn".

>> nd people that have used it with MCV++ use the
>> whole SGISTL rather than the MVC++ one.
You woudln't have to use the whole SGISTL.  but you will probalby have to edit the rope template to make it work.  I would not recommend that you switch to a different STL.  VC's STL is well tuned for VC.
briansharpeAuthor Commented:
Cool! Cool! namespaces!!!

We are writing to a generic &osteam interface already.  hense the filesaving uses the same routine and works fine.  but when we use it to save to a string, we use the stringstream object.  So we are doing it in a generic way, hense the shock when I found the stringstream was mind-numbingly slow for ~ >1meg.

Changing the namespace!  Wahhooo.  I'm going to try that!

At the mo, we write all the data to a tempfile, then suck it back up into memory and initialize the string in one big chunk which has made the process of saving a 4meg file to a string from about 10mins(stringstream)(actually, usually just hangs) to ~5seconds(fstream).  
It works fine but is far from elegant.  I'd rather avoid touching the filesystem for something that is primarily memorybased.

I'm going to give the namespace rope thing a go.
Axter, Thanks heaps for the suggestion about the ropes etc...  

Later you guys.
briansharpeAuthor Commented:
Really good common sense answer.
Understood where I was comming from and gave me a really good answer.

Don't get me wrong, the other people also gave really good answers, but ones which didn't help me.  I found this line of thinking cool.

So the solution was to use ropes?   I wish I had suggested that.
Thanks for the points.

Are you aware that you can split the points to a question.
Some questioners do this if they feel more then one expert contributed to the answer.

You can split the points in half, or you can split the points to a percentage that represents what you think each expert deserves.

Good luck on your project. :-)
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.