Solved

ostringstream preallocate

Posted on 2001-07-31
9
625 Views
Last Modified: 2008-02-01
I'm currently working on Pandromeda's MojoWorld project.

We're using Microsoft VC++6.0.
Within MojoWorld, we use ostreams to save the state of our planet.  This works fine for saving to ofstreams.
(When saving a planet to a file).
We also have a log, which users can save states of planets while they're changing them.  So to do this we use (binary) ostringstreams, and so just store this log of states as a vector or strings.

This works fine for small files ( < 1meg) but when we have meshes in our planes, file sizes can quickly grow to >4 megs.  As a result the ostringstream grinds to a very (extremly) slow crawl. ( around >~1.5megs ).
I suspect this is because we havn't preallocated memory and the memory allocation scheme in the stings is slow for >1.5megs.  

How can I preallocate memory for a ostringstream??  I've tried preallocating a string and then passing it in the constructor for ostringstream, but it seems to just clear it and start from scratch again!  

I want to avoid having to create my own allocator object as this seems like it should be a simple operation.....

HELP!!!
Thanks heaps.

Brian Sharpe.
Pandromeda.
0
Comment
Question by:briansharpe
  • 3
  • 3
  • 3
9 Comments
 
LVL 22

Expert Comment

by:nietod
ID: 6338958
>> I suspect this is because we havn't preallocated memory
>> and the memory allocation scheme in the stings
>> is slow for >1.5megs.  
yes, each time the string runs out of room, the string needs to expand and this requires a new allocation and then a copy of all the existing data.  So this gets slow, but tehoretically it shouldn't be that bad.   STL strings are supposed to grown in amortized constant time, which means that on average the large strings should not take longer to grow than the small strings.  This is accomplsihed by making the additional space allocated proportional to the space already used.  i.e. like when the string runs out of room always double the size.  This means that while a long string takes longer to grown, it needs to grow much less often than a small string, son avereage a long string is no slower than a short one.  

So this might be a factor, but it should not be a huge factor.  It think a bigger problem is likely to be memory swapping and the fact that you are allocating such huge chunks of memory.

Can you write your code in a stream-genral way.  i..e make your code output to a ostream of any type, not just a string stream.   The advantage of this is that you can try other strings types and see if you get better results.  For, example a fle stream might be much faster for this case.   Or you might try using a stream class of your own devising.  For example, you might write a stringstream-like class that uses a rope for storage rather than a string.  (Ropes are oftem much more efficient than strings for long text sequences.)
0
 
LVL 30

Expert Comment

by:Axter
ID: 6338977
FYI,
For more information on rope class, check out the following link:
http://www.sgi.com/tech/stl/Rope.html
0
 

Author Comment

by:briansharpe
ID: 6343241
Yea!!! Ropes!!!
I looked into them but they seem very tightly
bound to SGI's STL.  We're using Microsofts MCV++ STL
implementation.  Its tool late in the project for
us to think of changing that.

Has anybody ever tried implementing a rope in MCV++???
I've been looking but all I can see is the SGI one.  And people that have used it with MCV++ use the whole SGISTL rather than the MVC++ one.

I've looked harder and I also tried calling the
pubsetbuf method to set a huge buffer of about 1meg for the stringstream, but this still had little effect on its speed.  

Thanks heaps you guys.
Brian Sharpe.
0
 
LVL 30

Accepted Solution

by:
Axter earned 75 total points
ID: 6343433
>>I looked into them but they seem very tightly
>>bound to SGI's STL.  
I was experimenting with the hash_map class that is also tightly bound to SGI's STL, and I had the same problem.

I wanted the rest of my code to use the MSVC++ STL, so what I did was change the namespace for the SGI's STL.
It took me about 30 minutes of work making all the modifications, but it worked.  I was able to use the hash_map from the SGI's STL, and still able to have the rest of my code use the MSVC++ STL.
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 
LVL 22

Expert Comment

by:nietod
ID: 6344309
>> ts tool late in the project for
>> us to think of changing that.
If that's true, its probably too late for any solution.

But I doubt that it is too later.   How often does your code rely on the fact fact that the stream object is a string stream object.   usually this is relied on only in a small part of the code.  Usually the function that creates the string stream object, because this is usually the only palce that the string is extracted.    Most other portions of the code probalby treat the stream object as a general stream object, that is, they make no use of string-stream-specific behavior.  if so, Then "fixing" these portions is fast and easy.  You just change the decalred type to "ostream" from "ostringstreamn".

>> nd people that have used it with MCV++ use the
>> whole SGISTL rather than the MVC++ one.
You woudln't have to use the whole SGISTL.  but you will probalby have to edit the rope template to make it work.  I would not recommend that you switch to a different STL.  VC's STL is well tuned for VC.
0
 

Author Comment

by:briansharpe
ID: 6346755
Cool! Cool! namespaces!!!

We are writing to a generic &osteam interface already.  hense the filesaving uses the same routine and works fine.  but when we use it to save to a string, we use the stringstream object.  So we are doing it in a generic way, hense the shock when I found the stringstream was mind-numbingly slow for ~ >1meg.

Changing the namespace!  Wahhooo.  I'm going to try that!

At the mo, we write all the data to a tempfile, then suck it back up into memory and initialize the string in one big chunk which has made the process of saving a 4meg file to a string from about 10mins(stringstream)(actually, usually just hangs) to ~5seconds(fstream).  
It works fine but is far from elegant.  I'd rather avoid touching the filesystem for something that is primarily memorybased.

I'm going to give the namespace rope thing a go.
Axter, Thanks heaps for the suggestion about the ropes etc...  

Later you guys.
Brian.
0
 

Author Comment

by:briansharpe
ID: 6346767
Really good common sense answer.
Understood where I was comming from and gave me a really good answer.

Don't get me wrong, the other people also gave really good answers, but ones which didn't help me.  I found this line of thinking cool.

Later
Brian.
0
 
LVL 22

Expert Comment

by:nietod
ID: 6347069
So the solution was to use ropes?   I wish I had suggested that.
0
 
LVL 30

Expert Comment

by:Axter
ID: 6350538
briansharpe,
Thanks for the points.

FYI,
Are you aware that you can split the points to a question.
Some questioners do this if they feel more then one expert contributed to the answer.

You can split the points in half, or you can split the points to a percentage that represents what you think each expert deserves.

Good luck on your project. :-)
0

Featured Post

What Should I Do With This Threat Intelligence?

Are you wondering if you actually need threat intelligence? The answer is yes. We explain the basics for creating useful threat intelligence.

Join & Write a Comment

Suggested Solutions

Written by John Humphreys C++ Threading and the POSIX Library This article will cover the basic information that you need to know in order to make use of the POSIX threading library available for C and C++ on UNIX and most Linux systems.   [s…
This article shows you how to optimize memory allocations in C++ using placement new. Applicable especially to usecases dealing with creation of large number of objects. A brief on problem: Lets take example problem for simplicity: - I have a G…
The goal of the tutorial is to teach the user how to use functions in C++. The video will cover how to define functions, how to call functions and how to create functions prototypes. Microsoft Visual C++ 2010 Express will be used as a text editor an…
The viewer will be introduced to the member functions push_back and pop_back of the vector class. The video will teach the difference between the two as well as how to use each one along with its functionality.

743 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now