Reducing static library size

Dear experts,

I've been building my static library which has approximately 160,000 lines of codes.

However, I've observed 2 things,
1) The debug library is almost 1 GB
2) The release library is 500MB+

I find these sizes very prohibiting and makes it very hard to distribute via the network.

Is there something wrong?

I'm using Visual Studio 2005 SP1. My project is a pure C/C++ static library project.
I use templates quite extensively, it takes up about 30-40% of code memory approximately (due to smart pointers, data structures, etc etc etc)

I wonder if there's any options that I can set to reduce the static library size, since moving away from templates is next to impossible.

Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Some ideas :

(a) Most compilers have an option to optimize for size. For gcc eg. that's -Os, for VC++ that's /Os
(b) As you know, heavy use of templates can cause a lot of bloat in executable size. Wherever possible, consider minimizing the use of templates.
(c) Make sure to only link in that what you actually need (especially when it comes to external libraries).
(d) Consider splitting up the huge library in several smaller more focused libraries.
(e) Wherever possible, consider linking dynamically instead of statically.
(f) if you have any data in the binary, consider moving it out of the library into separate files, and load the data from those files.
Do you need to distribute both the debug and release libraries?  The debug libraries without source or .pdb are pretty useless.
thrawn80Author Commented:
Dear infinity80 and cup,

Thanks for your suggestions! I have been comtemplating whether to split this library into its namespace-sake components as individual libraries instead.

In the past, I didn't understand the linking processing at the end well. Now, if I'm not wrong, even though the dependencies to a certain extent are built into the static library, the final executable will contain only 1 unique code of the same class signature during linking.

But what I don't understand is that - how come game engines like Ogre3D can be have like 120,000+ lines of codes (estimated with a tool) and still have a relatively small static library?

I do agree that dynamic linking is much faster in terms of development but in this case, the static library is alike an application start; the final application doesn't need to define a WinMain(); it only needs to define the 2 declared and called functions in the library, ie, Init() and Release(). The static library will call these 2 functions that's expected to be implemented by the application itself otherwise linker errors will occur.

As for data, they're always loaded from external sources unless there's really by-design, no choice but to hard-code them.

And cup, yes, I need to distribute the codes as well :)

Is there any other things to look out for?

Cloud Class® Course: C++ 11 Fundamentals

This course will introduce you to C++ 11 and teach you about syntax fundamentals.

There are pros and cons for both techniques.  With static linking, you know that that is all you will ever need.  With dynamic linking, even though it is fast for development, it is slow on startup.  Also, depending on how you name your DLLs and where you put them could affect how your program loads etc.

Under the compile options, are you building as Multithreaded or Multithreaded DLL.  The latter will produce a smaller library but it is dependent upon some MSVC DLLs which may or may not be present on the target machine so you have to ship these too.  For instance, if you're building on VS6 and you use XP or Vista, the program will just fall over if they are not present.  Also, only the non debug versions are commercially redistributable.

Not only that: sometimes it is tied to some service packs.  For instance VS2005 ships with version 42 of the library, SP1 ships with version 762.  The number of problems you get with incompatible versions of DLL are unbelievable. Far better to stick to longer delivery times than mess with this DLL manifest nightmare.

Only other thing I can suggest is not using any I/O but that would only reduce it by about 20K.
Just one more note about dynamic linking : I did not intend it to be for the entire library, but rather for certain of the dependencies.

>> Is there any other things to look out for?

All the other things I can think of, impact the code (probably too much), like disabling exceptions and such.
thrawn80Author Commented:
Dear all,

If I'm not wrong, static library merely preprocesses the source codes and then compiles them into .obj files.
The lib.exe will simply chain up the entire bunch of object files. Of course, no linking is done.

Here's the thing - if i reduce the amount of preprocessing required, will it decrease the size of the static library in the end?

A few ways to decrease preprocessing:
1) Use forward declarations if possible
2) Do not include unnecessary headers

Anymore ways?

I bet if I can reduce the size of the object files, I'll eventually reduce the size of the static library altogether.

Please let me know if I'm right.

>> The lib.exe will simply chain up the entire bunch of object files.

That's pretty much what happens, yes.

>> I bet if I can reduce the size of the object files, I'll eventually reduce the size of the static library altogether.

Sure, that's the idea :)

>> 1) Use forward declarations if possible
>> 2) Do not include unnecessary headers

That's something you should always do. It can speed up compilation times by a big factor. However, don't expect it to have a big impact on object file size (unless you have code in your header files).

It is however worth investing some time in it to make sure there are no unnecessary includes.

Finally, allow me to expand on a previous point I made, which might not have made it across the way I intended :

>> (c) Make sure to only link in that what you actually need (especially when it comes to external libraries).

This not just refers to entire external libraries, but also to the libraries themselves. You can strip out certain features that you don't need from external libraries.

An example that comes to mind is the cURL library : you can compile it with and without SSL support. If you don't need SSL, then don't build in support for it. The same is true for all features of all libraries you use : get rid of whatever you don't need.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
thrawn80Author Commented:
Thanks Infinitely08 and Cup for your generous insights!

I guess I roughly know what to do know. Since it's just the static library size, it should be relatively easy to implement.

I'll start slow i guess, by confirming which are the libraries that are really not needed, especially the Platform SDK libs.

I'm not sure if we're still allowed to make comments here though but if we can, please continue to share your insights if possible.

Again, I thank the both for your generosity :)
>> I'm not sure if we're still allowed to make comments here though

The question is still open for discussion, so anybody can feel free to post new insights :) Should you have a further doubt or insight about this, please don't hesitate to ask here.
One of the problems I've found with splitting static libraries is that they have to be added to the linker in the order of dependency.  Sometimes, the same library needs to appear more than once.
thrawn80Author Commented:
It'd seem that the root cause of my large library size is due to my poor header inclusion management. :(

Perhaps I'll take some time off to re-look into the engine's headers.

Thanks again!
Interestingly had to deal with this very problem for a different reason recently. We are working in an environment where all our code is statically linked and one of the systems it needs to run on has a limited program area, the executable compiled for other platforms without this restriction (or rather a considerably larger restriction) did not fit inside that area. Most of what we did has already been mentioned but there are some other things to consider.

- Use of macros, could these be replaced by actual function calls.
- Disabling in-lining where performance is not critical (this was the biggest single win for us). Again this can be done through compiler command switches or just in the properties window in VS.
>> It'd seem that the root cause of my large library size is due to my poor header inclusion management. :(

That's interesting, that would imply either or both of the following :

(a) you have (a lot of) code in header files.
(b) the compiler adds a lot of extra debugging/symbol information for each of those unnecessary includes

Was this change in size due to removing some includes something you noticed for the debug version or the release version ?
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Visual C++.NET

From novice to tech pro — start learning today.