• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 396
  • Last Modified:

Implementing shared libraries


I have waht I think is a pretty basic question but I'm hoping I can get some details answers.

I'm running various Unix servers, including some FreeBSD implementations (3.4 version so this would mainly apply to FreeBSD but I assume it's generally the same for other Unix variants but please let me know if it isn't).

The utilities that come with Unix, such as bind, popper, ftpd, apache, sendmail, etc. always are much smaller in size than the later versions I compile and install to update the system.

I'm under the impression that what's going on here, is when I compile a newer version of, say Apache or Sendmail, by default the compilation is stand-alone, and in most cases, it could be compiled with some sort of shared library option which would dramatically reduce the size and memory overhead of the software?  In the case of Apache, where you might have 100+ daemons spawned, I've been told that if I recompiled the program to share memory/libs that it would be much more memory efficient.  How can I do this?

Specifically here are my questions:

1.  Is there a special module or set of libraries that must be installed to implement this?  Does it vary by OS/version?  Where can I get this code if I don't have it already?

2.  I'd like detailed instructions on how to compile the following programs to use these shared libraries: bind, ftpd/wuftp, popper, apache, sendmail, etc.

I'm assuming maybe it's a single extra compilation/linking command parameter?  Is it true that by default none of these systems compile in this manner?  

Any additional thoughts or comments on what I'm trying to do?  It's frustrating to replace a 300k 2.0 version with a 4320k 2.1 version of a program...

  • 6
  • 5
  • 2
  • +2
1 Solution
Whether an application is built against dynamic libraries or static libraries has everything to do with the link process. In general the configure process for most things will pick dynamic libs over a static link option unless forced to do otherwise or unless it thinks that dynamic libs won't work on some particular system. It's been an awful long time since I've used FreeBSD 3.4, but I don't think I've ever had a problem building Apache or Bind with dynamic libs on 4.0 or later.

Are you building your utilities directly from source or via the ports collection?
How can you tell?  When my Bind version is 8M and the bundled version with 3.4 is less than a MB and not much earlier, I know something is off...
It's easy to tell if something is dynamically linked or not. Just run ldd or file on the executable, i.e., 'ldd /usr/sbin/sendmail' or 'file /usr/sbin/sendmail'. Also be careful when comparing file sizes. For example, on a stock FreeBSD 4.4 system:

praetorian> ls -l /usr/sbin/sendmail
-r-xr-xr-x  1 root  wheel  4984 Oct 12 17:43 /usr/sbin/sendmail*

That's ridiculously small, let's look a bit further:

praetorian> file /usr/sbin/sendmail
/usr/sbin/sendmail: symbolic link to /usr/sbin/mailwrapper
praetorian> strings /usr/sbin/mailwrapper | grep sendmail

So the actual sendmail executable isn't what's in /usr/sbin and the real size of sendmail is:

praetorian> ls -l /usr/libexec/sendmail/sendmail
-r-sr-xr-x  1 root  wheel  415424 Oct 12 17:43 /usr/libexec/sendmail/sendmail
praetorian> ldd /usr/libexec/sendmail/sendmail
     libutil.so.3 => /usr/lib/libutil.so.3 (0x280c6000)
     libwrap.so.3 => /usr/lib/libwrap.so.3 (0x280cf000)
     libssl.so.2 => /usr/lib/libssl.so.2 (0x280d7000)
     libcrypto.so.2 => /usr/lib/libcrypto.so.2 (0x28103000)
     libsasl.so.8 => not found (0x0)
     libc.so.4 => /usr/lib/libc.so.4 (0x281ba000)

Now a 400Kb executable is lots more reasonable.

Free Tool: Site Down Detector

Helpful to verify reports of your own downtime, or to double check a downed website you are trying to access.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

maabuAuthor Commented:
Ahhh, here's what I get on apache:

# ldd httpd
        libcrypt.so.2 => /usr/lib/libcrypt.so.2 (0x280d3000)
        libc.so.3 => /usr/lib/libc.so.3 (0x280e8000)

So that looks good I believe...

# ldd named
        libc.so.3 => /usr/lib/libc.so.3 (0x281bf000)

named is still over 5MB in size.  Does that sound right for version 9.1.3?

# ldd /usr/sbin/sendmail
        libutil.so.2 => /usr/lib/libutil.so.2 (0x280d9000)
        libc.so.3 => /usr/lib/libc.so.3 (0x280e2000)

Hmmm, this looks good then?

So I assume that using Shared Libraries is just a standard thing?

One of the big things I noticed was when switching from FBSD's popper to Qualcomm's popper:

-r-xr-xr-x  1 root  wheel  - 36080 Dec 10  1999 popper

-rwxr-xr-x  1 root  wheel  - 571550 Jan 10 01:23 qpopper
A pretty dramatic increase... but a ldd reveals:

        libcrypt.so.2 => /usr/lib/libcrypt.so.2 (0x28083000)
        libc.so.3 => /usr/lib/libc.so.3 (0x28098000)
so I guess it's working... hmmmm

Why does your sendmail have so many extra libraries?  What goodies are you using?
Okay all of those are linked against dynamic libs. The size of Bind 9 sounds a bit on the big side compared to my copy on 4.4. I've got a system copy of Bind 8 that's 500Kb and a local version of Bind 9 that's 1.6Mb. That sounds about right to to since there's lots of stuff in Bind 9, like the crypto stuff for rndc & dynamic updates, that isn't in Bind 8. Your 8Mb copy sounds a bit on the large side, but I'd have to examine how it was built to determine if that's reasonable for a 3.4 system. It's possible that my 4.4 system has later versions of some routines in system libraries that Bind 9 requires. That could cause the activation of code from the Bind distribution to be built into your executable.

My sendmail copy has lots of other libraries included as it includes support for tcpwrappers, SSL encryption, and SASL authentication. Incidentally, that copy was built from the FreeBSD 4.4 sources by a tweak to the build options used by make world.
maabuAuthor Commented:

As an aside, how can I implement security in POP3 mail checking?  I couldn't find any support in the 3.4 popper and I couldn't figure out how to get qualcomm's qpopper to work either.
Are you speaking of using an encrypted POP connection? Part of that involves what the clients support. I'm a big fan of the Cyrus IMAP implementation, which includes POP, IMAPS and POPS support (the later are SSL encrypted connections).

The only disadvantage of the Cyrus implementation is that it works best when using SASL authentication rather than traditional Unix authentication. For my uses that's an advantage as I don't want ordinary users to have any access to my mail servers other than through IMAP or POP. By keeing their authentication tokens (username & password) in a SASL database rather than in passwd there's no way for a mail client user to break into the server or abuse their account because they don't have Unix accounts. Well there's always the risk of a root exploit if you don't keep the system up to date w/respect to security advisories, by I have less to worry about if my mail server only has a root account and the accounts for a couple of trusted admins.

Other advantages of Cyrus lie in it's handling of user INBOX's and its ability to apply mail quotas in a manner that is 'mail safe'. You can read more about Cyrus at http://asg.web.cmu.edu/ 
well done, jlevie, explained all things about dynamic linking.

About the size of bind/named, mine also has roughly 4.8MB.

To check what may cause the reason why some libs are linked static, while others are linked dynamic, check the result of the configure command: its output and generated files like config.status (sorry swapped out of mind how this can be checked with FreeBSD's ports).
There you should find why configure decided to use static vs. dynamic, or even it's own lib.
run strip against the binaries.  It's uncomon for out of the box executables to include the symbols table.  For a large binary this can cut out a good few hundred K.

maabuAuthor Commented:
How does strip work?

Jlevie, I think you earned the points but since we have such a cool thread going, let me ask you a tad about popper and what you think...

I am using Eudora (5.1) which supports a number of encryption mail checking options, which is why I installed Qualcomm's popper, but I can't seem to get any ssl or other encryption to work even though I have tons of crypt/ssl/openssl libs installed on my system (running apache w/mod_ssl with no troubles but could never get qpopper to use it - and qpopper is huge compared to the os's default pop3 client which is one reason that prompted me to investigate the whole shared lib issue).

I'll have to confess that it's been over 5 years since I last messed with Qualcomm's POP server. There wasn;t any SSL stuff in that version and I would have had no reason to use it then even if it had been available.

How far do you get in trying to enable SSL in popper? Will it build with the libs from a recent OpenSSL distribution? Or are you having problems with generating and/or using a certificate (assuming that you're using a self-signed certificate)?
maabuAuthor Commented:
I really don't know - I just made the version that was included.  Admittedly I'm really ticked off at how unnecessarily complicated the SSL/CA stuff is.  I know it does not need to be so complicated and I find it annoying.  Don't even get me started about the early days of integrating certs into Apache, which was a nightmare.

So basically I don't know.  All I know is I'd like to use ANY pop3 server that has some kind of encryption.. I'm open to ideas.
You have my sympathies with respect to certificates. And I felt the same way until I did enough research to more fully understand certificates and certificate authorities. Now that I understand it a bit better I'm not so sure that the process could be simplified and still meet the requirements. And yes the early days of certificates, before OpenSSL, was a royal pain. Especially since all of the certificates had to be obtained from a commercial vendor and they weren't wild about supporting anything but a Netscape or other commercial server.

Creating a local certificate authority and a self signed certificate can be made a lot easier with one of the script packages floating around on the Internet. I was less than thrilled with the way they worked so I wrote a perl script for creating self-signed certificates.

Setting up an email system for encrypted connections is a non-trivial task. Not only do you have the problem of making sure that the server (POP and SMTP) and the client share a common method, but you also need to generate a client side certificate and incorporate that into the client. None of this is documented very well and that's not so surprising considering the multitude of clients. If I were going to do an encrypted mail system I'd use the Cyrus implementation. There's a mailing list for Cyrus that is a very good resource if you run into problems.
maabuAuthor Commented:
On the client side, in Eudora for example, you can simply select the method, such as SSL.  Does a certificate still need to be installed on the client side?  This isn't necessary with SSL implementation in the browser, so why in email?

Is there a way to tell what support may be integrated into my POP3 servers?  Any URL to a page which more-clearly documents this process?

thanks very much for your comments - they are much appreciated!
"How does strip work?"
  strip file...

all strip does is remove extraneous info from a binary, be it executable or a plain object.  This useless info is normally just the symbols table, but will also include debug info if the file includes it.  I've seen on many occasions a 300K binary (especially gnu stuff) strip down to less than 100K.  If the code works, why carry loads of crap around with it.
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Cloud Class® Course: Python 3 Fundamentals

This course will teach participants about installing and configuring Python, syntax, importing, statements, types, strings, booleans, files, lists, tuples, comprehensions, functions, and classes.

  • 6
  • 5
  • 2
  • +2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now