Solved

Implementing shared libraries

Posted on 2002-03-14
15
374 Views
Last Modified: 2010-04-21
Hi,

I have waht I think is a pretty basic question but I'm hoping I can get some details answers.

I'm running various Unix servers, including some FreeBSD implementations (3.4 version so this would mainly apply to FreeBSD but I assume it's generally the same for other Unix variants but please let me know if it isn't).

The utilities that come with Unix, such as bind, popper, ftpd, apache, sendmail, etc. always are much smaller in size than the later versions I compile and install to update the system.

I'm under the impression that what's going on here, is when I compile a newer version of, say Apache or Sendmail, by default the compilation is stand-alone, and in most cases, it could be compiled with some sort of shared library option which would dramatically reduce the size and memory overhead of the software?  In the case of Apache, where you might have 100+ daemons spawned, I've been told that if I recompiled the program to share memory/libs that it would be much more memory efficient.  How can I do this?

Specifically here are my questions:

1.  Is there a special module or set of libraries that must be installed to implement this?  Does it vary by OS/version?  Where can I get this code if I don't have it already?

2.  I'd like detailed instructions on how to compile the following programs to use these shared libraries: bind, ftpd/wuftp, popper, apache, sendmail, etc.

I'm assuming maybe it's a single extra compilation/linking command parameter?  Is it true that by default none of these systems compile in this manner?  

Any additional thoughts or comments on what I'm trying to do?  It's frustrating to replace a 300k 2.0 version with a 4320k 2.1 version of a program...

Thanks!
0
Comment
Question by:maabu
  • 6
  • 5
  • 2
  • +2
15 Comments
 
LVL 40

Expert Comment

by:jlevie
Comment Utility
Whether an application is built against dynamic libraries or static libraries has everything to do with the link process. In general the configure process for most things will pick dynamic libs over a static link option unless forced to do otherwise or unless it thinks that dynamic libs won't work on some particular system. It's been an awful long time since I've used FreeBSD 3.4, but I don't think I've ever had a problem building Apache or Bind with dynamic libs on 4.0 or later.

Are you building your utilities directly from source or via the ports collection?
0
 
LVL 1

Expert Comment

by:wisdom042597
Comment Utility
How can you tell?  When my Bind version is 8M and the bundled version with 3.4 is less than a MB and not much earlier, I know something is off...
0
 
LVL 40

Expert Comment

by:jlevie
Comment Utility
It's easy to tell if something is dynamically linked or not. Just run ldd or file on the executable, i.e., 'ldd /usr/sbin/sendmail' or 'file /usr/sbin/sendmail'. Also be careful when comparing file sizes. For example, on a stock FreeBSD 4.4 system:

praetorian> ls -l /usr/sbin/sendmail
-r-xr-xr-x  1 root  wheel  4984 Oct 12 17:43 /usr/sbin/sendmail*

That's ridiculously small, let's look a bit further:

praetorian> file /usr/sbin/sendmail
/usr/sbin/sendmail: symbolic link to /usr/sbin/mailwrapper
praetorian> strings /usr/sbin/mailwrapper | grep sendmail
/usr/libexec/sendmail/sendmail

So the actual sendmail executable isn't what's in /usr/sbin and the real size of sendmail is:

praetorian> ls -l /usr/libexec/sendmail/sendmail
-r-sr-xr-x  1 root  wheel  415424 Oct 12 17:43 /usr/libexec/sendmail/sendmail
praetorian> ldd /usr/libexec/sendmail/sendmail
/usr/libexec/sendmail/sendmail:
     libutil.so.3 => /usr/lib/libutil.so.3 (0x280c6000)
     libwrap.so.3 => /usr/lib/libwrap.so.3 (0x280cf000)
     libssl.so.2 => /usr/lib/libssl.so.2 (0x280d7000)
     libcrypto.so.2 => /usr/lib/libcrypto.so.2 (0x28103000)
     libsasl.so.8 => not found (0x0)
     libc.so.4 => /usr/lib/libc.so.4 (0x281ba000)

Now a 400Kb executable is lots more reasonable.

0
 
LVL 1

Author Comment

by:maabu
Comment Utility
Ahhh, here's what I get on apache:

# ldd httpd
httpd:
        libcrypt.so.2 => /usr/lib/libcrypt.so.2 (0x280d3000)
        libc.so.3 => /usr/lib/libc.so.3 (0x280e8000)

So that looks good I believe...

# ldd named
named:
        libc.so.3 => /usr/lib/libc.so.3 (0x281bf000)

named is still over 5MB in size.  Does that sound right for version 9.1.3?

# ldd /usr/sbin/sendmail
/usr/sbin/sendmail:
        libutil.so.2 => /usr/lib/libutil.so.2 (0x280d9000)
        libc.so.3 => /usr/lib/libc.so.3 (0x280e2000)

Hmmm, this looks good then?

So I assume that using Shared Libraries is just a standard thing?

One of the big things I noticed was when switching from FBSD's popper to Qualcomm's popper:

-r-xr-xr-x  1 root  wheel  - 36080 Dec 10  1999 popper

-rwxr-xr-x  1 root  wheel  - 571550 Jan 10 01:23 qpopper
A pretty dramatic increase... but a ldd reveals:

qpopper:
        libcrypt.so.2 => /usr/lib/libcrypt.so.2 (0x28083000)
        libc.so.3 => /usr/lib/libc.so.3 (0x28098000)
so I guess it's working... hmmmm

Why does your sendmail have so many extra libraries?  What goodies are you using?
0
 
LVL 40

Expert Comment

by:jlevie
Comment Utility
Okay all of those are linked against dynamic libs. The size of Bind 9 sounds a bit on the big side compared to my copy on 4.4. I've got a system copy of Bind 8 that's 500Kb and a local version of Bind 9 that's 1.6Mb. That sounds about right to to since there's lots of stuff in Bind 9, like the crypto stuff for rndc & dynamic updates, that isn't in Bind 8. Your 8Mb copy sounds a bit on the large side, but I'd have to examine how it was built to determine if that's reasonable for a 3.4 system. It's possible that my 4.4 system has later versions of some routines in system libraries that Bind 9 requires. That could cause the activation of code from the Bind distribution to be built into your executable.

My sendmail copy has lots of other libraries included as it includes support for tcpwrappers, SSL encryption, and SASL authentication. Incidentally, that copy was built from the FreeBSD 4.4 sources by a tweak to the build options used by make world.
0
 
LVL 1

Author Comment

by:maabu
Comment Utility

As an aside, how can I implement security in POP3 mail checking?  I couldn't find any support in the 3.4 popper and I couldn't figure out how to get qualcomm's qpopper to work either.
0
 
LVL 40

Expert Comment

by:jlevie
Comment Utility
Are you speaking of using an encrypted POP connection? Part of that involves what the clients support. I'm a big fan of the Cyrus IMAP implementation, which includes POP, IMAPS and POPS support (the later are SSL encrypted connections).

The only disadvantage of the Cyrus implementation is that it works best when using SASL authentication rather than traditional Unix authentication. For my uses that's an advantage as I don't want ordinary users to have any access to my mail servers other than through IMAP or POP. By keeing their authentication tokens (username & password) in a SASL database rather than in passwd there's no way for a mail client user to break into the server or abuse their account because they don't have Unix accounts. Well there's always the risk of a root exploit if you don't keep the system up to date w/respect to security advisories, by I have less to worry about if my mail server only has a root account and the accounts for a couple of trusted admins.

Other advantages of Cyrus lie in it's handling of user INBOX's and its ability to apply mail quotas in a manner that is 'mail safe'. You can read more about Cyrus at http://asg.web.cmu.edu/  
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 51

Expert Comment

by:ahoffmann
Comment Utility
well done, jlevie, explained all things about dynamic linking.

About the size of bind/named, mine also has roughly 4.8MB.

To check what may cause the reason why some libs are linked static, while others are linked dynamic, check the result of the configure command: its output and generated files like config.status (sorry swapped out of mind how this can be checked with FreeBSD's ports).
There you should find why configure decided to use static vs. dynamic, or even it's own lib.
0
 
LVL 5

Expert Comment

by:ecw
Comment Utility
run strip against the binaries.  It's uncomon for out of the box executables to include the symbols table.  For a large binary this can cut out a good few hundred K.

0
 
LVL 1

Author Comment

by:maabu
Comment Utility
How does strip work?

Jlevie, I think you earned the points but since we have such a cool thread going, let me ask you a tad about popper and what you think...

I am using Eudora (5.1) which supports a number of encryption mail checking options, which is why I installed Qualcomm's popper, but I can't seem to get any ssl or other encryption to work even though I have tons of crypt/ssl/openssl libs installed on my system (running apache w/mod_ssl with no troubles but could never get qpopper to use it - and qpopper is huge compared to the os's default pop3 client which is one reason that prompted me to investigate the whole shared lib issue).

0
 
LVL 40

Expert Comment

by:jlevie
Comment Utility
I'll have to confess that it's been over 5 years since I last messed with Qualcomm's POP server. There wasn;t any SSL stuff in that version and I would have had no reason to use it then even if it had been available.

How far do you get in trying to enable SSL in popper? Will it build with the libs from a recent OpenSSL distribution? Or are you having problems with generating and/or using a certificate (assuming that you're using a self-signed certificate)?
0
 
LVL 1

Author Comment

by:maabu
Comment Utility
I really don't know - I just made the version that was included.  Admittedly I'm really ticked off at how unnecessarily complicated the SSL/CA stuff is.  I know it does not need to be so complicated and I find it annoying.  Don't even get me started about the early days of integrating certs into Apache, which was a nightmare.

So basically I don't know.  All I know is I'd like to use ANY pop3 server that has some kind of encryption.. I'm open to ideas.
0
 
LVL 40

Accepted Solution

by:
jlevie earned 300 total points
Comment Utility
You have my sympathies with respect to certificates. And I felt the same way until I did enough research to more fully understand certificates and certificate authorities. Now that I understand it a bit better I'm not so sure that the process could be simplified and still meet the requirements. And yes the early days of certificates, before OpenSSL, was a royal pain. Especially since all of the certificates had to be obtained from a commercial vendor and they weren't wild about supporting anything but a Netscape or other commercial server.

Creating a local certificate authority and a self signed certificate can be made a lot easier with one of the script packages floating around on the Internet. I was less than thrilled with the way they worked so I wrote a perl script for creating self-signed certificates.

Setting up an email system for encrypted connections is a non-trivial task. Not only do you have the problem of making sure that the server (POP and SMTP) and the client share a common method, but you also need to generate a client side certificate and incorporate that into the client. None of this is documented very well and that's not so surprising considering the multitude of clients. If I were going to do an encrypted mail system I'd use the Cyrus implementation. There's a mailing list for Cyrus that is a very good resource if you run into problems.
0
 
LVL 1

Author Comment

by:maabu
Comment Utility
On the client side, in Eudora for example, you can simply select the method, such as SSL.  Does a certificate still need to be installed on the client side?  This isn't necessary with SSL implementation in the browser, so why in email?

Is there a way to tell what support may be integrated into my POP3 servers?  Any URL to a page which more-clearly documents this process?

thanks very much for your comments - they are much appreciated!
0
 
LVL 5

Expert Comment

by:ecw
Comment Utility
"How does strip work?"
  strip file...

all strip does is remove extraneous info from a binary, be it executable or a plain object.  This useless info is normally just the symbols table, but will also include debug info if the file includes it.  I've seen on many occasions a 300K binary (especially gnu stuff) strip down to less than 100K.  If the code works, why carry loads of crap around with it.
0

Featured Post

Highfive + Dolby Voice = No More Audio Complaints!

Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

Join & Write a Comment

Attention: This article will no longer be maintained. If you have any questions, please feel free to mail me. jgh@FreeBSD.org Please see http://www.freebsd.org/doc/en_US.ISO8859-1/articles/freebsd-update-server/ for the updated article. It is avail…
Installing FreeBSD… FreeBSD is a darling of an operating system. The stability and usability make it a clear choice for servers and desktops (for the cunning). Savvy?  The Ports collection makes available every popular FOSS application and packag…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…

743 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now