Solved

Implementing shared libraries

Posted on 2002-03-14
15
382 Views
Last Modified: 2010-04-21
Hi,

I have waht I think is a pretty basic question but I'm hoping I can get some details answers.

I'm running various Unix servers, including some FreeBSD implementations (3.4 version so this would mainly apply to FreeBSD but I assume it's generally the same for other Unix variants but please let me know if it isn't).

The utilities that come with Unix, such as bind, popper, ftpd, apache, sendmail, etc. always are much smaller in size than the later versions I compile and install to update the system.

I'm under the impression that what's going on here, is when I compile a newer version of, say Apache or Sendmail, by default the compilation is stand-alone, and in most cases, it could be compiled with some sort of shared library option which would dramatically reduce the size and memory overhead of the software?  In the case of Apache, where you might have 100+ daemons spawned, I've been told that if I recompiled the program to share memory/libs that it would be much more memory efficient.  How can I do this?

Specifically here are my questions:

1.  Is there a special module or set of libraries that must be installed to implement this?  Does it vary by OS/version?  Where can I get this code if I don't have it already?

2.  I'd like detailed instructions on how to compile the following programs to use these shared libraries: bind, ftpd/wuftp, popper, apache, sendmail, etc.

I'm assuming maybe it's a single extra compilation/linking command parameter?  Is it true that by default none of these systems compile in this manner?  

Any additional thoughts or comments on what I'm trying to do?  It's frustrating to replace a 300k 2.0 version with a 4320k 2.1 version of a program...

Thanks!
0
Comment
Question by:maabu
  • 6
  • 5
  • 2
  • +2
15 Comments
 
LVL 40

Expert Comment

by:jlevie
ID: 6866366
Whether an application is built against dynamic libraries or static libraries has everything to do with the link process. In general the configure process for most things will pick dynamic libs over a static link option unless forced to do otherwise or unless it thinks that dynamic libs won't work on some particular system. It's been an awful long time since I've used FreeBSD 3.4, but I don't think I've ever had a problem building Apache or Bind with dynamic libs on 4.0 or later.

Are you building your utilities directly from source or via the ports collection?
0
 
LVL 1

Expert Comment

by:wisdom042597
ID: 6866412
How can you tell?  When my Bind version is 8M and the bundled version with 3.4 is less than a MB and not much earlier, I know something is off...
0
 
LVL 40

Expert Comment

by:jlevie
ID: 6866532
It's easy to tell if something is dynamically linked or not. Just run ldd or file on the executable, i.e., 'ldd /usr/sbin/sendmail' or 'file /usr/sbin/sendmail'. Also be careful when comparing file sizes. For example, on a stock FreeBSD 4.4 system:

praetorian> ls -l /usr/sbin/sendmail
-r-xr-xr-x  1 root  wheel  4984 Oct 12 17:43 /usr/sbin/sendmail*

That's ridiculously small, let's look a bit further:

praetorian> file /usr/sbin/sendmail
/usr/sbin/sendmail: symbolic link to /usr/sbin/mailwrapper
praetorian> strings /usr/sbin/mailwrapper | grep sendmail
/usr/libexec/sendmail/sendmail

So the actual sendmail executable isn't what's in /usr/sbin and the real size of sendmail is:

praetorian> ls -l /usr/libexec/sendmail/sendmail
-r-sr-xr-x  1 root  wheel  415424 Oct 12 17:43 /usr/libexec/sendmail/sendmail
praetorian> ldd /usr/libexec/sendmail/sendmail
/usr/libexec/sendmail/sendmail:
     libutil.so.3 => /usr/lib/libutil.so.3 (0x280c6000)
     libwrap.so.3 => /usr/lib/libwrap.so.3 (0x280cf000)
     libssl.so.2 => /usr/lib/libssl.so.2 (0x280d7000)
     libcrypto.so.2 => /usr/lib/libcrypto.so.2 (0x28103000)
     libsasl.so.8 => not found (0x0)
     libc.so.4 => /usr/lib/libc.so.4 (0x281ba000)

Now a 400Kb executable is lots more reasonable.

0
Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

 
LVL 1

Author Comment

by:maabu
ID: 6866573
Ahhh, here's what I get on apache:

# ldd httpd
httpd:
        libcrypt.so.2 => /usr/lib/libcrypt.so.2 (0x280d3000)
        libc.so.3 => /usr/lib/libc.so.3 (0x280e8000)

So that looks good I believe...

# ldd named
named:
        libc.so.3 => /usr/lib/libc.so.3 (0x281bf000)

named is still over 5MB in size.  Does that sound right for version 9.1.3?

# ldd /usr/sbin/sendmail
/usr/sbin/sendmail:
        libutil.so.2 => /usr/lib/libutil.so.2 (0x280d9000)
        libc.so.3 => /usr/lib/libc.so.3 (0x280e2000)

Hmmm, this looks good then?

So I assume that using Shared Libraries is just a standard thing?

One of the big things I noticed was when switching from FBSD's popper to Qualcomm's popper:

-r-xr-xr-x  1 root  wheel  - 36080 Dec 10  1999 popper

-rwxr-xr-x  1 root  wheel  - 571550 Jan 10 01:23 qpopper
A pretty dramatic increase... but a ldd reveals:

qpopper:
        libcrypt.so.2 => /usr/lib/libcrypt.so.2 (0x28083000)
        libc.so.3 => /usr/lib/libc.so.3 (0x28098000)
so I guess it's working... hmmmm

Why does your sendmail have so many extra libraries?  What goodies are you using?
0
 
LVL 40

Expert Comment

by:jlevie
ID: 6866622
Okay all of those are linked against dynamic libs. The size of Bind 9 sounds a bit on the big side compared to my copy on 4.4. I've got a system copy of Bind 8 that's 500Kb and a local version of Bind 9 that's 1.6Mb. That sounds about right to to since there's lots of stuff in Bind 9, like the crypto stuff for rndc & dynamic updates, that isn't in Bind 8. Your 8Mb copy sounds a bit on the large side, but I'd have to examine how it was built to determine if that's reasonable for a 3.4 system. It's possible that my 4.4 system has later versions of some routines in system libraries that Bind 9 requires. That could cause the activation of code from the Bind distribution to be built into your executable.

My sendmail copy has lots of other libraries included as it includes support for tcpwrappers, SSL encryption, and SASL authentication. Incidentally, that copy was built from the FreeBSD 4.4 sources by a tweak to the build options used by make world.
0
 
LVL 1

Author Comment

by:maabu
ID: 6866667

As an aside, how can I implement security in POP3 mail checking?  I couldn't find any support in the 3.4 popper and I couldn't figure out how to get qualcomm's qpopper to work either.
0
 
LVL 40

Expert Comment

by:jlevie
ID: 6867281
Are you speaking of using an encrypted POP connection? Part of that involves what the clients support. I'm a big fan of the Cyrus IMAP implementation, which includes POP, IMAPS and POPS support (the later are SSL encrypted connections).

The only disadvantage of the Cyrus implementation is that it works best when using SASL authentication rather than traditional Unix authentication. For my uses that's an advantage as I don't want ordinary users to have any access to my mail servers other than through IMAP or POP. By keeing their authentication tokens (username & password) in a SASL database rather than in passwd there's no way for a mail client user to break into the server or abuse their account because they don't have Unix accounts. Well there's always the risk of a root exploit if you don't keep the system up to date w/respect to security advisories, by I have less to worry about if my mail server only has a root account and the accounts for a couple of trusted admins.

Other advantages of Cyrus lie in it's handling of user INBOX's and its ability to apply mail quotas in a manner that is 'mail safe'. You can read more about Cyrus at http://asg.web.cmu.edu/ 
0
 
LVL 51

Expert Comment

by:ahoffmann
ID: 6867437
well done, jlevie, explained all things about dynamic linking.

About the size of bind/named, mine also has roughly 4.8MB.

To check what may cause the reason why some libs are linked static, while others are linked dynamic, check the result of the configure command: its output and generated files like config.status (sorry swapped out of mind how this can be checked with FreeBSD's ports).
There you should find why configure decided to use static vs. dynamic, or even it's own lib.
0
 
LVL 5

Expert Comment

by:ecw
ID: 6868156
run strip against the binaries.  It's uncomon for out of the box executables to include the symbols table.  For a large binary this can cut out a good few hundred K.

0
 
LVL 1

Author Comment

by:maabu
ID: 6869474
How does strip work?

Jlevie, I think you earned the points but since we have such a cool thread going, let me ask you a tad about popper and what you think...

I am using Eudora (5.1) which supports a number of encryption mail checking options, which is why I installed Qualcomm's popper, but I can't seem to get any ssl or other encryption to work even though I have tons of crypt/ssl/openssl libs installed on my system (running apache w/mod_ssl with no troubles but could never get qpopper to use it - and qpopper is huge compared to the os's default pop3 client which is one reason that prompted me to investigate the whole shared lib issue).

0
 
LVL 40

Expert Comment

by:jlevie
ID: 6870046
I'll have to confess that it's been over 5 years since I last messed with Qualcomm's POP server. There wasn;t any SSL stuff in that version and I would have had no reason to use it then even if it had been available.

How far do you get in trying to enable SSL in popper? Will it build with the libs from a recent OpenSSL distribution? Or are you having problems with generating and/or using a certificate (assuming that you're using a self-signed certificate)?
0
 
LVL 1

Author Comment

by:maabu
ID: 6870465
I really don't know - I just made the version that was included.  Admittedly I'm really ticked off at how unnecessarily complicated the SSL/CA stuff is.  I know it does not need to be so complicated and I find it annoying.  Don't even get me started about the early days of integrating certs into Apache, which was a nightmare.

So basically I don't know.  All I know is I'd like to use ANY pop3 server that has some kind of encryption.. I'm open to ideas.
0
 
LVL 40

Accepted Solution

by:
jlevie earned 300 total points
ID: 6871431
You have my sympathies with respect to certificates. And I felt the same way until I did enough research to more fully understand certificates and certificate authorities. Now that I understand it a bit better I'm not so sure that the process could be simplified and still meet the requirements. And yes the early days of certificates, before OpenSSL, was a royal pain. Especially since all of the certificates had to be obtained from a commercial vendor and they weren't wild about supporting anything but a Netscape or other commercial server.

Creating a local certificate authority and a self signed certificate can be made a lot easier with one of the script packages floating around on the Internet. I was less than thrilled with the way they worked so I wrote a perl script for creating self-signed certificates.

Setting up an email system for encrypted connections is a non-trivial task. Not only do you have the problem of making sure that the server (POP and SMTP) and the client share a common method, but you also need to generate a client side certificate and incorporate that into the client. None of this is documented very well and that's not so surprising considering the multitude of clients. If I were going to do an encrypted mail system I'd use the Cyrus implementation. There's a mailing list for Cyrus that is a very good resource if you run into problems.
0
 
LVL 1

Author Comment

by:maabu
ID: 6871665
On the client side, in Eudora for example, you can simply select the method, such as SSL.  Does a certificate still need to be installed on the client side?  This isn't necessary with SSL implementation in the browser, so why in email?

Is there a way to tell what support may be integrated into my POP3 servers?  Any URL to a page which more-clearly documents this process?

thanks very much for your comments - they are much appreciated!
0
 
LVL 5

Expert Comment

by:ecw
ID: 6876463
"How does strip work?"
  strip file...

all strip does is remove extraneous info from a binary, be it executable or a plain object.  This useless info is normally just the symbols table, but will also include debug info if the file includes it.  I've seen on many occasions a 300K binary (especially gnu stuff) strip down to less than 100K.  If the code works, why carry loads of crap around with it.
0

Featured Post

Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Let's say you need to move the data of a file system from one partition to another. This generally involves dismounting the file system, backing it up to tapes, and restoring it to a new partition. You may also copy the file system from one place to…
A metadevice consists of one or more devices (slices). It can be expanded by adding slices. Then, it can be grown to fill a larger space while the file system is in use. However, not all UNIX file systems (UFS) can be expanded this way. The conca…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.

820 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question