utmp/wtmp info is not showing

I'd like to add, reinstall is NOT an immediate
option, but will happen in near future (because
of the break-in). The machine has been temporarily disconnected from the network.

Okay, you can't fix it the right way right now. Fair enough, lets poke around a bit an see if something interesting turns up...

What are the permissions/ownership of utmp & wtmp? Also, does a ps -ef return usernames and the tty associated with the process?

I made utmp/wtmp both be 644 each time when trying
various ways of resetting/rebooting.

I should have mentioned that ps -aux, ps -ef etc.
works fine, showing uid's & ttys. That information
IS being added by init/login to utmp/wtmp, since
it shows up correctly if you run "strings" on those files.

I wrote the little C program with getlogin() to see
whether finger/who/w had been replaced. But
since getlogin() returns NULL I assume finger etc.
are also getting wrong info from utmp/wtmp.

"last" also shows no entries (it WAS showing corrupt info when I discovered the breakin).

Thanks for your help.

Typical permissions would likely be 664, owned by root group utmp (if 5.1 has that group, I can't remember).

It might be good to boot single user (or better yet from a resuce floppy), remove utmp & wtmp, touch utmp and wtmp, set owner/group and perms, then halt and reboot. It does sound like the utmp/wtmp files are still hosed somehow.

As to getlogin() the man page says "getlogin returns a pointer to a string containing the name of the user logged in on the controlling terminal  of  the process, or a null pointer if this information cannot be determined." So  if there was still something wrong with the begining of the utmp/wtmp files, getlogin() might return a null even if the files were now being updated (as they appear to be).

I will try that tomorrow and respond here.
Thanks.

You don't happen to have another 5.1 installation around, do you? If you did, it would be good to compare the size and checksums of libc on both. Most of the good stuff is in there and I've seen reports of them being modified to prevent detection during an attack.

Well, I tried a variety of combinations of booting single user, rescue disk and deleting/clearing utmp/wtmp at different times. No luck.

It appears that utmp entries are not being written properly. I'm tired of fighting it and will have to give up until I install RHL 6.1 (a big nuisance since that box has some active users and various add-on software).

Jlevie, your comments were still helpful and gave me some options. I'm going to go ahead and accept your comments as an answer. Thanks for the help.

If you'd like to continue to work this problem, I'm willing to help. There's some chance that we could figure out what the attackers screwed up and that might allow you to continue to use the system until it was more convient to re-install.

Most kind of you. If you get tired of it & stop posting I will fully understand.

I don't think it's anything the hackers did, anymore. They were from the far-east and as far as I can tell they ran a sniffer (rather obviously) and not much else, and did not do much to cover tracks. It really seems that shutdown or some program that writes utmp is not doing so correctly.

Here's a little C program I wrote:
---------------------------------------
   struct utmp *pu;

   utmpname("/var/run/utmp");
   setutent();
   while ((pu = getutent()) != NULL)
   {
      printf("%d %d %s %s %s %s\n", pu->ut_type, pu->ut_pid, pu->ut_line,
                pu->ut_id, pu->ut_user, pu->ut_host);
   }
   endutent();
-------------------------------------
and the first few lines of output:
6 731 tty3 3 LOGIN
6 727 tty2 2 LOGIN
6 734 tty1 1 LOGIN
8 746 ttyp0 09-a
======================
the ut_type,ut_pid, ut_line, ut_id all seem fine, but the ut_user field
is messed up. Those first 3 "LOGIN"s should be "root"s (me logging in).

Man pages say something about utmp depending on libc version. I vaguely remember changing libc versions, perhaps to compile a version of Gnu Pascal. But surely programs like finger, w, who and shutdown (which writes utmp) are statically linked and would not change if libc changed under them?

IF (a) I really changed libc AND (b) shutdown is dynamically linked with that new libc and thus writing in a new utmp format AND (c) finger, who, etc. are statically linked with old libc and expect an old utmp format THEN maybe, this is the problem.

Now "strings /var/run/utmp"still shows "LOGIN" as the user for those "root" logins above, but it shows correct userids in later entries. "Strings" also shows correct host fields. So that information IS getting into utmp.

My little test program also quits (getutent() returns NULL) after the first five entries, while utmp clearly has many more entries according to strings. So I wonder if  the FIRST PROGRAM that writes into the (initially empty) utmp is writing in a different format than others that add entries later (e.g. init, login?).

Maybe there's a way to check (a) which libc version I have and (b) which libc version finger/shutdown  etc. are linked with. Other than that I'm running out of ideas :-) I'll see if I can remember more of what I was doing with gnu pascal & libc. I don't remember changing any system files, only compiling a copy in a directory.

Long and boring but maybe more info is better than none. :-)

Off to bed now.. will check back if I hear that you've added something.

Ahh, now we're getting somewhere. I suspect that you did change libc, as that is the one common thread in all the problems. That was one of the reasons I suggested checking libc against a "known good copy". I can't absolutely speak for 5.1, but 6.0 and latter do use dynamiclly linked executables for finger, who, shutdown, and just about everything else. The only thigs that can't be dynamically linked are things that have to run before /usr would be mounted.

It's easy to see if they are and what libc they were linked with. Just run ldd on the executable of interest. It'll tell you what libraries are dynamic and what version the executable was lined against. For instance on my RH 6.1 laptop,

levie> ldd /usr/bin/who
        libc.so.6 => /lib/libc.so.6 (0x40018000)
        /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x40000000)
l

Jlevie>That was one of the reasons I suggested checking libc
Indeed you did, and I misread/misunderstood your advice. As it happens, I don't have an older RHL lying around (this one I installed over the net). At colleges we take what we can get for free :-)

Here's some ldd output:
-----------------------------------------------
%  ldd /sbin/shutdown /usr/bin/who
/sbin/shutdown:
        libutil.so.1 => /lib/libutil.so.1 (0x40006000)
        libc.so.6 => /lib/libc.so.6 (0x40009000)
        /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x00000000)
/usr/bin/who:
        libc.so.6 => /lib/libc.so.6 (0x40006000)
        /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x00000000)
-------------------------------------------------

And relevant files in /lib:
--------------------------------------------
% ls /lib/libc*
/lib/libc-2.0.7.so      /lib/libcom_err.so.2    /lib/libcrypt-2.0.7.so
/lib/libc.so.6          /lib/libcom_err.so.2.0  /lib/libcrypt.so.1
---------------------------------------------

I *could* go and do my homework and figure out what this means, but I'll ask here first: does this tell us anything?

I used to be handy around Unix in older BSD days ('85-'90), but I'm just now getting back into UNIX, and Linux has a lot of stuff that's killing me. Of course I never got much into dynamic linking back then, either.

Hum, now I found something:
----------------------------------------------
% ldd /sbin/init /bin/login /bin/ps
/sbin/init:
        libutil.so.1 => /lib/libutil.so.1 (0x40006000)
        libc.so.6 => /lib/libc.so.6 (0x40009000)
        /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x00000000)
/bin/login:
        not a dynamic executable
/bin/ps:
        libc.so.5 => not found
------------------------------------------------
At least now this explains why ps is showing uids where who, finger aren't, it looks like ps wants libc version 5. I'm not sure how it can run if libc version 5 is not found.

And login was probably statically linked with libc version 5, so it may be that after the first few entries (recorded by shutdown with libc 6), login adds records using libc 5. Programs like who, finger use libc6 so they can't get to the records written by login.


Any way to patch this mess? :-)

There's three ways you could fix it. The obvious would be to replace the newer libc with the original. The second way would be to compile the relevant utilities against the current libc. This will almost certainly require updating the header files so they match the new libc. It seems to me that the second method is likely to snowball. There's no telling what other problems you'll encounter if the header files change.
The last way would be to statically compile those against an original libc.a. I kinda like this one as the header files don't need to change and you won't break anything else that's currently working.

I think the last option is the one I like, best, also. I'll start looking for a RHL 5.1 version 5 libc.a.

Well, you certainly earned your points. This was my first use of experts exchange and if this is any indication of the kind of help one can get here, I'll be back.

Please consider this case closed. Thank you for your competence & patience.

You're most welcome, and thanks for the compliment. From what I've seen, I think you'll find that most of the "experts" are quite knowledgeable and very willing to help.