• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 797
  • Last Modified:

Got locked out of a Linux VM, how can I restore SSH and console access?

While using a routine connection WinSCP connection to a Linux VM (CentOS 4.0), I have a suspicion that I may have tripped some bad password counter when trying to save a file and now ALL SSH connections are being actively refused by the Linux VM. I have VM console access to this VM (older version, VM 1.0) but every console login I attempt (including root) just flashes a PAM error and refuses access. I also have access to the Linux server underlying the Linux VM running so I can also stop/start the VM by vmware-cmd utility. I just cannot access the VM itself. If this is a situation where ipchains/iptables has locked me out of the VM, is there anyway I can reset this externally? This is quite frustrating and, I know, a boneheaded situation to be in but I need to be very careful to not bork the VM any further than I already have because I have apps and users that I cannot disrupt too much. Maybe I have been rooted somehow, but since all the apps are running fine, I am hoping that there is no foul play involved here. I have searched for instances of similar VM lockouts but have not found any that sound like the ugly mess I have gotten into. I hope someone out there has maybe run into this before and can help because the situation is getting urgent for access to code and DB on the VM. I am also wondering if the vmware-cmd setrunuser command may be of any use in this situation but I am reluctant to try this until I am more sure what side-effect this has or if it even has any chance of working.
Any help that anyone can bring to this siutation would be deeply appreciated.
0
Tim YDC SupportStaff
Asked:
Tim YDC SupportStaff
  • 16
  • 9
  • 8
  • +1
1 Solution
 
PapertripCommented:
Are you running something like DenyHosts on the VM?
0
 
Tim YDC SupportStaffAuthor Commented:
Thanks for responding.
I do not have deep  Linux experience, if you are asking me if there is a service running called DenyHosts, then no I do not think so. But I do believe there are entries in deny.hosts files if that is what you meant.
Normally I think the ipchains/iptables is not running since there is another firewall service running on the
VM I am trying to access. Have I misunderstood your question?
0
 
PapertripCommented:
Even if you were I would be very surprised if that also locked out your system account, but I guess it's possible.  However if that were the case I don't think you'd see a PAM error, but I guess that would depend on exactly what caused all this.

What was the PAM error?
0
NEW Internet Security Report Now Available!

WatchGuard’s Threat Lab is a group of dedicated threat researchers committed to helping you stay ahead of the bad guys by providing in-depth analysis of the top security threats to your network.  Check out this quarters report on the threats that shook the industry in Q4 2017.

 
PapertripCommented:
No you pretty much answered my questions.

Check /etc/hosts.deny and see if your IP is in there.  After that look for /etc/denyhosts or /etc/init.d/denyhosts, or use locate (or find) and search for 'denyhosts'.  Again that shouldn't affect console access but one thing at a time.
0
 
Tim YDC SupportStaffAuthor Commented:
Well, unfortunately the console PAM error flashes extremely fast and reverts to login: but what I can see is simply " PAM error - aborting".....

I have tried a variety of logins and some flash with a blank and some flash with text error noted above.

And now, after many failed attemps I can see...
PAM failure, aborting: Critcal error - immediate abort...INIT: ID "1" spawning too fast: disabled for 5 minutes

This was not seen before and I am sure just a result of my messing around, the usual error just flashes as noted in first line. So , I have not code or other message to go on right now.

0
 
Tim YDC SupportStaffAuthor Commented:
Responding to your suggestions, clarification required...

Are you saying that I should check the /etc/hosts.deny on the VM that I cannot access or the Linux OS
that the VM is running on?  I can likely get to the latter but not the former or I wouldn't be in this mess :-)

Or am I missing something?

I am grateful that you are trying to help me out of this morass.
0
 
PapertripCommented:
Oh right, you are locked out of the VM :p

There must be some info I'm missing here, I would be shocked if some service running on the VM is blocking ssh AND console access (the latter being more surprising).  It's possible I guess for something to do that, but heh if that was setup on purpose then you got what you deserved ;)

You could try booting from a live CD and looking around.
0
 
Tim YDC SupportStaffAuthor Commented:
More clarification if that helps...

When I putty into the VM, i get login prompt and can enter "root" and when I enter psw, putty received network error about server refusing the connection.

Access via VM console is as noted above with PAM error flashing by without even seeing the password prompt and cannot be sure I have seen all of it. I have old slow eyes, I guess.
0
 
PapertripCommented:
Something must be altering PAM configs based on failed logins... Was this server built by someone else?  Are you using some vmware pre-built image or something?
0
 
Tim YDC SupportStaffAuthor Commented:
Following up again to Papertrip...

We have been using this VM normally for literally years (you can tell from the Cent OS version) without any major incident. There is nothing that I configured or am even aware of that I could have set to setup this lockout on purpose, unless it was out of pure ignorance or lack of experience on my part (always a possibility, I guess :-)) . I am not sure how booting to a live CD could help me access the VM, I can already access the underlying Linux OS that the VM is running on. Or am I not understanding your suggestion?

Maybe this is relevant but I was using a public wifi connection (which I have cautiously done before)
and WINscp and trying to make a quick in-and out update to a file. The file updated prompted for psw which I thought I entered successfully (cannot remember if I had to enter once or twice) but ever since this little update failure, now all SSH access is actively refused. Could someone have borked me during that brief a connection?
0
 
Tim YDC SupportStaffAuthor Commented:
I have an about an hour meeting I need to run an attend and then I will be back to trying to get this resolved. Thanks for your help, I will check for more ideas immediately when I get back.
0
 
PapertripCommented:
I'm assuming the virtual disks for your VM are some in proprietary format like .vmdk, so what I was getting at is if you can boot the VM off a live CD, you can mount the filesystems and check everything out.

In regards to your 2nd question, that is always a possibility.
0
 
Tim YDC SupportStaffAuthor Commented:
Following up again, thank you Papertrip...

Are you using some vmware pre-built image or something?

Response: No, this is a pretty simple Linux Lamp with MySql config, not anything pre-built. It was built by another individual who has left our organization a while ago. I have no reason to believe, nor I have observed, anything specialized in the configuration.

Something must be altering PAM configs based on failed logins..

Response: I would agree, something has or is mangling the PAM config, but what? If this was a result of a hack attempt (or maybe an interrupted one), what would they have tried doing to result in this outcome? Also, there does not seem to be any other mischief on the VM, but maybe that does not mean anything. If there was some IP flood when I was on the wifi, or other hack attempt, could I tell if they were successful or just hosed up my access?

I'm assuming the virtual disks for your VM are some in proprietary format like .vmdk, so what I was getting at is if you can boot the VM off a live CD, you can mount the filesystems and check everything out.
Well, there is another different  VM running on the underlying box, so I guess your suggestion is to copy the borked VM files to a disk that can be booted elsewhere and try to start that? Assuming that might work, what could I look for during the boot that might give me a clue to how to resolved the access lockout?
0
 
parparovCommented:
Can you try rebooting the VM into single user mode (catching the boot prompt and then typing something like 'linux single', or selecting 'append boot arguments' and adding 'single' to the string that appears?

That should take you to the root prompt w/o password and you can study the logs in /var/log and reset root passwords as well as check the pam config in /etc/pam* and fix it if anything in it looks fishy.

0
 
parparovCommented:
Clarification - all this should be done on VM console, of course.
0
 
Tim YDC SupportStaffAuthor Commented:
Follow up to Parparov:

Ah. Was not aware I could do that, I will attempt this tonight when I can reboot the VM without interrupting user activity. Even if this does not work from a remote VM console, it might if I can actually get a terminal physically attached to the base serve in the rack and try and interrupt the VM boot in the same way. Maybe this is some hope here, thanks for the idea, will try as soon as I can.
0
 
PapertripCommented:
Good call parparov, not sure why I went straight for the live cd and skipped over single-user, I think my brain was still trying to figure out why the hell this was even happening in the first place.
0
 
parparovCommented:
ydcsupport,

If the VM boots real fast, VMware should have a setting that delays the boot sequence by a desired time in order to be able to interrupt boot.

Also, an additional question: does VM use any external authentication service like NIS, Active Directory, WINS or whatever?

papertrip,
IMHO, PAM configuration got something wrong there. There is no other mechanism that can stop root from logging on console that I can think of. But only logs may tell for sure.
0
 
Tim YDC SupportStaffAuthor Commented:
Following up to parparov...

Also, an additional question: does VM use any external authentication service like NIS, Active Directory, WINS or whatever?

Response: No, not aware of any.

General Update:
After a few late hours and many failed attempts, I was finally able to use CentOs boot and GRUB in the correct fashion to allow a Linux single user boot, Yeah! But it was so late and exhaustion keep me from getting much further, so I will try again this evening and follow up on the log and PAM suggestions from earlier in the thread. If anyone can suggest more specific items to check, I would be grateful. Since the single user mode comes up in runlevel 1 instead of usual 3 there are some services that are not active in this mode that are in the other run level...am not sure what , if any, differences that may make.
0
 
parparovCommented:
the main thing is to check the /var/log/secure and likewise files. Also, posting pam config here may help.
0
 
PapertripCommented:
@ydcsupport were you able to get this resolved?
0
 
Tim YDC SupportStaffAuthor Commented:
Well, no. Have spent a bunch of hours looking around. And I am still perpelexed about what has happened.

In secure and other logs I see the point where the PAM started failing but there does not seem to
be any more helpful messages other than pam errors - critical processing terminated type text.

There is a pam.d directory (not in etc as usual) and there is no pam.conf anywhere that I can find. Could this file have been trashed somehow without any trace (or any trace that my limited experience could find).
What would happen is this file is gone? Will the VM behave as it does now?

There did seem to be a chroot jail mechanism in place but I cannot tell if that was involved in the issue or just ancillary information. But would the pam.d directory being off the root (instead of etc) implicate this in some fashion.

I have tried to compare this VM to one other with a similar (I stress similar) config. Would I be able to copy the pam.conf from my comparison box? Might it be better to somehow re-install PAM (assuming I could understand the pathing required). It that possible? Can I do this without a major disturbance to the operation of the box, since I am still quite sensitive to the users and apps?

Also, in single user mode, the PAM service is not running either so su sudo do not work?
What else could I be looking for, in addition to questions above, and thanks for any advice or suggestions.
0
 
parparovCommented:
maybe you can post both the errors from the log and the pam config that you have?
0
 
Tim YDC SupportStaffAuthor Commented:
I have attached snippets of the secure log and message log for the time of the failure. It does seem that suddenly the /etc/pam.conf got borked/deleted/damaged. I still have no real answers to the questions I posted earlier about the best way to resolve/restore/reinstall the PAM services.
Any light you can shed would be helpful....



secure-snippet.txt
0
 
Tim YDC SupportStaffAuthor Commented:
Also the messages log (same time period) messages-snippet.txt
0
 
Tim YDC SupportStaffAuthor Commented:
Also as mentioned above any pam.conf appears to have gone away ....
0
 
Tim YDC SupportStaffAuthor Commented:
Also, as noted previously, while these is no pam.d directory under etc, there is one under the main root...and in this dir are two files system-auth and vmware-guestd which look like what someone might expect to see in the pam.conf file it was there )accroding to all the reading about PAM that I have been doing) but I am not yet sure why these files are there or even if they are in play with all of this. Sigh.
0
 
parparovCommented:
Do you have a similar VM with working PAM configuration?
0
 
parparovCommented:
Also, in CentOS/RedHat there is no /etc/pam.conf , but /etc/pam.d should be present and replacing it
0
 
parparovCommented:
I think you can do one of the following things:

Transfer a working /etc/pam.d from another Linux

Replace the packages of botched pam on this machine.
Try:
rpm -qa |grep pam

Open in new window

to see the list of pam-related packages istalled
Then you can:
rpm -V package_name

Open in new window

for the listed packages to see their validity.
http://www.rpm.org/max-rpm/ch-rpm-verify.html explains the output; if everything ok, you should see dots with one or two chars only.
Then you can bring over the packages listed and botched, and re-install them.
0
 
maxchowCommented:
Hi, can you still able to get access to the system through the VM console or not?
0
 
Tim YDC SupportStaffAuthor Commented:
I have copied a pam.d directory from a similarly configured box and that seems to have re-enable the PAM service and I can access via VM Console now and via SSH. So, thanks very much to Papertrip and parparov for their kind guidance and patience in helping me through this.
0
 
Tim YDC SupportStaffAuthor Commented:
Thanks again!
0
 
PapertripCommented:
Awesome!
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

Featured Post

Upgrade your Question Security!

Your question, your audience. Choose who sees your identity—and your question—with question security.

  • 16
  • 9
  • 8
  • +1
Tackle projects and never again get stuck behind a technical roadblock.
Join Now