Solved

Got locked out of a Linux VM, how can I restore SSH and console access?

Posted on 2011-09-07
34
775 Views
Last Modified: 2012-05-12
While using a routine connection WinSCP connection to a Linux VM (CentOS 4.0), I have a suspicion that I may have tripped some bad password counter when trying to save a file and now ALL SSH connections are being actively refused by the Linux VM. I have VM console access to this VM (older version, VM 1.0) but every console login I attempt (including root) just flashes a PAM error and refuses access. I also have access to the Linux server underlying the Linux VM running so I can also stop/start the VM by vmware-cmd utility. I just cannot access the VM itself. If this is a situation where ipchains/iptables has locked me out of the VM, is there anyway I can reset this externally? This is quite frustrating and, I know, a boneheaded situation to be in but I need to be very careful to not bork the VM any further than I already have because I have apps and users that I cannot disrupt too much. Maybe I have been rooted somehow, but since all the apps are running fine, I am hoping that there is no foul play involved here. I have searched for instances of similar VM lockouts but have not found any that sound like the ugly mess I have gotten into. I hope someone out there has maybe run into this before and can help because the situation is getting urgent for access to code and DB on the VM. I am also wondering if the vmware-cmd setrunuser command may be of any use in this situation but I am reluctant to try this until I am more sure what side-effect this has or if it even has any chance of working.
Any help that anyone can bring to this siutation would be deeply appreciated.
0
Comment
Question by:ydcsupport
  • 16
  • 9
  • 8
  • +1
34 Comments
 
LVL 21

Expert Comment

by:Papertrip
ID: 36498297
Are you running something like DenyHosts on the VM?
0
 

Author Comment

by:ydcsupport
ID: 36498338
Thanks for responding.
I do not have deep  Linux experience, if you are asking me if there is a service running called DenyHosts, then no I do not think so. But I do believe there are entries in deny.hosts files if that is what you meant.
Normally I think the ipchains/iptables is not running since there is another firewall service running on the
VM I am trying to access. Have I misunderstood your question?
0
 
LVL 21

Expert Comment

by:Papertrip
ID: 36498343
Even if you were I would be very surprised if that also locked out your system account, but I guess it's possible.  However if that were the case I don't think you'd see a PAM error, but I guess that would depend on exactly what caused all this.

What was the PAM error?
0
 
LVL 21

Expert Comment

by:Papertrip
ID: 36498364
No you pretty much answered my questions.

Check /etc/hosts.deny and see if your IP is in there.  After that look for /etc/denyhosts or /etc/init.d/denyhosts, or use locate (or find) and search for 'denyhosts'.  Again that shouldn't affect console access but one thing at a time.
0
 

Author Comment

by:ydcsupport
ID: 36498403
Well, unfortunately the console PAM error flashes extremely fast and reverts to login: but what I can see is simply " PAM error - aborting".....

I have tried a variety of logins and some flash with a blank and some flash with text error noted above.

And now, after many failed attemps I can see...
PAM failure, aborting: Critcal error - immediate abort...INIT: ID "1" spawning too fast: disabled for 5 minutes

This was not seen before and I am sure just a result of my messing around, the usual error just flashes as noted in first line. So , I have not code or other message to go on right now.

0
 

Author Comment

by:ydcsupport
ID: 36498473
Responding to your suggestions, clarification required...

Are you saying that I should check the /etc/hosts.deny on the VM that I cannot access or the Linux OS
that the VM is running on?  I can likely get to the latter but not the former or I wouldn't be in this mess :-)

Or am I missing something?

I am grateful that you are trying to help me out of this morass.
0
 
LVL 21

Expert Comment

by:Papertrip
ID: 36498500
Oh right, you are locked out of the VM :p

There must be some info I'm missing here, I would be shocked if some service running on the VM is blocking ssh AND console access (the latter being more surprising).  It's possible I guess for something to do that, but heh if that was setup on purpose then you got what you deserved ;)

You could try booting from a live CD and looking around.
0
 

Author Comment

by:ydcsupport
ID: 36498512
More clarification if that helps...

When I putty into the VM, i get login prompt and can enter "root" and when I enter psw, putty received network error about server refusing the connection.

Access via VM console is as noted above with PAM error flashing by without even seeing the password prompt and cannot be sure I have seen all of it. I have old slow eyes, I guess.
0
 
LVL 21

Expert Comment

by:Papertrip
ID: 36498555
Something must be altering PAM configs based on failed logins... Was this server built by someone else?  Are you using some vmware pre-built image or something?
0
 

Author Comment

by:ydcsupport
ID: 36498583
Following up again to Papertrip...

We have been using this VM normally for literally years (you can tell from the Cent OS version) without any major incident. There is nothing that I configured or am even aware of that I could have set to setup this lockout on purpose, unless it was out of pure ignorance or lack of experience on my part (always a possibility, I guess :-)) . I am not sure how booting to a live CD could help me access the VM, I can already access the underlying Linux OS that the VM is running on. Or am I not understanding your suggestion?

Maybe this is relevant but I was using a public wifi connection (which I have cautiously done before)
and WINscp and trying to make a quick in-and out update to a file. The file updated prompted for psw which I thought I entered successfully (cannot remember if I had to enter once or twice) but ever since this little update failure, now all SSH access is actively refused. Could someone have borked me during that brief a connection?
0
 

Author Comment

by:ydcsupport
ID: 36498598
I have an about an hour meeting I need to run an attend and then I will be back to trying to get this resolved. Thanks for your help, I will check for more ideas immediately when I get back.
0
 
LVL 21

Expert Comment

by:Papertrip
ID: 36498625
I'm assuming the virtual disks for your VM are some in proprietary format like .vmdk, so what I was getting at is if you can boot the VM off a live CD, you can mount the filesystems and check everything out.

In regards to your 2nd question, that is always a possibility.
0
 

Author Comment

by:ydcsupport
ID: 36499334
Following up again, thank you Papertrip...

Are you using some vmware pre-built image or something?

Response: No, this is a pretty simple Linux Lamp with MySql config, not anything pre-built. It was built by another individual who has left our organization a while ago. I have no reason to believe, nor I have observed, anything specialized in the configuration.

Something must be altering PAM configs based on failed logins..

Response: I would agree, something has or is mangling the PAM config, but what? If this was a result of a hack attempt (or maybe an interrupted one), what would they have tried doing to result in this outcome? Also, there does not seem to be any other mischief on the VM, but maybe that does not mean anything. If there was some IP flood when I was on the wifi, or other hack attempt, could I tell if they were successful or just hosed up my access?

I'm assuming the virtual disks for your VM are some in proprietary format like .vmdk, so what I was getting at is if you can boot the VM off a live CD, you can mount the filesystems and check everything out.
Well, there is another different  VM running on the underlying box, so I guess your suggestion is to copy the borked VM files to a disk that can be booted elsewhere and try to start that? Assuming that might work, what could I look for during the boot that might give me a clue to how to resolved the access lockout?
0
 
LVL 9

Expert Comment

by:parparov
ID: 36499481
Can you try rebooting the VM into single user mode (catching the boot prompt and then typing something like 'linux single', or selecting 'append boot arguments' and adding 'single' to the string that appears?

That should take you to the root prompt w/o password and you can study the logs in /var/log and reset root passwords as well as check the pam config in /etc/pam* and fix it if anything in it looks fishy.

0
 
LVL 9

Expert Comment

by:parparov
ID: 36499483
Clarification - all this should be done on VM console, of course.
0
 

Author Comment

by:ydcsupport
ID: 36499526
Follow up to Parparov:

Ah. Was not aware I could do that, I will attempt this tonight when I can reboot the VM without interrupting user activity. Even if this does not work from a remote VM console, it might if I can actually get a terminal physically attached to the base serve in the rack and try and interrupt the VM boot in the same way. Maybe this is some hope here, thanks for the idea, will try as soon as I can.
0
 
LVL 21

Expert Comment

by:Papertrip
ID: 36499598
Good call parparov, not sure why I went straight for the live cd and skipped over single-user, I think my brain was still trying to figure out why the hell this was even happening in the first place.
0
Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

 
LVL 9

Expert Comment

by:parparov
ID: 36500290
ydcsupport,

If the VM boots real fast, VMware should have a setting that delays the boot sequence by a desired time in order to be able to interrupt boot.

Also, an additional question: does VM use any external authentication service like NIS, Active Directory, WINS or whatever?

papertrip,
IMHO, PAM configuration got something wrong there. There is no other mechanism that can stop root from logging on console that I can think of. But only logs may tell for sure.
0
 

Author Comment

by:ydcsupport
ID: 36505611
Following up to parparov...

Also, an additional question: does VM use any external authentication service like NIS, Active Directory, WINS or whatever?

Response: No, not aware of any.

General Update:
After a few late hours and many failed attempts, I was finally able to use CentOs boot and GRUB in the correct fashion to allow a Linux single user boot, Yeah! But it was so late and exhaustion keep me from getting much further, so I will try again this evening and follow up on the log and PAM suggestions from earlier in the thread. If anyone can suggest more specific items to check, I would be grateful. Since the single user mode comes up in runlevel 1 instead of usual 3 there are some services that are not active in this mode that are in the other run level...am not sure what , if any, differences that may make.
0
 
LVL 9

Expert Comment

by:parparov
ID: 36505731
the main thing is to check the /var/log/secure and likewise files. Also, posting pam config here may help.
0
 
LVL 21

Expert Comment

by:Papertrip
ID: 36520760
@ydcsupport were you able to get this resolved?
0
 

Author Comment

by:ydcsupport
ID: 36523043
Well, no. Have spent a bunch of hours looking around. And I am still perpelexed about what has happened.

In secure and other logs I see the point where the PAM started failing but there does not seem to
be any more helpful messages other than pam errors - critical processing terminated type text.

There is a pam.d directory (not in etc as usual) and there is no pam.conf anywhere that I can find. Could this file have been trashed somehow without any trace (or any trace that my limited experience could find).
What would happen is this file is gone? Will the VM behave as it does now?

There did seem to be a chroot jail mechanism in place but I cannot tell if that was involved in the issue or just ancillary information. But would the pam.d directory being off the root (instead of etc) implicate this in some fashion.

I have tried to compare this VM to one other with a similar (I stress similar) config. Would I be able to copy the pam.conf from my comparison box? Might it be better to somehow re-install PAM (assuming I could understand the pathing required). It that possible? Can I do this without a major disturbance to the operation of the box, since I am still quite sensitive to the users and apps?

Also, in single user mode, the PAM service is not running either so su sudo do not work?
What else could I be looking for, in addition to questions above, and thanks for any advice or suggestions.
0
 
LVL 9

Expert Comment

by:parparov
ID: 36523260
maybe you can post both the errors from the log and the pam config that you have?
0
 

Author Comment

by:ydcsupport
ID: 36527058
I have attached snippets of the secure log and message log for the time of the failure. It does seem that suddenly the /etc/pam.conf got borked/deleted/damaged. I still have no real answers to the questions I posted earlier about the best way to resolve/restore/reinstall the PAM services.
Any light you can shed would be helpful....



secure-snippet.txt
0
 

Author Comment

by:ydcsupport
ID: 36527063
Also the messages log (same time period) messages-snippet.txt
0
 

Author Comment

by:ydcsupport
ID: 36527072
Also as mentioned above any pam.conf appears to have gone away ....
0
 

Author Comment

by:ydcsupport
ID: 36527091
Also, as noted previously, while these is no pam.d directory under etc, there is one under the main root...and in this dir are two files system-auth and vmware-guestd which look like what someone might expect to see in the pam.conf file it was there )accroding to all the reading about PAM that I have been doing) but I am not yet sure why these files are there or even if they are in play with all of this. Sigh.
0
 
LVL 9

Expert Comment

by:parparov
ID: 36527194
Do you have a similar VM with working PAM configuration?
0
 
LVL 9

Expert Comment

by:parparov
ID: 36527198
Also, in CentOS/RedHat there is no /etc/pam.conf , but /etc/pam.d should be present and replacing it
0
 
LVL 9

Accepted Solution

by:
parparov earned 500 total points
ID: 36527216
I think you can do one of the following things:

Transfer a working /etc/pam.d from another Linux

Replace the packages of botched pam on this machine.
Try:
rpm -qa |grep pam

Open in new window

to see the list of pam-related packages istalled
Then you can:
rpm -V package_name

Open in new window

for the listed packages to see their validity.
http://www.rpm.org/max-rpm/ch-rpm-verify.html explains the output; if everything ok, you should see dots with one or two chars only.
Then you can bring over the packages listed and botched, and re-install them.
0
 
LVL 3

Expert Comment

by:maxchow
ID: 36535538
Hi, can you still able to get access to the system through the VM console or not?
0
 

Author Comment

by:ydcsupport
ID: 36544440
I have copied a pam.d directory from a similarly configured box and that seems to have re-enable the PAM service and I can access via VM Console now and via SSH. So, thanks very much to Papertrip and parparov for their kind guidance and patience in helping me through this.
0
 

Author Closing Comment

by:ydcsupport
ID: 36544444
Thanks again!
0
 
LVL 21

Expert Comment

by:Papertrip
ID: 36544445
Awesome!
0

Featured Post

Windows Server 2016: All you need to know

Learn about Hyper-V features that increase functionality and usability of Microsoft Windows Server 2016. Also, throughout this eBook, you’ll find some basic PowerShell examples that will help you leverage the scripts in your environments!

Join & Write a Comment

HOW TO: Connect to the VMware vSphere Hypervisor 6.5 (ESXi 6.5) using the vSphere (HTML5 Web) Host Client 6.5, and perform a simple configuration task of adding a new VMFS 6 datastore.
In this article, I will show you HOW TO: Perform a Physical to Virtual (P2V) Conversion the easy way from a computer backup (image).
Teach the user how to install log collectors and how to configure ESXi 5.5 for remote logging Open console session and mount vCenter Server installer: Install vSphere Core Dump Collector: Install vSphere Syslog Collector: Open vSphere Client: Config…
This Micro Tutorial walks you through using a remote console to access a server and install ESXi 5.1. This example is showing remote access and installation using a Dell server. The hypervisor is the very first component of your virtual infrastructu…

747 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

12 Experts available now in Live!

Get 1:1 Help Now