SSH Session terminates after one hour (even when active)

Why is "KeepAlive" set to "no"?

Did you try setting that to "yes"?

Do you have packet loss to the server from the network you are remotely logging in from?

I only just turned KeepAlive off recently since my hosting company said that having it on can sometimes cause disconnects, however turning it off made no difference.  Since I noticed no difference between having it on or off, I just left it off.

Also, there is no packet loss and I have tested it from multiple locations, servers and using various ssh clients.

100% of the time, it will disconnect my session after exactly one hour has passed on the first even 10 minutes following that hour.

So, for example if I login at 9:07a, my session will be terminated at 10:10a, or if I login at 11:23a, my session will terminate at 12:30p.  It's like something is running every 10 minutes, checking for ssh sessions that are greater than one hour old and killing them.

I have full root access to this server and I can find no crons that would do this though...

A few things I think could be responsible are PAM, syswatchd, but I don't know much about either...  I've also check all of the system logs and can find nothing in them to lead me in any direction.

Here's an example of 3 sessions from the wtmp log that were terminated after one hour - on the first even 10 minutes after that hour:
ppdev    pts/2        67.32.xx.xxx     Fri Jun  2 16:03 - 17:10  (01:07)
ppdev    pts/1        67.32.xx.xxx     Fri Jun  2 16:02 - 17:10  (01:07)
ppdev    pts/0        67.32.xx.xxx     Fri Jun  2 16:02 - 17:10  (01:07)

Thanks for your suggestion Ustas...

Any other ideas or suggestions will be greatly appreciated!

Thanks,
Jason

Note:  If needed I would be willing to give supervised access to my server to help me solve the problem.  Using something like XP's remote assistant or similar.

Some sshd have an 'autologout' feature. Try typing:

unset autologout

immediately after logon (or add it to your shell configuration):

http://www.cs.ucf.edu/~czou/linux/administrator.html#Remove%20%22auto-logout%22%20feature%20for%20SSH%20connection

Otherwise, it may be time to look at cron processes on your server:

crontab -e

It may be a process running every 10 minutes and checking for 'old' and 'stale' connections.

This issue could also apply to any of the more 'advanced' switches between you and the server concerned. Some of the Cisco switches can do something like this, and I know that there are other switches that offer a similar type of function.

Other tests to check on are:

Try creating a symbolic link to sshd called abcd, and run that instead of the sshd. If you use the sysV init process you will need to edit the init script to change it. If your problem goes away after this, then there is a cron (or other process) checking for 'old' sshd connections.

Next try using a different port to port 22....and see what happens. This may also help give an indication of where the problem is.

Do 'normal' logins get disconnected after 1 hour (and the extra up to 10 mins)? If so, then there may be some security enhancements installed to automatically log out anyone connected for over 1 hour?

HTH:)

Hi pjedmond,

I tried "unset autologout" after logging into a session, but it still terminated the session after one hour.

I have already checked all of the crons on the system... I went through /etc/passwd and ran a crontab -l -u <user> on every user and only two users have crons, both of which I am familiar with.  Neither of these two users have any crons running at 10 minute intervals.  I have even tried removing all crons just to be sure, but the connection still terminates.

I have also already tried using a different port 2222 for example, but the sessions still terminate.

This happens everytime for any user.  I'm not really sure what you mean by 'normal' logins, but it always happens no matter what...

This happens even when I am active in the session typing or whatever (vi'ing a file).  It will terminate in the middle of whatever I am doing.

I'm not really sure how to check the hardware between me and the system I am using...  I do know that it happens no matter where I connect from, so I would suspect that if it is hardware, it would have to be hardware at the location of my server -- but my hosting company has assured me there is nothing wrong on their side and they are telling me that I must have something mis-configured on my server.

I tried to create a soft link to sshd and restart it and also just tried copying sshd to another filename and running that, but I can't get it to run under any other name to test this.  Since there are no crons, I don't think the name of the process is what's causing it though.

By 'normal logins', I meant at the console, rather than remotely over ssh, in order to test whether it was a shell security feature rather than an ssh one. If it did, then we could isolate the issue to the shell rather than the sshd.

The fact that the rule is predictable virtually excludes any problem with the hardware, so this is a software or configuration problem.

OK - next trick - Try changing your default login shell to say a csh, and see if the problem still exists.

Also have a look through the /etc/bashrc, and ~/.bash_profile and ~/.bashrc for any autologout settings, or possible try backing up and replacing these files with blank files.

Hopefully from the above, we will be able to identify whether this is an issue with the shell or sshd.

Also try changing the loginfo setting in the sshd conf file to verbose, and see if anything else turns up.

HTH:)

It's been a long time since I used RH 7.2, but also might be worth having in the /etc/ directory for any login.conf type files and examining them.

HTH:)

Hi pjedmond,

Unfortunatly I have no way to test a 'normal login' not using ssh.

I changed the default shell and logged in, but that session still terminiated after one hour.

I also checked all of the ~ and /etc/ bash rc and profile files and found nothing relating to logouts.

I turned on verbose logging, but nothing seems to be showing up in any logs when the sessions terminate.  One time I saw where it regenerated the key after an hour, but that session didn't terminate until a few minutes later (first even 10 minutes), and I didn't see that for the other sessions, so I don't think that was it...

:|

anyone else have any suggestions?  This is still happening...

I have an update... I modified my sshd config to allow direct logins by root and these sessions do NOT get terminated...

However, if I login as any other user on the system, the session will be terminated after one hour.

What could be terminating sessions based on the user who is logged in?

Are there any  ~/.login files?

These files are executed on login, and may include instructions to 'timeout' a connection. These files are normally created by default from the /etc/skel  directory when you add a user. This process does not get used for root as root is created during the initial installation.

(   (()
(`-' _\
 ''  ''

Have you tried the 'standalone' sshd on a different port - lets say 2222? :

sshd -p 2222

And then try ssh to port 2222:

ssh -p 2222 ipaddress

Does it still log you out?

(   (()
(`-' _\
 ''  ''

Hi pj,

I checked for ~/.login files and there are none.  I also reviewed the bashrc and bash_profile files to make sure there was nothing in those that would cause a problem and there was not.

I just started up a standalone sshd on port 2222 and logged in.  I'll let you know in an hour if that makes a difference.

If that does work, what does that indicate?

Thank you,
jason

Hey pj,

No luck with connecting on port 2222 either.  It still disconnected after one hour.  I also had a session logged in as root on 2222 and that one did not disconnect, same as always.  

What would be checking to see which user is logged in and terminating the connection if it is anyone other than root?

Thanks,
jason

OK - this indicates to me that this issue is not sshd related, but possibly Bash related. Therefore:

Edit your /etc/bashrc and change the line that says:

     TMOUT=1800

to

     TMOUT=

This value could also be in the ~/.bashrc or~/.bash_profile

(   (()
(`-' _\
 ''  ''

hmm, well it's not in any of those.

~/.bashrc only contains a line that causes it to load /etc/bashrc/, and the ONLY thing in /etc/bashrc is:

ara:/etc root# cat bashrc
if [ -n "$PS1" ]; then PS1='\h:\w \u\$ '; fi

I will try adding this line to /etc/bashrc/ and see what happens.

I doubt this is it though, since the root .bashrc also loads the same /etc/bashrc, so in the case of bashrc, root should be loading the same one as all other users.

I'll reply back later and let you know if adding TMOUT=1800 made a difference though.

Thanks again!

-- j

>TMOUT=1800

That would make 30 mins....(rather than your 60mins..3600)

TMOUT=0 should stop you being disconnected....although I guess we'll find out in 30 mins:)

Other possibility is that there is some misconfiguration of something like a screensaver?

(   (()
(`-' _\
 ''  ''

..or you could just type:

export TMOUT=0

at the command line when you log in?

(   (()
(`-' _\
 ''  ''

ok, adding TMOUT=1800 did log me out after exactly 30 minutes.

So, I will try setting it to TMOUT=0 now and see if that makes a difference.

Thanks, hope this works! :)

Well, if it doesn't, then you'll get logged out after 0 seconds;) hehehehehehehehe;)

(   (()
(`-' _\
 ''  ''

:)  That didn't happen, it's still logged in.  But I'm waiting for the hour to pass to see if it still logs the account out after one hour.  If it does, I would think that would indicate that there is something else disconnecting the session...

I'll let you know in few minutes, have about 30 minutes to go:

jminton  pts/2        67.32.36.230     Fri Jul  7 16:25   still logged in

bad news... it still terminated the connection after one hour + the number of minutes needed to hit the first even 10 minutes... Just like it's allways done...

jminton  pts/2        67.32.36.230     Fri Jul  7 16:25 - 17:30  (01:04)

So, it was terminated after 1 hour 4 minutes since it needed 4 more minutes to hit an even 10...  Whatever it is, it checks every 10 minutes and kills ssh connections not owned by root...  :(

OK - I think that we've tried virtually everything that is part of sshd and bash, so I reckon that this now has to be part of a 'cron' task. I think that you've already checked whether root has a task that disconnects all other users.

 cat /var/spool/cron/*

should list all of the cron tabs (this is from a RHEL 3 setup and may be a slightly different path for you).

There is likely to be a script somewhere that runs every 10 minutes to disconnect non root users. It may be run as suid, or as guid users or something similar.

Another area to look at is the :

tail -f /var/log/messages

(probably as root, whwn another non-root process is being disconnected)

Another way to try and pin this down is to disable crond and atd:

/etc/init.d/crond stop
/etc/init.d/atd stop

This will at least confirm that these processes are responsible for the disconnection.

(   (()
(`-' _\
 ''  ''

no luck... :(

I had already checked all of the crons since the 10 minute increments led to that in the beginning, but i checked again just to be sure and nothing runs every 10 minutes.  Either way, I went ahead and disabled crond to be sure:

ara:/var/log root# /etc/init.d/crond stop
Stopping crond:                                            [  OK  ]


and I also logged in a root session and tail'd the message log, and at 1 hour 8 minutes at 16:40 it disconnected STILL:
jminton  pts/3        67.32.36.230     Sat Jul  8 15:31 - 16:40  (01:08)

so, whatever it is, it is not a cron.

One of the hunches I had from the start was that it might be one of the daemons... A few I think it could be are:

syswatchd
xinitd
healthd

but I have no idea how to know for sure or know how or why they are doing it...

That solved it, I stopped syswatchd and the sessions no longer disconnect...  I can't find any good reasons to leave it running so I'm going to hide it in /etc/init.d/ so it doesn't start up again.

Thanks for your help!

A pleasure, although I'm still intrigued as to why it should think the ssh process had hung?

(   (()
(`-' _\
 ''  ''

 

Not sure... However, once we narrowed it down to syswatchd I did some searching and found several others who had similar issues with other applications.  I saw were some said if they ran the process as nobody it wouldn't kill those for some reason, but was killing others.

syswatchd on my server is a perl script, but it's been encrypted, so I can't view it.