ssh issues to router

I will get a chance to get into the router via console. The router is at another state. For some reason, I am getting "connection refused" errors when trying to ssh in. It works just fine after a reboot, but after few times or few hours it will fail again with the above error. It happens daily. The only way to get back in is if I reboot the router.

The VTY lines look normal
line vty 0 4                  
 session-timeout 40                  
 exec-timeout 40            
 password 7 xxxxxxxxxxxx
 transport input ssh                  

I also have this:
service tcp-keepalives-in                        
service tcp-keepalives-out                        
service timestamps debug datetime msec                        
service timestamps log datetime msec                        
service password-encryption                        
no service password-recovery                        

Anything I should be checking for when I get into the console?
- I will check if the certificate is present
- If there are hung sessions

Anything else I could check to verify? There is no acl's on the vty lines as you can see, not sure what else to look for.

thank you