Solved

ssh suddenly not working

Posted on 2015-02-17
16
106 Views
Last Modified: 2015-09-08
All of a sudden I can no longer ssh to a host from outside the lan. I can ssh just fine inside the lan. I've been ssh'ing to this client host for over 5 years w/o problem. I must have done something to the network configuration, but I can't figure out what.

ssh webserver.ohprs.org

from my offsite workstation hangs forever (or a very long time). Running tcpdump on the target host gives:
$ tcpdump -n -l -tttt -i eth0 src 96.11.168.98
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
2015-02-18 02:15:13.345745 IP 96.11.168.98.40761 > 64.129.23.95.22: Flags [S], seq 139040590, win 4380, options [mss 1460,sackOK,TS val 2297114450 ecr 0,nop,wscale 6], length 0
2015-02-18 02:15:13.361599 IP 96.11.168.98.40761 > 64.129.23.95.22: Flags [.], ack 1922026094, win 69, options [nop,nop,TS val 2297114472 ecr 4294808446], length 0
2015-02-18 02:15:14.535933 IP 96.11.168.98.40760 > 64.129.23.95.22: Flags [F.], seq 3539628562, ack 1055638177, win 69, options [nop,nop,TS val 2297115648 ecr 4294751659], length 0

Open in new window

/var/log/messages has:
Feb 18 02:16:11 webserver sshd[2504]: Did not receive identification string from 96.11.168.98

Open in new window

All firewalls are off.

Also, I can't ssh to this host from other hosts outside the lan, not just 96.11.168.98

Also can't show web pages on this host (80,443), yet I do get response via telnet:
$ telnet webserver.ohprs.org 22
Trying 64.129.23.95...
Connected to webserver.ohprs.org.
Escape character is '^]'.
^]
telnet> quit
Connection closed.

$ telnet webserver.ohprs.org 80
Trying 64.129.23.95...
Connected to webserver.ohprs.org.
Escape character is '^]'.
^]
telnet> quit
Connection closed.

Open in new window


I need HELP. Any ideas?
0
Comment
Question by:jmarkfoley
  • 9
  • 7
16 Comments
 
LVL 25

Expert Comment

by:Zephyr ICT
Comment Utility
What does it say when you ssh with the -vvv switch, any relevant information?
Are you using keys or password for the ssh connection?
Are you using something like fail2ban?
0
 
LVL 1

Author Comment

by:jmarkfoley
Comment Utility
What does it say when you ssh with the -vvv switch, any relevant information?
> ssh -vvv mfoley@64.129.23.95
OpenSSH_5.8p1, OpenSSL 0.9.8r 8 Feb 2011
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: ssh_connect: needpriv 0
debug1: Connecting to 64.129.23.95 [64.129.23.95] port 22.
debug1: Connection established.
debug3: Incorrect RSA1 identifier
debug3: Could not load "/user/mfoley/.ssh/id_rsa" as a RSA1 public key
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug2: key_type_from_name: unknown key type '-----END'
debug3: key_read: missing keytype
debug1: identity file /user/mfoley/.ssh/id_rsa type 1
debug1: identity file /user/mfoley/.ssh/id_rsa-cert type -1
debug1: identity file /user/mfoley/.ssh/id_dsa type -1
debug1: identity file /user/mfoley/.ssh/id_dsa-cert type -1
debug1: identity file /user/mfoley/.ssh/id_ecdsa type -1
debug1: identity file /user/mfoley/.ssh/id_ecdsa-cert type -1

Open in new window

Are you using keys or password for the ssh connection?
keys
Are you using something like fail2ban?
no

Here's one where I am using password:
> ssh -vvv mfoley@webserver.ohprs.org
OpenSSH_6.7p1, OpenSSL 1.0.1j 15 Oct 2014
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: ssh_connect: needpriv 0
debug1: Connecting to webserver.ohprs.org [64.129.23.95] port 22.
debug1: Connection established.
debug1: identity file /home/mfoley/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mfoley/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mfoley/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mfoley/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mfoley/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mfoley/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mfoley/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mfoley/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.7

Open in new window

0
 
LVL 25

Expert Comment

by:Zephyr ICT
Comment Utility
Hmmm, did you play with tcp_wrappers or other security settings, for example changed rights on the user folder where the ssh keys are located?

Try resetting the security on your .ssh folder with something like this

chmod go-rwx ~./ssh

Open in new window



It looks similar to this as well.
0
 
LVL 25

Expert Comment

by:Zephyr ICT
Comment Utility
Also, check your logs if possible, maybe they can shed some light on the problem, should be in the /var/log/secure log I think ...
0
 
LVL 1

Author Comment

by:jmarkfoley
Comment Utility
spravtek:
Hmmm, did you play with tcp_wrappers or other security settings, for example changed rights on the user folder where the ssh keys are located?
I've done nothing that I know of on this host. I did change the domain name server, but same IP.
Try resetting the security on your .ssh folder with something like this
Current .ssh folder settings (again, I haven't changed these).
>ls -ld .ssh
drwx------ 2 mfoley users 4096 2012-04-05 11:52 .ssh/

Open in new window

Also, check your logs if possible, maybe they can shed some light on the problem, should be in the /var/log/secure log I think ...
I posted message log and tcpdump log in original posting. /var/log/messages has nothing. I even tried again.

Note that I can ssh to another computer in the same room from outside:  64.129.23.170, which also happens to be the domain name server.
0
 
LVL 1

Author Comment

by:jmarkfoley
Comment Utility
no pressure, but this is a big emergency in the office. Cannot ssh (not that big a deal), or access websites from the outside. Can do so from inside the office and from inside the building (building is managed by ISP). No idea what to do. Members cannot get to the website. HELP
0
 
LVL 25

Expert Comment

by:Zephyr ICT
Comment Utility
Hi, sorry for the late reply, busy life and all ...

Well ... It's a tricky thing to troubleshoot remotely, you mentioned you changed something in the network settings, what did you change, if you remember? Gateway, subnetmask, ??

So, you can't access the remote server at all? Makes it even harder to troubleshoot naturally. Without being able to see what is configured on the server it will be a big problem to troubleshoot ...
0
 
LVL 1

Author Comment

by:jmarkfoley
Comment Utility
Thanks for getting back -- sorry, I wasn't being critical of you with the "hurry up" thing ...

More information:

This host is a dual-NIC web server, eth0 is Internet facing at 64.129.23.95, eth1 on LAN at 192.168.0.3.
This host nfs mounts a partition from 192.168.0.5
The domain controller/DNS nameserver is at 192.168.0.2

I have the following set:

/usr/sbin/iptables --table nat --append POSTROUTING --out-interface eth0 -j MASQUERADE
/usr/sbin/iptables --append FORWARD --in-interface eth1 -j ACCEPT

When I have eth1 set to get it's IP address via dhcp, I can get to all the LAN hosts, and do the NFS mount, but cannot access the host from outside the LAN via 64.129.23.95. Locally, I can `host` to other local LAN hosts:
$ host mail
mail.hprs.local has address 192.168.0.2                         # this is the DC/DNS/nameserver

$ host ohprsstorage                                                           # this is the host with nfs partition
OHPRSstorage.hprs.local has address 192.168.0.5

Open in new window

i
When I have eth1 set to a static IP I can get to the host from the outside (via 64.129.23.95) at 22, 80 and 443, but the host cannot see any LAN host, including the nameserver:
$ host mail
Host mail not found: 3(NXDOMAIN)

$ host ohprsstorage
Host ohprsstorage not found: 3(NXDOMAIN)

Open in new window

ifconfig is:
eth0      Link encap:Ethernet  HWaddr 00:C0:A8:7B:93:1F
          inet addr:64.129.23.95  Bcast:64.129.23.127  Mask:255.255.255.192
          inet6 addr: fe80::2c0:a8ff:fe7b:931f/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2192 errors:0 dropped:0 overruns:0 frame:0
          TX packets:166 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:174090 (170.0 Kb)  TX bytes:59951 (58.5 Kb)
          Interrupt:17 Base address:0xd000

eth1      Link encap:Ethernet  HWaddr 60:A4:4C:61:9C:FE
          inet addr:192.168.0.3  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::62a4:4cff:fe61:9cfe/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:325 errors:0 dropped:0 overruns:0 frame:0
          TX packets:43 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:23237 (22.6 Kb)  TX bytes:3826 (3.7 Kb)
          Interrupt:49 Base address:0xa000

Open in new window

In both cases /etc/resolv.conf is

domain hprs.local
nameserver 192.168.0.2
nameserver 66.193.88.2

In both cases the default gateway is: 64.129.23.65

Summary: in the eth1/DHCP configuration I can resolve local LAN hosts, but cannot get to the host from the outside.
In the eth1/static configuration I can get to the hosts from the outside, but can't resolve local LAN hosts.

Does this tell you anything?
0
Windows Server 2016: All you need to know

Learn about Hyper-V features that increase functionality and usability of Microsoft Windows Server 2016. Also, throughout this eBook, you’ll find some basic PowerShell examples that will help you leverage the scripts in your environments!

 
LVL 25

Expert Comment

by:Zephyr ICT
Comment Utility
Ok, a dual-homed system, do you have ip-forwarding set up? Probably, since I see this in IP-tables, but just double-checking, usuallt it's enabled by doing following: "echo 1 > /proc/sys/net/ipv4/ip_forwarding"

Though it could be that after a reboot the ip_forwarding is disabled again, might need to add it to sysctl.conf to make it permanent.

Also, I would just set fixed IP-addresses on both interfaces, and enable ip-forwarding (if not set), so 1 interface you'd have the fixed public address (obviously) and on the other nic set a fixed IP in the LAN range.

I'll have another thinking session after my espresso shot or two.
0
 
LVL 1

Author Comment

by:jmarkfoley
Comment Utility
spravtek:
do you have ip-forwarding set up?
Yes:
$ cat /proc/sys/net/ipv4/ip_forward
1

Open in new window

Also, I would just set fixed IP-addresses on both interfaces, and enable ip-forwarding (if not set), so 1 interface you'd have the fixed public address (obviously) and on the other nic set a fixed IP in the LAN range.
That's exactly what I've come up with for the moment, but the problem is that for some reason the name server at 192.168.0.2 no longer resolves LAN hosts w/o FDQN:
$ host mail
Host mail not found: 3(NXDOMAIN)

$ host mail.hprs.local
mail.hprs.local has address 192.168.0.2

$ host charmaine
Host charmaine not found: 3(NXDOMAIN)

$ host charmaine.hprs.local
CHARMAINE.hprs.local has address 192.168.0.52

Open in new window

Which means I need to make entries in /etc/hosts for local hosts I need to get to (or reference them by FDQN in scripts)
0
 
LVL 1

Author Comment

by:jmarkfoley
Comment Utility
Here's a possible issue, in /etc/HOSTS (this is Slackware) I have:

    webserver.ohprs.org

whereas maybe it should be

    webserver.hprs.local

Not really sure the domainname part is used anywhere. I see the hostname part being parsed off in the rc.M script. Also, in /etc/hosts I have:

192.168.0.3             webserver.ohprs.org webserver.ohprs.local webserver

Maybe I should get rid of the "webserver.ohprs.org" alias. What do you think? I can try these variations after people leave the office today.
0
 
LVL 25

Expert Comment

by:Zephyr ICT
Comment Utility
Yeah, I don't see the need for the external address to be resolvable on the LAN, so that can go, the outside DNS is enough to make the website reachable isn't it ...
0
 
LVL 1

Author Comment

by:jmarkfoley
Comment Utility
I've been dabbling with Linux networking and iptables for a couple of years now, but this one has me totally flummoxed! I changed /etc/hosts to:

127.0.0.1               localhost
127.0.0.1               webserver.hprs.local
192.168.0.10            HP1102wMICR

/etc/HOSTNAME is:

webserver.hprs.local

and set eth1 back to DHCP instead of static. I killed the firewall completely and only set the following:

    /usr/sbin/iptables --table nat --append POSTROUTING --out-interface eth0 -j MASQUERADE
    /usr/sbin/iptables --append FORWARD --in-interface eth1 -j ACCEPT

Again, all LAN-side things worked fine; I could resolve hostnames for all LAN workstations. But again, absolutely no connectablily to 64.129.23.95 via ports 22, 80 or 443 from outside the LAN. Still getting the same tcpdump output and /var/log/messages entry as in my initial post. I do not get why I cannot ssh to this computer when using DHCP on eth1!? What does DHCP even have to do with ssh, esp. when ssh'ing to eth0? Aaaarrrggghhh!
0
 
LVL 25

Accepted Solution

by:
Zephyr ICT earned 500 total points
Comment Utility
But it does work when you set fixed IP on both interfaces? I think it's an internal thing, probably the ip-forwarding or some other magic isn't working when using DHCP, which is kinda logical if you think about it, though I understand the frustration :)

I would also add the webserver.hprs.local with it's fixed IP in the host file.
0
 
LVL 1

Author Comment

by:jmarkfoley
Comment Utility
Have not had a change to get back to this issue yet. Other fires to put out. Will try to do so in the next couple of days.
0
 
LVL 1

Author Comment

by:jmarkfoley
Comment Utility
Did a bunch of other things in the meantime, not purposefully related: upgraded OS, changed some FORWARD/POSTROUTING info in iptables, removed adding a default gateway for eth1, etc.

Whatever I did, it's back to working OK. Don't really know which thing or combination of things solved the problem. Wish I did so I don't accidently undo it!
0

Featured Post

Threat Intelligence Starter Resources

Integrating threat intelligence can be challenging, and not all companies are ready. These resources can help you build awareness and prepare for defense.

Join & Write a Comment

I have seen several blogs and forum entries elsewhere state that because NTFS volumes do not support linux ownership or permissions, they cannot be used for anonymous ftp upload through the vsftpd program.   IT can be done and here's how to get i…
Note: for this to work properly you need to use a Cross-Over network cable. 1. Connect both servers S1 and S2 on the second network slots respectively. Note that you can use the 1st slots but usually these would be occupied by the Service Provide…
This video gives you a great overview about bandwidth monitoring with SNMP and WMI with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're looking for how to monitor bandwidth using netflow or packet s…
Polish reports in Access so they look terrific. Take yourself to another level. Equations, Back Color, Alternate Back Color. Write easy VBA Code. Tighten space to use less pages. Launch report from a menu, considering criteria only when it is filled…

772 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

14 Experts available now in Live!

Get 1:1 Help Now