• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 3902
  • Last Modified:

Unable to access a single Domain Name. ARP timeouts in Sonicwall?

I have a tricky situation, and I've gone through virtually everything I can think of in order to resolve it.  

Starting about a month ago one particular company has been unable to send us e-mails, and neither of us can access the others' websites.  I'm not even sure what info would need to be provided, so I'll post all the relevant information I can think of:

System Info: Both Company's Systems Windows 2003, both running MS Exchange

1) Neither side can access the other's websites.  Lookups provide the IP information, but nothing occurs after that.  The same occurs for pinging (exchange or non-exchange ports).  

2) We can e-mail them, but their e-mails remain stuck in their MS Exchange Queue and eventually die.

3) I cannot telnet to their Exchange port, though I can telnet to the MX record with lowest priority (which does not show as being Microsoft Exchange anyway).  They cannot telnet to our mail server either.  

4) I have checked all firewalls, the main Domain controller, and the router for any rules or restrictions that might be blocking access.  None appears.

5) I've checked both companies for appearing on RBLs, neither appear on any.  

6) Only information I can see in any log, real time or no is an "arp timeout," in my Sonicwall log.  

For instance, if I attempt to access their website, I receive:

03/20/2006     12:58:40.112     ARP timeout     0.0.0.0   x.x.x.xxx (their ip number)

Where 0.0.0.0 is under source and their ip is under destination.

Prior to a month ago both sides could communicate properly, to my knowledge.  Also, nothing has changed on our end in our system setup/layout.

Any help on what I could try next would be greatly appreciated.

Thanks!
0
AdamRobinson
Asked:
AdamRobinson
  • 17
  • 12
  • 8
  • +5
2 Solutions
 
neoponderCommented:
Lets just forget exchange for a minute, if you cannot ping them than you have a network issue, assuming they are not blocking ICMP in their firewall.

If you ping a known good address from your firewall, does it respond? What about a trace route, does it fail? Where?
0
 
AdamRobinsonAuthor Commented:
Neo:

Ping, traceroute, etc. for any other address works correctly for all other sites, from both internally, and from the firewall itself.

If I do:

nslookup
set MX
their domain name

I cannot ping any of the addresses that are listed.  

0
 
AdamRobinsonAuthor Commented:
I should also clarify:  Both sides (ours and theirs) can be accessed normally (ping, http, telnet, mailed) from outside our respective domains.

0
Concerto Cloud for Software Providers & ISVs

Can Concerto Cloud Services help you focus on evolving your application offerings, while delivering the best cloud experience to your customers? From DevOps to revenue models and customer support, the answer is yes!

Learn how Concerto can help you.

 
neoponderCommented:
Do a tracert on the network address and see where it fails.  
0
 
AdamRobinsonAuthor Commented:
Tracing route to IP Address

* * * *  [Request Time Out]

0
 
neoponderCommented:
What about from a workstation from behind the firewall.
0
 
AdamRobinsonAuthor Commented:
Same thing.

0
 
RPPreacherCommented:
We ran into a similar issue which was specifically tied to a Sonicwall firewall.

It seems a recent update from Sonicwall makes Exchange look like a DoS attack.  Specifically, Exchange sends a SYN packets, the sonicwall ACKs, Exchange sends the message and THEN sends the RST packet to close the previous ICMP session.

It should work like this Exchange sends SYN, sonicwall ACKS, Exchange sends RST, Exchange send message.

Anyway, if either site is working with Sonicwall, you need to contact Sonicwall support.  If not, I would still follow this path for troubleshooting.
0
 
RPPreacherCommented:
Oh, missed the title... you are using a sonicwall... hehehe.
Guess I was right.
0
 
neoponderCommented:
Did you verify that they have not accidently posted Private IP's for the DNS servers?  Are they public?

If they are public, telent to their mx/mailserver

telnet (ipaddress) 25
see if you get a response from their mailserver.  They could have ICMP blocked
0
 
AdamRobinsonAuthor Commented:
RPPreacher:

Would this in some way disable our ability to view each other's websites, telnet, and ping?  

Is it possible Sonicwall has somehow blocked all access to/from their Domain without placing it in any log?  

0
 
AdamRobinsonAuthor Commented:
Neoponder:

As I said in the original, I can telnet to their mail server from OUTSIDE of my domain, but cannot telnet from within it.  By the same token, they can telnet to me from outside their domain, but cannot telnet from within it.

0
 
icanhelpCommented:
Have either of you changed your IP's recently, thereby changing your MX's to a internet authority to propogate downward?  It ususally only takes about a few days for the convergence between all DNS servers, however, this sounds like the problem area.

rc
0
 
neoponderCommented:
Either way, it does sound like an issue with the firewall.  It would make sense if the firewall shutdown access from that source for it to behave they way it is.  Your trace is dying at your Firewall, so it's not likley an BGP issue, etc.
0
 
neoponderCommented:
If you cannot telnet, and you are using the IP's, (Not the DNS address) I vote firewall issue.
0
 
AdamRobinsonAuthor Commented:
Neither company has changed IPs.  They recently upgraded to Windows 2003, etc., but that was over 6 months ago, to my knowledge.  

It's a little hard to get information from them as the people involved on their end are... erm... still getting used to...  yeah.  Not wanting to trash anyone.  's just say I have to figure it all out from this side. :/
0
 
RPPreacherCommented:
The Sonicwall marks the IP address as being a source of a denial of service attack.
One of our vendors is having the same issue with us.  They said they are working with Sonicwall on resolution but that the core of the issue is that Sonicwall identifies the Exchange ping without RST as a DoS attack.
0
 
RPPreacherCommented:
Oh, and it won't be by domain name.  It will be by destination IP address.
0
 
AdamRobinsonAuthor Commented:
Neoponder,

Ok.  Can you think of any suggestions on what I may be missing vis-a-vis the firewall?  I've checked every rule, every block list, etc.  I see absolutely nothing regarding their domains or IPs in any of it.  Could this be something hidden under another rule?  

I'm thinking you all may be on the right track given this popped up _randomly,_ but I've gone through every thing in the Sonicwall and found nothing that might indicate that's the issue.

0
 
neoponderCommented:
Plug a workstation in outside your firewall, if you can.  Then do an IP ping. if it works then you know the Firewall is the issue.  Do you manage your upstream router? or is it your ISP's?  If you manage it ping from there.
0
 
AdamRobinsonAuthor Commented:
RPP: Might be misreading you comment at 10:56 AM PST.  I've checked for both domain and IP address in everything as well.  

I will try to get in contact again with Sonicwall and see.  

Any other thoughts as to what I might try?

Appreciate the help as always.
0
 
neoponderCommented:
from an IP perspective, there are only a few things that would prevent packets from flowing from you to them:
1. errent static route redirecting their network to the wrong place
2. Incorrect ACL (either by and admin, or a dynamic one created by a firewall)
3. ISP issue with BGP configuration (which is really the same as 1. but beyond your control, useally)

THere is not much else to look at.  These things must be checked on both sides.  The fact that ping never makes it outside of your firewall leads me to belive that is where the issue is. If that is true the other company would see a trace die at your doorstep, assuming they are not having the same issue.
0
 
RPPreacherCommented:
I was just reviewing my notes from the issue we had.
I still think the tech on the other end is resolving.
I know that it has to do with the Exchange SYN/ACK/RST prior to connection.
The tech on the other end couldn't see a problem in his logs either but I sent our logs to Cisco TAC and isolated the issue to the Sonicwall.
I'm 100% sure this is your problem (the Sonicwall).
0
 
AdamRobinsonAuthor Commented:
Neoponder:

The other side is having the same issue as well, and are not on a Sonicwall.  Neverthless, I am going to dig a bit more into the Sonicwall and attempt to see if I can find anything else.
0
 
icanhelpCommented:
Are both companies running anti-spam front-end?  The reason I ask is that some mailserver security/spam filtering setups require that a reverse pointer record in DNS be shown else it thinks its a spam...can you let me know if this is the case?

rc
0
 
RPPreacherCommented:
You may want to check the domains on both ends filtering rating too
http://www.sonicwall.com/products/cfs.html
0
 
AdamRobinsonAuthor Commented:
RPPreacher:

That first link is great.  Not sure if it will help, but the setup is:

Our side: Sonicwall.  Norton AV 10.0.

Their side: Cisco PIX.  Norton AV 9.0.

Perfect list in your link of affected equipment/software.  :P
0
 
RPPreacherCommented:
We have a PIX 515e, Norton AV 10.
They have Sonicwall, unknown AV.

Mmmmm... curious, some deadly combination of Cisco, Sonicwall and Symantec?  I hate this shtuff.  I need a raise.
0
 
RPPreacherCommented:
If sonicwall gives you a solution, could you post it here?  I would like to help the fellow geek on the other end of our problem.
0
 
icanhelpCommented:
Do they have more than one ip bound to the nic on their exchange?  If they're sonicwall, and they're doing static nat, which mac will it use?
0
 
AdamRobinsonAuthor Commented:
RPPreacher:

If Sonicwall ever bothers to acknowledge my existence with anything other than general trouble ticket IDs, I'd be happy to.

Will return momentarily.
0
 
Danny_LaroucheCommented:
As per what Adam`s post it has nothing to firewall(ACL) issue since no "blocked trafic" appear in its log. The SYN RST issue isnt possible because the same problem occure using ICMP protocol. My though is that the problem is related to MTU or routing problem on either ISP side.

A good test would be to ask then to use denied port and see if denied trafic are logged or not. You may use sniffer in front of the firewall to doublecheck.  Did you tried a traceroute?  Are you linked to the same ISP? I ever see wrong netmask causing problem between users using adjacent netblock

Does the lowest MX is locate on the same subnet?
0
 
RPPreacherCommented:
>The SYN RST issue isnt possible because the same problem occure using ICMP protocol

Nice try.
A ping does SYN/ACK/RST
The Exchange connection does SYN/ACK/message/RST.  It is the message in the midst of the ACK/RST that causes the Sonicwall to think it is a DoS (at least that was Sonicwall's explanation and Cisco TAC concurred based on our PIX packet captures)

<sarcasm>But I'm certain that Cisco and Sonicwall are both wrong and you are right</sarcasm>
0
 
AdamRobinsonAuthor Commented:
Sonicwall suggested a Firmware upgrade, but can't get to fixing my account (I inherited this system and wasn't left the account information and have no way to contact who did it) until likely the end of the day.  Hate to leave a question going, but will have to try it tomorrow I guess..

0
 
jli168Commented:
Try adding the smtp rule to their exchange server.
0
 
dhoustonieCommented:
What is the MTU size on your router?
I ahve known some domains to not be able to be accessed due to MTU size incompatability.
I have heard of MTU size causing problems with emails coming through as well so if you can check in the sonicwall os for the info and post it here?


David Houston
0
 
Danny_LaroucheCommented:
RPPreacher: syn/ask (3 handshake process) belong to TCP protocol (connection based), not ICMP & UDP that are connectionless. many application using thoses protocols doesnt have to reply anyway.
0
 
CGretskiCommented:
Do their IP's resolve the same internally and externally ( are you hosting internal dns for their domain/ does your ISP's DNS have invalid records for them ) also, the IP's they resolve to - could they be covered by your subnet - and the traffic to them never attempting to leave your subnet.

Other than that, do you have a static route to their subnet which is pointing elsewhere?
0
 
AdamRobinsonAuthor Commented:
to jli168:

1) It is not just e-mails that are not working.  SMTP rules to exchange server won't (tried) fix the systemic problem.

to Dhoustonie:

Can the MTU size on the router change on the fly?  We had no problems until a month ago.  Nothing has changed on our end.  Again, it is not just e-mails that won't come through.

to CGretski:

IPs resolve the same internally and externally.  ISP's DNS has valid records for them.  IPs resolved to are not covered by our subnet.  Do not have a static route to their subnet pointing elsewhere.


Still awaiting Sonicwall's response.  They're being hideously slow.

0
 
dhoustonieCommented:
A route change, a firmware upgrade, yes it is possible.

David
0
 
RPPreacherCommented:
>They're being hideously slow.

Same experience here
0
 
Danny_LaroucheCommented:
What firmware version do you need and for what model?
0
 
AdamRobinsonAuthor Commented:
Danny,

Any more recent firmware upgrade for Sonicwall, and a TZ-170.

Sonicwall is giving me an insane runaround (talked to 6 people now, been requested to send the same information 3 times).

0
 
AdamRobinsonAuthor Commented:
Update:

It's most certainly an issue with the Sonicwall TZ-170.  I've gotten ping/http to work from our end now, but had to severely mess with some of the tables.  The odd part is nothing had been edited for months and this was working up until a month ago or so.  I will try to post back more as I get it figured out.

0
 
RPPreacherCommented:
Thanks for the update.  Was glad to help.
0
 
dhoustonieCommented:
What is the mtu size on the Sonicwall?

David
0
 
AdamRobinsonAuthor Commented:
Problem was the following, and why it was so hard to track down:

Sonicwall detected an attack (that wasn't an attack) and appears to have added a rule that DID NOT appear in the Access Rule sheet (we're on an older version of the firmware, so it's possible that it may appear in newer versions).

I essentially ended up having to completely ignore what the tables actually _said,_ and add extra access rules for all of the possible IPs they use (not fun, as they use quite a few :/ ).

The tricky part, and why it was hard to pin down, was it wasn't killing just a single IP, but a huge range: X.X.0.0

As far as I can tell, there's no way to make the Sonicwall NOT do this without upgrading to the most recent firmware, which I've given up on given their inability to work with me in any way.

Thanks to everyone for their help, especially RPPreacher.  Had a hard time dividing up the points, but put them where two commenters nearly nailed the issue.

Thanks,

Adam
 
0

Featured Post

 The Evil-ution of Network Security Threats

What are the hacks that forever changed the security industry? To answer that question, we created an exciting new eBook that takes you on a trip through hacking history. It explores the top hacks from the 80s to 2010s, why they mattered, and how the security industry responded.

  • 17
  • 12
  • 8
  • +5
Tackle projects and never again get stuck behind a technical roadblock.
Join Now