Solved

Riverbed Steelhead no longer showing up as a peer and is not optimizing traffic

Posted on 2013-05-21
21
3,039 Views
Last Modified: 2013-05-23
I have five steel heads. Four 250's and a 550 at the HQ. Everything was working fine until a few days ago. The four 250's all show up as each others peers. The 550 is not peering with any of the other SH's. I double checked the firewall to confirm the commands needed to allow peering to occur are there.

How can I trouble shoot this and how can I figure out why the SH's are not peering?

Thanks,

Justin
0
Comment
Question by:JustinGSEIWI
  • 12
  • 8
21 Comments
 
LVL 20

Expert Comment

by:agonza07
Comment Utility
Did anything change? Have you tried restarting the unit?

Are the in-path IPs pingable?
0
 
LVL 10

Expert Comment

by:wdurrett
Comment Utility
If you want to test, you can put in a static in-path rule from the 550 to the 250.  Put a reverse rule from the 250 to the 550.  See if any of the traffic gets picked up and optimized.  If not, there is some kind of communication block between the units.
0
 

Author Comment

by:JustinGSEIWI
Comment Utility
I can't think of anything that changed that may cause this although this is most likely the case and I am just not seeing the reason. I restarted the unit many times and all IP's are pingable.

I attached a pic of the in path rule page. Can you let me know the specific settings I need to set to test this?

Thanks,

Justin
sh.JPG
0
 
LVL 10

Expert Comment

by:wdurrett
Comment Utility
Insert a rule at spot 5.  Make it a fixed target rule.    Then fill in your in-path IP for the two steelheads.  The defaults for the port should be fine.

What type of firewalls do you have?  Most likely, the probes are being blocked.
0
 

Author Comment

by:JustinGSEIWI
Comment Utility
I added the in path rule as you described and they immediately peered. How do I trouble shoot why the auto peering is not working?

I have Cisco ASA's at each site and I configured them to allow probes a long time ago. I checked the config and the allow probes config is still there.

Thanks,

Justin
inpath.JPG
0
 
LVL 10

Expert Comment

by:wdurrett
Comment Utility
Take a look at the logging on your ASA and see if you can see the probes being blocked.  If you have a site-to-site VPN tunnel, you can try tearing down the tunnel to reset it.  The command is:

clear crypto isakmp sa

***Note: this will break the connection between your sites for several seconds while the tunnel is reset.

I have the exact same setup and one time I had to reset the tunnels for some odd reason.  After the reset, everything worked perfectly.

If you have a service contract, open a TAC case with Cisco.  The techs can help you troubleshoot the ASA.
0
 

Author Comment

by:JustinGSEIWI
Comment Utility
Thanks, I will look into that but I can only do that after hours.

I added one of the 250's to an in path rule for the 550 and it worked. So I tried adding the rest with an in path rule the same way and they are not peering still. Any thoughts on that? Anything we can try until I can try the clear crypto command?

Thanks,

Justin
0
 

Author Comment

by:JustinGSEIWI
Comment Utility
After I added the in path rules, only one steelhead peered with the one in HQ. The others did not. Also, after I added the in path rule, the Steelheads broke our phone system and we were unable to make internal calls and transfer calls. So I had to remove the in path rules.

I will try the clear crypto command soon. Is there anything else in the mean time? Trying to do the in path rules only worked for one SH and if I do that again, I need to make sure it does not intercept the VOIP traffic.

thanks,

Justin
0
 
LVL 10

Expert Comment

by:wdurrett
Comment Utility
HI Justin.

I am surprised that it broke your VoIP calls.  If you want to try again, you can insert an in-path exclude rule in spot 5 that will pass-through the VoIP traffic.  Then your other in-path rules would go in the spots below 5 to pick up other traffic.

Did you get a chance to reset the tunnels?

Wes
0
 

Author Comment

by:JustinGSEIWI
Comment Utility
I didn't get a chance to do the tunnel yet. I am going to try and talk with Cisco about it today.

How do I exclude VOIP traffic? How can I check any exclusions I may have now and make sure they are excluded when I add the in path rule? I know I have excluded traffic such as DFS and VOIP.

thanks,

Justin
0
Enabling OSINT in Activity Based Intelligence

Activity based intelligence (ABI) requires access to all available sources of data. Recorded Future allows analysts to observe structured data on the open, deep, and dark web.

 
LVL 10

Expert Comment

by:wdurrett
Comment Utility
Justin,

If you have things setup nicely, your VoIP traffic should be on a separate subnet.  Simply exclude that subnet and mark it as pass-through.  For instance, you would pass-through anything originating from the 10.10.2.0/24 subnet (if that was the subnet for VoIP).  Remember to put an exclusion in on both the 550 and the 250s.

If not,  need to define it and then exclude it.  Riverbed support can help out with this if you need to have them look at your environment.
0
 

Author Comment

by:JustinGSEIWI
Comment Utility
Cisco says they don't know of a "clear crypto isakmp sa" command. They did find information on a "clear crypto ipsec sa" command and a "clear isakmp sa" command. Those appear to be different. Are you sure the command you provided is for the ASA? What are your thoughts on the other commands?

Below is what Cisco said.

"

On this link you will be able to see information related to the “clear crypto ipsec sa” command:

http://www.cisco.com/en/US/docs/security/asa/asa82/command/reference/c3.html#wp2164191

It does mention that it will take down the IPSec peer connections, it is expected for them to come back up as soon as possible.

On this link you will be able to see information related to the “clear isakmp sa” command:

http://www.cisco.com/en/US/docs/security/asa/asa82/command/reference/c3.html#wp2166543

It mentions that it will clear the runtime SA database information but there is nothing related to connections or anything network impacting.  


"

Thanks,

Justin
0
 

Author Comment

by:JustinGSEIWI
Comment Utility
I just noticed your last comment.

My VOIP traffic is on the same subnet as the rest of my traffic. I will look at exluding the traffic but I am not familiar enough with it to know off the top of my head.

We let riverbed support lapse b/c of the great cost. That is why I am here asking questions.

Thanks,


Justin
0
 

Author Comment

by:JustinGSEIWI
Comment Utility
How do I force the SH's to peer but exclude DFS and VOIP traffic? Also, after I added the first in path rule, the subsequent in path rules did not peer the devices. Since that didn't work, I need an alternative method or need to troubleshoot.

thanks,

Justin
0
 
LVL 10

Expert Comment

by:wdurrett
Comment Utility
Justin,

You will need to define the traffic that you want to exclude.  Under Configure> Networking> Port Labels you can define a new Label and add the ports for that traffic.  Then in your in-path rules, you will add a rule to exclude it.  Please remember that rules are processed in order.

If you look at traffic between your sites under Reports> Networking> Current connections, is it all being passed through?

-Wes
0
 

Author Comment

by:JustinGSEIWI
Comment Utility
All traffic is being passed through since the SH at the Head Quarters office is not peering with the branch offices. There are many ports that the phone system uses so it look like it will be a pain to add but i'll try it.
0
 

Author Comment

by:JustinGSEIWI
Comment Utility
How come auto peering would suddenly stop working? I double checked the fireall and the correct commands to allow probes are in each firewall. It worked great until we had a power outage and then they no longer peer. However, the SH that won't peer appears to be online and functioning fine.

thanks,

Justin
0
 
LVL 10

Accepted Solution

by:
wdurrett earned 500 total points
Comment Utility
Sounds like you had a change in your firewalls when the rebooted.  Perhaps the running config was not saved?

You should contact Cisco TAC or Riverbed support.  They will be able to help you troubleshoot it if you cannot look at the logs on your FW and check the probe traffic.
0
 

Author Comment

by:JustinGSEIWI
Comment Utility
The Cisco config was saved and I can confirm the probes configuration is correct. I will check with Cisco when they call me back but I do not have riverbed support. Do you have any other suggestions? Otherwise I will escalate this question.

Thanks,

Justin
0
 
LVL 10

Expert Comment

by:wdurrett
Comment Utility
Justin,

You need to find out why your firewalls are blocking the probes.  Until that is solved, you will have issues.

-Wes
0
 

Author Closing Comment

by:JustinGSEIWI
Comment Utility
I finally got a hold of Cisco and they confirmed that the global policy was not enabled. I did not realize this. The allow probes commands were in the global policy. As soon as we enabled it with service-policy global_policy global, all of the steelheads peered.

Thanks  for your help.

Justin
0

Featured Post

6 Surprising Benefits of Threat Intelligence

All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

Join & Write a Comment

This is the first one of a series of articles I’ll be writing to address technical issues that are always referred to as network problems. The network boundaries have changed, therefore having an understanding of how each piece in the network  puzzl…
Quality of Service (QoS) options are nearly endless when it comes to networks today. This article is merely one example of how it can be handled in a hub-n-spoke design using a 3-tier configuration.
Viewers will learn how to properly install and use Secure Shell (SSH) to work on projects or homework remotely. Download Secure Shell: Follow basic installation instructions: Open Secure Shell and use "Quick Connect" to enter credentials includi…
In this tutorial you'll learn about bandwidth monitoring with flows and packet sniffing with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're interested in additional methods for monitoring bandwidt…

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now