Go Premium for a chance to win a PS4. Enter to Win

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 3692
  • Last Modified:

Riverbed Steelhead no longer showing up as a peer and is not optimizing traffic

I have five steel heads. Four 250's and a 550 at the HQ. Everything was working fine until a few days ago. The four 250's all show up as each others peers. The 550 is not peering with any of the other SH's. I double checked the firewall to confirm the commands needed to allow peering to occur are there.

How can I trouble shoot this and how can I figure out why the SH's are not peering?

Thanks,

Justin
0
JustinGSEIWI
Asked:
JustinGSEIWI
  • 12
  • 8
1 Solution
 
agonza07Commented:
Did anything change? Have you tried restarting the unit?

Are the in-path IPs pingable?
0
 
wdurrettCommented:
If you want to test, you can put in a static in-path rule from the 550 to the 250.  Put a reverse rule from the 250 to the 550.  See if any of the traffic gets picked up and optimized.  If not, there is some kind of communication block between the units.
0
 
JustinGSEIWIAuthor Commented:
I can't think of anything that changed that may cause this although this is most likely the case and I am just not seeing the reason. I restarted the unit many times and all IP's are pingable.

I attached a pic of the in path rule page. Can you let me know the specific settings I need to set to test this?

Thanks,

Justin
sh.JPG
0
Prepare for an Exciting Career in Cybersecurity

Help prevent cyber-threats and provide solutions to safeguard our global digital economy. Earn your MS in Cybersecurity. WGU’s MSCSIA degree program curriculum features two internationally recognized certifications from the EC-Council at no additional time or cost.

 
wdurrettCommented:
Insert a rule at spot 5.  Make it a fixed target rule.    Then fill in your in-path IP for the two steelheads.  The defaults for the port should be fine.

What type of firewalls do you have?  Most likely, the probes are being blocked.
0
 
JustinGSEIWIAuthor Commented:
I added the in path rule as you described and they immediately peered. How do I trouble shoot why the auto peering is not working?

I have Cisco ASA's at each site and I configured them to allow probes a long time ago. I checked the config and the allow probes config is still there.

Thanks,

Justin
inpath.JPG
0
 
wdurrettCommented:
Take a look at the logging on your ASA and see if you can see the probes being blocked.  If you have a site-to-site VPN tunnel, you can try tearing down the tunnel to reset it.  The command is:

clear crypto isakmp sa

***Note: this will break the connection between your sites for several seconds while the tunnel is reset.

I have the exact same setup and one time I had to reset the tunnels for some odd reason.  After the reset, everything worked perfectly.

If you have a service contract, open a TAC case with Cisco.  The techs can help you troubleshoot the ASA.
0
 
JustinGSEIWIAuthor Commented:
Thanks, I will look into that but I can only do that after hours.

I added one of the 250's to an in path rule for the 550 and it worked. So I tried adding the rest with an in path rule the same way and they are not peering still. Any thoughts on that? Anything we can try until I can try the clear crypto command?

Thanks,

Justin
0
 
JustinGSEIWIAuthor Commented:
After I added the in path rules, only one steelhead peered with the one in HQ. The others did not. Also, after I added the in path rule, the Steelheads broke our phone system and we were unable to make internal calls and transfer calls. So I had to remove the in path rules.

I will try the clear crypto command soon. Is there anything else in the mean time? Trying to do the in path rules only worked for one SH and if I do that again, I need to make sure it does not intercept the VOIP traffic.

thanks,

Justin
0
 
wdurrettCommented:
HI Justin.

I am surprised that it broke your VoIP calls.  If you want to try again, you can insert an in-path exclude rule in spot 5 that will pass-through the VoIP traffic.  Then your other in-path rules would go in the spots below 5 to pick up other traffic.

Did you get a chance to reset the tunnels?

Wes
0
 
JustinGSEIWIAuthor Commented:
I didn't get a chance to do the tunnel yet. I am going to try and talk with Cisco about it today.

How do I exclude VOIP traffic? How can I check any exclusions I may have now and make sure they are excluded when I add the in path rule? I know I have excluded traffic such as DFS and VOIP.

thanks,

Justin
0
 
wdurrettCommented:
Justin,

If you have things setup nicely, your VoIP traffic should be on a separate subnet.  Simply exclude that subnet and mark it as pass-through.  For instance, you would pass-through anything originating from the 10.10.2.0/24 subnet (if that was the subnet for VoIP).  Remember to put an exclusion in on both the 550 and the 250s.

If not,  need to define it and then exclude it.  Riverbed support can help out with this if you need to have them look at your environment.
0
 
JustinGSEIWIAuthor Commented:
Cisco says they don't know of a "clear crypto isakmp sa" command. They did find information on a "clear crypto ipsec sa" command and a "clear isakmp sa" command. Those appear to be different. Are you sure the command you provided is for the ASA? What are your thoughts on the other commands?

Below is what Cisco said.

"

On this link you will be able to see information related to the “clear crypto ipsec sa” command:

http://www.cisco.com/en/US/docs/security/asa/asa82/command/reference/c3.html#wp2164191 

It does mention that it will take down the IPSec peer connections, it is expected for them to come back up as soon as possible.

On this link you will be able to see information related to the “clear isakmp sa” command:

http://www.cisco.com/en/US/docs/security/asa/asa82/command/reference/c3.html#wp2166543 

It mentions that it will clear the runtime SA database information but there is nothing related to connections or anything network impacting.  


"

Thanks,

Justin
0
 
JustinGSEIWIAuthor Commented:
I just noticed your last comment.

My VOIP traffic is on the same subnet as the rest of my traffic. I will look at exluding the traffic but I am not familiar enough with it to know off the top of my head.

We let riverbed support lapse b/c of the great cost. That is why I am here asking questions.

Thanks,


Justin
0
 
JustinGSEIWIAuthor Commented:
How do I force the SH's to peer but exclude DFS and VOIP traffic? Also, after I added the first in path rule, the subsequent in path rules did not peer the devices. Since that didn't work, I need an alternative method or need to troubleshoot.

thanks,

Justin
0
 
wdurrettCommented:
Justin,

You will need to define the traffic that you want to exclude.  Under Configure> Networking> Port Labels you can define a new Label and add the ports for that traffic.  Then in your in-path rules, you will add a rule to exclude it.  Please remember that rules are processed in order.

If you look at traffic between your sites under Reports> Networking> Current connections, is it all being passed through?

-Wes
0
 
JustinGSEIWIAuthor Commented:
All traffic is being passed through since the SH at the Head Quarters office is not peering with the branch offices. There are many ports that the phone system uses so it look like it will be a pain to add but i'll try it.
0
 
JustinGSEIWIAuthor Commented:
How come auto peering would suddenly stop working? I double checked the fireall and the correct commands to allow probes are in each firewall. It worked great until we had a power outage and then they no longer peer. However, the SH that won't peer appears to be online and functioning fine.

thanks,

Justin
0
 
wdurrettCommented:
Sounds like you had a change in your firewalls when the rebooted.  Perhaps the running config was not saved?

You should contact Cisco TAC or Riverbed support.  They will be able to help you troubleshoot it if you cannot look at the logs on your FW and check the probe traffic.
0
 
JustinGSEIWIAuthor Commented:
The Cisco config was saved and I can confirm the probes configuration is correct. I will check with Cisco when they call me back but I do not have riverbed support. Do you have any other suggestions? Otherwise I will escalate this question.

Thanks,

Justin
0
 
wdurrettCommented:
Justin,

You need to find out why your firewalls are blocking the probes.  Until that is solved, you will have issues.

-Wes
0
 
JustinGSEIWIAuthor Commented:
I finally got a hold of Cisco and they confirmed that the global policy was not enabled. I did not realize this. The allow probes commands were in the global policy. As soon as we enabled it with service-policy global_policy global, all of the steelheads peered.

Thanks  for your help.

Justin
0

Featured Post

Visualize your virtual and backup environments

Create well-organized and polished visualizations of your virtual and backup environments when planning VMware vSphere, Microsoft Hyper-V or Veeam deployments. It helps you to gain better visibility and valuable business insights.

  • 12
  • 8
Tackle projects and never again get stuck behind a technical roadblock.
Join Now