?
Solved

Cisco 6509, Can't ping certain VLAN IP's

Posted on 2014-08-07
42
Medium Priority
?
1,667 Views
Last Modified: 2014-08-12
OK, so I believe this to be a weird issue.

Problem arose with 3 VLAN's not dishing out IP's via DHCP.
Went through a lot of troubleshooting and I realized I can't ping these certain VLAN IP's inside our Cisco 6509.

However, from my desktop I can.. I tried doing a shut/noshut on those VLAN's but still can't ping them internally..

If I assign a DHCP delivered IP to one the VLAN in question is still doesn't go... Any thoughts?
I'm thinking I have to reboot the core... Side story earlier this week we had a fiber issue so we brought up a redundant link with a Nortel switch. (The Nortel was connected with a Cisco and we're mostly a Cisco shop) When the fiber was fixed the primary went back up and we got calls that no one could go anywhere. So no problem I realized a broadcast storm/loop would happen so we unplugged the Nortel from the network and all was well... Could this be an underlying issue with these VLANS? These are the only VLAN's that are /20 others are much smaller.

Thanks,
Ron
0
Comment
Question by:PapaSmurff
  • 20
  • 14
  • 4
  • +2
42 Comments
 
LVL 21

Expert Comment

by:netcmh
ID: 40246778
I would run a configuration comparison exercise first - comparing current with pre-incident.

Then, check routes. Also, DHCP-relay configurations. If that turns up nothing, schedule a down time and reboot the core.
0
 

Author Comment

by:PapaSmurff
ID: 40246821
Thanks netcmh. I work for a school system so its just me making changes in the core and no changes have been made since the incident. I have my CCNA so I know just enough to be dangerous. If I can ping the VLAN gateway from my desktop I'd like to think the routes are correct, however, not being able to ping them from the core just seems strange. There are ip helper addresses on the VLAN's for the DHCP. What I'm looking for with the DHCP-replay configs?
Thanks again,R
0
 
LVL 1

Expert Comment

by:Andy Cantu
ID: 40246839
can you post the cleaned-up config from your 6509?  That should make it a lot easier for us to find the problem.  

Can a PC in each VLAN ping your DHCP server?

Thanks.
0
Microsoft Certification Exam 74-409

Veeam® is happy to provide the Microsoft community with a study guide prepared by MVP and MCT, Orin Thomas. This guide will take you through each of the exam objectives, helping you to prepare for and pass the examination.

 

Author Comment

by:PapaSmurff
ID: 40247069
Each VLAN that I can connect to can ping the DHCP server.

Even if I statically assign an IP address on a certain VLAN it would go anywhere or ping anything.
File attached.
R
hcrhs-mdf-6509-confg.txt
0
 
LVL 1

Expert Comment

by:Andy Cantu
ID: 40247085
Which are the problem VLANs?
0
 

Author Comment

by:PapaSmurff
ID: 40247137
530 531
0
 
LVL 1

Expert Comment

by:Andy Cantu
ID: 40247206
The problem seems to be in the route map called "Impulse" that's applied to those two VLAN interfaces using the command:
ip policy route-map impulse

Even though you have an access-list configured on the ingress of those two interfaces (access-lists 160&161) that allows everything you want it to allow, once the IOS is done checking against this access-list, it then moves on to the route-map, where everything you want to allow is DENIED.

This is route-maps configuration:
route-map impulse deny 10
 match ip address intranet
!
route-map impulse permit 20
 match ip address impulse_block
 set ip next-hop 10.10.40.22

The first part of the route map (starting with - route-map impulse deny 10) means that it's going to deny anything in the intranet access-list with a PERMIT statement.  The intranet access-list follows:
ip access-list extended intranet
 remark allow DNS
 permit udp any any eq domain
 remark allow DHCP
 permit udp any any eq bootpc
 remark allow access to AD server
 permit ip any host 10.10.40.2
 permit ip any host 10.10.40.3
 remark allow access to AV server
 permit ip any host 10.10.40.10
 remark allow access to WSUS server
 permit ip any host 10.10.40.119
 remark allow access to additional servers
 permit ip any host 10.10.40.58

As you can see, you're denying all the traffic you wanted to allow in the first place.

The second part of the route map says it's going to permit anything with a permit statement in the "impulse_block" access-list and then change the next-hop of that traffic to 10.10.40.22.  That access-list follows:
ip access-list extended impulse_block
 permit ip any host 198.31.193.211
 deny   udp any any eq domain
 deny   udp any any eq bootpc

Try removing the route map from those two interfaces like this:
int vlan 530
no ip policy route-map impulse
int vlan 531
no ip policy route-map impulse

That should solve your problem.  Also, I'd make sure that access-lists 161 and 160 are actually allowing or denying the correct traffic, since they will be what dictates what traffic can flow across VLANS 530 and 531.

Let me know if that helps.
Thanks.
0
 
LVL 21

Expert Comment

by:netcmh
ID: 40247213
Are 160, 161 and impulse configured correctly?
0
 
LVL 18

Expert Comment

by:Akinsd
ID: 40248907
Andy is 100% on point.
Your options include
1. Create a separate access-list and deny all the traffic you want to permit in the route map, and point the impulse route map to the ne ACL
2. If the ACL is not applied anywhere else (determine with the command "show ip interfaces"). Then change the permit statements to deny instead.
3. change the actions in the route map

When route maps reference ACLs, it processes statements that were permitted in the ACL. In other words, permit this traffic to be processed by route map. The actions specified in the Route map will then apply to objects it was permitted to check.
0
 
LVL 47

Expert Comment

by:Craig Beck
ID: 40250338
Are you saying that you can't ping 10.10.144.1 and 10.10.160.1 from the 6509 itself?
0
 

Author Comment

by:PapaSmurff
ID: 40250617
Craigbeck, yes. Cant ping from the 6509 but I can from my desktop computer. These vlan configs should be correct. They were worked fine for atleast a year. I haven't touched acls or vlan 530 531 configs. I'll try all suggests when I'm back at work though. Thanks all!
0
 
LVL 47

Expert Comment

by:Craig Beck
ID: 40250823
If you can't ping the IP locally from the switch itself there's something drastically wrong.  An ACL wouldn't usually stop the device itself from pinging itself, unless you're using VRF, and a route-map would only really tell the router which gateway to send packets to under certain conditions - that's not something that will happen here.
0
 
LVL 47

Expert Comment

by:Craig Beck
ID: 40250830
I just labbed it... the problem is your ACLs.

If you remove the 161 ACL from SVI 530 you can ping the 10.10.144.1 address.  The same is true with SVI 531 if you remove the 160 ACL; you can ping 10.10.160.1.
0
 

Author Comment

by:PapaSmurff
ID: 40250836
Interesting.. I'll test when I'm back in the office . Thanks much!!! :)
0
 

Author Comment

by:PapaSmurff
ID: 40255274
I did this: no ip access-list extended impluse_block

I still can't ping the 144.1 or 160.1 and DHCP still isn't handing out IP's. Any thoughts?
Thanks.
0
 
LVL 47

Expert Comment

by:Craig Beck
ID: 40255312
You removed the wrong bit.

You have this...
interface Vlan530
 ip address 10.10.144.1 255.255.240.0
 ip access-group 161 in
 ip helper-address 172.16.0.245
 ip flow ingress
 ip pim dense-mode
 ip policy route-map impulse
!
interface Vlan531
 ip address 10.10.160.1 255.255.240.0
 ip access-group 160 in
 ip helper-address 172.16.0.245
 ip flow ingress
 ip pim dense-mode
 ip policy route-map impulse
!
... so you need to do this...
int vl530
no ip access-group 160 in
int vl531
no ip access-group 161 in

Open in new window


...then try again.
0
 

Author Comment

by:PapaSmurff
ID: 40255368
For both it says:
Access-group 160/161 is not configured
0
 
LVL 47

Expert Comment

by:Craig Beck
ID: 40255377
Oops, that's my fault.  I got the ACL numbers the wrong way round.

Try this...
int vl530
no ip access-group 161 in
int vl531
no ip access-group 160 in

Open in new window

0
 

Author Comment

by:PapaSmurff
ID: 40255385
Thanks, yeah I just saw. So I applied those changes and can ping the interfaces now, however, I still don't get any DHCP leases when I try to attach to those two VLAN's... Any thoughts?
0
 
LVL 47

Expert Comment

by:Craig Beck
ID: 40255397
On VLAN 172 you have this...
interface Vlan172
 ip address 172.16.0.1 255.255.0.0
 ip helper-address 172.16.0.245
 ip pim dense-mode
!

If the DHCP server (172.16.0.245) uses 172.16.0.1 as its default gateway there should be no problem.  Can you confirm the DHCP server is using .1 as the default gateway?

Also, you don't need to stipulate the helper address on VLAN 172 as it's on the same subnet, therefore all broadcasts will reach the DHCP server via layer2 anyway.
0
 

Author Comment

by:PapaSmurff
ID: 40255412
Thanks craigbeck. Sorry, I meant to say no DHCP leases are given out to 530 & 531 still. Yes, the DHCP server is .245 with that gateway. (confirmed)
0
 
LVL 47

Expert Comment

by:Craig Beck
ID: 40255458
Ok so no issue from the network would cause this then.  Have you checked that the scopes for the 10.10.144.0/20 and 10.10.160.0/20 networks exist on the DHCP server, and that they're active?

What is the DHCP server OS?  Have you checked that there are no static routes on the DHCP server pointing those subnets to the wrong gateway?
0
 
LVL 1

Expert Comment

by:Andy Cantu
ID: 40255462
Have you tried removing the route map temporarily?
0
 

Author Comment

by:PapaSmurff
ID: 40255496
Andy, just tried still nothing but thanks.
0
 

Author Comment

by:PapaSmurff
ID: 40255692
Just to add:

I connected a windows 7 dell laptop to our guest wireless with wireshark, filtered with bootp (for dhcp) and I don't have anything in the logs..
0
 
LVL 18

Expert Comment

by:Akinsd
ID: 40255721
Something does not add up
You should at least see DHCP requests from your Win 7 since Wirreshark is running on it, unless it is set to static.
Filter by UDP 67 and 68 instead, and try again

Check that UDP ports 67 and 68 are not blocked on the firewall
0
 

Author Comment

by:PapaSmurff
ID: 40255779
Yeah, its wierd.. I did that.
So my filter is udp[68] == 67

I get nothing from our guest wireless but when I enable the local and plugin I get captures. It's not the local devices because there are no leases in the dhcp scope and I've tried multiple devices.
0
 

Author Comment

by:PapaSmurff
ID: 40255879
My bad someone did set this to a static. I'm getting what I need now.. what should I be looking for?
0
 
LVL 47

Expert Comment

by:Craig Beck
ID: 40255917
Do you have a wireless controller?
0
 

Author Comment

by:PapaSmurff
ID: 40255931
Yes, we use Cisco WCS. The only source that sticks out is 2.2.2.2 which is our controller I believe.

Whats the best way to upload a wireshark file? Default ext isn't allowed to attach.
0
 
LVL 47

Expert Comment

by:Craig Beck
ID: 40255975
On the controller you need to ensure that the interfaces which are attached to the WLAN are pointing to the 172.16.0.245 address.  The WLC is doing DHCP-Proxy, so when your client asks for an IP the WLC will modify the packet to make the client think the DHCP server is the WLC's Virtual interface IP (in your case configured as 2.2.2.2).

So, in WCS, browse to your WLC and click 'Interfaces' from the System menu.  In there, look at the interface that attaches to your Guest network and check the Primary DHCP Server entry.  If it's not 172.165.0.245 it won't be sending packets to your DHCP server.
0
 

Author Comment

by:PapaSmurff
ID: 40256061
Thanks craigbeck. I checked that already and just double checked for my sanity. This VLAN was working fine and suddenly just stopped working.
0
 

Author Comment

by:PapaSmurff
ID: 40256089
I never receive a "DHCP inform" packet on my guest wireless. On one of are others I do...
0
 
LVL 47

Expert Comment

by:Craig Beck
ID: 40256138
If you debug the client connecting to the guest network do you see anything on the WLC?

At the WLC CLI, use debug client <MAC> to get info relating to the association and authentication of that client.  In there you will see if an IP address is being requested, and what the response is.
0
 

Author Comment

by:PapaSmurff
ID: 40256187
I searched the client in WCS (gui) and apparently it thinks its in VLAN 526, w/ no ip address.

If I go into our wisms - > Controllers -> Interfaces -> the guest wireless is set for 530 and 531.. what I'm missing?

Thank-you!
0
 
LVL 47

Expert Comment

by:Craig Beck
ID: 40256201
The plot thickens!

Is your Guest WLAN using interface groups?

If WCS thinks the client is attached to VLAN 526 it can't be connecting to your Guest network if the WLAN is set to use a different VLAN interface.  I would log in locally to the WLC to make sure that the information is current as WCS polls the WLC at regular intervals - it's not always live data.
0
 

Author Comment

by:PapaSmurff
ID: 40256221
Craigbeck,
 Just want to thank you again for sticking with me. Hopefully we're getting to the bottom of this. I guess one of the other problems is I wear many different hats so I'm not an expert at anything. Our wireless infrastructure was setup by a contractor about 3 years ago so when problems like these arise  I can be somewhat clueless.

So I guess my question now is.. where would I find interface groups?
Ron
0
 
LVL 47

Accepted Solution

by:
Craig Beck earned 2000 total points
ID: 40256394
It's no problem - that's what we're here for!

You'd be better going to the WLC rather than via WCS as the configuration may not be current in WCS.  You have to audit the configuration to get WCS to learn the settings on the WLC, particularly if there is a mismatch noted in WCS.

To get to the WLC directly...  When you hit the Controllers page in WCS, make a note of the WLC IP then just browse to it via HTTPS.

When you're in the WLC, go to the WLANs tab then click on the Guest WLAN.  There will be an drop-down box called "Interface".  If the selection in that box has a (G) next to it, that's an interface group.
0
 

Author Comment

by:PapaSmurff
ID: 40256428
Craigbecks! You're the man! That was it.

So both VLAN's were reverted back to the old guest VLAN which is currently a disabled  DHCP scope.. and it's been disabled for a couple months. I have no idea how/why it was changed.. Would you have any ideas why this would've changed? There are only two of us how access this appliance and I doubt my colleague would've changed this.

Thanks again!!!! :)   \o/
0
 
LVL 47

Expert Comment

by:Craig Beck
ID: 40256638
I have no idea how or why a DHCP scope would be come disabled, but I'm glad you've fixed it :-)
0
 

Author Comment

by:PapaSmurff
ID: 40256649
The dhcp scope was disabled a while ago. That wasn't the surprise. The surprise was the interface in wcs was changed to the disabled scope. Could a mismatch reset ?
0
 
LVL 47

Expert Comment

by:Craig Beck
ID: 40256728
Ah ok.  WCS is a bit of a wild beast.  You can do two things usually...

1] Push the settings that WCS learned previously to the WLC, or
2] Pull the config from the WLC to WCS.

However, when you run a manual audit (when there is a mismatch for example) you get the option to replace the config that is pulled from the WLC with what's on the WCS at that particular time, or just pull the config from the WLC and update WCS so it matches what's on the WLC now.

So, if you make changes directly on the WLC the WCS doesn't always know about the change immediately, then when it automatically audits the WLC config it notices the mismatch.  You then run a manual audit because you see 'Mismatch' next to the WLC and tell the WCS to reapply the settings that it had (thinking that it contained the correct settings, and rightly so) and all hell breaks loose :-)

Moral of the story is:  If you have WCS, don't use the WLC's GUI or CLI to make changes.  ONLY use WCS.
0

Featured Post

What does it mean to be "Always On"?

Is your cloud always on? With an Always On cloud you won't have to worry about downtime for maintenance or software application code updates, ensuring that your bottom line isn't affected.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Configuring network clients can be a chore, especially if there are a large number of them or a lot of itinerant users.  DHCP dynamically manages this process, much to the relief of users and administrators alike!
I recently attended Cisco Live! in Las Vegas, a conference that boasted over 28,000 techies in attendance, and a week of hands-on learning hosted by a solid partner with which Concerto goes to market.  Every year, Cisco displays cutting-edge technol…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…

862 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question