Cisco 6509, Can't ping certain VLAN IP's

OK, so I believe this to be a weird issue.

Problem arose with 3 VLAN's not dishing out IP's via DHCP.
Went through a lot of troubleshooting and I realized I can't ping these certain VLAN IP's inside our Cisco 6509.

However, from my desktop I can.. I tried doing a shut/noshut on those VLAN's but still can't ping them internally..

If I assign a DHCP delivered IP to one the VLAN in question is still doesn't go... Any thoughts?
I'm thinking I have to reboot the core... Side story earlier this week we had a fiber issue so we brought up a redundant link with a Nortel switch. (The Nortel was connected with a Cisco and we're mostly a Cisco shop) When the fiber was fixed the primary went back up and we got calls that no one could go anywhere. So no problem I realized a broadcast storm/loop would happen so we unplugged the Nortel from the network and all was well... Could this be an underlying issue with these VLANS? These are the only VLAN's that are /20 others are much smaller.

Thanks,
Ron
PapaSmurffAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

netcmhCommented:
I would run a configuration comparison exercise first - comparing current with pre-incident.

Then, check routes. Also, DHCP-relay configurations. If that turns up nothing, schedule a down time and reboot the core.
0
PapaSmurffAuthor Commented:
Thanks netcmh. I work for a school system so its just me making changes in the core and no changes have been made since the incident. I have my CCNA so I know just enough to be dangerous. If I can ping the VLAN gateway from my desktop I'd like to think the routes are correct, however, not being able to ping them from the core just seems strange. There are ip helper addresses on the VLAN's for the DHCP. What I'm looking for with the DHCP-replay configs?
Thanks again,R
0
Andy CantuCommented:
can you post the cleaned-up config from your 6509?  That should make it a lot easier for us to find the problem.  

Can a PC in each VLAN ping your DHCP server?

Thanks.
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

PapaSmurffAuthor Commented:
Each VLAN that I can connect to can ping the DHCP server.

Even if I statically assign an IP address on a certain VLAN it would go anywhere or ping anything.
File attached.
R
hcrhs-mdf-6509-confg.txt
0
Andy CantuCommented:
Which are the problem VLANs?
0
PapaSmurffAuthor Commented:
530 531
0
Andy CantuCommented:
The problem seems to be in the route map called "Impulse" that's applied to those two VLAN interfaces using the command:
ip policy route-map impulse

Even though you have an access-list configured on the ingress of those two interfaces (access-lists 160&161) that allows everything you want it to allow, once the IOS is done checking against this access-list, it then moves on to the route-map, where everything you want to allow is DENIED.

This is route-maps configuration:
route-map impulse deny 10
 match ip address intranet
!
route-map impulse permit 20
 match ip address impulse_block
 set ip next-hop 10.10.40.22

The first part of the route map (starting with - route-map impulse deny 10) means that it's going to deny anything in the intranet access-list with a PERMIT statement.  The intranet access-list follows:
ip access-list extended intranet
 remark allow DNS
 permit udp any any eq domain
 remark allow DHCP
 permit udp any any eq bootpc
 remark allow access to AD server
 permit ip any host 10.10.40.2
 permit ip any host 10.10.40.3
 remark allow access to AV server
 permit ip any host 10.10.40.10
 remark allow access to WSUS server
 permit ip any host 10.10.40.119
 remark allow access to additional servers
 permit ip any host 10.10.40.58

As you can see, you're denying all the traffic you wanted to allow in the first place.

The second part of the route map says it's going to permit anything with a permit statement in the "impulse_block" access-list and then change the next-hop of that traffic to 10.10.40.22.  That access-list follows:
ip access-list extended impulse_block
 permit ip any host 198.31.193.211
 deny   udp any any eq domain
 deny   udp any any eq bootpc

Try removing the route map from those two interfaces like this:
int vlan 530
no ip policy route-map impulse
int vlan 531
no ip policy route-map impulse

That should solve your problem.  Also, I'd make sure that access-lists 161 and 160 are actually allowing or denying the correct traffic, since they will be what dictates what traffic can flow across VLANS 530 and 531.

Let me know if that helps.
Thanks.
0
netcmhCommented:
Are 160, 161 and impulse configured correctly?
0
AkinsdNetwork AdministratorCommented:
Andy is 100% on point.
Your options include
1. Create a separate access-list and deny all the traffic you want to permit in the route map, and point the impulse route map to the ne ACL
2. If the ACL is not applied anywhere else (determine with the command "show ip interfaces"). Then change the permit statements to deny instead.
3. change the actions in the route map

When route maps reference ACLs, it processes statements that were permitted in the ACL. In other words, permit this traffic to be processed by route map. The actions specified in the Route map will then apply to objects it was permitted to check.
0
Craig BeckCommented:
Are you saying that you can't ping 10.10.144.1 and 10.10.160.1 from the 6509 itself?
0
PapaSmurffAuthor Commented:
Craigbeck, yes. Cant ping from the 6509 but I can from my desktop computer. These vlan configs should be correct. They were worked fine for atleast a year. I haven't touched acls or vlan 530 531 configs. I'll try all suggests when I'm back at work though. Thanks all!
0
Craig BeckCommented:
If you can't ping the IP locally from the switch itself there's something drastically wrong.  An ACL wouldn't usually stop the device itself from pinging itself, unless you're using VRF, and a route-map would only really tell the router which gateway to send packets to under certain conditions - that's not something that will happen here.
0
Craig BeckCommented:
I just labbed it... the problem is your ACLs.

If you remove the 161 ACL from SVI 530 you can ping the 10.10.144.1 address.  The same is true with SVI 531 if you remove the 160 ACL; you can ping 10.10.160.1.
0
PapaSmurffAuthor Commented:
Interesting.. I'll test when I'm back in the office . Thanks much!!! :)
0
PapaSmurffAuthor Commented:
I did this: no ip access-list extended impluse_block

I still can't ping the 144.1 or 160.1 and DHCP still isn't handing out IP's. Any thoughts?
Thanks.
0
Craig BeckCommented:
You removed the wrong bit.

You have this...
interface Vlan530
 ip address 10.10.144.1 255.255.240.0
 ip access-group 161 in
 ip helper-address 172.16.0.245
 ip flow ingress
 ip pim dense-mode
 ip policy route-map impulse
!
interface Vlan531
 ip address 10.10.160.1 255.255.240.0
 ip access-group 160 in
 ip helper-address 172.16.0.245
 ip flow ingress
 ip pim dense-mode
 ip policy route-map impulse
!
... so you need to do this...
int vl530
no ip access-group 160 in
int vl531
no ip access-group 161 in

Open in new window


...then try again.
0
PapaSmurffAuthor Commented:
For both it says:
Access-group 160/161 is not configured
0
Craig BeckCommented:
Oops, that's my fault.  I got the ACL numbers the wrong way round.

Try this...
int vl530
no ip access-group 161 in
int vl531
no ip access-group 160 in

Open in new window

0
PapaSmurffAuthor Commented:
Thanks, yeah I just saw. So I applied those changes and can ping the interfaces now, however, I still don't get any DHCP leases when I try to attach to those two VLAN's... Any thoughts?
0
Craig BeckCommented:
On VLAN 172 you have this...
interface Vlan172
 ip address 172.16.0.1 255.255.0.0
 ip helper-address 172.16.0.245
 ip pim dense-mode
!

If the DHCP server (172.16.0.245) uses 172.16.0.1 as its default gateway there should be no problem.  Can you confirm the DHCP server is using .1 as the default gateway?

Also, you don't need to stipulate the helper address on VLAN 172 as it's on the same subnet, therefore all broadcasts will reach the DHCP server via layer2 anyway.
0
PapaSmurffAuthor Commented:
Thanks craigbeck. Sorry, I meant to say no DHCP leases are given out to 530 & 531 still. Yes, the DHCP server is .245 with that gateway. (confirmed)
0
Craig BeckCommented:
Ok so no issue from the network would cause this then.  Have you checked that the scopes for the 10.10.144.0/20 and 10.10.160.0/20 networks exist on the DHCP server, and that they're active?

What is the DHCP server OS?  Have you checked that there are no static routes on the DHCP server pointing those subnets to the wrong gateway?
0
Andy CantuCommented:
Have you tried removing the route map temporarily?
0
PapaSmurffAuthor Commented:
Andy, just tried still nothing but thanks.
0
PapaSmurffAuthor Commented:
Just to add:

I connected a windows 7 dell laptop to our guest wireless with wireshark, filtered with bootp (for dhcp) and I don't have anything in the logs..
0
AkinsdNetwork AdministratorCommented:
Something does not add up
You should at least see DHCP requests from your Win 7 since Wirreshark is running on it, unless it is set to static.
Filter by UDP 67 and 68 instead, and try again

Check that UDP ports 67 and 68 are not blocked on the firewall
0
PapaSmurffAuthor Commented:
Yeah, its wierd.. I did that.
So my filter is udp[68] == 67

I get nothing from our guest wireless but when I enable the local and plugin I get captures. It's not the local devices because there are no leases in the dhcp scope and I've tried multiple devices.
0
PapaSmurffAuthor Commented:
My bad someone did set this to a static. I'm getting what I need now.. what should I be looking for?
0
Craig BeckCommented:
Do you have a wireless controller?
0
PapaSmurffAuthor Commented:
Yes, we use Cisco WCS. The only source that sticks out is 2.2.2.2 which is our controller I believe.

Whats the best way to upload a wireshark file? Default ext isn't allowed to attach.
0
Craig BeckCommented:
On the controller you need to ensure that the interfaces which are attached to the WLAN are pointing to the 172.16.0.245 address.  The WLC is doing DHCP-Proxy, so when your client asks for an IP the WLC will modify the packet to make the client think the DHCP server is the WLC's Virtual interface IP (in your case configured as 2.2.2.2).

So, in WCS, browse to your WLC and click 'Interfaces' from the System menu.  In there, look at the interface that attaches to your Guest network and check the Primary DHCP Server entry.  If it's not 172.165.0.245 it won't be sending packets to your DHCP server.
0
PapaSmurffAuthor Commented:
Thanks craigbeck. I checked that already and just double checked for my sanity. This VLAN was working fine and suddenly just stopped working.
0
PapaSmurffAuthor Commented:
I never receive a "DHCP inform" packet on my guest wireless. On one of are others I do...
0
Craig BeckCommented:
If you debug the client connecting to the guest network do you see anything on the WLC?

At the WLC CLI, use debug client <MAC> to get info relating to the association and authentication of that client.  In there you will see if an IP address is being requested, and what the response is.
0
PapaSmurffAuthor Commented:
I searched the client in WCS (gui) and apparently it thinks its in VLAN 526, w/ no ip address.

If I go into our wisms - > Controllers -> Interfaces -> the guest wireless is set for 530 and 531.. what I'm missing?

Thank-you!
0
Craig BeckCommented:
The plot thickens!

Is your Guest WLAN using interface groups?

If WCS thinks the client is attached to VLAN 526 it can't be connecting to your Guest network if the WLAN is set to use a different VLAN interface.  I would log in locally to the WLC to make sure that the information is current as WCS polls the WLC at regular intervals - it's not always live data.
0
PapaSmurffAuthor Commented:
Craigbeck,
 Just want to thank you again for sticking with me. Hopefully we're getting to the bottom of this. I guess one of the other problems is I wear many different hats so I'm not an expert at anything. Our wireless infrastructure was setup by a contractor about 3 years ago so when problems like these arise  I can be somewhat clueless.

So I guess my question now is.. where would I find interface groups?
Ron
0
Craig BeckCommented:
It's no problem - that's what we're here for!

You'd be better going to the WLC rather than via WCS as the configuration may not be current in WCS.  You have to audit the configuration to get WCS to learn the settings on the WLC, particularly if there is a mismatch noted in WCS.

To get to the WLC directly...  When you hit the Controllers page in WCS, make a note of the WLC IP then just browse to it via HTTPS.

When you're in the WLC, go to the WLANs tab then click on the Guest WLAN.  There will be an drop-down box called "Interface".  If the selection in that box has a (G) next to it, that's an interface group.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
PapaSmurffAuthor Commented:
Craigbecks! You're the man! That was it.

So both VLAN's were reverted back to the old guest VLAN which is currently a disabled  DHCP scope.. and it's been disabled for a couple months. I have no idea how/why it was changed.. Would you have any ideas why this would've changed? There are only two of us how access this appliance and I doubt my colleague would've changed this.

Thanks again!!!! :)   \o/
0
Craig BeckCommented:
I have no idea how or why a DHCP scope would be come disabled, but I'm glad you've fixed it :-)
0
PapaSmurffAuthor Commented:
The dhcp scope was disabled a while ago. That wasn't the surprise. The surprise was the interface in wcs was changed to the disabled scope. Could a mismatch reset ?
0
Craig BeckCommented:
Ah ok.  WCS is a bit of a wild beast.  You can do two things usually...

1] Push the settings that WCS learned previously to the WLC, or
2] Pull the config from the WLC to WCS.

However, when you run a manual audit (when there is a mismatch for example) you get the option to replace the config that is pulled from the WLC with what's on the WCS at that particular time, or just pull the config from the WLC and update WCS so it matches what's on the WLC now.

So, if you make changes directly on the WLC the WCS doesn't always know about the change immediately, then when it automatically audits the WLC config it notices the mismatch.  You then run a manual audit because you see 'Mismatch' next to the WLC and tell the WCS to reapply the settings that it had (thinking that it contained the correct settings, and rightly so) and all hell breaks loose :-)

Moral of the story is:  If you have WCS, don't use the WLC's GUI or CLI to make changes.  ONLY use WCS.
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Routers

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.