Cisco 6509, Can't ping certain VLAN IP's

I would run a configuration comparison exercise first - comparing current with pre-incident.

Then, check routes. Also, DHCP-relay configurations. If that turns up nothing, schedule a down time and reboot the core.

Thanks netcmh. I work for a school system so its just me making changes in the core and no changes have been made since the incident. I have my CCNA so I know just enough to be dangerous. If I can ping the VLAN gateway from my desktop I'd like to think the routes are correct, however, not being able to ping them from the core just seems strange. There are ip helper addresses on the VLAN's for the DHCP. What I'm looking for with the DHCP-replay configs?
Thanks again,R

can you post the cleaned-up config from your 6509?  That should make it a lot easier for us to find the problem.  

Can a PC in each VLAN ping your DHCP server?

Thanks.

Each VLAN that I can connect to can ping the DHCP server.

Even if I statically assign an IP address on a certain VLAN it would go anywhere or ping anything.
File attached.
R
hcrhs-mdf-6509-confg.txt

Which are the problem VLANs?

530 531

The problem seems to be in the route map called "Impulse" that's applied to those two VLAN interfaces using the command:
ip policy route-map impulse

Even though you have an access-list configured on the ingress of those two interfaces (access-lists 160&161) that allows everything you want it to allow, once the IOS is done checking against this access-list, it then moves on to the route-map, where everything you want to allow is DENIED.

This is route-maps configuration:
route-map impulse deny 10
 match ip address intranet
!
route-map impulse permit 20
 match ip address impulse_block
 set ip next-hop 10.10.40.22

The first part of the route map (starting with - route-map impulse deny 10) means that it's going to deny anything in the intranet access-list with a PERMIT statement.  The intranet access-list follows:
ip access-list extended intranet
 remark allow DNS
 permit udp any any eq domain
 remark allow DHCP
 permit udp any any eq bootpc
 remark allow access to AD server
 permit ip any host 10.10.40.2
 permit ip any host 10.10.40.3
 remark allow access to AV server
 permit ip any host 10.10.40.10
 remark allow access to WSUS server
 permit ip any host 10.10.40.119
 remark allow access to additional servers
 permit ip any host 10.10.40.58

As you can see, you're denying all the traffic you wanted to allow in the first place.

The second part of the route map says it's going to permit anything with a permit statement in the "impulse_block" access-list and then change the next-hop of that traffic to 10.10.40.22.  That access-list follows:
ip access-list extended impulse_block
 permit ip any host 198.31.193.211
 deny   udp any any eq domain
 deny   udp any any eq bootpc

Try removing the route map from those two interfaces like this:
int vlan 530
no ip policy route-map impulse
int vlan 531
no ip policy route-map impulse

That should solve your problem.  Also, I'd make sure that access-lists 161 and 160 are actually allowing or denying the correct traffic, since they will be what dictates what traffic can flow across VLANS 530 and 531.

Let me know if that helps.
Thanks.

Are 160, 161 and impulse configured correctly?

Andy is 100% on point.
Your options include
1. Create a separate access-list and deny all the traffic you want to permit in the route map, and point the impulse route map to the ne ACL
2. If the ACL is not applied anywhere else (determine with the command "show ip interfaces"). Then change the permit statements to deny instead.
3. change the actions in the route map

When route maps reference ACLs, it processes statements that were permitted in the ACL. In other words, permit this traffic to be processed by route map. The actions specified in the Route map will then apply to objects it was permitted to check.

Are you saying that you can't ping 10.10.144.1 and 10.10.160.1 from the 6509 itself?

Craigbeck, yes. Cant ping from the 6509 but I can from my desktop computer. These vlan configs should be correct. They were worked fine for atleast a year. I haven't touched acls or vlan 530 531 configs. I'll try all suggests when I'm back at work though. Thanks all!

If you can't ping the IP locally from the switch itself there's something drastically wrong.  An ACL wouldn't usually stop the device itself from pinging itself, unless you're using VRF, and a route-map would only really tell the router which gateway to send packets to under certain conditions - that's not something that will happen here.

I just labbed it... the problem is your ACLs.

If you remove the 161 ACL from SVI 530 you can ping the 10.10.144.1 address.  The same is true with SVI 531 if you remove the 160 ACL; you can ping 10.10.160.1.

Interesting.. I'll test when I'm back in the office . Thanks much!!! :)

I did this: no ip access-list extended impluse_block

I still can't ping the 144.1 or 160.1 and DHCP still isn't handing out IP's. Any thoughts?
Thanks.

You removed the wrong bit.

You have this...

interface Vlan530
 ip address 10.10.144.1 255.255.240.0
 ip access-group 161 in
 ip helper-address 172.16.0.245
 ip flow ingress
 ip pim dense-mode
 ip policy route-map impulse
!
interface Vlan531
 ip address 10.10.160.1 255.255.240.0
 ip access-group 160 in
 ip helper-address 172.16.0.245
 ip flow ingress
 ip pim dense-mode
 ip policy route-map impulse
!

... so you need to do this...

int vl530
no ip access-group 160 in
int vl531
no ip access-group 161 in

Select allOpen in new window

...then try again.

For both it says:
Access-group 160/161 is not configured

Oops, that's my fault.  I got the ACL numbers the wrong way round.

Try this...

int vl530
no ip access-group 161 in
int vl531
no ip access-group 160 in

Select allOpen in new window

Thanks, yeah I just saw. So I applied those changes and can ping the interfaces now, however, I still don't get any DHCP leases when I try to attach to those two VLAN's... Any thoughts?

On VLAN 172 you have this...

interface Vlan172
 ip address 172.16.0.1 255.255.0.0
 ip helper-address 172.16.0.245
 ip pim dense-mode
!

If the DHCP server (172.16.0.245) uses 172.16.0.1 as its default gateway there should be no problem.  Can you confirm the DHCP server is using .1 as the default gateway?

Also, you don't need to stipulate the helper address on VLAN 172 as it's on the same subnet, therefore all broadcasts will reach the DHCP server via layer2 anyway.

Thanks craigbeck. Sorry, I meant to say no DHCP leases are given out to 530 & 531 still. Yes, the DHCP server is .245 with that gateway. (confirmed)

Ok so no issue from the network would cause this then.  Have you checked that the scopes for the 10.10.144.0/20 and 10.10.160.0/20 networks exist on the DHCP server, and that they're active?

What is the DHCP server OS?  Have you checked that there are no static routes on the DHCP server pointing those subnets to the wrong gateway?

Have you tried removing the route map temporarily?

Andy, just tried still nothing but thanks.

Just to add:

I connected a windows 7 dell laptop to our guest wireless with wireshark, filtered with bootp (for dhcp) and I don't have anything in the logs..

Something does not add up
You should at least see DHCP requests from your Win 7 since Wirreshark is running on it, unless it is set to static.
Filter by UDP 67 and 68 instead, and try again

Check that UDP ports 67 and 68 are not blocked on the firewall

Yeah, its wierd.. I did that.
So my filter is udp[68] == 67

I get nothing from our guest wireless but when I enable the local and plugin I get captures. It's not the local devices because there are no leases in the dhcp scope and I've tried multiple devices.

My bad someone did set this to a static. I'm getting what I need now.. what should I be looking for?

Do you have a wireless controller?

Yes, we use Cisco WCS. The only source that sticks out is 2.2.2.2 which is our controller I believe.

Whats the best way to upload a wireshark file? Default ext isn't allowed to attach.

On the controller you need to ensure that the interfaces which are attached to the WLAN are pointing to the 172.16.0.245 address.  The WLC is doing DHCP-Proxy, so when your client asks for an IP the WLC will modify the packet to make the client think the DHCP server is the WLC's Virtual interface IP (in your case configured as 2.2.2.2).

So, in WCS, browse to your WLC and click 'Interfaces' from the System menu.  In there, look at the interface that attaches to your Guest network and check the Primary DHCP Server entry.  If it's not 172.165.0.245 it won't be sending packets to your DHCP server.

Thanks craigbeck. I checked that already and just double checked for my sanity. This VLAN was working fine and suddenly just stopped working.

I never receive a "DHCP inform" packet on my guest wireless. On one of are others I do...

If you debug the client connecting to the guest network do you see anything on the WLC?

At the WLC CLI, use debug client <MAC> to get info relating to the association and authentication of that client.  In there you will see if an IP address is being requested, and what the response is.

I searched the client in WCS (gui) and apparently it thinks its in VLAN 526, w/ no ip address.

If I go into our wisms - > Controllers -> Interfaces -> the guest wireless is set for 530 and 531.. what I'm missing?

Thank-you!

The plot thickens!

Is your Guest WLAN using interface groups?

If WCS thinks the client is attached to VLAN 526 it can't be connecting to your Guest network if the WLAN is set to use a different VLAN interface.  I would log in locally to the WLC to make sure that the information is current as WCS polls the WLC at regular intervals - it's not always live data.

Craigbeck,
 Just want to thank you again for sticking with me. Hopefully we're getting to the bottom of this. I guess one of the other problems is I wear many different hats so I'm not an expert at anything. Our wireless infrastructure was setup by a contractor about 3 years ago so when problems like these arise  I can be somewhat clueless.

So I guess my question now is.. where would I find interface groups?
Ron

Craigbecks! You're the man! That was it.

So both VLAN's were reverted back to the old guest VLAN which is currently a disabled  DHCP scope.. and it's been disabled for a couple months. I have no idea how/why it was changed.. Would you have any ideas why this would've changed? There are only two of us how access this appliance and I doubt my colleague would've changed this.

Thanks again!!!! :)   \o/

I have no idea how or why a DHCP scope would be come disabled, but I'm glad you've fixed it :-)

The dhcp scope was disabled a while ago. That wasn't the surprise. The surprise was the interface in wcs was changed to the disabled scope. Could a mismatch reset ?

Ah ok.  WCS is a bit of a wild beast.  You can do two things usually...

1] Push the settings that WCS learned previously to the WLC, or
2] Pull the config from the WLC to WCS.

However, when you run a manual audit (when there is a mismatch for example) you get the option to replace the config that is pulled from the WLC with what's on the WCS at that particular time, or just pull the config from the WLC and update WCS so it matches what's on the WLC now.

So, if you make changes directly on the WLC the WCS doesn't always know about the change immediately, then when it automatically audits the WLC config it notices the mismatch.  You then run a manual audit because you see 'Mismatch' next to the WLC and tell the WCS to reapply the settings that it had (thinking that it contained the correct settings, and rightly so) and all hell breaks loose :-)

Moral of the story is:  If you have WCS, don't use the WLC's GUI or CLI to make changes.  ONLY use WCS.