[Webinar] Streamline your web hosting managementRegister Today

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 765
  • Last Modified:

multicast traffic cisco

I have a strange issue here and am out of ideas:

1) We have 2 facilities, 1 HQ and 1 remote office
2) We have 2 different ISP's providing 2 different 10MB point to point links (one is kept offline for a backup)
3) Our multicast application works correctly at HQ, NOT at our remote office
4) The multicast server, working HQ users using the multicast application, and one ISP share the same switch
5) Both ISP's have been tested, neither is working for the multicast application (I assume that this would rule out the PTP link)

When I first started troubleshooting this, I assumed that all multicast traffic had stopped, but we put wireshark (packet sniffer) at the remote office, and it's receiving multicast traffic from server "A" but not server "B".  Server "B's" application will not work at the remote site, but works fine at our HQ.

The bizarre thing is this application worked fine before we had a power outage at the remote office, afterwords it stopped working at the remote office.

Any ideas or thoughts would be appreciated!
0
bschwarting
Asked:
bschwarting
  • 37
  • 15
  • 13
  • +2
3 Solutions
 
Rick_O_ShayCommented:
Is it possible the IGMP config got wiped out or something at the remote site? IGMP snooping should be on for any port that may want to join a multicast stream so usually it is configured on all the switch ports. The other piece is a multicast routing protocol like PIM on the routers but if you are seeing the multicast on the capture then that is getting there ok.
0
 
bschwartingAuthor Commented:
igmp snooping is disabled on both sides.  all multicast traffic is currently being repeated to every port.  we don't have any IGMP config.
0
 
mikebernhardtCommented:
So both servers are working locally? Did it ever work correctly at the remote office? Is there some access list or firewall in between that might allow one but not the other?
0
The new generation of project management tools

With monday.com’s project management tool, you can see what everyone on your team is working in a single glance. Its intuitive dashboards are customizable, so you can create systems that work for you.

 
bschwartingAuthor Commented:
both are working locally at the HQ.  it worked correctly before a power outage according to some CSR's.  it's possible they didn't realize it wasn't working.  no firewall exists between these locations.
0
 
mikebernhardtCommented:
I am assuming that the L3 multicast configuration in the routers is correct? That whatever is configured to make the one work is duplicated for the other?
0
 
bschwartingAuthor Commented:
we have L3 switches, cisco 4507, and it's working correctly on those, yes. the app works fine here.
0
 
mikebernhardtCommented:
What about the remote end. Is it configured to deal with server B's multicast like it is for Server A?

You could do show ip mroute on all the routers between the server and the client and see if you have similar output for both streams.
0
 
bschwartingAuthor Commented:
remember, there is no "config" on either end.  IGMP snooping is disabled on both ends.  the mulitcast traffic is being forwarded to all ports.

that is the way I understand it.  is that correct?
0
 
mikebernhardtCommented:
I mean at layer 3. Unless both sites are tied together on one big subnet, the routers in between have to have a Layer 3 multicast routing configuration it won't be routed between sites.
0
 
bschwartingAuthor Commented:
sorry, should have given more info.  it is all on one subnet so there are no routers in between the sites.
0
 
Steve JenningsIT ManagerCommented:
server B is sending multicast to the backup ISP
0
 
mikebernhardtCommented:
But you said that the backup ISP is... backup. So how is it going to get to the remote office?

did it ever work?
0
 
bschwartingAuthor Commented:
i tried disabling one ISP (the backup) and it was the same thing.

yes, it used to work great.
0
 
Rick_O_ShayCommented:
Is the switchport block multicast setting on for any interface?
0
 
bschwartingAuthor Commented:
Not sure what that command is but I assume it would be obvious.
0
 
bschwartingAuthor Commented:
My switch has been up for three years, any chance just a reboot would fix this?  I haven't yet since this is our core switch and has hundreds of devices connected to it.
0
 
Rick_O_ShayCommented:
That's always a possibility but you saw the multicast reaching the remote site correctly right?
0
 
bschwartingAuthor Commented:
Yeah, server A is getting there fine.
0
 
mikebernhardtCommented:
If it's all on one subnet, and server A is getting there, then either the same MAC is being used for something else or the switch is hosed up and should be rebooted.

Duplicate MAC is possible depending on what other multicast IP addresses you have running:
http://www.ciscosistemi.com/en/US/docs/internetworking/technology/handbook/IP-Multi.html#wp1020628
0
 
bschwartingAuthor Commented:
clearing the mac-address-table during the day shouldn't interrupt anything, correct?
0
 
mikebernhardtCommented:
No, the only effect is that everything will be broadcast for a bit while the switch rebuilds its table. But my point was to look at your multicast addresses and make sure that you don't have a multicast IP address that could be stomping on the one you're having trouble with. One way to do this is to figure out the mac address of server B's multicast address and then see if the switch thinks that address belongs to a specific port. Since it should not, that would cause a problem. The mac address table is filled by looking at SOURCE addresses, not destination addresses. Or maybe the multicast mac address was configured statically to a certain port- that would mean no more broadcasting since the switch knows which port it should land on.
0
 
bschwartingAuthor Commented:
yeah, I got what you meant, it just got me thinking.  the switch had been up for 3 years, so maybe i had a corrupt mac table.  clearing didn't do anything, just wanted to try it.  I've also checked out your duplicate theory and I'm not finding anything.
0
 
mikebernhardtCommented:
What is your topology between the 2 sites? Can you post a drawing?
0
 
bschwartingAuthor Commented:
let me attach this image and see if it sheds any new light on this.  the attached image is the config screen.  the 234.5.6.2 is the one we are having trouble seeing at the remote office.  230.0.0.1 is working correctly at HQ and the remote office.
Capture.JPG
0
 
mikebernhardtCommented:
So if my calculations are correct, the destination MAC address should be 0100.5e05.0602

Can you attach a drawing of your topology? I want to see all the equipment between the sender and the remote office.
0
 
bschwartingAuthor Commented:
how is this for a quick draft?


topology.jpg
0
 
mikebernhardtCommented:
So 230.0.0.1 is the PC at the remote office? If not, what's the address of that one?

I assume that you have some kind of virtual ethernet between the 2 sites?
0
 
bschwartingAuthor Commented:
no, PC at the remote office and Server are both on 10.9.1.0 network.  same vlan, subnet mask, gateway, etc...
0
 
bschwartingAuthor Commented:
230.0.0.1 is one of the multicast addresses
0
 
mikebernhardtCommented:
Where it says PC w/ multicasting app, does that mean it's listening only, or sending too? If sending, what address is it sending to?

How are the remote office and main office connected that allows them to share a subnet? MPLS, VPN, bridge over WAN, or what?
0
 
bschwartingAuthor Commented:
All connections are point to point from both ISP's.  They provide a port that we plug into, usually off of their router/switch or a fiber to Ethernet converter.

The PC application is a listener only.  When it is installed it asks for two IP addresses, which are the A and B server.
0
 
mikebernhardtCommented:
So it's probably MPLS. you might talk to them about the problem. MPLS doesn't usually pass multicast, so if it working for Server A, they may have done something to make it work that is not/no longer in place for Server B.
0
 
bschwartingAuthor Commented:
just got a response from one ISP, "No I do not have any storm control set on that switch."
0
 
mikebernhardtCommented:
On what switch? We need to know how the multicast that works is getting from the main site to the remote site. Do they have a mac address statically set?
0
 
bschwartingAuthor Commented:
on the ISP switch.

nothing is statically set.
0
 
bschwartingAuthor Commented:
FYI, I rebooted both core switches this past Friday and it didn't help.
0
 
bschwartingAuthor Commented:
Still no solution on this one.  We are getting AT&T involved now.  I'm open to any new ideas.
0
 
harbor235Commented:


What AT&T product do you have connecting the two sites?

harbor235 ;}
0
 
harbor235Commented:

I would debug IGMP and use wireshark at the endpoints to see if the joins and leaves are occurring.
I assume you are using igmp version 2?  What IOS are you using ? I also assume that you are not doing any storm/broadcast/multicast suppression?

harbor235 ;}

0
 
bschwartingAuthor Commented:
Waiting to hear back from the ATT rep now on what type of equipment it is, it's something they supplied.

How can I do IGMP debug if I don't have any IGMP config?
0
 
harbor235Commented:

IGMP is on by default

harbor235 ;}
0
 
bschwartingAuthor Commented:
This is the only thing we have on our switch.
4507-1stFloor#show run | incl igmp
no ip igmp snooping vlan 2
ip igmp snooping vlan 2 immediate-leave
4507-1stFloor#

Open in new window

0
 
harbor235Commented:


Right, but thats in the running config, IGMP functionality is on by default, there is no command to activate it. Try it and you will see the IGMP messages and Wireshark will pick them up on the host side as well.

harbor235 ;}
0
 
bschwartingAuthor Commented:
ok, so what is the best syntax for igmp debug?


debug igmp vlan 2

Open in new window

0
 
harbor235Commented:


"debug ip igmp" is good, there are a couple of other switches but this will let you know whats going on.
You verify what your host is doing as well with wireshark, this should give you better insight to whats going on, if something is failing you will see it.

harbor235 ;}
0
 
bschwartingAuthor Commented:
When I run that command, it shows that it is on, shouldn't I see output in the terminal window?
4507-2ndFloor#debug ip igmp
IGMP debugging is on

Open in new window

0
 
harbor235Commented:


remember to add "terminal monitor" from exec mode and "logging console" from global config mode
if you are not on the console and your are telnett'd or ssh'd in.

harbor235 ;}
0
 
bschwartingAuthor Commented:
ok, I'm in telnet and have done both, and still nothing is coming back.  I don't guess the order matters does it?
4507-2ndFloor#debug ip igmp
IGMP debugging is on

4507-2ndFloor#terminal monitor

4507-2ndFloor(config)#logging console

Open in new window

0
 
harbor235Commented:

ok, if they are enabled then you need to initiate receiving a multicast stream, remember what debugging igmp will show you, it will show you the joins and leaves to available multicast groups.

harbor235 ;}
0
 
bschwartingAuthor Commented:
I'm not sure what, but I am missing something.  I have the debugging on, and logging level set to debugging, and terminal monitor turned on, and still nothing while running the streaming application.
4507-2ndFloor#show logging
Syslog logging: enabled (0 messages dropped, 1 messages rate-limited, 0 flushes, 0 overruns)
    Console logging: level debugging, 37 messages logged
    Monitor logging: level debugging, 3 messages logged
        Logging to: vty2(1)
    Buffer logging: level debugging, 39 messages logged
    Exception Logging: size (8192 bytes)
    Trap logging: level informational, 44 message lines logged

Log Buffer (128000 bytes):

Open in new window

0
 
harbor235Commented:


And you entered "logging console" from global config mode for your session?

harbor235 ;}
0
 
bschwartingAuthor Commented:
yeah, just verified it again, and did it again.
0
 
bschwartingAuthor Commented:
let me walk through exactly what I do, just to make sure I'm not missing something.
telnet to switch
login as root

4507-2ndFloor#no debug all
All possible debugging has been turned off

4507-2ndFloor#terminal monitor
4507-2ndFloor#config t
Enter configuration commands, one per line.  End with CNTL/Z.
4507-2ndFloor(config)#logging console
4507-2ndFloor(config)#exit
4507-2ndFloor#debug ip igmp
IGMP debugging is on

Open in new window

0
 
harbor235Commented:



your log buffer is very small 128K, try making it at least 1M (1000000).

I just completed teh same process on my 3845 and no issues,

term mon
(config) logging console
debug ip bgp events

So we need to make sure if there are actual IGMP events

What does show ip igmp <do a question mark here, look for relevant switches>

harbor235 ;}


 
0
 
harbor235Commented:


show ip igmp snooping   - what relevant output do we get?   Can you post your config?


harbor235 ;}
0
 
bschwartingAuthor Commented:
looks like it's disabled on vlan2, which is where the issues are.
4507-2ndFloor#show ip igmp snooping
Global IGMP Snooping configuration:
-----------------------------------
IGMP snooping             : Enabled
IGMPv3 snooping           : Enabled
Report suppression        : Enabled
TCN solicit query         : Disabled
TCN flood query count     : 2

Vlan 1:
--------
IGMP snooping                  : Enabled
IGMPv2 immediate leave         : Disabled
Explicit host tracking         : Enabled
Multicast router learning mode : pim-dvmrp
CGMP interoperability mode     : IGMP_ONLY

Vlan 2:
--------
IGMP snooping                  : Disabled
IGMPv2 immediate leave         : Enabled
Explicit host tracking         : Enabled
Multicast router learning mode : pim-dvmrp
Vlan 4:
--------
IGMP snooping                  : Enabled
IGMPv2 immediate leave         : Disabled
Explicit host tracking         : Enabled
Multicast router learning mode : pim-dvmrp
CGMP interoperability mode     : IGMP_ONLY

Vlan 5:
--------
IGMP snooping                  : Enabled
IGMPv2 immediate leave         : Disabled
Explicit host tracking         : Enabled
Multicast router learning mode : pim-dvmrp
CGMP interoperability mode     : IGMP_ONLY

Vlan 10:
--------
IGMP snooping                  : Enabled
IGMPv2 immediate leave         : Disabled
Explicit host tracking         : Enabled
Multicast router learning mode : pim-dvmrp
CGMP interoperability mode     : IGMP_ONLY

Vlan 11:
--------
IGMP snooping                  : Enabled
IGMPv2 immediate leave         : Disabled
Explicit host tracking         : Enabled
Multicast router learning mode : pim-dvmrp
CGMP interoperability mode     : IGMP_ONLY

4507-2ndFloor#

Open in new window

0
 
bschwartingAuthor Commented:
show ip igmp ?
4507-2ndFloor#show ip igmp ?
  filter     IGMP filter information
  groups     IGMP group membership information
  interface  IGMP interface information
  profile    IGMP profile information
  snooping   Snooping info on Catalyst Vlans
  udlr       IGMP undirectional link multicast routing information

Open in new window

0
 
harbor235Commented:


Do you have multicast routing enabled? If you need multicast traffic to be availble across subnets then you need to enable multicast routing, enable pim on interfaces you want multicast to work.

harbor235 ;}
0
 
harbor235Commented:

enable snooping on vlan 2 as well

harbor235 ;}
0
 
bschwartingAuthor Commented:
it is all on the same subnet, so I shouldn't need multicast routing, correct?
0
 
harbor235Commented:


You do if you expect the multicast traffic to be availble over multiple subnets. An RPF check needs to be done from the receivers top the source which is going to use mroutes.


harbor235 ;}
0
 
bschwartingAuthor Commented:
if I'm on the same subnet though, I don't need it, correct?
0
 
harbor235Commented:


If the source and receiver are on the same subnet then you do not need it

harbor235 ;}
0
 
bschwartingAuthor Commented:
ok, thanks, just trying to rule out all the possibilities here.  i think that is why we had igmp snooping turned off on vlan 2 as well.  that way the multicast would act just like broadcast, and not be limiting us either.
0
 
bschwartingAuthor Commented:
Just had our ISP out.  He setup a port mirror and did a sniffer on our port.  He is seeing the traffic here also, but it isn't making it all the way across the point to point.  He has gone to the 1st hop and saw the traffic there as well.  He is going the second and final hop tomorrow.  Will update as I know more!
0
 
bschwartingAuthor Commented:
Traffic is stopping at the last hop, he is checking into that issue now.
0
 
bschwartingAuthor Commented:
From what I can gather, Cisco has some reserved multicast IP's.  We changed it to another multicast address and the application is work again.  Thanks all for the help!
0
 
harbor235Commented:


great news !!


harbor235 ;}
0
 
bschwartingAuthor Commented:
Multicast reservation issue
0

Featured Post

Hire Technology Freelancers with Gigs

Work with freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely, and get projects done right.

  • 37
  • 15
  • 13
  • +2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now