Network packet storm created by backup over routed Cisco network
Posted on 2008-10-20
We have an obvious problem in that whenever a backup is initiated using IBM Tivoli and its new super fast VTL tape library, network problems occur due to some sort of packet / broadcast storm.
We have recently moved this backup service to a building 100 metres away connected by 1GB fibre in order to serve as an immediate off-site backup for the time being until we sort out replication. The backup server now sits on another subnet and must now traverse a 'vlan interface' default gateway in order to do all of its backup to all the servers in the computer room (when the server was local a gateway was obviously not required).
Our HQ building and the large branch office building are connected via a routed vlan. The vlan at the branch building is vlan 29 (on the 172.29 subnet) - and the HQ building here with the server room is on vlan 20 (172.20 subnet). In order for this backup server to reach vlan 20 from its access vlan 29 Cisco 3560 switch, is must now go to the 172.29.1.252 address of the vlan 29 interface before leaping onto vlan 20. The vlan 29 interface is configured on our core switch (comprising 6 stacked 3750s as one logical unit) in the HQ central computer room. The core switch acts as the server vlan db - whereas the branch buildings floor switches are the client vlan db.
So as soon as the backup starts off from access vlan 29 addressing all of its servers to backup on vlan 20 all hell breaks loose on the vlan 20 network - which unfortunately includes some WAN bridged connections on the same subnet which now disconnect because, being on the same subnet and not partitioned off, also take on board all this excess traffic, swamp the slow bridged 2mb link, and drop off.
Of course I will partiion off these critical bridged connections in response to this now - but we never had this problem before when the backup server was plugged into access vlan 20 at HQ straight on the same subnet as the servers. So what has inter-vlan routing go to do with changing things so dramatically like this? And how can I solve it?