reesehenry
asked on
Cisco ASA 5505, Cisco AP541N and Apple TV Devices
Hi experts, I have weird issues with an ASA 5505, AP541N and Apple TV environment.
I have an ASA 5505 unlimited sec plus with 2 vlans (Work and Home).
Work VLAN (192.168.29.0/28) is tied to Work SSID on the AP541N
Home VLAN (192.168.28.0/28) is tied to Home SSID on the AP541N
I have a trunk port to serve the AP541N that allowed both Work VLAN and Home VLAN.
The AP541N management port is on the Work VLAN (call it 192.168.29.2).
The environment works fine for over 2 years, until.... the owner introduced a few Apple TV devices on the Home SSID/VLAN. After a few hours of working properly, the Home SSID will lose ability to connect to the Internet. But yet, I can still get a DHCP address from the ASA.
Rebooting the AP will not do anything. However, if I reboot the ASA - then all the SSID/VLANs seemed to work for a few more hours then - the Home SSID will lose ability to connect to the Internet. A reboot of the ASA works like a charm.
Any thoughts on any special requirements the Apple TV requires. My guess is that the ASA sees some traffic pattern that it doesn't like on the Apple TV devices and shut that down (similar to shunning - although it's not, since if I do sh shun it doesn't lists anything).
ASA Version 8.3(2) ------- CISCO TAC suggested an upgrade to 8.4(2) and we did - NO CHANGE IN BEHAVIOR
!
interface Vlan1
nameif inside
security-level 100
ip address 192.168.29.1 255.255.255.0
!
interface Vlan2
nameif outside
security-level 0
ip address dhcp setroute
!
interface Vlan3
nameif DMZ
security-level 50
ip address 192.168.28.1 255.255.255.0
!
interface Ethernet0/0
switchport access vlan 2
!
interface Ethernet0/1
switchport access vlan 3
!
interface Ethernet0/2
switchport access vlan 3
!
interface Ethernet0/3
switchport access vlan 3
!
interface Ethernet0/4
switchport trunk allowed vlan 1,3
switchport mode trunk
!
interface Ethernet0/5
switchport access vlan 3
!
interface Ethernet0/6
!
interface Ethernet0/7
switchport access vlan 3
!
boot system disk0:/asa832-k8.bin CHANGED
ftp mode passive
dns server-group DefaultDNS
domain-name corp.xxxxxx.com
object network obj_any
subnet 0.0.0.0 0.0.0.0
object network DMZ_LAN
subnet 192.168.28.0 255.255.255.0
object network Internal_LAN
subnet 192.168.29.0 255.255.255.0
object-group network internal
network-object 192.168.29.0 255.255.255.0
object-group network xxxxx
network-object 10.138.0.0 255.255.240.0
access-list outside_cryptomap extended permit ip 192.168.29.0 255.255.255.0 10.1 38.0.0 255.255.240.0
pager lines 24
logging enable
logging buffered debugging
logging asdm informational
mtu inside 1500
mtu outside 1500
mtu DMZ 1500
no failover
icmp unreachable rate-limit 1 burst-size 1
asdm image disk0:/asdm-645-204.bin CHANGED
no asdm history enable
arp timeout 14400
nat (inside,outside) source static internal internal destination static xxxxx
xxx
nat (inside,DMZ) source static Internal_LAN Internal_LAN
!
object network obj_any
nat (inside,outside) dynamic interface
object network DMZ_LAN
nat (DMZ,outside) dynamic interface
timeout xlate 3:00:00
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
dynamic-access-policy-reco
aaa authentication ssh console LOCAL
aaa authentication serial console LOCAL
aaa authentication http console LOCAL
http server enable 4443
http 10.138.0.0 255.255.224.0 outside
http 192.168.29.0 255.255.255.0 inside
no snmp-server location
no snmp-server contact
snmp-server enable traps snmp authentication linkup linkdown coldstart
crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac
crypto ipsec transform-set ESP-AES-128-SHA esp-aes esp-sha-hmac
crypto ipsec transform-set ESP-AES-128-MD5 esp-aes esp-md5-hmac
crypto ipsec transform-set ESP-AES-192-SHA esp-aes-192 esp-sha-hmac
crypto ipsec transform-set ESP-AES-192-MD5 esp-aes-192 esp-md5-hmac
crypto ipsec transform-set ESP-AES-256-SHA esp-aes-256 esp-sha-hmac
crypto ipsec transform-set ESP-AES-256-MD5 esp-aes-256 esp-md5-hmac
crypto ipsec transform-set ESP-3DES-MD5 esp-3des esp-md5-hmac
crypto ipsec transform-set ESP-DES-SHA esp-des esp-sha-hmac
crypto ipsec transform-set ESP-DES-MD5 esp-des esp-md5-hmac
crypto ipsec security-association lifetime seconds 28800
crypto ipsec security-association lifetime kilobytes 4608000
crypto map outside_map 1 match address outside_cryptomap
crypto map outside_map 1 set peer xxx.x.xxx.xxx
crypto map outside_map 1 set transform-set ESP-3DES-SHA
crypto map outside_map interface outside
crypto ca trustpoint _SmartCallHome_ServerCA
crl configure
crypto ca certificate chain _SmartCallHome_ServerCA
certificate ca 6ecc7aa5a7032009b8cebcf4e9
308205ec 308204d4 a0030201 0202106e cc7aa5a7 032009b8 cebcf4e9 52d49130
0d06092a 864886f7 0d010105 05003081 ca310b30 09060355 04061302 55533117
30150603 55040a13 0e566572 69536967 6e2c2049 6e632e31 1f301d06 0355040b
13165665 72695369 676e2054 72757374 204e6574 776f726b 313a3038 06035504
0b133128 63292032 30303620 56657269 5369676e 2c20496e 632e202d 20466f72
20617574 686f7269 7a656420 75736520 6f6e6c79 31453043 06035504 03133c56
65726953 69676e20 436c6173 73203320 5075626c 69632050 72696d61 72792043
65727469 66696361 74696f6e 20417574 686f7269 7479202d 20473530 1e170d31
30303230 38303030 3030305a 170d3230 30323037 32333539 35395a30 81b5310b
30090603 55040613 02555331 17301506 0355040a 130e5665 72695369 676e2c20
496e632e 311f301d 06035504 0b131656 65726953 69676e20 54727573 74204e65
74776f72 6b313b30 39060355 040b1332 5465726d 73206f66 20757365 20617420
68747470 733a2f2f 7777772e 76657269 7369676e 2e636f6d 2f727061 20286329
3130312f 302d0603 55040313 26566572 69536967 6e20436c 61737320 33205365
63757265 20536572 76657220 4341202d 20473330 82012230 0d06092a 864886f7
0d010101 05000382 010f0030 82010a02 82010100 b187841f c20c45f5 bcab2597
a7ada23e 9cbaf6c1 39b88bca c2ac56c6 e5bb658e 444f4dce 6fed094a d4af4e10
9c688b2e 957b899b 13cae234 34c1f35b f3497b62 83488174 d188786c 0253f9bc
7f432657 5833833b 330a17b0 d04e9124 ad867d64 12dc744a 34a11d0a ea961d0b
15fca34b 3bce6388 d0f82d0c 948610ca b69a3dca eb379c00 48358629 5078e845
63cd1941 4ff595ec 7b98d4c4 71b350be 28b38fa0 b9539cf5 ca2c23a9 fd1406e8
18b49ae8 3c6e81fd e4cd3536 b351d369 ec12ba56 6e6f9b57 c58b14e7 0ec79ced
4a546ac9 4dc5bf11 b1ae1c67 81cb4455 33997f24 9b3f5345 7f861af3 3cfa6d7f
81f5b84a d3f58537 1cb5a6d0 09e4187b 384efa0f 02030100 01a38201 df308201
db303406 082b0601 05050701 01042830 26302406 082b0601 05050730 01861868
7474703a 2f2f6f63 73702e76 65726973 69676e2e 636f6d30 12060355 1d130101
ff040830 060101ff 02010030 70060355 1d200469 30673065 060b6086 480186f8
45010717 03305630 2806082b 06010505 07020116 1c687474 70733a2f 2f777777
2e766572 69736967 6e2e636f 6d2f6370 73302a06 082b0601 05050702 02301e1a
1c687474 70733a2f 2f777777 2e766572 69736967 6e2e636f 6d2f7270 61303406
03551d1f 042d302b 3029a027 a0258623 68747470 3a2f2f63 726c2e76 65726973
69676e2e 636f6d2f 70636133 2d67352e 63726c30 0e060355 1d0f0101 ff040403
02010630 6d06082b 06010505 07010c04 61305fa1 5da05b30 59305730 55160969
6d616765 2f676966 3021301f 30070605 2b0e0302 1a04148f e5d31a86 ac8d8e6b
c3cf806a d448182c 7b192e30 25162368 7474703a 2f2f6c6f 676f2e76 65726973
69676e2e 636f6d2f 76736c6f 676f2e67 69663028 0603551d 11042130 1fa41d30
1b311930 17060355 04031310 56657269 5369676e 4d504b49 2d322d36 301d0603
551d0e04 1604140d 445c1653 44c1827e 1d20ab25 f40163d8 be79a530 1f060355
1d230418 30168014 7fd365a7 c2ddecbb f03009f3 4339fa02 af333133 300d0609
2a864886 f70d0101 05050003 82010100 0c8324ef ddc30cd9 589cfe36 b6eb8a80
4bd1a3f7 9df3cc53 ef829ea3 a1e697c1 589d756c e01d1b4c fad1c12d 05c0ea6e
b2227055 d9203340 3307c265 83fa8f43 379bea0e 9a6c70ee f69c803b d937f47a
6decd018 7d494aca 99c71928 a2bed877 24f78526 866d8705 404167d1 273aeddc
481d22cd 0b0b8bbc f4b17bfd b499a8e9 762ae11a 2d876e74 d388dd1e 22c6df16
b62b8214 0a945cf2 50ecafce ff62370d ad65d306 4153ed02 14c8b558 28a1ace0
5becb37f 954afb03 c8ad26db e6667812 4ad99f42 fbe198e6 42839b8f 8f6724e8
6119b5dd cdb50b26 058ec36e c4c875b8 46cfe218 065ea9ae a8819a47 16de0c28
6c2527b9 deb78458 c61f381e a4c4cb66
quit
crypto isakmp enable outside
crypto isakmp policy 10
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 86400
telnet timeout 5
ssh 192.168.29.0 255.255.255.0 inside
ssh timeout 15
console timeout 0
management-access inside
dhcpd auto_config outside
!
dhcpd address 192.168.29.5-192.168.29.9 inside
dhcpd dns 10.138.4.9 4.2.2.2 interface inside
dhcpd wins 10.138.4.9 10.138.0.8 interface inside
dhcpd domain corp.xxxxxx.com interface inside
dhcpd enable inside
!
dhcpd address 192.168.28.100-192.168.28.
dhcpd dns 75.75.75.75 75.75.76.76 interface DMZ
dhcpd enable DMZ
!
threat-detection basic-threat
threat-detection statistics access-list
no threat-detection statistics tcp-intercept
webvpn
username xxxx password xxx encrypted privilege 15
username xxxx password xxx encrypted privilege 15
tunnel-group xxx.x.xxx.xxx type ipsec-l2l
tunnel-group xxx.x.xxx.xxx ipsec-attributes
pre-shared-key *****
!
!
prompt hostname context
call-home
profile CiscoTAC-1
no active
destination address http https://tools.cisco.com/its/service/oddce/services/DD CEService
destination address email callhome@cisco.com
destination transport-method http
subscribe-to-alert-group diagnostic
subscribe-to-alert-group environment
subscribe-to-alert-group inventory periodic monthly
subscribe-to-alert-group configuration periodic monthly
subscribe-to-alert-group telemetry periodic daily
Cryptochecksum:460e2aba96d
: end
I have an ASA 5505 unlimited sec plus with 2 vlans (Work and Home).
Work VLAN (192.168.29.0/28) is tied to Work SSID on the AP541N
Home VLAN (192.168.28.0/28) is tied to Home SSID on the AP541N
I have a trunk port to serve the AP541N that allowed both Work VLAN and Home VLAN.
The AP541N management port is on the Work VLAN (call it 192.168.29.2).
The environment works fine for over 2 years, until.... the owner introduced a few Apple TV devices on the Home SSID/VLAN. After a few hours of working properly, the Home SSID will lose ability to connect to the Internet. But yet, I can still get a DHCP address from the ASA.
Rebooting the AP will not do anything. However, if I reboot the ASA - then all the SSID/VLANs seemed to work for a few more hours then - the Home SSID will lose ability to connect to the Internet. A reboot of the ASA works like a charm.
Any thoughts on any special requirements the Apple TV requires. My guess is that the ASA sees some traffic pattern that it doesn't like on the Apple TV devices and shut that down (similar to shunning - although it's not, since if I do sh shun it doesn't lists anything).
ASA Version 8.3(2) ------- CISCO TAC suggested an upgrade to 8.4(2) and we did - NO CHANGE IN BEHAVIOR
!
interface Vlan1
nameif inside
security-level 100
ip address 192.168.29.1 255.255.255.0
!
interface Vlan2
nameif outside
security-level 0
ip address dhcp setroute
!
interface Vlan3
nameif DMZ
security-level 50
ip address 192.168.28.1 255.255.255.0
!
interface Ethernet0/0
switchport access vlan 2
!
interface Ethernet0/1
switchport access vlan 3
!
interface Ethernet0/2
switchport access vlan 3
!
interface Ethernet0/3
switchport access vlan 3
!
interface Ethernet0/4
switchport trunk allowed vlan 1,3
switchport mode trunk
!
interface Ethernet0/5
switchport access vlan 3
!
interface Ethernet0/6
!
interface Ethernet0/7
switchport access vlan 3
!
boot system disk0:/asa832-k8.bin CHANGED
ftp mode passive
dns server-group DefaultDNS
domain-name corp.xxxxxx.com
object network obj_any
subnet 0.0.0.0 0.0.0.0
object network DMZ_LAN
subnet 192.168.28.0 255.255.255.0
object network Internal_LAN
subnet 192.168.29.0 255.255.255.0
object-group network internal
network-object 192.168.29.0 255.255.255.0
object-group network xxxxx
network-object 10.138.0.0 255.255.240.0
access-list outside_cryptomap extended permit ip 192.168.29.0 255.255.255.0 10.1 38.0.0 255.255.240.0
pager lines 24
logging enable
logging buffered debugging
logging asdm informational
mtu inside 1500
mtu outside 1500
mtu DMZ 1500
no failover
icmp unreachable rate-limit 1 burst-size 1
asdm image disk0:/asdm-645-204.bin CHANGED
no asdm history enable
arp timeout 14400
nat (inside,outside) source static internal internal destination static xxxxx
xxx
nat (inside,DMZ) source static Internal_LAN Internal_LAN
!
object network obj_any
nat (inside,outside) dynamic interface
object network DMZ_LAN
nat (DMZ,outside) dynamic interface
timeout xlate 3:00:00
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
dynamic-access-policy-reco
aaa authentication ssh console LOCAL
aaa authentication serial console LOCAL
aaa authentication http console LOCAL
http server enable 4443
http 10.138.0.0 255.255.224.0 outside
http 192.168.29.0 255.255.255.0 inside
no snmp-server location
no snmp-server contact
snmp-server enable traps snmp authentication linkup linkdown coldstart
crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac
crypto ipsec transform-set ESP-AES-128-SHA esp-aes esp-sha-hmac
crypto ipsec transform-set ESP-AES-128-MD5 esp-aes esp-md5-hmac
crypto ipsec transform-set ESP-AES-192-SHA esp-aes-192 esp-sha-hmac
crypto ipsec transform-set ESP-AES-192-MD5 esp-aes-192 esp-md5-hmac
crypto ipsec transform-set ESP-AES-256-SHA esp-aes-256 esp-sha-hmac
crypto ipsec transform-set ESP-AES-256-MD5 esp-aes-256 esp-md5-hmac
crypto ipsec transform-set ESP-3DES-MD5 esp-3des esp-md5-hmac
crypto ipsec transform-set ESP-DES-SHA esp-des esp-sha-hmac
crypto ipsec transform-set ESP-DES-MD5 esp-des esp-md5-hmac
crypto ipsec security-association lifetime seconds 28800
crypto ipsec security-association lifetime kilobytes 4608000
crypto map outside_map 1 match address outside_cryptomap
crypto map outside_map 1 set peer xxx.x.xxx.xxx
crypto map outside_map 1 set transform-set ESP-3DES-SHA
crypto map outside_map interface outside
crypto ca trustpoint _SmartCallHome_ServerCA
crl configure
crypto ca certificate chain _SmartCallHome_ServerCA
certificate ca 6ecc7aa5a7032009b8cebcf4e9
308205ec 308204d4 a0030201 0202106e cc7aa5a7 032009b8 cebcf4e9 52d49130
0d06092a 864886f7 0d010105 05003081 ca310b30 09060355 04061302 55533117
30150603 55040a13 0e566572 69536967 6e2c2049 6e632e31 1f301d06 0355040b
13165665 72695369 676e2054 72757374 204e6574 776f726b 313a3038 06035504
0b133128 63292032 30303620 56657269 5369676e 2c20496e 632e202d 20466f72
20617574 686f7269 7a656420 75736520 6f6e6c79 31453043 06035504 03133c56
65726953 69676e20 436c6173 73203320 5075626c 69632050 72696d61 72792043
65727469 66696361 74696f6e 20417574 686f7269 7479202d 20473530 1e170d31
30303230 38303030 3030305a 170d3230 30323037 32333539 35395a30 81b5310b
30090603 55040613 02555331 17301506 0355040a 130e5665 72695369 676e2c20
496e632e 311f301d 06035504 0b131656 65726953 69676e20 54727573 74204e65
74776f72 6b313b30 39060355 040b1332 5465726d 73206f66 20757365 20617420
68747470 733a2f2f 7777772e 76657269 7369676e 2e636f6d 2f727061 20286329
3130312f 302d0603 55040313 26566572 69536967 6e20436c 61737320 33205365
63757265 20536572 76657220 4341202d 20473330 82012230 0d06092a 864886f7
0d010101 05000382 010f0030 82010a02 82010100 b187841f c20c45f5 bcab2597
a7ada23e 9cbaf6c1 39b88bca c2ac56c6 e5bb658e 444f4dce 6fed094a d4af4e10
9c688b2e 957b899b 13cae234 34c1f35b f3497b62 83488174 d188786c 0253f9bc
7f432657 5833833b 330a17b0 d04e9124 ad867d64 12dc744a 34a11d0a ea961d0b
15fca34b 3bce6388 d0f82d0c 948610ca b69a3dca eb379c00 48358629 5078e845
63cd1941 4ff595ec 7b98d4c4 71b350be 28b38fa0 b9539cf5 ca2c23a9 fd1406e8
18b49ae8 3c6e81fd e4cd3536 b351d369 ec12ba56 6e6f9b57 c58b14e7 0ec79ced
4a546ac9 4dc5bf11 b1ae1c67 81cb4455 33997f24 9b3f5345 7f861af3 3cfa6d7f
81f5b84a d3f58537 1cb5a6d0 09e4187b 384efa0f 02030100 01a38201 df308201
db303406 082b0601 05050701 01042830 26302406 082b0601 05050730 01861868
7474703a 2f2f6f63 73702e76 65726973 69676e2e 636f6d30 12060355 1d130101
ff040830 060101ff 02010030 70060355 1d200469 30673065 060b6086 480186f8
45010717 03305630 2806082b 06010505 07020116 1c687474 70733a2f 2f777777
2e766572 69736967 6e2e636f 6d2f6370 73302a06 082b0601 05050702 02301e1a
1c687474 70733a2f 2f777777 2e766572 69736967 6e2e636f 6d2f7270 61303406
03551d1f 042d302b 3029a027 a0258623 68747470 3a2f2f63 726c2e76 65726973
69676e2e 636f6d2f 70636133 2d67352e 63726c30 0e060355 1d0f0101 ff040403
02010630 6d06082b 06010505 07010c04 61305fa1 5da05b30 59305730 55160969
6d616765 2f676966 3021301f 30070605 2b0e0302 1a04148f e5d31a86 ac8d8e6b
c3cf806a d448182c 7b192e30 25162368 7474703a 2f2f6c6f 676f2e76 65726973
69676e2e 636f6d2f 76736c6f 676f2e67 69663028 0603551d 11042130 1fa41d30
1b311930 17060355 04031310 56657269 5369676e 4d504b49 2d322d36 301d0603
551d0e04 1604140d 445c1653 44c1827e 1d20ab25 f40163d8 be79a530 1f060355
1d230418 30168014 7fd365a7 c2ddecbb f03009f3 4339fa02 af333133 300d0609
2a864886 f70d0101 05050003 82010100 0c8324ef ddc30cd9 589cfe36 b6eb8a80
4bd1a3f7 9df3cc53 ef829ea3 a1e697c1 589d756c e01d1b4c fad1c12d 05c0ea6e
b2227055 d9203340 3307c265 83fa8f43 379bea0e 9a6c70ee f69c803b d937f47a
6decd018 7d494aca 99c71928 a2bed877 24f78526 866d8705 404167d1 273aeddc
481d22cd 0b0b8bbc f4b17bfd b499a8e9 762ae11a 2d876e74 d388dd1e 22c6df16
b62b8214 0a945cf2 50ecafce ff62370d ad65d306 4153ed02 14c8b558 28a1ace0
5becb37f 954afb03 c8ad26db e6667812 4ad99f42 fbe198e6 42839b8f 8f6724e8
6119b5dd cdb50b26 058ec36e c4c875b8 46cfe218 065ea9ae a8819a47 16de0c28
6c2527b9 deb78458 c61f381e a4c4cb66
quit
crypto isakmp enable outside
crypto isakmp policy 10
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 86400
telnet timeout 5
ssh 192.168.29.0 255.255.255.0 inside
ssh timeout 15
console timeout 0
management-access inside
dhcpd auto_config outside
!
dhcpd address 192.168.29.5-192.168.29.9 inside
dhcpd dns 10.138.4.9 4.2.2.2 interface inside
dhcpd wins 10.138.4.9 10.138.0.8 interface inside
dhcpd domain corp.xxxxxx.com interface inside
dhcpd enable inside
!
dhcpd address 192.168.28.100-192.168.28.
dhcpd dns 75.75.75.75 75.75.76.76 interface DMZ
dhcpd enable DMZ
!
threat-detection basic-threat
threat-detection statistics access-list
no threat-detection statistics tcp-intercept
webvpn
username xxxx password xxx encrypted privilege 15
username xxxx password xxx encrypted privilege 15
tunnel-group xxx.x.xxx.xxx type ipsec-l2l
tunnel-group xxx.x.xxx.xxx ipsec-attributes
pre-shared-key *****
!
!
prompt hostname context
call-home
profile CiscoTAC-1
no active
destination address http https://tools.cisco.com/its/service/oddce/services/DD CEService
destination address email callhome@cisco.com
destination transport-method http
subscribe-to-alert-group diagnostic
subscribe-to-alert-group environment
subscribe-to-alert-group inventory periodic monthly
subscribe-to-alert-group configuration periodic monthly
subscribe-to-alert-group telemetry periodic daily
Cryptochecksum:460e2aba96d
: end
Gareth Gudger
I had an issue like this a while back and oddly it required me to update the software on the AP to the latest version. Are you on the most current AP build?
I wonder if there is an issue with NAT.
Can you post the show xlate output from the ASA?
Can you post the show xlate output from the ASA?
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
This was one of the last issues my predecessor dealt with before leaving. I have no idea how we fixed it, I'm just doing this to close the question.