Outlook 2000 Client Hangs when opening Exchange 5.5 mailbox over WAN
Background:
The central site has a 5.5 Exchange Server (SP4) running on a BDC named CORP (NT 4.0 with SP6a and Post6a security Rollup).
The remote sites (Dallas, Austin, and San Antonio) are all connected via a WAN using Netscreen 10's and 5's and VPN tunnels. Each of the remote sites has an NT4.0 server with WINS doing a push-pull replication with CORP which is also running WINS.
This configuration has been in place for at least 3 years and the remote workstations used to be Windows 98. Over time, many of them have been upgraded to Windows XP Professional. Outlook 2000 has always been used as the client and we were not aware of any major problems until about 6 weeks ago. Note: the private information store on the EXCHANGE server has grown to about 7GB over the years.
The Problem:
On a random but pretty frequent basis, users are going into OUTLOOK 2000 client at a remote site. The OUTLOOK screen loads but never completes (the number of messages in the inbox does not display). The hour glass stays until the user brings up task manager and kills OUTLOOK. Here's the interesting part. The user is unable to open his/hermailbox until we (the tech support people) do the following: at the 'local' site in Houston, we open OUTLOOK 2000 client, go into their mailbox, and sort their mail messages (usually by subject and then back to date received - but it really doesn't matter as long as we change something.). After this, the user at the remote site is able to successfully open their mailbox. Also, even if the mailbox opens, there will be some emails with attachments (usually larger attachments 100K+) that will not open - goes into the "Not Responding" stall.
So, is this a RPC over TCP/IP problem, a corrupted views problem, a router config problem ? Any insights greatly appreciated. PS: Yes I know that Server 2003 with Exchange 2003 and RPC via HTTPS is great, but my config will not support this right now. (Still have plenty of 98 PC's not to mention no AD or Exchange 2003).
The central site has a 5.5 Exchange Server (SP4) running on a BDC named CORP (NT 4.0 with SP6a and Post6a security Rollup).
The remote sites (Dallas, Austin, and San Antonio) are all connected via a WAN using Netscreen 10's and 5's and VPN tunnels. Each of the remote sites has an NT4.0 server with WINS doing a push-pull replication with CORP which is also running WINS.
This configuration has been in place for at least 3 years and the remote workstations used to be Windows 98. Over time, many of them have been upgraded to Windows XP Professional. Outlook 2000 has always been used as the client and we were not aware of any major problems until about 6 weeks ago. Note: the private information store on the EXCHANGE server has grown to about 7GB over the years.
The Problem:
On a random but pretty frequent basis, users are going into OUTLOOK 2000 client at a remote site. The OUTLOOK screen loads but never completes (the number of messages in the inbox does not display). The hour glass stays until the user brings up task manager and kills OUTLOOK. Here's the interesting part. The user is unable to open his/hermailbox until we (the tech support people) do the following: at the 'local' site in Houston, we open OUTLOOK 2000 client, go into their mailbox, and sort their mail messages (usually by subject and then back to date received - but it really doesn't matter as long as we change something.). After this, the user at the remote site is able to successfully open their mailbox. Also, even if the mailbox opens, there will be some emails with attachments (usually larger attachments 100K+) that will not open - goes into the "Not Responding" stall.
So, is this a RPC over TCP/IP problem, a corrupted views problem, a router config problem ? Any insights greatly appreciated. PS: Yes I know that Server 2003 with Exchange 2003 and RPC via HTTPS is great, but my config will not support this right now. (Still have plenty of 98 PC's not to mention no AD or Exchange 2003).
RPC over HTTP only exists in Windows 2003 and Outlook 2003. So that is not the problem. It sounds like the caching is the problem. I have found some issues with mobile users under different subnets causing the cached creds to block the connection. This is resolved by either registering a new DNS entry or creating a new profile.
ASKER
UPDATE: thanks to help from a Microsoft support person who could read the tea-leaves of a NETMON log, we now know the problem. The Netscreen VPN tunnels are not or will not send packets through that are between 1419 and 1450 bytes in length. You can test it be using the ping command with the -l (for length) option. Note that longer packets (1451 and up) work just fine. Hmmm... so everytime an OUTLOOK session just happens to get a packet in reply that is 1419 to 1450 in length, it will stall out with the client waiting forever for the packet to arrive but the server had sent it.
Now, it get's interesting. The router at the main 'hub' where the exchange server is located is a Netscreen 10 running software 3.0.1r6.0. There are command line options to set the MTU (Maximum Transmission Unit) as follows:
set flow tcp-mss (value)
set flow gre-out-tcp-mss (value)
set flow gre-in-tcp-mss (value)
I have set all these values to 1400 (the Netscreen allows a max of 1420).
I have 1 remote (Dallas) that is a Netscreen 10 and 2 that are Netscreen 5's running software 2.6.1r11.1.
On the Netscreen 5's, they appear to only have the command line option "set flow tcp-mss (value)". I have also set this to 1400.
Here's the puzzle. Before I made these changes, one of the Netscreen 5 VPN tunnels would work (ie, it will allow packets to go through between 1419 and 1450). After making the changes and rebooting the routers, it continues to work - BUT all the other VPN tunnels continue to fail in the same manner as before !!!!
I am going to tear out the VPN tunnels and rebuild them between the hub and the spokes. Also, FYI, I can set up a VPN tunnel between the remotes directly (ie, from Austin to San Antonio direct), and it will allow the 'large' packets to go through OK too !!!
I am continuing to work on the problem. Any Netscreen/router experts are welcome to add their expertise to this more specific problem.
Now, it get's interesting. The router at the main 'hub' where the exchange server is located is a Netscreen 10 running software 3.0.1r6.0. There are command line options to set the MTU (Maximum Transmission Unit) as follows:
set flow tcp-mss (value)
set flow gre-out-tcp-mss (value)
set flow gre-in-tcp-mss (value)
I have set all these values to 1400 (the Netscreen allows a max of 1420).
I have 1 remote (Dallas) that is a Netscreen 10 and 2 that are Netscreen 5's running software 2.6.1r11.1.
On the Netscreen 5's, they appear to only have the command line option "set flow tcp-mss (value)". I have also set this to 1400.
Here's the puzzle. Before I made these changes, one of the Netscreen 5 VPN tunnels would work (ie, it will allow packets to go through between 1419 and 1450). After making the changes and rebooting the routers, it continues to work - BUT all the other VPN tunnels continue to fail in the same manner as before !!!!
I am going to tear out the VPN tunnels and rebuild them between the hub and the spokes. Also, FYI, I can set up a VPN tunnel between the remotes directly (ie, from Austin to San Antonio direct), and it will allow the 'large' packets to go through OK too !!!
I am continuing to work on the problem. Any Netscreen/router experts are welcome to add their expertise to this more specific problem.
ASKER
Ok, after all the pain, the problem turned out to be a configuration setting in the Netscreen(s). Without the following settings, packets of size 1419 to 1450 would not be sent through the VPN tunnel. So, here are the settings added (at the Command Interface - ie, in a telnet session with the Netscreen):
Netscreen 10:
set flow tcp-mss 1420 (note: 1420 is the max value allowed)
set flow gre-in-tcp-mss 1420 (note: don't know if this is necessary, but put it in anyway)
set flow gre-out-tcp-mss 1420 (note: don't know if this is necessary, but put it in anyway)
set flow max-frag-pkt-size 1360 (60 less than tcp-mss - this was the key missing last piece of the puzzle).
save
Netscreen 5:
set flow tcp-mss 1420 (note: 1420 is the max value allowed)
set flow max-frag-pkt-size 1360 (60 less than tcp-mss - this was the key missing last piece of the puzzle).
save
I am officially asking that this call be closed. Thanks very much for those that worked on the problem - it was almost the death of me !!!
Netscreen 10:
set flow tcp-mss 1420 (note: 1420 is the max value allowed)
set flow gre-in-tcp-mss 1420 (note: don't know if this is necessary, but put it in anyway)
set flow gre-out-tcp-mss 1420 (note: don't know if this is necessary, but put it in anyway)
set flow max-frag-pkt-size 1360 (60 less than tcp-mss - this was the key missing last piece of the puzzle).
save
Netscreen 5:
set flow tcp-mss 1420 (note: 1420 is the max value allowed)
set flow max-frag-pkt-size 1360 (60 less than tcp-mss - this was the key missing last piece of the puzzle).
save
I am officially asking that this call be closed. Thanks very much for those that worked on the problem - it was almost the death of me !!!
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Do you have WebMail setup on your server? If so it might be worth getting the users to try accessing their mailbox like that as a test. This would rule Outlook in or out as the issue.
Also, there is an anoyying setting in Outlook where it integrates with Messenger (Tools, Options, Other) - it does not sound like the issue but might help.
Stuart