Outlook 2010 and 2007 sp3 VERY slow with install of new Exchange 2013 server to Exchange 2007 environment.
We are in the process of migrating to Exchange 2013 from Exchange 2007. I have installed the new Exchange 2013 on a fresh install of Windows server 2012.
I setup a third party cert from godaddy and assigned it, then removed all the built in certs. Everything is working after reboot. OWA works, ECP works, activesync works, email works between the servers and the few mailboxes we have moved. Send and receive connectors are setup, edge sync was recreated to allow co-existence... no problems there.
Everything appears fine, but when i launch outlook 2010 or 2007sp3, it sits there for 30 seconds "loading profile", then once it loads, it takes about 20-40 seconds to switch between emails. I moved three mailboxes, used three different computers to access them, they all do the same thing when accessing email from outlook. We even used a laptop that had outlook 2007 installed, same result. OWA access and use works fine, no slowness.
I disabled IPV6, no change.
I started up wireshark and watched outlook load, there is a ton of back and forth file transfer actions. Also, it seems the larger the email, the longer the wait.
This is just odd. Why would everything appear to be working, no errors in the logs, but outlook be painfully slow to use.
Also, on two of the mailboxes that were migrated, the contacts and junk email folders were duplicated. Seems the old system folders were recreated, so we had to move the content from the old to the new folders. We were able to delete the old folders without issue. Two other accounts that got moved, didnt have the duplicate folder issue.
Also, i should note that the users devices and clients were successfully redirected to the new server without issue.
I should note that i have been administering exchange since 5.5, so i am not new to the scene, but this is a new version, so i may have missed something simple.... I say that because an extensive internet search has revealed nothing but issues with the outlook client, which isn't the case based on several different computers having the issue(i could be wrong though).
Thanks ahead
I setup a third party cert from godaddy and assigned it, then removed all the built in certs. Everything is working after reboot. OWA works, ECP works, activesync works, email works between the servers and the few mailboxes we have moved. Send and receive connectors are setup, edge sync was recreated to allow co-existence... no problems there.
Everything appears fine, but when i launch outlook 2010 or 2007sp3, it sits there for 30 seconds "loading profile", then once it loads, it takes about 20-40 seconds to switch between emails. I moved three mailboxes, used three different computers to access them, they all do the same thing when accessing email from outlook. We even used a laptop that had outlook 2007 installed, same result. OWA access and use works fine, no slowness.
I disabled IPV6, no change.
I started up wireshark and watched outlook load, there is a ton of back and forth file transfer actions. Also, it seems the larger the email, the longer the wait.
This is just odd. Why would everything appear to be working, no errors in the logs, but outlook be painfully slow to use.
Also, on two of the mailboxes that were migrated, the contacts and junk email folders were duplicated. Seems the old system folders were recreated, so we had to move the content from the old to the new folders. We were able to delete the old folders without issue. Two other accounts that got moved, didnt have the duplicate folder issue.
Also, i should note that the users devices and clients were successfully redirected to the new server without issue.
I should note that i have been administering exchange since 5.5, so i am not new to the scene, but this is a new version, so i may have missed something simple.... I say that because an extensive internet search has revealed nothing but issues with the outlook client, which isn't the case based on several different computers having the issue(i could be wrong though).
Thanks ahead
Henry Dunn
Most likely you didn't change your OAB to point to the existing servers and when the new server was installed your clients are downloading a new OAB. One of the steps before you install the first Exchange 2013 servers into your org is to set all of the existing clients OAB for the existing server. There's a few powershell commands that will allow you to quickly determine what DBs aren't set for an OAB.
ASKER
The exchange 2007 server has the default OAB it created assigned to all databases. This was done when we migrated to 2007 from 2003 a while back.
When i built the new server, once it was online, it had created it own OAB, so i set the new databases on that server to point to the new OAB.
This should take care of the OAB for the clients.
However, you did highlight the fact that, i should of mentioned that all the existing outlook clients with mailboxes on the 2007 server are working fine. Its only the clients that are attached to mailboxes that have been migrated to the new exchange 2013 server. This is cu5 btw.
When i built the new server, once it was online, it had created it own OAB, so i set the new databases on that server to point to the new OAB.
This should take care of the OAB for the clients.
However, you did highlight the fact that, i should of mentioned that all the existing outlook clients with mailboxes on the 2007 server are working fine. Its only the clients that are attached to mailboxes that have been migrated to the new exchange 2013 server. This is cu5 btw.
ASKER
I forgot to add
Outlook is still slow when i use the outlook /safe to launch in safe mode. I also used the /rpcdiag to see the stats and it shows very high avg resp values... like 200+ for each one.
Outlook is still slow when i use the outlook /safe to launch in safe mode. I also used the /rpcdiag to see the stats and it shows very high avg resp values... like 200+ for each one.
If you set the new DBs to their own OAB, I think the clients, once migrated, are downloading a new OAB and this is unneccesary. What type of traffic did you see in wireshark? Find the signature for OAB file transfer and you'll know whether the migration is triggering the downloads. The traffic is what's making your clients run slow. Where is the traffic coming from? I assumed the 2013 server, right?
ASKER
Yea, the traffic is to and from the 2013 server. Its not only that the client is very slow to load, but its also slow if you are in the inbox and are clicking on different emails( with the reading pane at the bottom ). I left it up all night and it was still 10-20 seconds to move between emails.
The traffic is showing as TCP and TLS1 only.
I should mention that my client workstation is plugged into a gig switch which is uplinked into the 10 gig the 2013 virtual host is plugged into. The other clients are on a 10/100 which is uplinked to the same 1 gig switch as mine, so bandwidth shouldnt be an issue. ( i did a network test file transfer from my machine to the server and it was running at 1gig speeds ).
The traffic is showing as TCP and TLS1 only.
I should mention that my client workstation is plugged into a gig switch which is uplinked into the 10 gig the 2013 virtual host is plugged into. The other clients are on a 10/100 which is uplinked to the same 1 gig switch as mine, so bandwidth shouldnt be an issue. ( i did a network test file transfer from my machine to the server and it was running at 1gig speeds ).
Do you have a unique namespace for the Exchange 2013 server? Have you changed all of the URLs with Exchange 2013 to match that unique namespace?
Removing the default SSL certificates was a mistake, because Exchange communicates internally using them. Therefore the first thing you should do is generate a new self-signed SSL certificate.
In EMS, run new-exchangecertificate, no other switches or commands. You will be prompted about overwriting the default SMTP certificate. Say yes to that and then restart the transport service.
Which build of Exchange 2013 are you using?
Simon.
Removing the default SSL certificates was a mistake, because Exchange communicates internally using them. Therefore the first thing you should do is generate a new self-signed SSL certificate.
In EMS, run new-exchangecertificate, no other switches or commands. You will be prompted about overwriting the default SMTP certificate. Say yes to that and then restart the transport service.
Which build of Exchange 2013 are you using?
Simon.
You can use the public certs for SMTP though. You don't NEED self signed for anything.
You can indeed use public certificates for SMTP, however if you have multiple servers then for Exchange to send email between the servers you need to have the server's FQDN on the certificate.
As you cannot get internal host names on public certificates any longer that means using a self signed certificate.
The receive connectors which receive email from the other Exchange servers can only accept blank, NetBIOS or the FQDN of the server as the address on them - using an external name isn't possible.
Simon.
As you cannot get internal host names on public certificates any longer that means using a self signed certificate.
The receive connectors which receive email from the other Exchange servers can only accept blank, NetBIOS or the FQDN of the server as the address on them - using an external name isn't possible.
Simon.
ASKER
Mail is still working, but i will re-add the cert as directed. The issue was going on before i deleted the certificates. I probably should of said this earlier...
The certificates probably work because our internal domain name is the same as our external domain name. This was setup back in 2000....
Exchange 2013 CU5
The certificates probably work because our internal domain name is the same as our external domain name. This was setup back in 2000....
Exchange 2013 CU5
ASKER
We do have three servers, two 2007 and one 2013, the 2007 stuff will be decommissioned once the 2013 environment is working 100%
You cant get an internal hostname on a public certificate any longer? Or you cant get non valid internal hostnames? The change was that you couldn't get .local or .internal or similar non valid TLDs on certs anymore. And I'm not even sure if that's gone into action yet, I think it's next year that that's an official rule. At any rate, if your domain is a tld or child of your registered tld, getting a cert for your servers FQDN should be no problem. So in that case, the receive connectors can be configured with FQDN and they'll be fine. If you need internal mail routing to be secured via public certs, which is a requirement for some orgs, this is the method you'll have to use.
Ah, you beat me. If your internal domain name matches your external, you're fine. I wouldn't worry too much about recreating that self signed cert, it wont resolve your issue most likely.
ASKER
I created a new account with a new email box on the new 2013 server and it has the same issue. its got to be something with the server or its settings... its a virtual, so i doubt its anything outside the OS.
At this point, i am tempted to setup another exchange mailbox server and see if that one has the same issues....
At this point, i am tempted to setup another exchange mailbox server and see if that one has the same issues....
ASKER
before you ask... vmware 5.5
Sounds like that might be helpful. What's the resources on the current box and what's the utilization levels?
ASKER
12GB memory and 4 processors. utilization is 3.5-4Ghz and host memory is at 10GB but thats because of exchange...
Yes Exchange does that. Check the logs as well. Maybe theres a clue as to what the problem could be.
ASKER
yea, i did that and cleared out an error it had with the edge sync.. i had to resubscribe it, but thats all green now. There are two errors related to the performance monitor, but after looking them up online, everyone has them... apparently they should be informational not warning... but that is MS for ya.
ASKER
i should note ... apparently when you add a 2013 server to a 2007 environment and you are using a 2007 edge server( separate from the mailbox server.. ), you must re-subscribe the edge server to the 2007 hub and then manually run the start-sync on both the hub and the new exchange 2013 servers.
ASKER
We called Microsoft and opened a ticket with the exchange team, they have been attempting many things over the last two weeks with no change in situation.
We did install an outlook 2010 client on the exchange server and it works properly. The avg response times are correct as well.
Last thing we did was pull a packet capture between workstation client and server( both sides ), they are "analysing" that now.
I also opened a ticket with VMWare, perhaps there is an issue on their side being that this is the new vmx10 hardware on 5.5 vsphere with 2012 server and exchange 2013.
Its also possible this a 2007 to 2013 specific issue, perhaps there is something in our AD.....
Everything else in the exchange 2007 and 2013 environment is working 100%, so this is just very odd. My IT test subjects are sticking it out with webmail for now...
We did install an outlook 2010 client on the exchange server and it works properly. The avg response times are correct as well.
Last thing we did was pull a packet capture between workstation client and server( both sides ), they are "analysing" that now.
I also opened a ticket with VMWare, perhaps there is an issue on their side being that this is the new vmx10 hardware on 5.5 vsphere with 2012 server and exchange 2013.
Its also possible this a 2007 to 2013 specific issue, perhaps there is something in our AD.....
Everything else in the exchange 2007 and 2013 environment is working 100%, so this is just very odd. My IT test subjects are sticking it out with webmail for now...
ASKER
So, no solution to this problem yet. MS has had three technicians work on this, each asking for the same things(which is really irritating) and then some new things.
Upgraded both the test machine client and the server client to 2013 and both remained the same. So at this point they think its network related. I dont think so since my test machine is three switches away from the server and i tested on a terminal server that was on the same esxi host(only vmware networking at play there ).
I brought up another exchange 2013 server and finally pointed everything over to it for another user, same issue. I cant tell 100% that its not talking to the other new server, thought the avg response info shows it only talking to the new server.
I am wondering if this is an AD issue of some kind. We have been running this domain and exchange since windows 2000. The client on the exchange server kinda rules that out and does point to a networking issue though... but why does exchange 2007 work fine(along with 60 other servers). The only answer i can come up with is 2007 is RPC and 2013 is https...
Very frustrating at this point as this project is being delayed by weeks.
Upgraded both the test machine client and the server client to 2013 and both remained the same. So at this point they think its network related. I dont think so since my test machine is three switches away from the server and i tested on a terminal server that was on the same esxi host(only vmware networking at play there ).
I brought up another exchange 2013 server and finally pointed everything over to it for another user, same issue. I cant tell 100% that its not talking to the other new server, thought the avg response info shows it only talking to the new server.
I am wondering if this is an AD issue of some kind. We have been running this domain and exchange since windows 2000. The client on the exchange server kinda rules that out and does point to a networking issue though... but why does exchange 2007 work fine(along with 60 other servers). The only answer i can come up with is 2007 is RPC and 2013 is https...
Very frustrating at this point as this project is being delayed by weeks.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
no change, i cant see network latency being an issue anyways, the server is 3 1Gb switches away and clients only access via 100Mb switches, so i essentially have the biggest pipe besides the servers.
The link is definitely chatty when you look at wireshark or net mon while outlook opens and functions.
The link is definitely chatty when you look at wireshark or net mon while outlook opens and functions.
Can you upload 30 seconds of wire shark trace somewhere?
Preferably taken from the client while in outlook connected to exchange 2013
Preferably taken from the client while in outlook connected to exchange 2013
ASKER
Wow, thanks for the call! I applied the fix to the wrong network adapter per our call.
Once it was applied to the correct network adapter, the slowness went away and outlook is just as fast as it used to be now. I cant believe microsoft hasnt issued a server side fix for this...
I am going to let them know and demand they produce a fix, either server or client.
Again, i appreciate you following up on my inability to follow instructions! I really do hope they fix this soon, i cant imagine organizations switching over to 2013 and dealing with this problem.
Once it was applied to the correct network adapter, the slowness went away and outlook is just as fast as it used to be now. I cant believe microsoft hasnt issued a server side fix for this...
I am going to let them know and demand they produce a fix, either server or client.
Again, i appreciate you following up on my inability to follow instructions! I really do hope they fix this soon, i cant imagine organizations switching over to 2013 and dealing with this problem.
ASKER
Thanks again!