Link to home
Create AccountLog in
Avatar of swatgear
swatgear

asked on

Moving mailboxes from Ex 2000 to Ex 2003 is REALLY slow.

This weekend I moved about 18 mailboxes from an Exchange 2000 machine to an Exchange 2003 machine.

I've moved mailboxes in the past (in the same direction between the same servers) but I never paid attention to the data throughput because it seemed to take a reasonable amount of time. For example, about two weeks ago I moved my own mailbox which is about 900MB and it didn't take very long at all. Maybe 10 minutes if I recall correctly.

This weekend on the other hand took substantially longer. I was getting about 1.5MB/sec on average. Sometimes it would spike to 4MB but for the most part it hovered between 1MB/sec and 2MB/sec.

Why would the data transfer rate be so low?
Avatar of abhaigh
abhaigh
Flag of United Kingdom of Great Britain and Northern Ireland image

what were you using to move the mailboxes?

how large were the mailboxes being moved?

how many items were in each of them?

what sort of network connection did you have between the old server, the machine running mailmig or exmerge, and the new server?

were any disck-intensive operations being run at the same time?

one of the things that directly impacts the speed at which a mailbox will migrate is the size of the mailbox * the number of items in it - I have seen large mailboxes with a low number of large items migrate a lot faster than a small mailbox with a large number of small items in it - it's all in how the migration tools work
Avatar of swatgear
swatgear

ASKER

Hi abhaigh,

1. Exchange's built-in tool.

2. From less than one megabyte to almost 3.5GB.

3. From 100 to ~74,000.

4. Both machines are plugged into a switch.

5. Possibly. The 2000 machine was doing some swapping in the beginning because it only has 2GB of RAM but after turning off some services the throughput stayed the same even though the swap usage went way down.


Does the speed I was getting not seem low? Since I didn't actual record the throughput of my original mailbox move I have to rely on my memory (ha!) but I'm almost certain that my move did not go nearly as slow as all the others.

If I did more than one box at a time I got a better speed (the 1 to 2MB). If I did only a single box I would get anywhere from 250KB/sec to 1MB/sec. So it seems the higher speed was just a matter of doing several boxes at the same time.
If both devices were connected to the same switch, it makes me wonder what broadcast traffice wis impacting the transfer.

If connected to two different switches, what network traffic was happening between the two devices.

There are a number of things that could impact on data rate, mailbox properties are only one of them - in fact - how far away topologically, are your dc/gc's? That would impact on the transfer rate as well as mailmig needs to talk to the dc/gc in order to function correctly
Everything is plugged into the same two switches (that are teamed together) but I don't know which server is plugged into which switch. All the servers are about as close as they can be to each other.
Actually...  on what machine were you running the mailmig tool?

you say over the weekend, you didn't vpn into the system and run it from a machine on the end of the vpn did you?
I was on the network, at my desk, using remote software and executing the migration from the Ex 2003 machine. No VPN involved.
then it's hard t odetermine exactly what caused the slowdown - the data transfer should've been much faster than that, but without knowing what was going on on both machines and the gear between them, we can do nothing but speculate as to why your dtr was so low
I move mailboxes all the time. A 2gb mailbox can take two hours or more, particularly if the server is poorly specified.

Start perfmon and then change it to the report view and watch the counter with the disk queue in the middle. If it goes above 1 then the server is being thrashed.

I have attached a screenshot from a server I worked on at the weekend.
The top server is the new server, the bottom one is the old one. It took eight hours to move 40gb of data off that server.

Simon.
Disk-Bottleneck-Screenshot.jpg
@Sembee

Thanks for the info! But I'm afraid it's left me a bit confused.

I was unable to take a screenshot but it seems that my new server is thrashing during the transfer. I moved a smaller mailbox (of an employee that is no longer here) back to the old server. It has 7,791 messages and is 242MB big. The average disk queue length was about 1.4 on the new server (sometimes below 1, but usually closer to 2) whereas the old server rarely went above 1.

But then I transfered a 250MB file from my own workstation to the new server (this was initiated from the new server's desktop) and the new server had disk queue lengths of 50-60! But I also had throughput of about 10MB/sec so it would seem that the disk queue length is not the problem.

I also monitored the the % Processor Time on both computers during the mail transfer and while the new server didn't seem to be working hard at all, the old one was at about 50% usage almost the whole time.


Does any of this shed any light?
How is your storage configured? DO you have the databases and transaction logs on the same database or something like that?

Going above 1 is not that bad, I will often see it go up to two or three when the migration is under way, but normal use it should be less than 1 most of the time.

Simon.
the partition you have the new info store on is your bottleneck

what is the configuration of that partition? single drive, raid array?

are the transaction logs on the same partition? I'm betting they are, and if I'm right - move them to another partition and your dtr rate wil improve

Yeah you're right. Since this is only a temporary place for my mailboxes (I'm doing a swing upgrade) I put all the logs and stores on the same partition which happens to be a RAID5 array. I'll move the logs to another partition (which is on a separate disk) and test this out again.

I'll report back after I make the adjustment and move the mailbox box to the new server.
Well, I just moved the logs and moved the mailbox back to the new server and I didn't see any improvement.

I watched the disk queue length on the two partitions separately. The partition which the logs (RAID1 array) had very low (not above 1) disk queue length. The RAID5 array however had anywhere from 0.1 to 40. It often jumped back and forth between 2 and some number over 10.

Data transfer rate is still about 900kB to 1.5MB.


????
What is running on the servers? I have seen AV software bring the transfer rate down as well.

Simon.
Both servers have Symantec Corporate Edition AV on them but I don't think that's interfering because the utilization for those programs stays the same before, during, and after the transfer.

Otherwise they don't do much else.
I operate a simple rule when it comes to Symantec's products.
If those are installed they are the problem. Until they are removed (not disabled - removed) I will continue to blame those.
I only do one thing with Symantec's products and that is remove them. I find that the server is a lot more responsive without them and I can usually save the client some money by writing off whatever is left on them and buying something else.

In my experience with Symantec AV you wouldn't see much more use of the resources because it will be using enough of it as it is.

Simon.
try disabling your AV, or at least excluding those directories from it, and try again then
I'm already excluding the appropriate directories. I even tried disabling AV during my original move (this past weekend) because it was using so much memory but even after it was shutdown the transfer speed stayed the same (even though the memory usage dropped a lot).

Also I made a mistake in saying that AV was on both servers. The new one didn't have any AV installed during the time of the original move. I thought I'd installed it already but I hadn't.

@sembee

I know, I've heard several anecdotal stories about the poor performance and reliability of Symantec's products but I've never had a problem with them in any way in the last 6 years of being a customer.

I have no idea what is causing the slow transfer speed but I'm not convinced that it's AV, simply because you don't like Symantec.



Are there any utilities that can tell me how much disk usage each process is generating?

Are there perhaps any connectors that need to be created (that might not be) that is preventing the system from moving quickly?


I know we've gone over many different possibilities and it's very difficult to diagnose something through a forum but does anyone have any other ideas?
My dislike of Symantec products I don't hide, but it isn't based on no evidence. Time and time again Symantec products are the cause of problems with Exchange servers. I work with many many Exchange servers and a common cause of issues is Symantec. I remove Symantec and the server is much better.

I also have a favourite saying with the IT industry.
"Prior behaviour is not an indication that the behaviour was correct or that the item will continue to work correctly". That is particularly the case with AV software which by its nature changes every few days at most.

There is nothing that you have to do to the server to move the mailboxes, it should go across without any problems.

You can get issues with the network card and/or switch with negotiation, particularly when you are shifting large amounts of data which can put strain on the server. If the network utilisation less than 5% then you may have a problem there.

Simon.
Yeah, I understand. I just didn't want the conversation to stop at the mention of Symantec's products.

When I upgrade the old server to 2003 and move all the boxes back, I'll uninstall all of Symantec's products and see how it goes.
ASKER CERTIFIED SOLUTION
Avatar of swatgear
swatgear

Link to home
membership
Create an account to see this answer
Signing up is free. No credit card required.
Create Account