Import PST to O365 without duplicates

in doing a on premise exchange to Office 365 hosted exchange I setup the Office 365 environment and then forwarded the on premise inbound email to both on premise and O365 boxes until time of conversion, about 5 days.  Then I proceeded to go ahead and export the on premise email.  Then when it was time for conversion, I did another export with a date range getting the week that I was forwarding.  The reason I did this was to get the email that was replied to, read, or otherwise dealt with during the 5 days of forwarding.

So now that the conversion is done, I connect to the O365 mailbox and then import that 5 days of email, choosing to replace duplicate items.  My assumption was that if an email had been read and replied to, the import would over write the unread version that had been forwarded and import the sent one in sent items.
It did import the emails but it duplicated them - I had a ton of emails that were read and then unread - exact emails.

is there a way to import these and over write the unread one in the inbox?

Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Jackie Man IT ManagerCommented:
You need to set a cut off date for migration to be a day before you stop using the on-premise exchange.

Although it could be done manually, but it will be a tedious job for you and your IT.

So, it would be better to do it using third party service like the one below.
Jeffrey Kane - TechSoEasyPrincipal ConsultantCommented:
It's really too bad that you didn't read the full documentation prior to setting up your migration because none of what you did was necessary.  The native Office 365 migration tool will automatically resync a migrated mailbox on a continual basis until the migration batch has been stopped.  

Because of this there is no need to forward messages -- as you obviously know now.

Depending on how many mailboxes we are talking about, if not an outrageous amount, my suggestion is to use a script to remove the duplicates.   One that has been known to work is here:


Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
RFloyd30Author Commented:
Nice, thank you...  I will look into the migration utility.  Some of these did not work in the past so I have not revisited them.  For example the PST upload tool.  I started it yesterday and did a small test first and found that the import worked.  But it put the entire imported mailbox under the Inbox so no I have to move all the mail to the appropriate folder(s) plus I don't think it did anything with calendar and contacts.  I believe in the CSV you can use a simple / for location and it does it from the "top" level.  But testing will have to be done.

I will try the script for sure and read up on the migration tool.

Thank you both for suggestions!

Take care,
Starting with Angular 5

Learn the essential features and functions of the popular JavaScript framework for building mobile, desktop and web applications.

Jeffrey Kane - TechSoEasyPrincipal ConsultantCommented:
Let's be clear here -- you don't want to create user mailboxes from .pst files when you have a functioning Exchange Server.  

You should just be doing a direct migration.

If users have .pst files that you also want to merge into their Office 365 mailboxes that is a separate issue and should be done AFTER you have migrated all the Exchange mailboxes.
RFloyd30Author Commented:
a little off subject of the post, but thanks anyway.  The migration is good to go.  I do tons of these.  I really wanted to know why the Outlook import, when choosing "do not create duplicates," does so.  I thought it might be an attribute such as "read" or "unread" but today I find that it does not matter, the import duplicated them no matter what state they were in.  Odd.
Edwin HofferTechnical ExpertCommented:
Hello RFloyd30,

There is a best way to Import PST to O365. It's a trick I want to share so whenever a user want to Import PST files in O365. He never have to face the problems of file Duplication. You can also share this to others to solve and get through it.
1.Open Control panel
2.Go to Mail(Microsoft Outlook 2016) (32-bit).
3.A Dialog box will appear showing Mail Setup-office 365 Email.
4.In Profile Tab Click on show Profiles.
5.A New Dialog box will appear Click on Always use this profile radio box.
6.It will show a drop-down Menu having 2 options.
Office 365 Email and Outlook. Select Outlook and Click on OK.
8.Open Outlook and go to File then Open & Export.
9.Now click on Import and Export tab it would result in showing a new dialog box of Import and Export Wizard.
10.Choose Export to a file and Click Next.
11.Choose your Account "".
12.Click on Next a new dialog box of Export Outlook Data File would appear.
13.Click on Browse and save a path location for for the file to save and Choosing on Replace duplicates with items exported Radio box and click on Finish.
14.A new Dialog box will appear Request for setting up a password. Try to ignore it by clicking on OK.
15.Now mail Inbox will appear.Exit the Outlook.and again go to Control panel and follow Steps 1 to 6.
16.This time select Office 365 Email and Click on OK.
17.Now Follow same step as in 7 to 9.
18.Now select Import from another program or file and click Next.
19.A Dialog box of Import a file will appear choose on Outlook data file (.pst).
20.Click on Next.
21.Go to Browse and select the file from location which was saved earlier.
22.Choose by clicking on Radio button of Do not Import Duplicate.
23.Click on Next.Then Finish.

So this is the Permanent solution to get this problem rectified.

And for quick and easy solution we can also use 3rd party software. The one I would prefer to use is SysTools Outlook Duplicates Remover. Which can efficiently used in Deduplication of emails, contacts, calendars, journals, tasks, notes. After that you can import pst to office 365 using Office 365 Restore Tool to restore data from Outlook to Office 365 which provide selective data restoration instead of importing complete data of PST to the Office 365 user account, it also provides option of date-wise filters & perform selective import of data items from PST file to Office 365.
Jeffrey Kane - TechSoEasyPrincipal ConsultantCommented:
RFloyd30 --  to answer your question: the reason that it creates duplicates is that with the Office 365 .pst import tool the process actually recreates the messages rather than copying them as would be done if you used Outlook's import feature.  

The Office 365 .pst import tool is not designed to be used for live mailbox migration -- it was created to help administrators upload ARCHIVE mailboxes more easily.

As I originally stated your methodology isn't the correct way to migrate mailboxes from Exchange into Office 365.  So I don't think my comments are off topic at all.
E ATech LeadCommented:
RFloyd30Author Commented:
I appreciate all the input.  The problem is simple but still no explanation.  Most wanted to tell me how to import the mail from on premise exchange to O365.  That was not the question.

The question is why when importing a PST and you tell it to "REPLACE DUPLICATES WITH Import" it DOES NOT.
For example, I import an archive to O365.  Done on that and it works.  But the cutover is a few days later and in my case I've forwarded the on premise email to the O365 email box.  So for those few days, the same email in the on premise mailbox is in the O365 mailbox.  Since the user was using the on premise email, they have opened, replied and done whatever to the emails.  But in the O365 mailbox, they are sitting there unread.  So if I export those few days to a PST, I want to import those to the O365 mailbox (after the cutover) and OVERWRITE.  So if john doe sent an email to this user and it was replied to, the email is read and a message is in the sent folder.  I wanted to import to replace the inbox email with it being read and import the reply in the sent folder.

Yes my bad for not using the O365 migration tool to keep the email boxes in sync.  But the question is why the Outlook Import does not replace?  Since this is a small exported PST I am simply doing the import from Outlook client itself.

Thanks again for the time and replies!!!
Jeffrey Kane - TechSoEasyPrincipal ConsultantCommented:
So this it the first time you have said you are using Outlook for the .pst import.

That being said, it should update the read/unread flag when you import and replace duplicates.  As long as you are logged in with the mailbox owner's username and password and aren't doing it as a delegate of the owner.

If you were logged in as the mailbox owner then the only workaround I can think of at this time would be to delete the messages from the O365 mailbox and then import again.
Jackie Man IT ManagerCommented:
The problem is " the cutover is a few days later"... and the migration tool is not that smart to compare all your emails on a one-to-one basis or it will take a very very long time to do the migration.

So, the principle is to take a definite cutover date so that only the emails on or before the cutover date will be migrated to o365.
RFloyd30Author Commented:
My fault for not explaining better.  We are past this migration now and all is good.  Not a high volume email firm compared to others so manually deleting the duplicates was fairly easy.
But yes, we used Outlook to import the small PST and did it logged in as the user who owned the mailbox.  
Going forward, I think I will use the migration tool early before the cut over weekend and allow the O365 tool to import the large PSTs into the mailboxes.  Then do a export via date range at time of cutover to get the email period without the forwarding of email.  This should solve the issue.
Also will look at the migration tool that will keep the mailboxes in sync.
Thanks to all for contribution to this.  No matter how long you do this job, there is always something to learn and I thank you for your time and input!
Jeffrey Kane - TechSoEasyPrincipal ConsultantCommented:
Again, you do not need to worry about that "export via date range at time of cut-over".

If you use the native migration tool you just switch the MX record and keep the migration batches running.  It will bring over anything that is missing and then a day or two later you can shut it off.

After switching the MX record you will need to create new Outlook Profiles for everyone -- when you do this, there may be a message or two that are missing right away, but those will sync within the next 24 -48 hours, so I always recommend doing this on a Friday to have the least amount of impact.  That way, come Monday morning user's mailboxes will be ready to go.

The native tools really do work -- so don't try to overthink this because you are just making more work for yourself.

Good luck!

It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Office 365

From novice to tech pro — start learning today.