Solved

Exchange PF/System folders replication issue

Posted on 2009-04-14
43
894 Views
Last Modified: 2012-06-22
Hi.

Last month I did an Exchange migration from an old exchange 2003 box to a new exchange 2003. The old is a DC and so I wanted to get exchnage off it. All went fine thanks to the experts on here as it is my first migration. However, at the final hurdle I have an issue.

At the start of the migration I replicated the public folders and required system folders, seemed to go fine so I pressed ahead with the rest of migration. Mailboxes moved, SMTP moved, RUS etc. I left it as is for a few weeks then to make sure everything was ok. Was away working on a client site last week and came in today ready to start decommissioning the old box.

However today I notice that replication of the OAB isnt working and hasnt since the beginning of the migration. On the new server the last update received date is a month ago. This is the same for the public folders though we dont use them here. So it appears replication stopped during the migration and I am suspecting maybe it has to do with SAV or SMSME.

As a consequence of this Outloook this morning gives 0x8004010F which references the Microsoft Exchange offline address book. The OABs on the new server in ESM - Server - First admin group - public folder store - public folder instances has an age limit of 30 days and so would appear it was removed over the weekend, hence the outlook sync error as the OAB (all versions) on the new server is now empty.

So, I need to know how to monitoring the replication and what exactly to look for to nail down why the replication is failing. Need to get a good copy of the oab replicated over before decommissioning. I looked in message tracking and see the traffic being sent from serverold-IS to servernew-IS however I need some guidance on what specifiically I need to look for, I assume under diagnostic logging?

Thanks
0
Comment
Question by:BGilhooley
  • 26
  • 16
43 Comments
 
LVL 65

Expert Comment

by:Mestha
Comment Utility
There is little that you can do.
The main thing that I will do is check message tracking on both server to ensure that the traffic is leaving the original server and is being delivered in to the store correctly on the second server. I have seen replication fail before, with NDRs generated, which of course you cannot see.

Simon.
0
 
LVL 1

Expert Comment

by:jnicpon
Comment Utility
A good first place to look is ion your new server's ESM to see what the default OAB is and how it is assigned. You should rehome it with the ESM gui to the new server and rebuild it. Verify that the OAB setting on each store, as well as in any policy objects that you've created.
0
 
LVL 1

Author Comment

by:BGilhooley
Comment Utility
Simon,

Little that I can do about the replication? How do I get over this so? Looks to me when I query the message tracking centre for IS traffic that its being properly sent/delivered.

Whats next to get a good copy of OAB on new server? Is it safe to do as suggested above? The default offline address list server is currently shown as the old exchange server (also a DC/GC).

Thanks
0
 
LVL 1

Author Comment

by:BGilhooley
Comment Utility
On the mesagae tracking, the last line of the message for the receiving new server is 'SMTP:message queued for local delivery'.
 Is this successful ? or should it also have 'message delivered locally to store' like when a user receives mail and its delivered to the mailbox store successfully..
0
 
LVL 65

Expert Comment

by:Mestha
Comment Utility
That is not a successful delivery.
You should have entries on the originating server which shows the message being delivered out, then on the receiving server the message being delivered to the store. If the last line is not delivered to the store then the replication traffic hasn't been delivered correctly. AV is the usual cause, I have seen that block the replication traffic in the past.

Simon.
0
 
LVL 1

Author Comment

by:BGilhooley
Comment Utility
Ok thanks Simon.

So just check on this again this morning. At 6AM replication traffic was sent. The last line on the old server is:
'message transfered to oldserver.domain.com through SMTP'

New server receives this but as mentioned above it only gets as far as 'message queued for local delivery'. I will disable AV and see how I go today..
0
 
LVL 65

Expert Comment

by:Mestha
Comment Utility
Something is blocking the traffic. It is being stopped or blocked at the queue level.

Simon.
0
 
LVL 1

Author Comment

by:BGilhooley
Comment Utility
Have had the file level AV disabled on this server all day and still have something blocking traffic at the same level. The only other thing is the SMSME which I cant really turn off for hours on end or else I leave server too open. Have just researched the diagnostic logging which can be done so I will try that tomorrow to see can I get a lower level clue to where the issue is...thanks for the help so far Simon, stayed tuned.
0
 
LVL 65

Expert Comment

by:Mestha
Comment Utility
Is that a Symantec product?
If so, then it is the cause of the problem. I work on a simple basis with Symantec's products - if they are installed then they are the cause of the problem, until shown otherwise by their complete removal. In most cases the problem does indeed go away when they are removed.

Simon.
0
 
LVL 1

Author Comment

by:BGilhooley
Comment Utility
Symantec yes Simon,

Yesterday I disabled the file level which is Symantec Endpoint.
For mail security Symantec Mail Security for MS Exchange is installed. Unfortunately for me these products are bought and paid for a while ago so I have to stick with what i got. No money in budget.

Something I am notcing from the diagnostic logging is no inbound replication messages showing on the new server. Outbound ones are. On the old server I see both inbound and outbound replication messages logged. Is this consistent with the theory that its likely a Symantec issue?

By the by i am just off a support call with Symantec and they say that they have no known cases of SMSME causing replication issues. Surely they would have something if it was an issue with their product?

Also, how best to try and force the replication when I want it to run? My understanding is that if the public folder store replication is set to 2 hour intervals and a subfolder is the set to 'always run' that that means it should replicate every 2 hours. However, i dont see consistent replication traffic at consistent intervals, just random. If I knew when it would take place I could at least disable the Mail security for that time period and see what happens. Is it just unpredictable and thats that?



0
 
LVL 65

Expert Comment

by:Mestha
Comment Utility
Rule number one with product support - they will NEVER admit to a problem with their product. Of course Symantec support are going to say they are not aware of problems, if they were to say otherwise that would cause them problems.
Rule number two is that they routinely lie, see rule number one.

Public Folder replication does its own thing. It will just replicate in effect when it feels like it. With Exchange 2003 you can force the content to go across using the Send Content now command, but that will send everything across.

While I appreciate that the Symantec product is paid for, while it is installed it will always be the primary place to look for any problems. In some cases you can remove it, reboot and then reinstall and find the problems will go away, however as I wrote above, I have a lot of experience with the Symantec products causing problems. Due to the way that AV integrates with Exchange, disabling it is not enough, it has to be removed.

Simon.
0
 
LVL 1

Author Comment

by:BGilhooley
Comment Utility
By the way I should mention this though I dont think it is related but Im open to correction. (I just am havin no luck with this Exchange stuff!).

In ESM/Administrative Groups/First Administrative group when I try to expand the public folders when connected to the new server I get the following error:

"Could not expand the root folder, ensure that your default virtual server is running and is configured correctly" it mentions IIS lockdown and URL scanning tools. Error ID is c103b405.

This had been a issue for me pre-migration on the old server however i resolved it by editing urlscan.ini. Had posted a question here actually:  http://www.experts-exchange.com/Software/Server_Software/Email_Servers/Exchange/Q_23953180.html.

i wasnt expecting the same issue to reappear on the new one but it has. So in ESM I cant connect to the new server to view public or system folders, only the original server.  I dont have urlscan or any IIS lockdown tools running on the new server so its not going to be the same fix as before unfortunately. Its really a seperate question but just addking it in case.

 
0
 
LVL 1

Author Comment

by:BGilhooley
Comment Utility
Do you think that removing the file level AV is more likely a fix than the mail security Simon? Having no AV for a while leaves me feeling less vunerable than no mail security.
0
 
LVL 1

Author Comment

by:BGilhooley
Comment Utility
Have uninstalled the Endpoint...waiting game.....
0
 
LVL 65

Expert Comment

by:Mestha
Comment Utility
Is there anything else installed on the server? URLSCAN can be installed with other things, but on a native Exchange server it shouldn't be the cause of the problem.

Simon.
0
 
LVL 1

Author Comment

by:BGilhooley
Comment Utility
No nothing else apart from exchange tools, best practice and troubleshooter. I purposely kept the server free of other things to avoid conflicts/issues but not working out like that. No sign of URLSCAN installed anywhere. No incoming replication traffic showing for problem server since 4:30 this afternoon
0
 
LVL 1

Author Comment

by:BGilhooley
Comment Utility

Well removing the Symantec Endpoint made no difference. I see a few replication messages from last night in the message tracking centre but same issue, not being delivered to store. I suppose my only option now for tonight is to remove the mail security and see what happens, will just need to disconnect the WAN port so I dont get hit with spam/mail viruses.
0
 
LVL 1

Author Comment

by:BGilhooley
Comment Utility
Something interesting also is if I add an item to public folder (on the new problem server) from an Outlook client it does replicate to the public folder on the old server.
0
 
LVL 65

Expert Comment

by:Mestha
Comment Utility
That would tally up, because the replication traffic is not getting out of the old server. If you are receiving external email correctly then I would expect to see replication traffic come through as well.

Simon.
0
 
LVL 1

Author Comment

by:BGilhooley
Comment Utility
I think it is getting out of the old server Simon, its when it hits the new server that the traffic isnt delivered to the public store, it is just queued for local delivery. I have just disabled the port forward and on the WAN port and now have both the AV and mail security out of the picture on the new server so will wait and see what replicates tonight. Original server seems to attempt replication every night anyway...
0
 
LVL 1

Author Comment

by:BGilhooley
Comment Utility

No dice last night with the mail security taken out of picture. This mornng trawling google I came across this: http://forums.msexchange.org/m_1800475205/mpage_1/key_/tm.htm#1800506154

Seems to be a very similar case to mine Simon which you were also helping out on and he claims the resolution was removing the integrated windows authentication setting from the SMTP virtual server was the answer, could this possibly be the solution?
0
IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

 
LVL 65

Expert Comment

by:Mestha
Comment Utility
Exchange to Exchange email uses integrated authentication, so I would be very surprised if removing integrated would fix the problem. It isn't a setting I have ever had to change, other than turning it back on again!

If the content is leaving the old server and is then sitting on the new server, you should be able to see the content in the queues.

Simon.
0
 
LVL 1

Author Comment

by:BGilhooley
Comment Utility
Figured as much, that would make sense, maybe he ticked them on is what he meant. I dont have anything sitting in the local delivery queue on the new server (or the old). Have attached a screenshot just to confirm of the message tracking I see of traffic from oldserver-IS to newserver-IS.

Anything else this could possibly be? Permissions or something?
IS-traffic.JPG
0
 
LVL 65

Expert Comment

by:Mestha
Comment Utility
Has the hierarchy gone across? If you connect to that new server specifically, do you see the list of folders?

Simon.
0
 
LVL 1

Author Comment

by:BGilhooley
Comment Utility

Simon,

No the hierarchy doesnt go from old to new server. I have tried many times the send hierarchy option but no success since the beginning of migration.


Also, in the application log I notice now the following (new server):
--------------------------------------------------------------------
Event Type:      Error
Event Source:      MSExchangeIS Public Store
Event Category:      Replication Errors
Event ID:      3093
Date:            4/20/2009
Time:            4:55:14 PM
User:            N/A
Computer:      FSMAILSVR
Description:
Error -2147221233 reading property 0x674b0014 on object type tbtMsgFolder from database "First Storage Group\Public Folder Store (FSMAILSVR)".
---------------------------------------------------------------

Event Type:      Error
Event Source:      MSExchangeIS Public Store
Event Category:      Replication Errors
Event ID:      3093
Date:            4/20/2009
Time:            6:08:42 PM
User:            N/A
Computer:      GGMAILSR
Description:
Error -2147221233 reading property 0x67480014 on object type tbtOwningFolders from database "First Storage Group\Public Folder Store (GGMAILSR)".
------------------------------------------------------

The same have been logged on the old server also and from what i read it most likely is nothing, one of those this behaviour can generally be ignored messages but in this instance is it a clue to anything?





0
 
LVL 65

Expert Comment

by:Mestha
Comment Utility
If the hierarchy hasn't gone across then Exchange doesn't know where to put the data. It basically goes in to this black hole.

I can find nothing about about those errors that would be of any consequence. Logs of interest would be on the new server.
If you force the hierarchy to go across, do you see it in Message Tracking? From memory the subject is different.

The fault may well be with the new server.

Simon.
0
 
LVL 1

Author Comment

by:BGilhooley
Comment Utility
Sorry for delay was out to a client site most of last 2 days.

If I do the send hierarcy I just get the same thing, message queued for local delivery on the new server. The hierarchy did go across initially at the start of migration but not for weeks has it worked.

Replication traffic flows freely in and out of old server but only out of new server. Doesnt deliver on new.
0
 
LVL 65

Expert Comment

by:Mestha
Comment Utility
I have to presume then that the new server has a fault and I would be looking to move all the data back and remove it gracefully.

Simon.
0
 
LVL 1

Author Comment

by:BGilhooley
Comment Utility

Is there a way to create a new OAB on the new server some other way? All the public folder content I need went across at the start, it is just the OAB I have the issue with.
0
 
LVL 1

Author Comment

by:BGilhooley
Comment Utility
Simon,

Some extra info which may be of note. In ESM - First Admin Group - Servers - NEWSERVER - Public folder store I get the following message:

The object is no longer availabe. ID:80040e19.

Hmmm. I assume also this is linked to why i cant view the public/system folders on the new server under First admin group - servers - folders.

Could this potential permissions or whatever it is problem also cause the replication to fail? i have no issue accessing these folders on the old server and I think I recall changing something related to permissions in IIS when i said configuring the webmail on the new server (I know I should have that documented somewhre but I cant find it. Is it documented anywhere the default permissions the default website should have for exchange?)

Thanks again

0
 
LVL 65

Expert Comment

by:Mestha
Comment Utility
You should be able to see public folders for all Exchange 2003 servers through ESM on either server.

Public Folder access for ESM does go through IIS, and if the permissions are wrong or there is a require SSL setting set on one of the virtual directories it can cause a problem. If you have played around with the authentication settings then reset the virtual directories: http://support.microsoft.com/default.aspx?kbid=883380

As for recreating the OAB, that isn't very easy, because it is coming from the other server. You can recreate the system folders, but I think you would need to do that on both servers so that they are in sync.

Simon.
0
 
LVL 1

Author Comment

by:BGilhooley
Comment Utility

Ok, will try the reset this evening when users gone. This morning I also just installed exchange on a free PC and added it as a replication partner just to see does it have any issue replicating, theres nothing on it at all except exchange so unless theres an issue with the origional server it should work fine.
0
 
LVL 1

Author Comment

by:BGilhooley
Comment Utility

Well the replication to my new test server worked fine, both hierarchy and content. Going to strip my problem server down to the bare bones tonight (uninstall everything not exchange critical) and see can I get the darn thing to replicate. that and resetting the IIS
0
 
LVL 65

Expert Comment

by:Mestha
Comment Utility
That would tend to point to my thought above - that the server you were trying to replicate to is at fault.

Simon.
0
 
LVL 1

Author Comment

by:BGilhooley
Comment Utility
One thing I forgot to mention that was installed also is Exclaimer mail utilities (not the Av or Spam componenents just the disclaimer). I am uninstalling that now..
0
 
LVL 1

Author Comment

by:BGilhooley
Comment Utility

Right, I removed exclaimer, found some registry entries for Symantec mail security, deleted those. Have mailed Symantec support for an ftp link to their utility to COMPLETELY remove Symantec Endpoint, from my reading tonight seemingly add/remove programs doesnt remove it completely.

The only progress I have after all this is that I can now see public folders for the problem Exchange server through ESM on either server. No more errors when looking at those in ESM.

I am still getting these erros logged regularly which must be a clue to something:

--------------------------------------------------------------------
Event Type:      Error
Event Source:      MSExchangeIS Public Store
Event Category:      Replication Errors
Event ID:      3093
Date:            4/24/2009
Time:            7:55:14 PM
User:            N/A
Computer:      FSMAILSVR
Description:
Error -2147221233 reading property 0x674b0014 on object type tbtMsgFolder from database "First Storage Group\Public Folder Store (FSMAILSVR)".
---------------------------------------------------------------

In looking for info on this I came across this:

http://www.experts-exchange.com/Software/Server_Software/Email_Servers/Exchange/Q_21997518.html?sfQueryTermInfo=1+10+object+tbtowningfold

A very similar situation to mine in which case the solution was AV removal but unfortunately its not been so straight forward for me, however it has led me back to trying to lget the removal tool from Symantec,

I am desperately clutching at straws here also but having the FQDN for the Default SMTP Virtual Server the same for both the old and the new server wouldnt affect replication in any way would it? i changed the new one to match what the old one announces as,   i.e   server.ourdomain.com.

Thanks

0
 
LVL 1

Author Comment

by:BGilhooley
Comment Utility

By the way I didnt do the IIS reset seeing as I had back access again to the PF's on the problem server, should I do it anyway, could it in anyway possible resolve the replication issue?
0
 
LVL 65

Accepted Solution

by:
Mestha earned 500 total points
Comment Utility
Same FQDN on thee SMTP VS can cause problems going across routing groups, but shouldn't cause a problem with a single routing group. However I would change it just in case. Run IISRESET from a command prompt to ensure the change is written to the IIS metabase.

Running the IIS folder reset procedure will not do any harm. It doesn't take long.

Simon.
0
 
LVL 1

Author Comment

by:BGilhooley
Comment Utility

That was it Simon, after all that, having the same FQDN. Changed it, sent the hierarchy and the magic happened. Suppose I should have tried that earlier, occured to me a few days ago but then in a previous post I had read that it wouldnt make a difference to anything that I sort of put it to the back of my thinking. Only question now is will have a different FQDN on that server affect our outgoing mail for a day or two while I let the replication settle down? Is there a good chance it will be rejected by some mail servers of just a slight one?

Thanks so much for your help on this, I've learned more than I thought I'd ever need about Exchange replication! I will be accepting the above answer.
0
 
LVL 65

Expert Comment

by:Mestha
Comment Utility
As long as the FQDN resolves on the internet then it wouldn't cause a problem. If your intention is to remove the other server then one option as a temporary measure would be to route outbound email out through your ISPs SMTP Server using a smart host.

Simon.
0
 
LVL 1

Author Comment

by:BGilhooley
Comment Utility
Thats working fine through the connector yes.

Now that my OAB is successfully replicated is it just straight forward to just change the default offline address list to the new server, anything to watch for?
0
 
LVL 65

Expert Comment

by:Mestha
Comment Utility
Once the content has been replicated across, simply switch the OAB generation server and then wait for at least one successful OAB generation cycle to complete.

Simon.
0
 
LVL 1

Author Closing Comment

by:BGilhooley
Comment Utility
Thanks Simon for the guidance.
0

Featured Post

Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

Join & Write a Comment

Easy CSR creation in Exchange 2007,2010 and 2013
Following basic email etiquette rules will help you write a professional email and achieve a good, lasting impression with your contacts.
In this video we show how to create an Address List in Exchange 2013. We show this process by using the Exchange Admin Center. Log into Exchange Admin Center.: First we need to log into the Exchange Admin Center. Navigate to the Organization >> Ad…
how to add IIS SMTP to handle application/Scanner relays into office 365.

728 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now