Introduction
During the course of a migration for Mailboxes and Public Folders to Exchange Online from Exchange 2010, we experience numerous issues with the Public Folder migration part of the project. What started as an authentication change from Microsoft (Basic Auth disablement) and TLs changes (TLS 1.0/1.1 blocked) ended up sending us into a bit of a rabbit hole. Instead of directly migrating to Exchange Online, we would move to Exchange 2016 first and then to Exchange Online. In this two-part series we will cover a series of unfortunate events as well as they solutions and lessons learned along the way. The hope is that those stuck in this scenario, or a similar scenario will see the light and end up migrating successfully to their target environment.
Note: Attempts to move to Exchange Online were initially working and only failed after TLS/Basic Auth changes in Exchange Online.
Scenario Environment
In our scenario environment, we have three Exchange 2010 servers and one Exchange 2016 server in two different AD sites. Initials syncs from the two servers in the same site as the Exchange 2016 server fail. However, we are able to replicate from the server in the other AD site, which coincidentally or not is in a child domain.
Server names, left to right, are Ex10Srv-01, Ex10Srv-02 and Ex10Srv-03. Total Public Folder data is over 1.5TB and consists of 20k folders.
Sample (Failure 1)
New-MigrationBatch -Name PFMigration -SourcePublicFolderDatabase (Get-PublicFolderDatabase -Server Ex10Srv-01) -CSVData ([System.IO.File]::ReadAllBytes('FolderToMailboxMapPath.csv')) -NotificationEmails damian@PowerShellGeek.com
Sample (Failure 2)
New-MigrationBatch -Name PFMigration -SourcePublicFolderDatabase (Get-PublicFolderDatabase -Server Ex10Srv-02) -CSVData ([System.IO.File]::ReadAllBytes('FolderToMailboxMapPath.csv')) -NotificationEmails damian@PowerShellGeek.com
Sample (Success!)
New-MigrationBatch -Name PFMigration -SourcePublicFolderDatabase (Get-PublicFolderDatabase -Server Ex10Srv-03) -CSVData ([System.IO.File]::ReadAllBytes('FolderToMailboxMapPath.csv')) -NotificationEmails damian@PowerShellGeek.com
Failure Messages: Migrations Batches 1 and 2: (MAPIExceptionNetworkError)
When the third source was chosen, the job began migrating data properly and no more MAPI errors, we think we are in a good place. Little did we know that there were trying times ahead.
Issues Experienced
First Issue – Incorrectly Created Mailboxes
The first issue we experience cause strange issues during the first migration attempt. So much so I wrote a blog post to cover it, which you can read about here:
Public folders ‘Error Unexpected Mailbox State’
I won’t rehash any other details as that article is complete.
Issue with Date Stamps on the Public Folders
On the next attempt to get a clean migration migration from 2010 to 2016 , the batch was not going to complete in time so we stopped the migration batch:
Stop-MigrationBatch PFMigration
This step apparently caused an unknown, to us at the time, issue that would need to be resolved.
We did leave the original mailboxes behind in hopes of reusing them for the next migration. However, this seem to fail and we were left wondering why. What we noticed was a disconnect where some folders would get to a synced state and others would not. In the Exchange 2016 console all Public Folders were shown as ‘synced’. Once in this state, we decided to perform a complete migration, but it those mailbox migrations never completed. With some digging we found that this ‘LastSuccessfulSyncTime’ property was disjointed between all Public Folder mailboxes. Some were current (within a day of the restart of the migration), while others were stamped from the last migration attempt, as shown below:
Get-MigrationUSer | Where Identity -like ‘PFMailbox*’ | ft Identity,LastSuccess*
In order to make this work, we had to force a sync for each individual mailbox and not the batch itself. Forcing the batch to sync did NOT resolve the issue. How can we resume these? There is no ‘Resume-MigrationUser’ cmdlet. However, there is a Resume-PublicFolderMailboxMigrationRequest cmdlet. If we run the Get version of that cmdlet to see what requests are present and, in this case, we have one per Public Folder Mailbox destination:
When combined the Get and Resume cmdlets together, they force each user (Public Folder Mailbox) to do a manual incremental sync.
Get-PublicFolderMailboxMigrationRequest | Resume-PublicFolderMailboxMigrationRequest
Now, this takes time. Public Folder migration requests seem to get queued and are not the highest priority for Exchange to process. Be patient and wait. After kicking off these migrations and waiting patiently, results started to come in (see improved dates):
Eventually the property in question, LastSuccessfulSyncTime, corrects itself and all of individual Public Folder mailboxes are now syncing correctly. However, we are still not out of the woods.
Mail Enabled Public Folder Issue (Near Transient AD Error)
Next up we have an issue where we were able to sync 26 of 27 Public Folder mailboxes, however, the main or hierarchy mailbox, would not sync and noticed that the Incremental syncs were always failing with the same error. It appears that there was a disjointed mail enabled Public Folder that was cause an error and an Active Directory transient error right after. The assumption at the time was that they were both connected and that this folder was causing issues.
We had one of the client’s engineers with AD permissions remove this object, which is located in Active Directory under Microsoft Exchange System Objects [MESO] – do not forget to check the ‘Advanced Features’ box in ADUC.
Removing the object still did not help as the migration batch would not complete. Since it would not complete, we removed the migration batch (again), Public Folder Mailboxes (again) and databases (again). Then we started all over with a new migration batch, only to have another issue. However, with the next batch we did not see this ‘mail enabled’ Public Folder issue again.
Note: On this issue, the errors ONLY occurred AFTER a full synchronization. When an Incremental sync completed, the error was then thrown.
SACL – Transient Active Directory Issue
Now this one is a bit more common and many Exchange administrators may have already run into this on their own, but we did not realize this was already an issue in our migration. With all the other issues we had experienced so far, this one had slipped through the cracks because it was adjacent to the previous mail enabled folder issue and simply combined those two as one issue in our minds.
The issue also stopped our migrations from completing properly. With some review of Event Logs we found these issues:
Inside of the Application Event Log, there is also an event (2112) that shows what DCs are considered reliable and that Exchange will use for it’s operating procedures. Inside of this event are numerous fields that we can interpret, but SACL is the one we need. Active Director servers need to have this assigned otherwise Exchange will not use that server. In our case, there were three separate Domain Controllers in the same site as the source Public Folder server and none had the correct value for SACL set.
In the below screenshot, the servers listed all have a SACL of ‘1’, which is the value we desire. However, none of these servers were in the same site as the Public Folder server:
Transient Error – No PF Mail Enabled Errors
Resolution : The fix was to resolve any Group Policy issues which would then force the SACL attribute of ‘1’ on those Domain Controllers. A good write-up on this was written up by fellow Microsoft MVP Gareth Gudger:
https://supertekboy.com/2018/01/06/msexchange-adaccess-event-id-2112
Public Folder Migration Validation Scripts
Because of all of the issues we had experience, we decided to see what Microsoft tools were available to validating our setup and Public Folder data. We were able to find the below are the results of this script [ https://learn.microsoft.com/en-us/exchange/troubleshoot/public-folders/public-folder-migration-fails] and run them against our Public Folders.
The script found some orphan folders, which we cleaned out of Active Directory, but there were no other obvious errors that we could detect at this point in time.
Ghost Mail Enabled Public Folders
When this last, and it was the last, Migration Batch was nearing complete, we happened to see this interesting message:
Looks good, the migration should complete successful now. However, we found that there were additional error messages.
12/14/2022 1:17:00 AM [EXCH01] Public folder "/ PublicFolder1" could not be mail-enabled. The error is as follows: "No mail public folder was found in Active Directory with object ID '6c41c2ea-85fb-4071-97ab-3cb9e13a5efc'". 12/14/2022 1:17:22 AM [EXCH01] Public folder "/PublicFolder2" could not be mail-enabled. The error is as follows: "No mail public folder was found in Active Directory with object ID '6e53bc3c-6992-41f7-a70f-9f13108d2eeb'". 12/14/2022 1:17:24 AM [EXCH01] Public folder "/ PublicFolder3" could not be mail-enabled. The error is as follows: "No mail public folder was found in Active Directory with object ID '56b76aa1-779d-4065-ad93-86f876577c38'".
Looks like a repeat of the previous issue, and we make sure to clean these folders up in Active Directory. After this we ran PowerShell to complete the batch:
Complete-MigrationBatch PFMigration
The batch did complete successfully and with no more issues.
Conclusion
Public Folders have existed for a long time in Exchange and while at one point in time they seemed like a more useful feature, they have also become a thorn in the side of some administrators and / or organizations. This means that at some point in time they will need to be migrated to either another version of Exchange or to Exchange Online. While these migrations have commonly gone well for most organizations, problems still do arise and need to be dealt with. This article has tried to expose these issues and how to accommodate for them. In the second part of this series we will cover some tips and tricks for making the migration run smoother.
———————————————————————————————————–
Comments? Questions?
Feel free to leave your Comments below! Learn to more efficiently utilize PowerShell to manage Exchange Server, Exchange Online, Microsoft Defender for Office or Microsoft Purview Compliance portals by picking up frequently updated eBooks: