Exchange 2010 to 2016 Migrations – Loadbalancer Fun

Over the past three months I have worked with multiple migrations to Office 365 and Exchange 2016. Each environment had its unique challenges and problems to overcome, but between myself and the IT team of the company migrating, the issues were all resolved. The commonality for all of these migrations was the load balancer. That does not mean that loadbalancers are necessarily an problem with Exchange. It just means that they are a layer of complexity that one needs to be aware of when deploying a highly available solution. Here is a list of the issues we had and worked around:

• Load balancer – incorrect encryption
• Load balancer – incorrect mechanism for load balancing
• Load balancer – URLs not configured correctly on Exchange 2010

The problem is that the list above is putting the cart before the horse. The solution before we review the symptoms of the problem. The similarities for all of the scenarios are Exchange 2016 behind load balancers, with the mailboxes for the users on Exchange 2010.
Scenario One
For this scenario, my client and I followed a normal process for a client access change from Exchange 2010 to Exchange 2016:

• Preconfiguring the rules on the new load balancers. These load balancers were configured using templates provided by the vendor for Exchange 2016.
• Changed URLS on Exchange 2010 as needed – OWA, OAB, ActiveSync, Webservices, Outlook Anywhere, and AutoDiscover (removed setting)
• Changed URLS on Exchange 2016 as needed – OWA, OAB, ActiveSync, Webservices, Outlook Anywhere, and AutoDiscover (carried this over from Exchange 2010)
• Changed DNS/firewall rules as needed

Once all of these changes were made, we testes these connections:

• Internal OWA – check that the Exchange 2016 servers answer the connection and that the old Exchange 2010 interface shows for the mailbox. Verify no double authentication.
• External OWA – check that the Exchange 2016 servers answer the connection and that the old Exchange 2010 interface shows for the mailbox. Verify no double authentication.
• Internal Outlook Anywhere – Verify that Outlook connects and can function normally. Verify that AutoDiscover works, OOF works, address book works and more.
• External Outlook Anywhere – Verify that Outlook connects and can function normally. Verify that AutoDiscover works, OOF works, address book works and more.
• ActiveSync – Test connection to mailbox with multiple devices, preferably at least Android and iPhone, if not other device types.

For this scenario, the breakdown was only external Outlook Anywhere. Continually prompted for credentials. First troubleshooting step is to simplify the connection path for Outlook to the mailbox. This means eliminating the loadbalancer from the path by redirecting traffic from the Firewall to one Exchange 2016 server. With this change, Outlook was able to connect without issue. Because the loadbalancer was configured with the vendor templates, we know that the issue is with the template. After some digging, it was apparent that the encryption being used was incorrect.
The vendor HLB was Kemp in this scenario. The original template configuration configured the encryption to look like this:

The solution was to change the encryption to the following:

Once the setting was changed, Outlook was able to connect from an external connection, through the load balancer.
Scenario Two
Scenario two is almost identical to Scenario One as the client was going from Exchange 2010 to 2016. The only difference was they were using Netscalers for their load balancing.
During testing of the new setup, external Outlook clients would get stuck in perpetual prompting for credentials. Reviewing logs from the Exchange servers, I noticed an error I had seen in previous migrations. Microsoft has an support article on this scenario:
Outlook Anywhere users prompted for credentials when they try to connect to Exchange Server 2013 or Exchange Server 2016
The key is to review the log files present in this folder: Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\RpcHttp. In the log files should something like “NegotiateSecurityContext failed with for host ‘mail.contoso.com’ with status ‘InvalidToken’ “. This is occurring because the Exchange 2010 servers are installed on Windows 2008 R2. Patch the servers in question, reboot and try your connection again. Which, unfortunately, did not resolve this issue.
Next step was to circumvent the loadbalancers. Which yielded the same results as Scenario One. The client began reviewing the template used from the vendor and found the settings to be correct per the template. He then reviewed the settings for his Exchange 2010 rules and noticed a difference in the way load balancing was configured.
With a bit of persistence, the client was able review the logs on the loadbalancers and found that Outlook anywhere connections were being split between the two Exchange 2016 servers. So the sessions for Outlook anywhere were not persistent or ‘sticky’. He modified the rules to add ‘SourceIP for persistence and Outlook Anywhere issues went away. They now have 1500+ mailboxes connecting to the loadbalancers without any issues or pop-ups for authentication.
Scenario Three
For this last scenario, we had introduced Exchange Server 2016 to provide coexistence for their migration to Office 365. The server would handle client access traffic like AutoDiscover, OWA, Outlook Anywhere, OAB, ActiveSync and more. We preconfigured the Exchange 2016 server
During testing, we experienced the same issue as in Scenario Two where the Invalid Token and Outlook prompts occurred. We also applied the same hotfixes and this did not resolve the issue. We did some digging and found that there were some misconfigured URLs on the Exchange 2010 servers that had the CAS Array name in them instead of the server or share name as we used on other URLs. The CAS array name was pointed, in DNS to the loadbalancer. The load balancer did not have a rule for this traffic and was thus dropping it.
In order to make this scenario work, you have two choices. One is to create a rule set for this namespace (casarray.domain.com) to point back to the Exchange 2010 servers, or the better choice, is to change all URLs on the Exchange 2010 servers so that they would not use the casarray name anymore. Once the change is made to handle this, Outlook was able to connect and not continually prompt for credentials.
Conclusion
Ironically as I was writing this article, I was assisting another customer with their cutover to Exchange 2016 and we experience another misconfiguration. In their case, connection persistence was not enabled on their F5 (yet another load balancing vendor). Once persistence was in place, connections worked as expected from Exchange 2016 to 2010.
These issues just highlight the importance of a correct Loadbalancer configuration. It can impact users and it can be a roadblock until the issue gets resolved.
F5 Persistence Information – Manual Chapter: Enabling Session Persistence
My final word or advice would be, if you can test it test it. Do this a week before a cutover. Make sure you work out the bugs in your configuration and especially make sure ALL workloads work through the load balancers. This includes:

• Outlook Anywhere
• OWa
• OAB
• ActiveSync

The top one is usually the one that may require tweaking.