Real Life Exchange 2010 Site Switchover - Lesson Learned

by Henrik Walther [Published on 9 Oct. 2011 / Last Updated on 9 Oct. 2011]

Had to perform an Exchange 2010 site switchover for a customer that had lost all servers in the primary datacenter this weekend (not only Exchange but everything!). 5 months ago their messaging solution was based on a non-redundant Exchange 2003 configuration (talk about good timing). So the the customer uses an Exchange 2010 active/passive user distribution datacenter model with a shared namespace similar to scenario 1, which I describe in part 1 of my “Designing a Site Resilient Exchange 2010 Solution” articles series. Because the customer have LAN quality communication between the datacenters, they use a stretched Active Directory site/subnet. In addition, the DAG runs in DAC mode. To get Exchange 2010 up and running in the failover datacenter, I basically went through the steps described in part 12 of my “Planning, Deploying, and Testing an Exchange 2010 Site-Resilient Solution sized for a Medium Organization” articles series. First, I used “Stop-DatabaseAvailabilityGroup DAG01 –MailboxServer <DAG member server> –ConfigurationOnly” to stop the respective Mailbox servers located in the primary datacenter that participated in the DAG. I used the “ConfigurationOnly” parameter since all Domain Controllers in the primary datacenter was unavailable. After having stopped the Mailbox servers in the primary datacenter, I used “Get-DatabaseAvailabilityGroup DAG01 | FL Name,StoppedMailboxServers,StartedMailboxServers” to verify the servers had been stopped and that the Mailbox servers in the failover datacenter was started. Then I stopped the cluster service on the Mailbox s

Had to perform an Exchange 2010 site switchover for a customer that had lost all servers in the primary datacenter this weekend (not only Exchange but everything!). 5 months ago their messaging solution was based on a non-redundant Exchange 2003 configuration (talk about good timing).

So the the customer uses an Exchange 2010 active/passive user distribution datacenter model with a shared namespace similar to scenario 1, which I describe in part 1 of my “Designing a Site Resilient Exchange 2010 Solution” articles series. Because the customer have LAN quality communication between the datacenters, they use a stretched Active Directory site/subnet. In addition, the DAG runs in DAC mode.

To get Exchange 2010 up and running in the failover datacenter, I basically went through the steps described in part 12 of my “Planning, Deploying, and Testing an Exchange 2010 Site-Resilient Solution sized for a Medium Organization” articles series.

First, I used “Stop-DatabaseAvailabilityGroup DAG01 –MailboxServer <DAG member server> –ConfigurationOnly” to stop the respective Mailbox servers located in the primary datacenter that participated in the DAG. I used the “ConfigurationOnly” parameter since all Domain Controllers in the primary datacenter was unavailable.

After having stopped the Mailbox servers in the primary datacenter, I used “Get-DatabaseAvailabilityGroup DAG01 | FL Name,StoppedMailboxServers,StartedMailboxServers” to verify the servers had been stopped and that the Mailbox servers in the failover datacenter was started.

Then I stopped the cluster service on the Mailbox servers in the failover datacenter as this is necessary prior to restoring the DAG using the “Restore-DatabaseAvailabilityGroup” cmdlet.

After having stopped the cluster service, I could concentrate on restoring the DAG in the failover datacenter. For this, I used the following command:

Restore-DatabaseAvailabilityGroup DAG01 –ActiveDirectorySite <AD site>

I used the “ActiveDirectorySite” parameter since a “MailboxServer” parameter doesn’t exist for the “Restore-DatabaseAvailabilityGroup cmdlet (see http://technet.microsoft.com/en-us/library/dd351169.aspx).

The resulted in an error message similar to the following:

Confirm
Are you sure you want to perform this action?
Restoring Mailbox servers for Active Directory site "<AD site>" in database availability group "DAG01".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [?] Help (default is "Y"): y

WARNING: Server '<Exchange Servers>' was marked as stopped in database availability group 'DAG01' but couldn't be removed
from the cluster. Error: A server-side database availability group administrative operation failed. Error: The
operation failed. CreateCluster errors may result from incorrectly configured static addresses. Error: An error
occurred while attempting a cluster operation. Error: Cluster API '"EvictClusterNodeEx('<Exchange Servers>’) failed
with 0x46. Error: The remote server has been paused or is in the process of being started"' failed. [Server: <Exchange Servers>]

WARNING: The operation wasn't successful because an error was encountered. You may find more details in log file
"C:\ExchangeSetupLogs\DagTasks\dagtask_2011-10-08_21-22-17.549_restore-databaseavailabilitygroup.log".


I suspected this was because Exchange tried to communicate with a Domain Controller in the failed datacenter, so I made sure Exchange used a domain controller in the failover datacenter using this command:

Set-ADServerSettings –PreferredServer <Domain Controller in Failover Datacenter>

Then I ran Restore-DatabaseAvailabilityGroup DAG01 –ActiveDirectorySite <AD site> again and voilá I got this result:

Confirm
Are you sure you want to perform this action?
Restoring Mailbox servers for Active Directory site "<AD site>" in database availability group "DAG01".
[Y] Yes  [A] Yes to All  [N] No  [L] No to All  [?] Help (default is "Y"): y
WARNING: The Exchange Trusted Subsystem is not a member of the local Administrators group on specified witness server
file02.domain.com.

And databases began to mount on DAG member servers in the failover datacenter.

So what have I learned? Remember to use the “Set-ADServerSettings” cmdlet to make sure Exchange tries to communicate with a Domain Controller in the failover datacenter, when you’re dealing with a stretched AD site/subnet.

Until later,

Henrik Walther
Technology Architect/Writer/MS Vendor
MCM: Exchange Server | MVP: Exchange Architecture

Add Review or Comment

Featured Links