Saturday, 15 June 2013

Backup Exec 2012 backup of Hyper-V 2008 R2 cluster fails with error 0x1398

We have a two node Hyper-V R2 environment at our customer premises which is backed up by Symantec Backup Exec 2012. Node 2 in a cluster was experiencing difficulties and we evicted the node, reinstalled the operating system and rejoined the node in the cluster. After that, virtual machines that were running on this node could not be backed and this was logged in backup exec log:

Final error: 0x1398 - The operation failed because either the specified cluster node is not the owner of the group, or the node is not a possible owner of the group.

The error is described in this Symantec article:
http://www.symantec.com/business/support/index?page=content&id=TECH160511

When examining the Failover Cluster configuration, we ran the powershell script that displays the possible owners list for CSV volumes:

PS C:\Windows\system32> Get-ClusterSharedVolume | Get-ClusterOwnerNode

ClusterObject                                               OwnerNodes
-------------                                               ----------
EXCHANGE_LUN                                                {vmhost1}
VM1_LUN                                                     {vmhost1}
VM2_LUN                                                     {vmhost1}


The node vmhost2 was missing from the list so we added it back with this commands:

Set-ClusterOwnerNode -Resource "EXCHANGE_LUN" -Owners vmhost1,vmhost2
Set-ClusterOwnerNode -Resource "VM1_LUN" -Owners vmhost1,vmhost2
Set-ClusterOwnerNode -Resource "VM2_LUN" -Owners vmhost1,vmhost2

Re-running the first script now shows this:

PS C:\Windows\system32> Get-ClusterSharedVolume | Get-ClusterOwnerNode

ClusterObject                                               OwnerNodes
-------------                                               ----------
EXCHANGE_LUN                                                {vmhost1, vmhost2}
VM1_LUN                                                     {vmhost1, vmhost2}
VM2_LUN                                                     {vmhost1, vmhost2}


The backup of virtual machines on previously evicted node is now running without any problems.