Hyper-V & SCVMM Error: 12700 / (0x8005) Migration Failed

Today I was testing the migration between my Hyper-V server clusters and the non-clusters Hyper-V.

The first step was to migrate from the cluster to a non-cluster server using SCVMM. The process went very well and the machine was successfully migrated between both hosts...

Then I decide to do the other way around... When the process was at 49% it crashes with the following error:

Error (12700)
VMM cannot complete the Hyper-V operation on the hv01srv.domain.tld server because of the error:
Import failed because the virtual machine must have a new identifier. Select a new identifier and try the import again.
(Unknown error (0x8005))
Recommended Action
Resolve the issue in Hyper-V and then try the operation again.

Additionally you get the EVENTID 18070 on the host where you perform the operation:
Log Name: Microsoft-Windows-Hyper-V-VMMS-Admin
Source: Microsoft-Windows-Hyper-V-VMMS
Date:
Event ID: 18070
Task Category: None
Level: Error
Keywords:
User: SYSTEM
Computer: hv01srv.domain.tld
Description: Import failed because the virtual machine must have a new identifier. Select a new identifier and try the import again.


GREAT!!!!

Before telling you how to solve this, let me tell you that you can always rollback the migration process by right-clicking in the VM and choose repair and then select the Undo option and press OK. So don't panic!!!

This problem occurred because when I did the migration from the HV Cluster to the Non-HV server, the file {GUIG}.xml) that identifies the VM was not deleted from cluster node. The file is in the "C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines\{GUIG}.xml"

Note: By default the ProgramData Folder is hidden, to view this folder you must activate the option "Show hidden folders and files" under folders option menu.

Solution:
To solve this error, delete the duplicated file in the destination server (if you're in a cluster configuration you may need to check all the nodes before retry the migration).

How do I know the file to delete?

Simple, on the source server open the:

"C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines\"

Do the same for the destination server and open:

"C:\ProgramData\Microsoft\Windows\Hyper-V\Virtual Machines\"

Now, find the file that is duplicated in both sides:

After identification of the given xml file, on the destination server, delete the duplicated file and retry the migration again and you should be able to succeed this time.

Finishing the migration process at 99% after the change...

I ran search for a fix or explanation for this behavior, but I did not find anything that explains this, I'll try to update this post latter with a possible MS fix or explanation.

:)

Hyper-V & SCVMM Error: 0x80070057 Import / Export Virtual Machine

Today I tried to import a VM using the Hyper-V mmc console that I exported some time ago to later use. When I tried to do that I got the error:

Hyper-V Manager
A server error occured while attempting to import the virtual machine.
Failed to import the virtual nachine from directory %directorypath%.
Error: One or more arguments are invalid (0x80070057).

The solution to fix this error is to clear the ScopeOfResidence value and you should be able to import the same machine again with no errors.

To do that, go to the the VM path folder, then open the {GUID}.exp file (generally under "drive:\pathtovm\vmname\Virtual Machines\"). Open the .exp file
In this case clear the value for 9c62b993-f3e9-44fc-80e1-f798f8bbe17f :

Save the file and import the machine again.

DONE!

TIP: Suppose that you sysprep a VM and exported that VM to latter use. Now you duplicated that exported VM into 10 VMs (VM01, 02, 03 ...). When you try to import that VM you get this error, what to do?

2 options:

You can correct the ScopeOfResidence value for the original VM (the exported VM) and then duplicate that VM to 10 VMs as you did.

Or

You can correct the ScopeOfResidence value on the first duplicated VM (VM01) and then copy that *.exp file to all other VMs (VM02, 03 ...). Will this cause trouble with duplicated *.exp file names? No as long as you don't use the option "Reuse old virtual machine IDs", because when you import that VM, Hyper-V will change the value of those files to a different {GUID}.xml file.

After solving this one, I went to internet and did a search for a MS KB about this problem, and I found additional information under KB968968 that states this problem for Hyper-V and System Center Virtual Machine Manager (SCVMM) scenarios.

Additionaly they provide a script to use on a Hyper-V server before you import a virtual machine to the Hyper-V server

*****************************************************

Option Explicit

Dim WMIService
Dim VMList
Dim VM
Dim VMSystemGlobalSettingData
Dim VMManagementService
Dim Result

'Get instance of 'virtualization' WMI service on the local computer
Set WMIService = GetObject("winmgmts:\\.\root\virtualization")

'Get a VMManagementService object
Set VMManagementService = WMIService.ExecQuery("SELECT * FROM Msvm_VirtualSystemManagementService").ItemIndex(0)

'Get all the MSVM_ComputerSystem object
Set VMList = WMIService.ExecQuery("SELECT * FROM Msvm_ComputerSystem")

For Each VM In VMList
if VM.Caption = "Virtual Machine" then
Set VMSystemGlobalSettingData = (VM.Associators_("MSVM_ElementSettingData", "Msvm_VirtualSystemGlobalSettingData")).ItemIndex(0)
VMSystemGlobalSettingData.ScopeOfResidence = ""
Result = VMManagementService.ModifyVirtualSystem(VM.Path_.Path, VMSystemGlobalSettingData.GetText_(1))
end if
Next

*****************************************************

Hyper-V Resources & Documentation

Check some Microsoft useful documentation for Hyper-V in Windows 2008 and R2.

Hyper-V TechNet

Hyper-V Server 2008 Getting Started Guide

Microsoft Hyper-V Server 2008 Configuration Guide

Hyper-V Planning and Deployment Guide

Hypervisor Functional Specification

Virtualization with Windows Server 2008 Hyper-V

Overview & Architecture

Understanding Networking with Hyper-V

Hyper-V Security Guide

Microsoft Hyper-V Server 2008 - English

Microsoft® Hyper-V™ Server 2008 R2 RC

Windows Server 2008 R2 RC Evaluation Virtual Hard Drive Images (for Hyper-V)

Step-by-Step Guide for Testing Hyper-V and Failover Clustering

Windows Server High Availability with Microsoft MPIO

Windows Server Hyper-V Management Pack for System Center Operations Manager 2007

Last Friday Microsoft released the Windows Server Hyper-V Management Pack for System Center Operations Manager 2007.

Overview

This management pack supports monitoring of Windows Server Hyper-V systems. This includes monitoring coverage of Hyper-V host servers, including critical services and disks, and Hyper-V virtual machines, including virtual components and virtual hardware.
Feature Summary
- Management of critical Hyper-V services that affect virtual machines and host server functionality

- Management of host server logical disks that affect virtual machine health

- Full representation of virtualization in a single Hyper-V host server, including virtual networks, virtual machines, and guest computers

- Monitoring of virtual machine hardware components that affect availability.

Download it here

SQL Server Management Studio Slow Startup

This is an old one, but time to time I get the question
"Do you know how to speed up the SQL Server Management Studio?"
Normally I redirect the user to Microsoft KB555686. Today I decided to blog this and in the future I will redirect the users to this one :).

The problem is when you click the SSMS it performs several validations. For example the .Net runtime tries to contact "crl.microsoft.com" in order to ensure that the Certificate is valid. This can represent a problem if your internet connection is not available or if your firewall is blocking it.

How to solve this:
- If the server is allowed to access the internet, make sure that your firewall rules allows the access to the crl.microsoft.com website, additionally you may want to check if your firewall is not too busy to respond the server which may cause some delay as well.

- If your server doesn't have IE connectivity or you don't want to allow that access or perform the validation at crl.microsoft.com, you can configure the IE Advanced Options (Clear the "Check for Publishers certificate revocation list"

- Additionally, you may remove the splash option using the SSMS shortcut. Add the flag "/nosplash" at the end of the path to the SSMS.


- As described in the KB555686, also check the additional configurations:

1 Go to "Start" -> "Programs" -> "Microsoft SQL Server 2005".
2 Open "SQL Server Management Studio".
3 Navigate to "Tools" -> "Options" -> "Help" -> "Online".
4 Mark the checkbox "Try local first, not online".
5 Press on "Ok" button.6. Restart SQL Server Management Studio".

1 Go to "Start" -> "Programs" -> "Microsoft SQL Server 2005".
2 Open "SQL Server Error and Usage Reporting"
3 Uncheck the following checkboxes:
a."Send error reports for all components and instances of SQL 2005 to Microsoft or your corporate error reporting server."
Note: Using local "corporate error reporting server", may require to enable this feather.
b. "Send feather usage reports for all components and instances of SQL 2005 to Microsoft."
4 Press on "Ok" button.

Have fun :)

Two-Node Failover Cluster in Windows Server 2008 r2

Introduction:
In this blog entry I decided to create a demo to help you to get starting with Microsoft Cluster Services. I'm not going into much detail on every aspect of clustering but you should be aware that clustering has some special requirements for apps and hardware, make sure you follow them before starting to build or think to create one.

Description:
A failover cluster is a group of independent computers that work together to increase the availability of applications and services. The clustered servers (called nodes) are connected by physical cables and by software. If one of the cluster nodes fails, the other node begins to provide the service that fail in the first one (this process is known as failover). By doing this the users experience a minimum of disruptions in service and the business doesn't stop because of the failed node.

Clusters are used to provide high availability to mission-critical applications. Failover clustering can help you build redundancy into your network and eliminate single points of failure for apps, services that are critical to your business. In the network infrastructure that connects your cluster nodes, avoid having single points of failure. You can connect your cluster nodes by multiple, distinct networks. Alternatively, you can connect your cluster nodes with one network that is constructed with teamed network adapters, redundant switches, redundant routers, or similar hardware that removes single points of failure.

Cool!!! Well... yes it's nice to have a group of servers to provide availability but nothing comes for free, and clusters can be expensive.

In today’s world there're many apps that are mission-critical for business needs, and without those apps running your company may lose lots of money because of that. Mail Servers, Database Servers, Virtualization Servers, Print Servers are just few examples in many other possible scenarios where clusters are used.

Two Node Cluster:

- Before starting, check Failover Cluster Requirements here.

- In this demo we'll use 2 servers with identical hardware and OS (Windows 2008 r2) and the same SP level.

- Both nodes are part of Active Directory Domain as member servers.

- The account that I'll use to configure the Cluster does not need to be a Domain Admins account, but needs to be a Domain Users account that is in the Administrators group on each clustered server (or the group that the account is a member of). In addition, if the account is not a Domain Admins account, the account (or the group that the account is a member of) must be given the Create Computer Objects permission in the domain. If your Domain Admin don't want to give you that privilege you must request or pre-create the Computer Object in a Disabled state. After creating the Computer Account you must give 'Full Control' permission to the user (or to the group that the account is a member of) for that Computer account prior to creating the cluster.

- The cluster should have in both nodes 2 NICs, one for Public communications (the clients use this interface to communicate with the cluster), and other for Private communications (only between both nodes "Heartbeat").

- For this two-node cluster, the quorum configuration will be Node and Disk Majority. Node and Disk Majority means that the nodes and the witness disk each contain copies of the cluster configuration, and the cluster has quorum as long as a majority (two out of three) of these copies are available that provides the specific permissions and privileges that are necessary for the service (similar to the local system context, but with reduced privileges).

- Remember that you must use shared storage that is compatible with Windows Server 2008. If you are using Serial Attached SCSI or Fiber Channel, in all clustered servers, the mass-storage device controllers that are dedicated to the cluster storage should be identical. They should also use the same firmware version (With Windows Server 2008, you cannot use parallel SCSI to connect the storage to the clustered servers.).

Heve Fun :)

System Center Operations Manager 2007 R2 Documentation

System Center Operations Manager 2007 R2 Documentation available for download at:
MS Download Center

or

For information about the specific guides available in the library, check:
Technet Web Site

The Story Ends Bad for Scott/VMware FUD Fiasco Final

For those who don't known recently some videos were anonymously posted at YouTube “trying” to demonstrate how Hyper-V was unstable and a bad option for costumers.

After looking at the Videos Microsof Guys decided to reproduce the scenario in their lab.

The Video: No Facts Just FUD.
Not much.
Where are the details?
There are no facts provided.
What Hyper-V build was this? The beta? What was the configuration being tested? Who posted this? Why didn't they contact Microsoft support? I mean there's literally no data other than a defamatory statement at the beginning of the video implying that Hyper-V had something to do with some downtime at TechNet/MSDN.

In short, Hyper-V had nothing to do with the outage.
So, why would someone create such a video? Let's dig a little deeper.
The Poster.
The poster, who doesn't appear on the video, doesn't state what company he works for or provide any context. Gee, I wonder where he works?

Introducing Scott Drummonds, VMware Product Marketing.
Gosh, I wonder why Scott didn't mention he works for VMware?
Very professional Scott.
No signs of desperation there at all.

VMware has admitted their mistake, apologized and removed the video.

At the end it they discover that the Lab didn’t meet the minimum requirements for running windows Server 2008 and the patch to solve the issue was release in April 2008.

I wonder why VMware guys are afraid of...
Hyper-V is in the right track. Congratulations Microsoft.

Hyper-V & SCVMM Error: Host status of 'Not Responding'

Host status of 'Not Responding'

This is a very common issue,
Although can be very hard to troubleshoot, where's a nice article that describes step-by-step how to troubleshoot this error.

Important Updates:
· 958124 A wmiprvse.exe process may leak memory when a WMI notification query is used heavily on a Windows Server 2008-based or Windows Vista-based computer

· 954563 Memory corruption may occur with the Windows Management Instrumentation (WMI) service on a computer that is running Windows Server 2008 or Windows Vista Service Pack 1

· 955805 Certain applications become very slow on a Windows Server 2008-based or Windows Vista S955805-based computer when a certificate with SIA extension is installed

· 961983 Description of the hotfix rollup package for System Center Virtual Machine Manager 2008: April 14th, 2009

Nice job :)

Anti-Virus Settings for Microsoft OS

Here's a nice document from Microsoft that you should read before configuring your Anti-virus settings.

This document has recommendations that may help you protect a computer that is running Windows Server 2008, Windows Server 2003, Microsoft Windows 2000, Windows XP, or Windows Vista from viruses. This article also contains information to help you minimize the effect of antivirus software on system and network performance.

Click KB822158

Updated Read-Only Domain Controller (RODC) Branch Office Guide

Planning for RODC?
Before any implementation have a look at this updated guide at:

RODC) Branch Office Guide
or using
Technet

This guide describes new features in Windows Server 2008 that can provide benefits for Active Directory deployments that include branch offices. It explains how to assess an existing deployment of domain controllers in branch offices to determine whether deploying read-only domain controllers (RODCs) in existing or future branch offices is appropriate for your organization. For more general information about how to install and configure an RODC, see Planning and Deploying Read-Only Domain Controllers. For more information about deploying an RODC in a perimeter network (also known as DMZ), see Active Directory Domain Services in the Perimeter Network (Windows Server 2008).

Windows 7 and Windows Server 2008 R2 RTM dates

It's official, finally Microsoft gave us a date for Windows 7 and Windows 2008 R2 RTM
According with MS Press:
Windows 7 is On The Way – Oct. 22 Announced as Release Date

Microsoft will deliver Release to Manufacturing (RTM) Windows 7 and Windows Server 2008 R2 code to partners in the second half of July, additionally Windows 7 will become generally available on Oct. 22, 2009, and Windows Server 2008 R2 will be broadly available at the same time.

Read more at:
MS Press

Active Directory Database Corruption/Recovery

Active Directory Database corruption and you don't know what to do?

Did you got stuck with a domain controller that at startup is showing a message "Directory is Rebuilding Indices" and after a long time it fails?

Did you found corruption messages in event log:
NTDS ISAM Event ID: 467 database corruption Error,
NTDS Replication Event ID: 1084 Replication Error
NTDS Replication Event ID: 2108 Replication Error
NTDS General Internal Event ID: 1173 Processing Warning
………………………………………………….and others...

What about descriptions:
- 8451 The replication operation encountered a database error.
-1414 JET_errSecondaryIndexCorrupted, Secondary index is corrupt. The database must be defragmented

Yep, these are some of the corruption errors that you may find if your Active Directory (AD) Database (DB) is "dead"

1 - Before proceed, let me tell you that in scenarios like this one, you should always try to get the best help possible. Best help means calling to Microsoft PSS. They have the necessary experience and documentation to help you with those problems.

2 - The recovery solutions posted bellow does not guarantee that your problem will be fixed. The posted solutions are general recommendations that you may decide to follow or not. Use the information on this post at your own RISK!!! And remember to ALWAYS test in lab environments before going to production.

That said what options do I have when this happens?

Note: You should always try to find the root cause for these types of problems. The root problems are normally related with hardware problems, antivirus configurations, Virus, Power outages, etc... If you don't identify the root cause, there's a good probability that you may end up where you started.

Now it's time to recover...

******************************************************************
Scenario 1: Recover From Backup *******************************
******************************************************************
1. Get your DC backup and try to restore the DC using the latest backup. At minimum you need the system state backup to recover the AD DB. To restore the System State data on a domain controller, you must first start your computer in Directory Services Restore Mode (ADRM). This will allow you to restore the SYSVOL directory and Active Directory service database.

2. To access Directory Services Restore Mode, reboot the server press F8 during startup and select it from the list of startup options. If you're using a third party backup solution please consult the vendor documentation for Domain Controllers backup/recover scenarios, if you’re using NTBackup from Microsoft Windows check the basics here and here.

Note: Your DC backup is only valid if it’s within the forest tombstone lifetime.

3. After system restore, reboot the server and if everything is ok, find the root cause of the problem and fix it.

******************************************************************
Scenario 2: Rebuild the Domain controller **********************
******************************************************************
If you've more than one Domain controller you may try to rebuild the DC that is having problems and then re-promote it again.

1. Remove active directory from the DC. You can do that formatting the hard drive, replacing the drive with a new one (backup the files that you need before formatting the drive). Normally this is done by using the dcpromo /forceremoval, but in corruption scenarios that shouldn't work. Just MAKE SURE that the DC and related Active Directory configuration IS OUT of the DC and is NEVER AGAIN related or CONNECTED to the same network where the ORIGINAL HEALTHY DCs are. Is very important to guarantee this step or you may end up in a complete forest corruption scenario. Perhaps formatting the drive is the best option here... Just in case :)

2. The second step relates to seizing process. No, it's NOT "Transfer", it's SEIZE ROLES, transfers are only possible when the DCs that have FSMO roles are online, but that’s not the case because we formatted the drive, right?

If your "formatted" DC held any FSMO roles, you must seize them to another online DC. To identify if your “formatted” DC had any FSMO roles in it go to command prompt and type (first install support tools from your windows cd\Support directory):

netdom query fsmo

This command will return the FSMO owners for the forest and the domain were you're performing the query. In a forest, there are at least five FSMO roles that are assigned to one or more domain controllers, 2 are Forest wide and 3 exist in each domain in the forest.

The five FSMO roles are:
Schema Master (Forest)
Domain naming master (Forest)
Infrastructure Master (Domain)
Relative ID (RID) Master (Domain)
PDC Emulator (Domain)
To know more about these roles click here:

If the command returns the "formatted" DC as owner of any of these FSMO roles, you need to Seize them to a different, online domain controller to know more about that process check:
Using Ntdsutil.exe to transfer or seize FSMO roles to a domain controller

3. The next step is to perform metadata cleanup. You need to manually remove all remnant entries of the corrupted DC from AD database. To do that follow:
How to remove data in Active Directory after an unsuccessful domain controller demotion
You should also remove any DNS related entry to that DC.

4. Okay, take a deep breath... Wait or force replication so every DC knows about the changes that you made.

5. After knowing that changes were successfully replicated to all existing DCs it should be safe to promote the server back to domain controller. But wait!!! Did you already determine the root cause of the problem? No!!! Fix that first. You don't want to end up in the initial scenario right?!

******************************************************************
Scenario 3: Manually FIX AD Database ************************** ******************************************************************

This option should be used as last resource. As I said at the beginning of this post, follow it or not at your own RISK!!! Remember to ALWAYS test in lab environments before going to production.

To manually fix the AD DB check ALL STEPS

1. Reboot the server and press F8. Choose Directory Services Restore Mode from the Menu.
2. Check the physical location of the NTDS folder (Normally at %WINDIR%\NTDS\).
3. Perform a backup of the NTDS folder (copy the folder to a different drive or to the same drive with a different name eg: NTDSBK). If something goes wrong you can always replace the original files with this copy/backup.

4. Check the permissions for the "NTDS" folder.

Windows Server 2003
Default permissions are:
System Full Control This folder, subfolders and files
Administrators Full Control This folder, subfolders and files
Creator Owner Full Control Subfolders and Files only
Local Service Create Folders / Append Data This folder and subfolders

5. Check the %WINDIR%\Sysvol\Sysvol folder to make sure it is shared.

6. Check the permissions on the %WINDIR%\Sysvol\Sysvol share. Compare them with other online DC.

Note: You may not be able to change the permissions on these folders if the Active Directory database is unavailable because it is damaged, however it is best to know if the permissions are set correctly before you start the recovery process, as it may not be the database that is the problem.

7. Make sure there is a folder in the Sysvol share labeled with the correct name for the domain.

8. Open a command prompt and run NTDSUTIL to verify the paths for the NTDS.dit file. They should match the physical structure from Step 2
From command prompt type:

ntdsutil files info

Output that is similar to the following appears:
C:\ NTFS (Fixed Drive) free(850.4 Mb) total(10.2Gb)
DS Path Information:
Database : C:\WINDOWS\NTDS\ntds.dit - 10.1 Mb
Backup dir : C:\WINDOWS\NTDS\dsadata.bak
Working dir: C:\WINDOWS\NTDS
Log dir : C:\WINDOWS\NTDS - 20.2 Mb total
temp.edb - 1.1 Mb
res2.log - 10.0 Mb
res1.log - 10.0 Mb
edb00001.log - 10.0 Mb
edb.log - 10.0 Mb

This information is pulled directly from the registry subkey: "HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\NTDS\Parameters". Wrong paths can lead to incorrect start up of Active Directory. If this is your case check KB240362.
Type Quit to end the NTDSUTIL session.

9. Rename the edb.chk file and try to boot to Normal mode. If that fails, proceed with the next steps.

10. Reboot the server and press F8. Choose Directory Services Restore Mode from the Menu. From command prompt use the ESENTUTL to check the integrity of the database. You can use NTDSUTIL to check the integrity however ESENTUTL is usually more reliable.

To perform the integrity check, start the command prompt, type the following command:

esentutl /g "path\ntds.dit" /!10240 /8 /o

The output will tell you if the database is inconsistent and may produce a jet_error 1206 stating that the database is corrupt. If the database is inconsistent or corrupt it will need to be recovered or repaired. To recover the database type the following at the command prompt:
NTDSUTIL
Files
Recover

If this fails with an error, type quit back at the command prompt and repair the database using ESENTUTL. Type the following command:

esentutl /p "path\ntds.dit" /!10240 /8 /o

Note: If you do not put the switches at the end of the command you will most likely get a Jet_error 1213 "Page size mismatch" error.

11. Delete the log files inside NTDS directory, but do not delete or move the ntds.dit file.

12. The NTDSUTIL tool needs to be run again to check the Integrity of the database and to perform Semantic Database analysis. To check the integrity, at the command prompt type:

NTDSUTIL
Files
Integrity

The output should tell you that the integrity check completed successfully and prompt that you should perform a Semantic Database Analysis.
Type quit.

To perform the Semantic Database Analysis type the following at the NTDSUTIL Prompt type:
Semantic Database Analysis
Go

The output will tell you that the Analysis completed successfully.
Type quit and close the command prompt.

NOTE: If you get errors running the Analysis then type the following at the semantic checker prompt:
semantic checker: go fix

This puts the checker in Fixup mode, which should fix whatever errors there were.

13. Okay, take a deep breath... Review all steps...

14. Reboot the server to Normal Mode.

Hopefully one of these options will fix your problem :)

Additional Information:
Complete a Semantic Database Analysis for the Active Directory
Error Message: Lsass.exe - System Error : Security Accounts Manager
Windows 2000 DCs Unable to Boot into Active Directory
Use Ntdsutil to Manage Active Directory Files from the Command

This posting is provided "AS IS" with no warranties, and confers no rights.