flyinweb's blog - 云计算与虚拟化

VMware vCenter Converter导入计算机时出错

shanyiwan@live.com() — Fri, 03 Feb 2012 03:20:11 GMT

VMware vCenter Converter导入计算机时出错，对话框内容如下：

---------------------------
VMware vCenter Converter
---------------------------
出现错误。此错误的原因可能由网络连接故障所致。
向导将立即关闭。
请检查网络连接并重试。
---------------------------
确定
---------------------------

解决办法：

重启以下服务：

VMware vCenter Converter Integrated Server

VMware vCenter Converter Integrated Worker

使用VM Explorer备份与管理VMware ESX虚拟机

shanyiwan@live.com() — Tue, 15 Nov 2011 08:44:53 GMT

　　市场上各种VMware工具中，有一来自TriLead公司叫做VM Explorer的工具，他们提供了完整产品的免费版本，我喜欢免费的VMware工具，所以测试了它。VM Explorer也叫做VMX，它能做开发虚拟环境以外的更多事情。事实上，专业版本的VMX可以替换VMware Consolidated Backup（VCB），帮助运行VMware ESX的小型企业。

　　VMX有一个图形用户界面和作业调度程序，能够直接复制到NAS或SAN，并且提供像任务历史、备份完成邮件、虚拟机子机恢复、SSH及访问虚拟机的远程桌面协议、克隆和快照管理，这些都是VCB不提供的功能。对于喜欢尝试新免费VMware工具的管理者来说，VMX本身是一个非常有用的备份和管理工具。

什么是TriLead的VM Explorer？

　　我发现TriLead是一家开发程序的公司，不是典型的VMware Application开发公司。不过他们仍然开发出了非常有用的工具。这家公司将这款工具描述成“简化VMware环境管理、备份和灾难恢复任务的虚拟化管理工具”。下面是这款工具提供的主要功能：
?同时连接多个VMware ESX与Linux Servers的能力。
?直接从一台服务器复制文件（如虚拟子机配置与虚拟磁盘文件）到另一台的能力。
?备份和存储虚拟子机到（附属到ESX主机的）SAN、另一台ESX主机、Linux主机或者运行VMX的Windows工作站的能力。
?设定时间备份虚拟子机的能力。
?文件浏览及拖放文件。
?在命令行使用SSH客户端配置ESX和ESXi主机。
?创建和删除VMware ESX快照的VMware ESX快照管理。
?压缩迁移或ESXi支持文件的能力。

　　尽管这款工具可免费下载使用，不过免费版本不包括一些功能。完整功能版本的包括作业调度、全部技术支持和无限设备（关于完整版本与免费版本的总体比较参见VMX edition comparison chart）。

　　下载和安装VM Explorer：在TriLead VM Explorer download页面下载VM Explorer。

　　VMX文件只有3.1MB，下载和安装都非常快。下载后，启动VMX窗口只有不到30秒的时间。

使用VM Explorer

　　要使VMX发挥作用，第一件事是添加主机。我添加了ESX主机，如图所示：

　　然后进入数据中心视图，点击新服务器，VMX迅速显示服务器的类型及其所有存储卷、所以运行其上的虚拟子机及其参数，如图：

　　在这里，使用SSH制表（自动登录的）连接到ESX或Linux主机，检查为主机制订的预期备份（使用的是专业版本），或者查看子虚拟机。当我点击一个子虚拟机，我可以得到操作系统类型、虚拟硬件、目前性能参数、子VMDK的虚拟存储信息，并且能通过远程桌面协议（RDP）或SSH直接连接到虚拟机的能力。

　　如果右击虚拟机，出现以下几种选项：

　　如图所见，你能在File Explorer里找到子虚拟机的VMX与VMDK文件，还可以关闭、开启、关掉子操作系统、注销虚拟机、克隆、备份或管理每个虚拟子机的快照。

　　我尤其喜欢从界面克隆和备份虚拟子机。下面是一个虚拟子机的备份窗口：

　　记住，这些备份是每台虚拟机的完全备份，不需要任何子机停机时间。这是由于VMX对子虚拟机进行快照和备份。同样注意如何使用变量为虚拟机名字和日期及时间创建自动目录。这对预定备份选项尤其有用。

　　最后，你不需要将子虚拟机备份到本地Windows PC。你可以直接备份到Linux主机或ESX主机上的SAN连接。

　　有了这些功能，对于每天有预期备份到Linux SAN或iSCSI SAN的拥有少量ESX主机的小型企业来说，VMX是一种不错的选择。

VM Explorer专业版本中使用作业调度

　　我想实验作业调度，所以我获取了一个评估密匙激活这个工具。然后右击一台虚拟子机，点击“Create New Backup Schedule”。

　　从上面的图片中可以看出，我能为虚拟子机快速地创建一个简单的预定备份。

　　下面是预定备份工作显示的作业调度窗口：

　　此外，VM Explorer也会告诉你如何为ESX和ESXi主机启动SSH Access。这意味着不是仅仅告诉你启用SSH就能执行某个任务，它还在帮助条选项里提供了步骤说明。如图所示：

　　许多公司都提供了一些免费的VMware工具，不过多数都充斥着是完整版本的广告，或者只提供了一些有限的功能。TriLead的免费版本相对于其完整版也很有用，VM Explorer是我所推荐的非常有用的工具之一。

主机效能越来越强大，为了要能妥善使用系统的效能，虚拟主机（vm）的使用势在必行，逐渐把系统使用率较低的主机转换成vm集中在一起，虽然说把鸡蛋都丢同一个篮子风险性较高，不过对于节能省碳来说真的比较好，风险上的问题可以透过架构来减低。而我在家里使用ESXi也很长一段时间了，可以让我开很多台Linux来玩，不过一直以来我只有对档案做备份，整个系统却未曾备份过，若真的需要灾难复原会花上较多的时间，因此找到了可以备份整个vm的软件，也就是【VM Explorer】。

VM Explorer：

【VM Explorer】有免费及付费的版本（比较），功能上有些许的差异，不过对个人用户来说，免费版本就已经非常的方便，而且已经可以做到Online Backup（Hot Backup），当vm在开机运作的同时也可直接做备份，就不用再另外找时间关机备份了！而免费版本做可惜的就是无排程备份的功能，不然就真的太方便啰！

【主界面】

软件的安装就不多介绍了，主要就功能其实也非常简单，就当作让大家预览一下画面吧！这软件除了能备份还原vm之外，其实也可以做简易的ESX与ESXi管理，可以加入很多Host在软件接口当中，进而达到不同主机不同vm的集中备份功能。

【新增ESX/ESXi主机】

首先必须先「Add a new Server」，ESX的主机必须要支持ssh的管理功能，ESXi 4.1已经默认就有支持直接开启SSH，ESXi 4.0可以参考之前的文章开启，填写完信息后进行「Test Connection」，确认联机没问题后就可以完成新增主机。

【ESX/ESXi状态】

VM Explorer可以看到一些很基本的信息，包含每个vm的基本设定，ESX主机的使用状态，还有每一个vm的运作情形，也可以知道每一台vm的vmdk运用。

【File Explorere管理Datastore】

「File Explorer」能直接读取Datastore当中的档案，可以透拖曳的动作达到文件传输的功能，而且我想可以支持不同ESX主机之间的传输，不过这方面我还没有做过测试^^。

【vm状态管理】

vm状态的查看，Setting之中有包含ssh的设定，不过设定之后我也不知道要做什么，因为在最后ssh那页依然无法连接。

【vm的备份】

要开始一个备份相当的简单，又见点选要备份的vm，从选单内点选「Backup」。

设定备份的目的地，如果目的地已经有目录或是有档案都有不同的设定，这种设定适合排程的工作，可以自动判断该如何处理备份的档案，尤其最后一项的「Nbr. of backup to keep」更是很好的一种备份设定，可以主动的帮你保留最近几代的数据，不用自己再去删除过多的备份，像我的PC目前使用【EASEUS】来备份到我的NAS，不过我都要定期的上去删除较早的备份。

选择要备份的vmdk，每一台主机可能不只一个vmdk，通常还会把系统使用的独立分开，备份时就可以分开选择。若vmdk的Mode为预设的状态，也就是persistent的模式，那么就可以支持online备份。

若是vmdk的Mode并非预设的情况，像我通常都是使用independent-persistent的设定，就无法在开机的状态下直接备份。而Passthrough的device也无法支持，但是Passthrough的装置就比较没有vm毁掉数据回不来的问题。

设定Snapshot

网络传输的选项，传输时可以设定压缩，最后的档案也建议可以压缩，这样可以较省备份空间。

设定完成就开始备份啰！我觉得这样的备份方式真的颇方便，而且设定真的很简单，相信已经有不少人已经开始使用，我还算是后知后觉得啦！

【vm的还原】

我们可以从主接口选择「Restore a Virtual Machine」，之后则会跳到「File Explorer」，开启之前的备份后，可以发现备份的档案之中有一个xml的档案，直接点选右键选择「Restore Backup」。

还原时可以设定直接覆盖原有的vm，也可以另外的还原成一份新的vm，有点类似Clone的功能。而这软件原本就有Clone的功能，透过备份还原算是比较麻烦的方式，不过证明这软件的确有方便的复制功能。

Cluster name resource failed registeration in DNS

shanyiwan@live.com() — Tue, 01 Nov 2011 06:28:16 GMT

For clustering the Cluster resource name must have full access to the Virtual Cluster Names so when failover takes place DNS entries can be updated.

However if you used a underscore _ a big NO! NO! with NetBIOS and DNS names. Solution: wipe and start over.
If you used a dash -, an exceptable NetBIOS name, your still screwed and got this error. BUG in Windows 2008 R2. You can add the name manualy to the DNS, but it will not be automatic.

Following errors were recorded in event logs when it registrations fails:

场景1：

Cluster network name resource 'SQL Network Name (VirutalClusterName)' failed registration of one or more associated DNS name(s) for the following reason:
DNS signature failed to verify.

Ensure that the network adapters associated with dependent IP address resources are configured with at least one accessible DNS server
In DNS Management (dnsmgmt.msc):

Find the VirtualClusterName that is failing to register.
Right-Click Properties.
Select Security Tab.
Click Add.
Click Object Types.
Check off "Computers"; uncheck other options selected.
Enter in the name of the cluster (a.k.a Cluster Name Object (CNO)).
Click Check Names; Verify that the entry has been found.
Click OK.
Give the CNO FULL Control over this record.
Click OK.

场景2：

群集网络名称资源“群集名称”注册一个或多个相关联的 DNS 名称失败。原因如下:
DNS 服务器失败。

请确保与从属 IP 地址相关联的网络适配器已配置为至少可以访问一个 DNS 服务器。

Hyper-V V2: Guest Only External Networks + Add Roles Wizard Changes

shanyiwan@live.com() — Thu, 27 Oct 2011 10:11:51 GMT

The Hyper-V best practice in traditional production environments is that any physical network interface used by guest virtual machines is dedicated and isolated to guest only traffic and not shared with the management operating system (host). This recommendation was made for several reasons – security is the primary reason since virtual machines are considered less trusted than the management partition isolating the network traffic reduces the risk that a malicious guest could take advantage of a remote security exploit to take over the physical machine. Following this practice also reduces the risk of a guest virtual machine saturating the network preventing the server administrator from being able to log onto the physical machine and take appropriate action. In Server 2008 (Hyper-V V1) you accomplished this by unbinding TCP (as well as any other network protocol) on the virtual adapter exposed by Hyper-V – in Server 2008 R2 we have added a new feature which be default does not create the virtual adapter on the management partition (of course there’s an option to have it the old way). In addition this functionality was pushed into the Add Roles Wizard when you create your first virtual network. Here’s some screen captures to illustrate.

Server 2008 (V1)	Server 2008 R2 (V2) One Physical Interface	Server 2008 R2 (V2) Two Physical Interfaces
Add Role Wizard – Virtual Network Page	Add Role Wizard – Virtual Network Page	Add Role Wizard – Virtual Network Page
Allows the creation of a new virtual network at install time even if the server has only one network interface.	When there is only one physical interface the ability to create a new virtual network at install time is disabled – you can create a new network post install.	When there are two or more interfaces you can select the interface(s) for which you want virtual networks created – you must leave at least one interface unchecked.
Hyper-V Virtual Network Manager Post Role Install	Hyper-V Virtual Network Manager Post Role Install	Hyper-V Virtual Network Manager Post Role Install
Post install the virtual network is created and bound to the physical interface.	No virtual network is created – however you can now create an interface which can/should be shared with the management partition/operating system.	Post install a new virtual network is created and bound to the physical interface but no virtual interface is exposed to the management partition/operating system.
Network Connections On The Host	Network Connections On The Host	Network Connections On The Host
You can see both the physical and virtual adapters are available – the physical interface will only have the Microsoft Virtual Network Switch Protocol bound to it and the virtual interface will have TCP and other network services bound.	Since there are no virtual networks created by default you don’t have any virtual interfaces exposed and the physical interface is not bound to the Microsoft Virtual Network Switch Protocol.	There are still just two interfaces on the management partition/operating system* both are physical (the fact that one is disconnected is because it really is disconnected on my server).*
Hyper-V Virtual Network Manager Creating a New Shared Virtual Network
When creating a new virtual network checking the “Allow management operating system to share this network adapter” checkbox will create a new virtual interface on the management partition/operating system.
Network Connections On The Host
A new virtual interface is created - the physical interface will only have the Microsoft Virtual Network Switch Protocol bound to it and the virtual interface will have TCP and other network services bound.

Soure From:http://blogs.msdn.com/b/taylorb/archive/2009/01/12/hyper-v-v2-guest-only-external-networks-add-roles-wizard-changes.aspx

Changing the IP Address of an Exchange Cluster

shanyiwan@live.com() — Thu, 27 Oct 2011 10:01:06 GMT

I was recently asked about how to change the IP address of an Exchange 2007 cluster. Here is the information that I shared:

NOTE: Be sure to test this before trying in production. Each environment and application may respond differently to changes.

For Windows 2003 Cluster
1. Set Cluster Admin against the local node (.)
2. Take the Cluster Groups Offline
3. Change the cluster IP Address resource objects to new addresses within each group
4. Change Cluster Service to Manual and stop on both nodes
5. Change physical IP Address/Subnet/Gateway on each node
6. Check the IP address under HKLM\Cluster\Network Interfaces and HKLM\Cluster\Network by pointing to the present ip replacing the old IP address.
7. Cleared the DNS entries for the cluster pertaining to the old IP addresses.
8. Reboot the server (cluster service should start)
9. Bring resources online

For Windows 2008 Cluster
1. Open Failover Cluster Management
2. Take the Exchange Cluster Groups offline (if applicable)
3. Select the Cluster Name in the left pane. In the middle pane, scroll down to CLUSTER CORE RESOURCES.
4. Take Cluster core resource Name and IP address resource offline (right click and select Take this resource offline)
5. Open the Network Properties on each physical node and Change IP addresses of the Public Network
6. Register the new IP addresses in DNS (IPConfig /RegisterDNS) on both physical nodes (make sure DNS is up to date)
7. Back in Failover Cluster Management. Select the cluster object and then Networks. Select each cluster network and make sure that the NODE IP address is up-to-date.
8. Under Cluster Core Resources, go to the properties of the Cluster Name’s IP Address resource.
9. In the NETWORK drop down, select the appropriate Subnet object.
NOTE: If the object does not exist yet, refresh the cluster object, check network settings, and make sure that the Cluster Network is set to “allow clients to connect through this network”
10. In the Static Address field, change the IP address and apply the settings.
NOTE: Check the dependencies tab too
11. Go to the properties of the Cluster Name resource and make sure that the IP address is refreshed with the new IP address
12. Start Cluster Core services IP and name resources
13. Update the IP address of the Exchange Cluster Group Resource objects (if applicable)
14. Start Application Cluster Group’s IP and name (if applicable) (and all other offline resources)
15. Select Failover Cluster Management in the top left pane and then select Validate a Configuration… Check for errors and mis-configured objects.

FAQ
Do I need to reboot the server nodes?
You might if you plan to move between subnets and/or AD Sites or if the server is not recognizing the network after the IP address change. Otherwise, no.

Can I switch it back to the old IP address?
Yes but you may need to clear out any caching of that IP address from WINS, DNS, and Network Devices (i.e. routers, switch ports, etc) before switching it back.

Is this something that I can do in production?
Yes but you should limit your changes to a cluster. The purpose of a cluster is to provide high availability and in this scenario, you need to bring down the resources to complete the change.

How can I use Cluster.exe to update the IP address of the cluster?
You can use Cluster.exe to update the cluster resources that may reflect the IP address but not necessarily the IP address associated with the NIC.

What else should I consider when I change the IP address of the cluster?
If you are changing subnets, the server may belong to another AD site. Know the affect on Exchange and the clients connecting to it before you make that move.
Also, make sure that the switch port is not statically mapped to an IP address or subnet

Unknown processor error when performing Live Migration on Windows Server 2008 R2 Failover Cluster

shanyiwan@live.com() — Thu, 27 Oct 2011 03:53:50 GMT

[Window Title]
错误

[Main Instruction]
检查目标节点上的虚拟机兼容性时出错

[Content]
虚拟机“Win08R2Web”与目标节点“node140.zzyddc.com”不兼容。
Hyper-V 检测到:
虚拟机“Win08R2Web”与物理计算机“NODE140”不兼容。(虚拟机 ID B577EECE-FCFE-4DF6-AE30-3C1D64EA471D)

虚拟机“Win08R2Web”当前正在使用 Intel 处理器，但物理计算机“NODE140”具有的处理器为未知。如果虚拟机和物理计算机的处理器来自不同的供应商，则无法将正在运行或已保存的虚拟机迁移到物理计算机。但如果关闭了该虚拟机，则可以将其移动到此节点。(虚拟机 ID B577EECE-FCFE-4DF6-AE30-3C1D64EA471D)

[确定]

While initiating a live migration of a running virtual machine under Hyper-V R2 the error message below is displayed:

“there was an error checking for virtual machine compatibility on the target node
The virtual machine <name> is not compatible with the target node <nodename>. Hyper-V has detected that: The virtual machine <name> is not compatible with physical computer <servername>. The virtual machine <name> is currently using an Intel processor, but physical computer <servername> has an Unknown processor.

Solution: make sure Intel Virtualization Technology & Execute Disable Bit (or AMD equivalent) are both enabled on the target Hyper-V server.

Unknown processor error when performing Live Migration on Windows Server 2008 R2 Failover Cluster

Symptoms

On failover cluster running on windows 2008 R2 server , when a live migration of...

On failover cluster running on windows 2008 R2 server , when a live migration of a Virtual Machine from one node to the other is performed the following error may be generated:

There was an error checking for virtual machine compatibility on the target node

The virtual machine 'GuestName' is not compatible with the target node 'Win2008R2-2.contoso.com'.
Hyper-V has detected that:
The virtual machine 'GuestName' is not compatible with physical computer 'Win2008R2-2'. (Virtual machine ID D0734038-F3OD-4FEC-A36F-99AD 14665885)

The virtual machine 'GuestName' is currently using an Intel processor, but physical computer 'Win2008R2-2’ has an Unknown processor. A running or saved virtual machine cannot be migrated to a physical computer that has a processor from a different vendor. However, you can move the virtual machine to this node if you shut down the virtual machine. (Virtual machine ID D0734038-F3OD-IFEC-A36F-99AD 14665885)

Cause
Virtulatization service is failing to start on the node or the HyperV Role is no...

Virtulatization service is failing to start on the node or the HyperV Role is not installed correctly

Resolution
Option one: Verify hardware-assisted virtualization setting are enabled in the b...

Option one:

Verify hardware-assisted virtualization setting are enabled in the bios on node we are trying to migrate to:
a) Create a new Virtual machince on Node and store Virtual Machine Configuration file on the local hard drive
(Do Not store in Virtual Machine Configuration file Custer Shared Volume Folder)
b) Start this Virtual machine to verify that we are able to start this virtual machine succesfully

Option two:

Remove the Hyper-V role and reboot the server
Add the Hyper-V role

More Information
It has been observed that when an image with Hyper-V role enabled is deployed, t...

It has been observed that when an image with Hyper-V role enabled is deployed, there are certain Bios updates that may disable the hardware-assisted virtualization.

Windows Server 2008 R2 & Microsoft Hyper-V Server 2008 R2 - Hyper-V Live Migration Overview & Architecture:
http://www.microsoft.com/downloads/details.aspx?FamilyID=fdd083c6-3fc7-470b-8569-7e6a19fb0fdf&displaylang=en

Hyper-V: Using Live Migration with Cluster Shared Volumes in Windows Server 2008 R2:
http://technet.microsoft.com/en-us/library/dd446679(WS.10).aspx

NOTE: If the error you are receiving is similiar to the above but references "The virtual machine "guestName" is using processor-specific features not support on physical computer 'ClusternodeName' then refer to the following document on enabling Processor Compatibility mode:

http://download.microsoft.com/download/F/2/1/F2146213-4AC0-4C50-B69A-12428FF0B077/VM%20processor%20compatibility%20mode.doc

What is a Microsoft Failover Cluster Virtual Adapter

shanyiwan@live.com() — Tue, 25 Oct 2011 10:20:25 GMT

What is a Microsoft Failover Cluster Virtual Adapter anyway?

A question often asked is, "What is the Microsoft Cluster Virtual Adapter and what can I do with it?" The typical, and correct answer, is to leave it alone and let it just work for you. While that answer satisfies most, others may want just a little more by way of an explanation, so hopefully, this blog will provide that.

The networking model in Windows Server 2008 Failover Clustering was rewritten to accommodate new functionality which included being able to obtain IP addresses from DHCP servers and being able to locate Cluster nodes on separate, routed subnets. Additionally, communications went from being UDP Broadcast transmissions to UDP Unicast with a smattering of TCP connections thrown in for good measure. What this all adds up to is more reliable and robust communication connectivity within the Cluster, no matter where the Cluster nodes were located. It no longer matters if Cluster nodes are located in the same physical rack in the same datacenter or in a server rack in a server room in a remote datacenter located at the end of an OC3 WAN connection. This now makes the Cluster more tolerant of single points of failure, e.g. Network Interface Card (NIC) card (and hence the new driver name 'Network Fault-Tolerant or NetFT.sys). The only real minimum requirement is multiple (at least two), redundant communication paths between all nodes in the Cluster. This way, the Cluster network driver (NETFT.SYS) could build a complete routing structure to provide the redundant communication connectivity the Cluster would need to keep applications and services highly available.

Note: Not having at least two networks available for cluster communications will result in a Warning (violation of a 'best practice') being recorded during the Cluster validation process. This is noted in the hardware requirements under Network Adapters and cable section.

To provide some examples of this new functionality and still not get deep into the new networking model, I generated a cluster log from a cluster node so I could illustrate how this new network model is reflected as the cluster service starts. In the cluster log, several entries are associated with NETFT. Some of these include, but may not be limited to, the following:

NETFT - Network Fault-Tolerant
TM - Topology Manager (discovers and maintains the cluster network topology. Reports failures of any networks or network interfaces. configures the Microsoft Failover Cluster Virtual Adapter)
IM - Interface Manager (Responsible for any network interfaces that are part of a cluster configuration)
NETFTAPI - NETFT Application Programming Interface (API)
FTI - Fault-Tolerant Interface

As the cluster service starts, there are events registered indicating NETFT is preparing for communications with other pieces of the cluster architecture -

00000784.000007cc::2009/01/30-14:26:38.199 INFO [NETFT] FTI NetFT event handler ready for events.
00000784.000007b0::2009/01/30-14:26:39.369 INFO [NETFT] Starting NetFT eventing for TM
00000784.000007b0::2009/01/30-14:26:39.369 INFO [NETFT] TM NetFT event handler ready for events.
00000784.000007b0::2009/01/30-14:26:39.369 INFO [CS] Starting IM
00000784.000007b0::2009/01/30-14:26:39.369 INFO [NETFT] Starting NetFT eventing for IM
00000784.000007b0::2009/01/30-14:26:39.369 INFO [NETFT] IM NetFT event handler ready for events.

As connectivity is established with other nodes in the cluster, routes are added -

00000784.00000648::2009/01/30-14:26:39.744 INFO [NETFT] Added route <struct mscs::FaultTolerantRoute>
00000784.00000648::2009/01/30-14:26:39.744 INFO <realLocal>172.16.0.181:~3343~</realLocal>
00000784.00000648::2009/01/30-14:26:39.744 INFO <realRemote>172.16.0.182:~3343~</realRemote>
00000784.00000648::2009/01/30-14:26:39.744 INFO <virtualLocal>fe80::2474:73f1:4b12:8096:~3343~</virtualLocal>
00000784.00000648::2009/01/30-14:26:39.744 INFO <virtualRemote>fe80::8b6:30ea:caa3:8da7:~3343~</virtualRemote>
00000784.00000648::2009/01/30-14:26:39.744 INFO <Delay>1000</Delay>
00000784.00000648::2009/01/30-14:26:39.744 INFO <Threshold>5</Threshold>
00000784.00000648::2009/01/30-14:26:39.744 INFO <Priority>99</Priority>
00000784.00000648::2009/01/30-14:26:39.744 INFO <Attributes>1</Attributes>
00000784.00000648::2009/01/30-14:26:39.744 INFO </struct mscs::FaultTolerantRoute>

Additional events are registered as the routes to the nodes become 'reachable' -

00000784.0000039c::2009/01/30-14:26:39.759 DBG [NETFTAPI] Signaled NetftRemoteReachable event, local address 172.16.0.181:003853 remote address 172.16.0.182:003853
00000784.0000039c::2009/01/30-14:26:39.759 DBG [NETFTAPI] Signaled NetftRemoteReachable event, local address 172.16.0.181:003853 remote address 172.16.0.182:003853
00000784.0000039c::2009/01/30-14:26:39.759 DBG [NETFTAPI] Signaled NetftRemoteReachable event, local address 172.16.0.181:003853 remote address 172.16.0.182:003853
00000784.000004f4::2009/01/30-14:26:39.759 INFO [FTI] Got remote route reachable from netft evm. Setting state to Up for route from 172.16.0.181:~3343~ to 172.16.0.182:~3343~.
00000784.000002f4::2009/01/30-14:26:39.759 INFO [IM] got event: Remote endpoint 172.16.0.182:~3343~ reachable from 172.16.0.181:~3343~
00000784.000002f4::2009/01/30-14:26:39.759 INFO [IM] Marking Route from 172.16.0.181:~3343~ to 172.16.0.182:~3343~ as up
00000784.000001f8::2009/01/30-14:26:39.759 INFO [TM] got event: Remote endpoint 172.16.0.182:~3343~ reachable from 172.16.0.181:~3343~
00000784.00000648::2009/01/30-14:26:39.759 INFO [FTW] NetFT is ready after 0 msecs wait.
00000784.00000648::2009/01/30-14:26:39.759 INFO [FTI] Route is up and NetFT is ready. Connecting to node W2K8-CL2 on virtual IP fe80::8b6:30ea:caa3:8da7%15:~3343~
00000784.0000061c::2009/01/30-14:26:39.759 INFO [CONNECT] fe80::8b6:30ea:caa3:8da7%15:~3343~: Established connection to remote endpoint fe80::8b6:30ea:caa3:8da7%15:~3343~.

A consequence of the changes made to the Cluster networking model is the fact that the Cluster network driver now manifests itself as a network adapter, a hidden adapter, but an adapter nonetheless.

While this is hidden from normal view (by default) in Device Manager (must select “Show hidden devices” to see it), it is plainly visible when listing the network configuration of a Cluster node using the ipconfig /all command line.

Like other adapters, the Microsoft Failover Cluster Virtual Adapter has a MAC address and both IPv4 and IPv6 addresses assigned to it. The IPv4 address is an Automatic Private Internet Protocol Addressing (APIPA) address and the IPv6 address is a non-routable Link-Local address, but that does not matter as all cluster communications are tunneled through the networks supported by the physical NICs as shown here using the route information obtained during the cluster service startup.

The MAC address that is assigned to the Microsoft Failover Cluster Virtual Adapter is based on the MAC address of one of the physical NICs

The Cluster network driver (netft.sys) is a kernel mode driver and is started and stopped by the Cluster Service.

The Cluster network driver has an entry under HKLM\System\CurrentControlSet\Services.

Additionally, there is an entry for the Microsoft Failover Cluster Virtual Adapter in the routing table for each Cluster node. Here are sample outputs for the three sections of the route print command executed on a Cluster node. The first part shows the listing of all the interfaces on the node. Interface 15 is the Microsoft Failover Cluster Virtual Adapter.

This next screen shows the IPv4 Route Table which reflects three entries for the Microsoft Failover Cluster Virtual Adapter.

And finally, the adapter appears in the IPv6 Route Table (If 15).

So, how can one get in trouble? Here are a couple of ways:

Disable the Microsoft Failover Cluster Virtual Adapter.
Sysprep an installation of Windows Server 2008 with the Failover Cluster feature installed. This will cause an error in the Cluster Validation Process.
Modifying any properties of the adapter.

Hopefully, this gives you a better feel for this new functionality in Windows Server 2008 Failover Clusters, and like I stated at the beginning of the blog, the correct answer is to not do anything to the adapter - just let it work for you. Thanks and we hope this has been helpful.

Hyper-V Live Migration: A Step-by-Step Guide

shanyiwan@live.com() — Tue, 25 Oct 2011 09:46:48 GMT

Live migration is probably the most important technology that Microsoft has added to Hyper-V in Windows Server 2008 R2. It enables virtual machines (VMs) to be moved between Hyper-V hosts with no downtime. Using live migration, you can migrate all VMs off the Hyper-V host that needs maintenance, then migrate them back when the maintenance is done. In addition, live migration enables you to respond to high resource utilization periods by moving VMs to hosts with greater capacities, thereby enabling the VM to provide end users with high levels of performance even during busy periods.

Live migrations can be manually initiated, or if you have System Center Virtual Machine Manager 2008 R2 and System Center Operations Manager 2007, you can run automated live migrations in response to workload. You need to complete quite a few steps to set up two systems for live migration, and I’ll guide you through the process. First, I’ll explain how live migration works. Then I’ll cover some of the hardware and software prerequisites that must be in place. Finally, I’ll walk you through the important points of the Hyper-V and Failover Clustering configuration that must be performed to enable live migration.

How Live Migration Works

Live migration takes place between two Hyper-V hosts. Essentially, the VM memory is copied between Hyper-V hosts. After the memory is copied, the VM on the new host can access its virtual hard disk (VHD) files and continue to run. Both hosts access shared storage where the VM’s VHD files are stored. When you initiate a live migration, which Figure 1 shows,

the following steps occur:

1. A new VM configuration file is created on the target server.

2. The source VM’s initial memory state is copied to the target.

3. Changed memory pages on the source VM are tagged and copied to the target.

4. This process continues until the number of changed pages is small.

5. The VM is paused on the source node.

6. The final memory state is copied from the source VM to the target.

7. The VM is resumed on the target.

8. An Address Resolution Protocol (ARP) is issued to update the network routing tables.

Requirements for Live Migration

On the hardware side, you need two x64 systems with compatible processors. It’s best if the host processors are identical, though it’s not required. However, they do need to be from the same processor manufacturer and family—you can’t perform a live migration when one host has an AMD processor and the other host has an Intel processor. Learn more about Hyper-V processor compatibility in the Microsoft white paper “Virtual Machine Processor Compatibility Mode.”

In addition, each of the servers should be equipped with at least three NIC cards, running at 1GHz: one for external network connections, one for iSCSI storage connectivity, and one for node management. Ideally, you’d have another NIC dedicated to the live migration, but the live migration can also occur over the external network connection—it will just be a little slower. It’s important to note that if you’re implementing a server consolidation environment, you will want additional NICs for the network traffic of the VMs.

On the software side, all the nodes that take part in live migration must have Server 2008 R2 x64 installed. This can be the Standard, Enterprise, or Datacenter editions. Live migration is also supported by the Hyper-V Server 2008 R2 product. In addition, the Hyper-V role and the Failover Cluster feature must be installed on all servers participating in live migration.

You also need shared storage, which can be either an iSCSI SAN or a Fibre Channel SAN. In this example, I used an iSCSI SAN. Be aware that the iSCSI SAN must support the iSCSI-3 specifications, which includes the ability to create persistent reservations, something that live migration requires. Some open-source iSCSI targets such as OpenFiler don’t have that support at this time. If you’re looking to try this for a local test and don’t want to buy an expensive SAN, you might want to check out the free StarWind Server product at www.starwind.com.

Failover Cluster Networking Configuration

Failover clustering is a requirement for live migration. You can live-migrate VMs only between the nodes in the failover cluster. The first step in creating a failover cluster is to configure the networking and storage. You can see an overview of the network configuration used to connect the Windows servers in the cluster to the external network and to the shared storage in Figure 2.

In Figure 2 the servers are using the network with the subnet of 192.168.100.xxx for client connections. The iSCSI SAN is running on an entirely separate physical network which was configured using the 192.168.0.xxx IP addresses. You can use different values for either of these IP address ranges. I selected these values to more clearly differentiate between the two networks. Ideally, you would also have additional NICs for management and an optional live migration connection, but these aren’t strictly required. Live migration can work with a minimum of two NICs in each server.

Storage Configuration

I used a LeftHand Networks ISCSI SAN for Hyper-V live migration as well as a test SQL Server implementation. On the iSCSI SAN I created four LUNs. One LUN was sized at 500MB to be used for the cluster quorum. Another was sized at 1024GB to be used for 10 VMs. Two other LUNS were for the test SQL Server implementation and consisted of a 200MB LUN for the Distributed Transaction Coordinator and a 500GB LUN for SQL Server data files.

After creating the LUNs, I configured the iSCSI Initiator on both the Windows Server nodes. To add the iSCSI targets, I selected the Administrative Tools, iSCSI Initiator option, then on the Discovery tab I chose the Discover Portal option. This displayed the Discover Portal dialog box where I entered the IP address and iSCSI port of the SAN. In my case, this was 192.168.0.1 and 3260, respectively.

Next, in the Connect to Target dialog box, I supplied the target name of the iSCSI SAN. This name came from the properties of the SAN and varies depending on the SAN vendor, the domain name, and the names of the LUNs that are created. I checked the option Add this connection to the list of Favorite Targets. After completing the ISCSI configuration, the iSCSI Initiator Targets tab was populated with the LUNs.

Finally, using Disk Administrator I assigned drive letters to the LUNs. I opened Disk Management and used Q for the quorum, R for DTC, S for SQL Server, and V for the VMs. You need to make the assignments on one node, then bring the disks offline and make identical assignments in the second node. Figure 3 shows the completed Disk Management disk assignments for one of the nodes.

Adding the Hyper-V Role and Failover Clustering Feature

The next step is to add the Hyper-V role, then the Failover Clustering feature. You add both by using Server Manager. To add the Hyper-V role, select Administrative Tools, Server Manager, then click the Add Role link. From the Select Server Roles dialog box, select Hyper-V, then click Next. You’ll be prompted with Create Virtual Networks. This essentially creates a bridge between the Hyper-V VMs and your external network.

Select the NICs that you want to use for your VM traffic. Be careful not to choose the NICs that are used for the iSCSI SAN connection. Click Next to complete the Add Role Wizard. The system then reboots. You need to perform this process for all of the nodes in the cluster.

Next, add the Failover Cluster Feature by using the Administrative Tools, Server Manager, Add Feature option. This starts the Add Features wizard. Scroll through the list of features and select the Failover Clustering feature. Click Next to complete the wizard. This process must be completed on all nodes.

Configuring Failover Clustering

Next, create a Failover Cluster. You can do this on any of the cluster nodes. Select the Administrative Tools, Failover Cluster Manager option to start the Failover Cluster Management console. Then select theValidate a Configuration link to start the wizard, which displays the Select Servers or Cluster dialog box.

Enter the fully quailed names of all the nodes that will belong to the cluster, then click Next. Click Next through the subsequent wizard screens to run the cluster validation tests, which check the OS level, network configuration, and storage of all cluster nodes. A summary of the test results is displayed. If the validation tests succeed, you can continue and create the cluster. If there are errors or warnings, you can display them in the report, correct them, and rerun the validation tests.

After the validation tests have run, you create the cluster using theCreate a Cluster link from the Failover Cluster Management console. Like the validate option, the Create a Cluster option first starts by displaying a Select Servers dialog box where you enter the names of all cluster nodes. Clicking Next displays the Access Point for Administering the Cluster dialog box, which you can see in Figure 4.

You use the dialog box labeled Access Point for Administering the Cluster to assign the cluster a name and an IP address. The name and IP address both must be unique in the network. In Figure 4, you can see that I named the cluster WS08R2-CL01 and gave the cluster an IP address of 192.168.100.200. With Windows Server 2008 R2 you can choose to have the IP address assigned by DHCP, but I prefer to use manually assigned IP address for my server systems because it allows all of my servers to always have the same IP addresses, which is handy for troubleshooting problems.

Clicking Next displays the Confirmation screen where you review your cluster creation selections. You can page back and make changes. Click Next again to create the cluster. A summary screen then displays the configuration of the new cluster. This action configures the cluster on all of the selected clustered nodes.

The Create Cluster wizard automatically selects the storage for your quorum, but it doesn’t always choose the quorum drive that you want. You can check and change the quorum configuration by right-clicking the name of the cluster in the Failover Cluster Management console, then selecting More Actions, Configure Cluster Quorum Settings from the context menu. This displays the Select Quorum Configuration dialog box. A wizard automatically chooses the best quorum type, depending mainly on the number of nodes in the cluster. In my two-node cluster, it selected the Node and Disk Majority quorum type.

Next, the Configure Storage Witness dialog box is displayed. Here I changed the original value to the Q drive that I wanted to use as the quorum by selecting a check box. Clicking Next saves the cluster quorum changes. If you would like to know more about configuring Windows Server 2008 R2 failover clustering, the Windows IT Pro website has articles and FAQs that can help; you can start with “4 Failover Clustering Hassles and How to Avoid Them,” InstantDoc ID 103534.

Enabling Cluster Shared Volumes

The next step in cluster configuration is to enable Cluster Shared Volumes. The Cluster Shared Volumes feature lets multiple cluster nodes simultaneously access the shared storage locations, but it’s not enabled by default. To do so, use the Failover Cluster Management console and right-click the name of the cluster at the top of the navigation pane, then select Enable Cluster Shared Volumes from the context menu. This displays the summary pane for Cluster Shared Volumes, which initially is blank.

To select a shared storage location to be used by Cluster Shared Volumes, click the Add Storage option in the Action pane. This displays the Add Storage dialog box. The storage for Cluster Shared Volumes has to be visible to the cluster and it can’t be used for other purposes. Select the box next to the storage location you want to use. I selected the V drive, which is actually a LUN on the LeftHand Networks SAN. Click OK to enable Cluster Shared Volumes for that drive. This also results in the creation of a mount point on all the cluster nodes. By default, the mount point is labeled C:\ClusterStorage\Volume1.

Creating VMs on Cluster Shared Volumes

At this point, failover clustering is configured on all the nodes in the cluster and the Cluster Shared Volumes feature has been enabled, allowing all of the nodes to simultaneously access the storage. The next step is to create VMs that can take advantage of this infrastructure. Hyper-V VMs can be created using either the Hyper-V Manager or System Center Virtual Machine Manager. To create a new VM using Hyper-V Manager, click the Administrative Tools, Hyper-V Manager option at the Start menu, then select New from the Action pane to start the New Virtual Machine wizard. Figure 5 shows the dialog box you will see, labeled Specify Name and Location.

In Figure 5, you can see that the VM will be named vWS08-SQL01. Also note that the value for the VM location has been set to the Cluster Shared Volumes mount point: C:\ClusterStorage\Volume1. This causes the VM configuration files to be created on the shared storage.

Click Next to assign RAM to the VM. Click Next again to select the network connection for the VM. Assigning a network for the VM is optional. However, if you do select an external network, be sure that the external network connection is named the same on all of your Hyper-V nodes. In my case, I used the external network name of External Virtual Network on all of my Hyper-V cluster nodes.

Click Next to display the Connect Virtual Hard Disk dialog box. Here, again, it’s important to create the VHD files on the Cluster Shared Volumes storage. Initially, the dialog displays the Hyper-V Manager default values for name and location. I used the value of vWS08-SQL01.vhd for the VHD file and changed the location to C:\ClusterStorage\Volume1. Click Next to specify the guest OS installation options. All guest OSs including Linux can take advantage of live migration. The rest of the process for creating a VM is exactly like creating a normal VM.

When you complete the New Virtual Machine wizard, the VM will be created on the Cluster Shared Volumes storage. The next step is to start the VM and install the guest OS and the application that you want to run on the VM.

Enabling VMs for Live Migration

Open the Failover Cluster Management console, then navigate to the Services and Applications node under the cluster name and right-click to display the context menu. Select Configure Service or Application to start the High Availability wizard. On the Select Service or Application dialog box, select Virtual Machine from the list of services displayed, then click Next. This displays the Select Virtual Machine dialog box.

Scroll though the list of VMs until you find the one you want to enable for live migration. I selected the VM vWS08-SQL01 created earlier. The VM can’t be running while you perform this operation—it must be in the Off or Saved state.

Select the check box in front of the VM name, then click Next until you complete the wizard. A confirmation screen is displayed and the summary dialog box reports the status. If you see “Success” in the description, then the VM has been successfully enabled for live migration. If not, you need to review the VM properties and make sure all of the VM assets can be accessed on all of the nodes in the cluster.

Ready, Set, Migrate!

That’s all there is to configuring the Hyper-V live migration environment. At this point, you can initiate a live migration using the Failover Cluster Manager. To start a live migration, expand the Services and Applicationsnode, then select the VM node displayed beneath it. This displays the summary pane, which shows the VMs that have been enabled for clustering, along with their current status, which Figure 6 shows.

In Figure 6, you can see that VM vWS08-SQL01 is currently running and that the current owner is node WS08R2-S1. To initiate a live migration, go to the Action pane and select the Live migrate virtual machine option shown in the upper third portion of the Action pane. A menu flyout prompts you for the name of the target node. In this example, the menu flyout shows 1 – Live migrate to node WS08R2-S2. Clicking this option starts the live migration. The summary window is updated with the status of the running live migration.

The running status is displayed until the live migration finishes. The length of time it takes to complete depends on the size and activity of the VM, as well as the speed and activity of the network connection between the Hyper-V host systems. Typically, my network live migrations take between about 10 seconds and a minute. When the live migration has been completed, the summary pane is redisplayed and the Current Owner value is updated with the name of the target node.

The Virtual Promised Land

Live migration addresses the issues of planned host downtime and lays the foundation for the dynamic datacenter. Although there are quite a few steps in the process, if you carefully navigate the critical points in the process, you will reach the promised land of Hyper-V live migration.

Windows Server 2008 Failover Clusters: Networking

shanyiwan@live.com() — Wed, 19 Oct 2011 09:13:11 GMT

The Windows Server 2008 Failover Clustering feature provides high availability for services and applications. To ensure applications and services remain highly available, it is imperative the cluster service running on each node in the cluster function at the highest level possible. Providing redundant and reliable communications connectivity among all the nodes in a cluster plays a large role in ensuring for the smooth functioning of the cluster. Configuring proper communications connectivity within a failover cluster not only provides access to highly available services required by clients but also guarantees the connectivity the cluster requires for its own internal communications needs. The sections that follow discuss Windows Server 2008 Failover Clustering networking features, functionality and recommended processes for the proper configuration and implementation of network connectivity within a cluster.

The following sections provide the information needed to understand failover cluster networking and to properly implement it.

Windows Server 2008 Failover Cluster networking features

Windows Server 2008 Failover Clustering introduces new networking capabilities that are a major shift away from the way things have been done in legacy clusters (Windows 2000\2003 and NT 4.0). Some of these take advantage of the new networking features that are included as part of the operating system and others are a result of feedback that has been received from customers. The new features include:

A new cluster network driver architecture
The ability to locate cluster nodes on different, routed networks in support of multi-site clusters
Support for DHCP assigned IP addresses
Improvements to the cluster health monitoring (heartbeat) mechanism
Support for IPv6

New cluster network driver architecture

The legacy cluster network driver (clusnet.sys) has been replaced with a new NDIS level driver called the Microsoft Failover Cluster Virtual Adapter (netft.sys). Whereas the legacy cluster network driver was listed as a Non-Plug and Play Driver, the new fault tolerant adapter actually appears as a network adapter when hidden devices are displayed in the Device Manager snap-in (Figure 1).

Figure 1: Device Manger Snap-in

The driver information is shown in Figure 2.

Figure 2: Microsoft Failover Cluster Virtual Adapter driver

The cluster adapter is also listed in the output of an ipconfig /all command on each node (Figure 3).

Figure 3: Microsoft Failover Cluster Virtual Adapter configuration information

The Failover Cluster Virtual Adapter is assigned a Media Access Control (MAC) address that is based on the MAC address of the first enumerated (by NDIS) physical NIC in the cluster node (Figure 4) and uses an APIPA (Automatic Private Internet Protocol Addressing) address.

Figure 4: Microsoft Failover Cluster Virtual Adapter MAC address

The goal of the new driver model is to sustain TCP/IP connectivity between two or more systems despite the failure of any component in the network path. This goal can be achieved provided at least one alternate physical path is available. In other words, a network component failure (NIC, router, switch, hub, etc…) should not cause inter-node cluster communications to break down, and communication should continue making progress in a timely manner (i.e. it may have a slower response but it will still exist) as long as an alternate physical route (link) is still available. If cluster communications cannot proceed on one network, the switchover to another cluster-enabled network is automatic. This is one of the primary reasons that each cluster node must have multiple network adapters available to support cluster communications and each one should be connected to different switches.

The failover cluster virtual adapter is implemented as an NDIS miniport adapter that pairs an internally constructed virtual route with each network found in a cluster node. The physical network adapters are exposed at the IP layer on each node. The NETFT driver transfers packets (cluster communications) on the virtual adapter by tunneling through the best available route in its internal routing table (Figure 5).

Figure 5: NetFT traffic flow diagram

Here is an example to illustrate this concept. A 2-Node cluster is connected to three networks that each node has in common (Public, Cluster and iSCSI). The output of an ipconfig /all command from one of the nodes is shown in Figure 6.

Figure 6: Example Cluster Node IP configuration

Note: Do not be concerned with the name ‘Microsoft Virtual Machine Bus Network Adapter’ as these examples were derived from cluster nodes running as Guests in Hyper-V.

The Microsoft Failover Cluster Virtual Adapter configuration information for each node is shown in Figure 7. Keep in mind; the default port for cluster communication is still TCP\UDP: 3343.

Figure 7: Node Failover Cluster Virtual Adapter configuration information

When the cluster service starts, and a node either Forms or Joins a cluster, NETFT, along with other components, is responsible for determining the node’s network configuration and connectivity with other nodes in the cluster. One of the first actions is establishing connectivity with the Microsoft Failover Cluster Virtual Adapter on all nodes in the cluster. Figure 8 shows an example of this in the cluster log.

Figure 8: Microsoft Failover Cluster Virtual Adapter information exchange

Note: You can see in Figure 8 that the endpoint pairs consist of both IPv4 and IPv6 addresses. The NETFT adapter prefers to use IPv6 and therefore will choose the IPv6 addresses for each end point to use.

As the cluster service startup continues, and the node either Forms or Joins a cluster, routing information is added to NETFT. Using the three networks mentioned previously, Figure 9 shows each route being added to a cluster.

Route between 1.0.0.31 and 1.0.0.32

Route between 192.168.0.31 and 192.168.0.32

Route between 172.16.0.31 and 172.16.0.32

Figure 9: Routes discovered and added to NETFT

Each ‘real’ route is added to the ‘virtual’ routes associated with the virtual adapter (NETFT). Again, note the preference for NETFT to use IPv6 as the protocol of choice.

The capability to place cluster nodes on different, routed networks in support of Multi-Site Clusters

Beginning with Windows Server 2008 failover clustering, individual cluster nodes can be located on separate, routed networks. This requires that resources that depend on IP Address resources (i.e. Network Name resources), implement an OR logic since it is unlikely that every cluster node will have a direct local connection to every network the cluster is aware of. This facilitates IP Address and hence Network Name resources coming online when services\applications failover to remote nodes. Here is an example (Figure 10) of the dependencies for the cluster name on a machine connected to two different networks.

Figure 10: Cluster Network Name resource with an OR dependency

All IP addresses associated with a Network Name resource, which come online, will be dynamically registered in DNS (if configured for dynamic updates). This is the default behavior. If the preferred behavior is to register all IP addresses that a Network Name depends on, then a private property of the Network Name resource must be modified. This private property is called RegisterAllProvidersIP (Figure 11). If this property is set equal to 1, all IP addresses will be registered in DNS and the DNS server will return the list of IP addresses associated with the A-Record to the client.

Figure 11: Parameters for a Network Name resource

Since cluster nodes can be located on different, routed networks, and the communication mechanisms have been changed to use reliable session protocols implemented over UDP (unicast), the networking requirements for Geographically Dispersed (Multi-Site) Clusters have changed. In previous versions of Microsoft clustering, all cluster nodes had to be located on the same network. This required ‘stretched’ VLANs be implemented when configuring multi-site clusters. Beginning with Windows Server 2008, this requirement is no longer necessary in all scenarios.

Support for DHCP assigned IP addresses

Beginning with Windows Server 2008 Failover Clustering, cluster IP address resources can obtain their addressing from DHCP servers as well as via static entries. If the cluster nodes themselves have at least one NIC that is configured to obtain an IP addresses from a DHCP server, then the default behavior will be to obtain an IP address automatically for all cluster IP address resources. The new ‘wizard-based’ processes in Failover Clustering understand the network configuration and will only ask for static addressing information when required. If the cluster node has statically assigned IP addresses, the cluster IP address resources will have to be configured with static IP addresses as well. Cluster IP address resource IP assignment follows the configuration of the physical node and each specific interface on the node. Even if the nodes are configured to obtain their IP addresses from a DHCP server, individual IP address resources can be changed to static addresses (Figure 12).

Figure 12: Changing DHCP assigned to Static IP address

Improvements to the cluster ‘heartbeat’ mechanism

The cluster ‘heartbeat’, or health checking mechanism, has changed in Windows Server 2008. While still using port 3343, it is no longer a broadcast communication. It is now unicast in nature and uses a Request-Reply type process. This provides for higher security and more reliable packet accountability. Using the Microsoft Network Monitor protocol analyzer to capture communications between nodes in a cluster, the ‘heartbeat’ mechanism can be seen (Figure 13).

Figure 13: Network Monitor capture

A typical frame is shown in Figure 14.

Figure 14: Heartbeat frame from a Network Monitor capture

There are properties of the cluster that address the heartbeat mechanism; these include SameSubnetDelay, CrossSubnetDelay, SameSubnetThreshold, and CrossSubnetThreshold (Figure 16).

Figure 16: Properties affecting the cluster heartbeat mechanism

The default configuration (shown here) means the cluster service will wait 5.0 seconds before considering a cluster node to be unreachable and have to regroup to update the view of the cluster (One heartbeat sent every second for five seconds). The limits on these settings are shown in Figure 17. Make changes to the appropriate settings depending on the scenario. The CrossSubnetDelay and CrossSubnetThreshold settings are typically used in multi-site scenarios where WAN links may exhibit higher than normal latency.

Figure 17: Heartbeat Configuration Settings

These settings allow for the heartbeat mechanism to be more ‘tolerant’ of networking delays. Modifying these settings, while a worthwhile test as part of a troubleshooting procedure (discussed later), should not be used as a substitute for identifying and correcting network connection delays.

Support for IPv6

Since the Windows Server 2008 OS will be supporting IPv6, the cluster service needs to support this functionality as well. This includes being able to support IPv6 IP Address resources and IPv4 IP Address resources either alone or in combination in a cluster. Clustering also supports IPv6 Tunnel Addresses. As previously noted, intra-node cluster communications by default use IPv6. For more information on IPv6, please review the following:

Microsoft Internet Protocol Version 6

Implementing networks in support of Failover Clusters

The main consideration when designing Failover Cluster networks is to ensure there is built-in redundancy for cluster communications.This is typically accomplished by having a minimum of two physical Network Interface Cards (NICs) installed in each node that will be part of the cluster.These cards must be supported by two separate and distinct buses (e.g.Two PCI NICs).Many people think a single multi-port NIC card meets this requirement – it does not as this configuration creates a single point of failure for all cluster communications.The best configuration would be two multi-port NICs running on separate buses and having fault tolerance implemented by way of NIC Teaming software (provided by 3^rd Party vendors.) and being physically connected to separate network switches.

Note:NIC Teaming is not supported on iSCSI connections.Please review the iSCSI Cluster Support: Frequently Asked Questions.The appropriate fault-tolerant mechanism for iSCSI connectivity would be multi-path software. Please review the Microsoft Multi-path I/O: Frequently Asked Questions.

There are two primary design scenarios when planning for Failover Cluster network connectivity.In the first scenario (and the most common), all nodes in the cluster are located on the same networks.In the second scenario, nodes in the cluster are located on separate and distinct routed networks (this is very common in multi-site cluster implementations).Figure 18 shows an example of the second scenario.

Figure 18:Multi-site cluster (network connectivity only)

Note:Even though it is supported to locate cluster nodes on separate, routed networks, it is still supported to connect nodes in a multi-site cluster using stretched Virtual Local Area Networks (VLAN).This configuration places the nodes on the same network(s).

It is important in any cluster that there are no NICs on the same node that are configured to be on the same subnet.This is because the cluster network driver uses the subnet to identify networks and will use the first one detected and ignore any other NICs configured on the same subnet on the same node.The cluster validation process will register a Warning if any network interfaces in a cluster node are configured to be on the same network.The only possible exception to this would be for iSCSI (Internet Small Computer System Interface) connections.If iSCSI is implemented in a cluster, and MPIO (Multi-Path Input/Output) is being used for fault-tolerant connections to iSCSI Storage, then it is possible that the network interfaces could be on the same network. In this configuration, the iSCSI network in the Failover Cluster Manager should be configured such that cluster would not use it for any cluster communications.

Note:Please consult the iSCSI Cluster support: Frequently Asked Question.

As previously mentioned, Windows Server 2008 accommodates cluster nodes being located on separate, routed networks by including a new logic, called an OR logic, when it comes to IP Address resources.Figure 19 illustrates this.

Figure 19:IP Address Resource OR logic

When a Network Name resource is configured with an OR dependency on more than one IP Address resource, this means at least one of the IP Address resources must be able to come Online before the Network Name resource can come Online.Since a Network Name resource can be associated with more than one IP Address, there is a property of a Network Name resource that can be modified so DNS registrations will occur for all of the IP Addresses.The property is called RegisterAllProvidersIP (See Figure 20).

Figure 20:Network Name resource properties

Note:In Figure 20 above, Failover Cluster PowerShell cmdlets were used to access cluster configuration information.This is new in Windows Server 2008 R2.For more information, review the TechNet Cmdlet Reference.

The default registration behavior is to register only the IP Address that can come Online on the node.Implementing this other behavior by modifying the setting to (1) can assist name resolution in a multi-site cluster scenario.

Note:Please review KB 947048 for other things to consider when deploying failover cluster nodes on different, routed subnets (multi-site cluster scenario).

While Failover Clusters require a minimum of two NICs to provide reliable cluster communications, there are scenarios where more NICs may be desired and\or required based on the services or applications that are running in the cluster.One such scenario we already mentioned – iSCSI connectivity to storage.The other scenario involves Microsoft’s virtualization technology – Hyper-V.

The integration of Failover Clustering with Hyper-V was introduced in Windows Server 2008 (RTM) in the form of making Virtual Machines highly available in a cluster by being able to move (Failover) the Virtual Machines between the nodes in the cluster using a process called Quick Migration.In Windows Server 2008 R2, additional capabilities were introduced including Live Migration and Cluster Shared Volumes (CSV).These features improved the high availability story for Virtual machines, but also introduced new networking requirements.The inner workings of Hyper-V networking will not be discussed here.For more information, please download this whitepaper (http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=3fac6d40-d6b5-4658-bc54-62b925ed7eea).

The networking requirements in a Hyper-V Cluster supporting Live Migration and using Cluster Shared Volumes (CSV) can add up quickly as illustrated in Figure 21.

Figure 21: Hypothetical Networking Requirements

For more information on Live Migration and Cluster Shared Volumes in Windows Server 2008 R2, visit the Microsoft TechNet site.

Using Cluster Shared Volumes in a Failover Cluster in Windows Server 2008 R2

Hyper-V:Using Live Migration with Cluster Shared Volumes in Windows Server 2008 R2

Troubleshooting cluster networking issues

As previously stated, it is important that redundant and reliable cluster communications connectivity exist between all nodes in a cluster.However, there may be times when communications connectivity within a cluster gets disrupted either because of actual network failures or because of misconfiguration of network connectivity.A loss of communications connectivity with a node in a cluster can result in the node being removed from cluster membership.When a node is removed from cluster membership, it will terminate its cluster service to avoid problems or conflicts as other nodes in the cluster take over the services or applications and resources that were hosted on the node that was removed.The node will attempt to rejoin the cluster when the cluster service restarts.This problem can also have broader effects because the loss of a node in a cluster affects ‘quorum’.Should the number of nodes participating in a cluster fall below a majority; all highly available services will be taken Offline until ‘quorum’ is re-established (The quorum model,No Majority: Disk Only, is the one exception.However, this model is not recommended).

Here are some recommended troubleshooting procedures for cluster connectivity issues:

1.Examine the system log on each cluster node and identify any errors reportinga loss of communications connectivity in the cluster or even broader network related issues.Here are some example cluster related error messages you may encounter:

Figure 22:Cluster Network Connectivity error messages

Source:http://technet.microsoft.com/en-us/library/cc773562(WS.10).aspx

Figure 23:Network Connectivity and Configuration error messages

Source:http://technet.microsoft.com/en-us/library/cc773417(WS.10).aspx

2.If the system logs provide insufficient detail, generate the cluster logs and inspect the contents for more detailed information concerning the loss of network connectivity.

Note: Generate the cluster logs by running this PowerShell cmdlet –

3.Verify the configuration of all networks in the cluster.

4.Verify the configuration of network connectivity devices such as Ethernet switches.

5.Run an abbreviated cluster validation process by selecting only the Network tests.

The tests that are executed are shown here:

The desired end result is this:

As an example, here is the section in the validation report that shows the results for the List Network Binding Order test –

Some of the common issues seen with respect to the network validation tests include, but may not be limited to:

·Multiple NICs on a cluster node configured to be on the same subnet.

·Excessive latency (usually > 2 seconds) in ping tests between interfaces on cluster nodes.

·Warning that the firewall has been disabled on one or more nodes.

6.Conduct simple networking tests, such as a ‘ping’ test, across all networks enabled for cluster communications to verify connectivity between the nodes.Use network monitoring tools such as Microsoft’s Network Monitorto analyze network traffic between the nodes in the cluster (Refer to Figures 13 and 14).

7.Evaluate hardware failures related to networking devices such as Network Interface Cards (NICs), network cabling, or network connectivity devices such as switches and routers as needed.

8.Review the change management log (if one exists in your organization) to determine what, if any, changes were made to the nodes in the cluster that may be related to the disruption in communications connectivity.

9.Consider opening a support incident with Microsoft because if a node is removed from cluster membership, this means there were no networks configured on that node that could be used to communicate with other nodes in the cluster.If there are multiple networks configured for cluster use, as recommended, then cluster membership loss indicates a problem that affects all the networks or the system’s ability to send or receive heartbeat messages.

Note:For additional information on Troubleshooting Windows Server 2008 consult TechNet.

Hopefully, the information provided in this three part blog was helpful and will assist in properly configuring network connectivity in Windows Server 2008 Failover Clusters.

The Windows Server 2008 Failover Clustering: Networkingthree-part blog series has been out for a little while now.Hopefully, it has been helpful.Little did I know there would be an opportunity to write another part.This segment will be short as it covers a very specific scenario.One that we rarely see, but we have encountered it enough that I felt it might be worth writing about it.

There are applications written to access resources that are being hosted in Microsoft clusters running on Windows Server 2008 (RTM + R2).The resource could be a File Server, could be a SQL database, or whatever.The point is that the required resource is being hosted in a Failover Cluster.It is hoped that applications that need to function in this manner are written properly to locate the required resource being hosted in a cluster.By that I mean I would expect an application to be written in a manner where it would first query a name server (DNS server) and then use the information obtained to make a proper connection to the required cluster resource.In a Failover Cluster, that connection point is known as a Client Access Point (CAP).A CAP consists of a Network Name (NetBIOS) resource and one or more IP Address resources.The default behavior in a Windows Server 2008 cluster is to dynamically register CAP information in a DNS server provided it is configured to support Dynamic Updates.This occurs when the CAP is brought Online in the cluster. There are applications that are not written in this manner.There are some application that are written in such a way that they will make a local connection on a cluster node by binding to the first network adapter and then use the IP address configured for that adapter.The end result is in a cluster, the first connection listed in the binding order by default is the Microsoft Failover Cluster Virtual Adapter. This adapter uses an IP address that is drawn from the APIPA (Automatic Private IP Addressing) address range which is non-routable and not registered in DNS.

To assist with helping make these types of applications work better, we can use a utility that has been released for public download on the Microsoft MSDN site.The utility is called ‘nvspbind.’So, the first step is to download and install the utility on each cluster node. The options we will be using are shown in Figure 1.

Figure 1:Options for nvspbind

First we need to identify the adapter that is the Microsoft Failover Cluster Virtual Adapter by using the nvspbind /n command (Figure 2).The adapter is ‘Local area connection* 9’.

Figure 2:Identify the Microsoft Failover Cluster Virtual Adapter

Next, we use the 'nvspbind /o ms_tcpip’to determine the binding order for IPv4 (Figure 3).

Figure 3: Listing the bindings for IPv4

We can see here, that the adapter is listed at the top of the binding order for IPv4 which is causing the problem for some applications.We need to move the adapter down in the binding order so we will use the following command to accomplish that –

C:\nvspbind /- “local area connection* 9” ms_tcpip (Figure 4).

Figure 4: Moving the adapter down in the binding order for IPv4

Note:The adapter can be moved further down by using /-- if desire.

Once the adapter has been positioned correctly in the binding order, the application can be tested to see if it now works as desired.

To further highlight the effect of this utility, we can inspect the registry.First, we need to locate some information for the Microsoft Failover Cluster Virtual Adapter.Navigating to the following registry key (Figure 5), and locate the adapter –

HKEY_LOCAL_MACHINE\SYSTEM\CurrenControlSet\Class\{4D36E972-11CE-BFC1-08002BE10318}

Figure 5:Microsoft Failover Cluster Virtual Adapter NetCfgInstanceId

The same information shown in Figure 5 is also displayed in Figure 2.

With the information in hand, navigate to the following registry key (Figure 6) to verify the adapter is no longer listed at the top of the binding order.

Figure 6: HKLM\SYSTEM\CurrentControlSet\services\Tcpip\Linkage

Microsoft Hyper-V Server 2008 R2 with Failover Cluster

shanyiwan@live.com() — Thu, 13 Oct 2011 02:34:35 GMT

Microsoft Windows Hyper-V Server 2008 R2 is a free version much like Windows Server 2008 R2 Core, except it comes with the Hyper-V role preconfigured.

The only thing you will have to take in consideration before installing such a server is that Hyper-V Server 2008 R2 only comes with a PowerShell/CMD-prompt. So to configure such a server you should learn some Windows commands and PowerShell cmdlets.

First thing you would do it simply installing the server from the DVD.
Configure a domain membership, configure network settings, configure hostname, enable remote desktop etc.
These basic settings are all configured from the CMD-window that autostarts with the server.

If you’re running Windows 7 there is a Remote Server Administration Toolkit which is highly recommended to install to remotely set things up on the Hyper-V server. This can be downloaded from here:

http://www.microsoft.com/downloads/details.aspx?FamilyID=7D2F6AD7-656B-4313-A005-4E344E43997D&displaylang=en

Next thing to do is to install the failover cluster role on the server.

Start /w ocsetup FailoverCluster-Core
cmd.exe /c ocsetup FailoverCluster-Core-WOW64

With this cmdlet you will activate remote control of the Disk Management.

netsh advfirewall firewall set rule group=”Remote Volume Management” new enable=yes

More can be found here:

http://blogs.technet.com/askds/archive/2008/06/05/how-to-enable-remote-administration-of-server-core-via-mmc-using-netsh.aspx

Note, if you get any errors when connecting with Disk Management via MMC or if the service on the server stops when connecting, then thats probably because you haven’t enabled outgoing “Remote Volume Management” from the client you are connected from. Enable outgoing “Remote Volume Management” from Windows Firewall with Advanced Security and restart the service on the server:

net start vds

Now that we have configured some basic setting, lets configure our SAN/iSCSI. Depending on what hardware you’re using, configurations are different. But this is a pretty easy straight-forward step.

I read a really good article on how to set my servers up, unfortunately it wasn’t up-to-date, I reckon there were some minor differences. However, you might wanna read it thru:
http://www.servercare.nl/Lists/Posts/Post.aspx?ID=61

When you’ve configured your iscsi targets and authentication etc.

Lets go remotely (RDP) onto the server and from cmd type:

iscsicpl

Go the tab named Discovery and enter the IP of the NAS/SAN.

Click Discover portal..

Go back to the Targets tab and you will see your discovered device.

Press Connect

Now press the Advanced.. button

Enter the CHAP-authentication information that you configured your device with.

Run MMC and add the Snap-In: Group Policy Object Editor and connect it to the remote server and configure the following:

Computer Management > Administrative Templates > System > Device Installation:
ENABLE “Allow remote access to the Plug and Play interface”

Do all the above steps for the second server that you will use in the cluster.

The drive should now show up in Disk Management which you remotely connect to from a client in the SAME domain as the server, which is important. So create some partitions of your own choice.

Update 2010-03-25:
IMPORTANT!
For Live Migration to work (Possibly failover too) you must have configured the Hyper-V Virtual Network Adapters you are going to use BEFORE creating a cluster. These virtual network adapters must have the same name on both your servers – yes you will have to set up your two servers to be identical. Otherwise you will get errors like, Migration attempt failed, and you will find yourself lost cause it doesn’t say why anywhere.

More information about this issue can be found here:
http://www.hodgkins.net.au/2010/03/how-to-fix-hyper-v-migration-attempt-failed/

Now fire up Failover Cluster Manager from Administrative Tools on your client computer. On the right collumn you’ll see Validate a Configuration.., Create a Cluster and Manage a Cluster. Lets begin with Creating a cluster. Input the hostnames of your two servers, i.e:

server1;server2

Click Next.
Accept all the default settings and Next, Next finnish etc. Except where you will be choosing a name for the cluster. You might choose a good name for it, like Cluster_companyname

Now lets configure the Quorum settings.
Right click your clustername in the left collumn and go to More Actions… > Configure Cluster Quorum Settings…
Go on with the default settings and select your iscsi quorum drive.
I created two iscsi targets, one for quorum (2 GB) and one for storage (1,90 TB).

When you’ve setup Quorum go to the left collumn and select your clustername, on the middle collumn you’ll see Enable Cluster Shared Volumes…, so lets click there and select the iscsi storage drive.

Open Hyper-V manager from Administrative Tools and create a new virtual machine.

Once installed, shut it down and go back to Failover Cluster Manager.
Right click your clustername and select Configure a Service or Application….
On the next window you will be presented a number of services, go to the very bottom and select Virtual Machine and select your newly created virtual machine that you created in Hyper-V Manager.

This will create a so called Highly Available virtual machine that will failover to the next running node if one node fails.