Looking for advice or recommendations. Here's the setup:
We had a vSphere 4.1 implementation, with a cluster consisting of four ESXi 4.1 hosts that utilized two different SANs. The first SAN is an Equallogic PS5XXX series that we own. The second SAN is a Falconstor NSS appliance from which the hosts access two LUNs that we "lease" from a larger campus department. Each of the ESXi hosts had four vmkernel ports dedicated for iSCSI. All of these port groups use the same vSwitch. There are four vmnics utilized by the vSwitch, with a one-to-one mapping for the vmnic to the port group. Cue the esxcfg-vmknic output:
~ # esxcfg-vmknic -l
Interface Port Group/DVPort IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type
...
vmk1 Software iSCSI3 IPv4 172.17.28.xxx 255.255.255.224 172.17.28.xxx 00:50:56:72:3d:95 1500 65535 true STATIC
vmk2 Software iSCSI1 IPv4 172.16.52.xxx 255.255.255.0 172.16.52.xxx 00:50:56:75:fc:66 1500 65535 true STATIC
vmk3 Software iSCSI2 IPv4 172.16.52.xxx 255.255.255.0 172.16.52.xxx 00:50:56:7e:54:0f 1500 65535 true STATIC
vmk4 Software iSCSI4 IPv4 172.17.28.xxx 255.255.255.224 172.17.28.xxx 00:50:56:7b:92:0d 1500 65535 true STATIC
...
The esxcfg-vswitch output:
~ # esxcfg-vswitch -l
...
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch1 128 9 128 1500 vmnic3,vmnic7,vmnic4,vmnic5
PortGroup Name VLAN ID Used Ports Uplinks
Software iSCSI4 0 1 vmnic5
Software iSCSI2 0 1 vmnic3
Software iSCSI1 0 1 vmnic7
Software iSCSI3 0 1 vmnic4
...
Interfaces vmk2 and vmk3 are used to connect to the Equallogic, and all devices are connected the same subnet (172.16.52.0/24). Interfaces vmk1 and vmk4 are used to connect to the Falconstor. These interfaces (subnet 172.17.28.xxx/27) are not on the same subnet as the Falconstor (subnet 172.17.10.0/24), so the iSCSI traffic is routed. In order to connect to the Falconstor, I added a route with esxcfg-route:
~ # esxcfg-route -l
VMkernel Routes:
Network Netmask Gateway Interface
172.17.28.xxx 255.255.255.224 Local Subnet vmk1
172.16.51.0 255.255.255.0 Local Subnet vmk6
172.16.52.0 255.255.255.0 Local Subnet vmk2
172.16.251.0 255.255.255.0 Local Subnet vmk0
172.17.10.0 255.255.255.0 172.17.28.xxx vmk1 <<== added here
default 0.0.0.0 172.16.251.1 vmk0
All the vmkernel ports were bound to the ESXi software iSCSI inititiator, which allowed us to multi-path to each SAN. Now, I am aware that iSCSI port binding does not support routing (http://kb.vmware.com/kb/1009524), however, under 4.1 it seemed to work. Earlier this week, I began to upgrade to vSphere 5.0 update 2, and after performing an upgrade in place of one of the ESXi hosts, I found that the connections to the Equallogic came back fine, but the connection to the Falconstor did not. In checking the new "Network Configuration" tab under the Software iSCSI Initiator properties window, I see that vmk1 and vmk4 show a "Path Status" of "Not Used". The route I added is still in place. I double-checked the connection properties for the Falconstor iSCSI target (CHAP is used, so I re-entered the secrets). So I'm left with the fact that it really now is the case that routing isn't supported in this configuration.
I'm wondering what my configuration options are right now, assuming we will still want to utilize the storage from the Falconstor? The physical nics are Broadcom 5709, which support TOE and show as hardware iSCSI adapters. All hosts are licensed for vSphere Enterprise (so no dvSwitches). I don't believe there is any flexibility for changing adapter subnets in order to avoid having to route (the hardware is located in a datacenter that my department does not manage).
Can the two vmkernel ports used for the Falconstor be unbound from the software initiator while the other two used for the Equallogic remain bound? Can the hardware initiator be used in conjunction with the software initiator? I'd like to be able to continue to take advantage of multipathing to both SANs, or will that be lost?
Thanks