Ohh and one really important fact i mssed out.
CPU IO GOES THROUGH THE ROOF!
I had several ESXi5 Servers drop off the IB network for minutes at a time as there were NO CPU CYCLES LEFT to let the 1Gb Nic allow management traffic through! Good work Mellanox!! With scathing sarcasm !
SRP IS BETTER!
Having tested IB_ISCSI on ESSX 5 i now conclude the CPU overhead is worse than i thought. Copying from 1 ISCSI SAN, a VMFS folder to a 2nd SAN of same configuration on 40GB IB takes 8 mins for a simple 40GB VM. may as well be ethernet at that speed!
The same 40GB on IB_SRP on same sans with SRP enabled only took 2 mins! 4 x Faster than ISCSI. And that was during the ISCSI move on a 2nd server. When SRP SAN is not moving ISCSI Volumes it only takes 1.05minutes! to move 40GB
consider if you have to move VM's with snapshots which could be as much as 120GB - it takes a very long time.! CPU Use during these moves is peeking @100% of Dual Xeon5450's Average is about 6000-8000Mhz during the whole time its being moved! OUCH!
Despite any imporovements in ESX5 over 4.1U1. i cant run my SQL and MSSQL and high perf clouds on ISCSI at all! So dumpng ESXi5 untull either a SRP driver comes out. out VMWare do something about lack of SRP.
Also i would like to note im using RAID6 with only 8 drives with these current test SANS. SAS6gb/s on LSI9280-8i and they achieve 1400MB/s read and 800MB/s write on SRP.
IB_ISCSI on same SAN with same array can barley achieve 600MB/Read 300MB/sWrite! ISCSI SUCKS! Sharing that for 4 ~6 esxi nodes is pathetic.
So all of those who want SRP for esxi5 send me an email admin@v365.com.au and lets develop our own SRP driver.