However, ESXi booted until "cnic_register loaded successfully" and just hang there.
This is only because you have given the onboard VGA display as the passthrough. You need to disable this device as the passthrough. However, you cant do it because, ESX will not proceed. You might want to disable the vtd in your bios. Boot to the ESX, remove the onboard vga card as passthrough and reboot again.