When an ESXi server crashing it generates what is called a PSOD (Purple screen of death) the fact that the ESXi host will resond to Ping and SSH show that the actual host has not crashed but your issue is somewhere else.
if VM's respond to ping then that again is evidence that the Host has not crashed, as a crashed host will bring down the guests, until they are restarted on another host if you have HA configured.
what edition of vSphere are you using? do you have vCenter installed? What is your network topology? how loaded are these hosts? what are the hosts CPU type? number and core count? how much memory, what is the shared storage or is it local?
What exactly do you mean, "This is not a purely network? Sake of Heartbeat"
there are many questions that need to be answered before we can start to look into this for you.