Have a read of the following Blog post on the subject,
http://professionalvmware.com/2009/09/how-to-shrink-a-vmdk-file-in-esx/
but be aware that this is not supported. and it is at your own risk. so before this is done on a production guest. TEST TEST and TEST again, better still, if you need to reduce the size of the VMDK just do a Conversion of the Guest and select the resize disk option during the configureation stage. this is Tested, and more importantly SUPPORTED.
the ability to resize VMDKs was removed from VMFSTools in VI3.x because of the number of issues caused by users resizing VMDK's and chopping of data from the end of disks.