I tried to install RHEL 5 x86_64 in a VMware Server 2.0 Beta 2 virtual machine. The host system has a 64 bit CPU and the virtual machine had been configured for a 64 bit OS. Nevertheless the installer failed with:

Your CPU does not support long mode. Use a 32 bit distribution

Some poking around turned up this VMware document. In short I had to go into the BIOS on my host system and enable Virtualization Technology. On my Dell server that was disabled by default.