Are you using latest Update 5?
Is it possible some GUI components are corrupted? So you may uninstall/reinstall the product and retry. Did you see any error message before it hung?
Is it a RedHat Enterprise Server 5.3 64bit specific issue?
Can you open result (generated by command line) from amplxe-gui?