If you are in fact not specifying any options to either ifort or gfortran, you would certainly expect your completely different choices of optimization to have an effect.
You didn't say whether you are running on a 32-bit platform, where gfortran would default to x87 code. Depending on your platform, more consistent options between gfortran and ifort might be something like:
gfortran -march=pentium4 -mfpmath=sse -O2....
ifort -fp-model source .....
[bash]$ ifort -help optwill show you optimization-related options. The correct syntax for the particular option suggested is (note the hyphen binding "fp" to "model")
[bash]$ ifort -fp-model source file1.f file2.f ...where you substitute for "file1",... the names of your source files, and ".f90" in place of ".f" if your sources are free-format Fortran.