only when -ipo is used it will slow down a lot, but not so much with only -O2.
it's more likely a bug in the compiler. is it slow in compiling or linking? If it's in compiling, could you send the code?
you can use "Private" post to upload code if you'd like or file a ticket to Intel Premier Support https://premier.intel.com/