Yes, this could potentially be impactful to performance.
Normally, you shouldn't need to specify -lm at all to the compiler. It will link it in automatically. If you do have to do it explicitly, you should always link in system libraries last in order. In your example above, I believe: