PGO-Gen works, PGO-Use doesn't?

[Alex_K_6]

Well this is strange:

http://software.intel.com/sites/default/files/forum/356681/capture.png

So when compiling with -Qprof-gen, the program compiles and runs fine, when using -Qprof-use (or without PGO at all) the program doesn't compile. I tried using -FORCE in the linker, but to no avail.

[Georg_Z_Intel]

Hello, it doesn't seem to be directly related to the compiler at all. Is quantize_sse2.asm some generated assembly file? Or any of the included assembly files therein? The one you provided indeed has undefined symbols (at least they're not marked as external as far as I can see). But it's just a small window to look through and without more data it's not possible to help here. You say that just enabling the /Qprof-gen option causes the problem. So, compare the assembly files with and without this option (quantize_sse2.asm and all other files that are included). Best regards, Georg Zitzlsberger

[Alex_K_6]

Georg Zitzlsberger (Intel) wrote:

Hello,

it doesn't seem to be directly related to the compiler at all. Is quantize_sse2.asm some generated assembly file? Or any of the included assembly files therein?
The one you provided indeed has undefined symbols (at least they're not marked as external as far as I can see). But it's just a small window to look through and without more data it's not possible to help here.

You say that just enabling the /Qprof-gen option causes the problem. So, compare the assembly files with and without this option (quantize_sse2.asm and all other files that are included).

Best regards,

Georg Zitzlsberger

Everything is there already, so the files aren't generated by the compiler. It's part of the libvpx package and all it does is get compiled (i.e no files generated). This is why I'm confused why Qprof-gen is the only way to be able to compile the package?

[Alex_K_6]

Here's the package (sorry for the double post, I got a forum error uploading the first time).

[Georg_Z_Intel]

Hello, Thank you for the package. The assembly asm_enc_offsets.asm is generated from asm_enc_offsets.c which is supposed to define all the undefined references in question. Please provide me both assemblies for turned-on & -off /Qprof-gen option. Please also add the full compiler calls with all options (esp. macro definitions). I'm also not sure whether you set INLINE_ASM. Thank you & best regards, Georg Zitzlsberger

[SergeyKostrov]

>>...I tried using -FORCE in the linker, but to no avail... That looks very strange to me because you have a compiler error at a line 58 in quantize_sse2.asm file and the error clearly says '...undefined symbol...(first use)'.

[Georg_Z_Intel]

Hello Sergey, that error comes from YASM, not from the compiler. However, the file asm_enc_offsets.asm is generated by the compiler and included in quantize_sse2.asm. Hence it is worth to look at asm_enc_offsets.asm and compare its different builds. Best regards, Georg Zitzlsberger

[Alex_K_6]

Hi Georg,

Here are the two files, but it doesn't seem like they'll be much help

[Georg_Z_Intel]

Hello, that's what I've expected: in one case all the symbols are defined, in the other, none. Initially I got it kind of wrong: I assumed that /Qprof-gen caused the problem which is not true. What you wrote suggests that *nothing* works except using /Qprof-gen. Is that true? Please still provide me the full command line for building asm_enc_offsets.c for both, using /Qprof-gen and not using PGO at all. Optionally you could also provide it for /Qprof-use. Thank you & best regards, Georg Zitzlsberger

[Alex_K_6]

Georg Zitzlsberger (Intel) wrote:

Hello,

that's what I've expected: in one case all the symbols are defined, in the other, none.
Initially I got it kind of wrong: I assumed that /Qprof-gen caused the problem which is not true. What you wrote suggests that *nothing* works except using /Qprof-gen. Is that true?

Please still provide me the full command line for building asm_enc_offsets.c for both, using /Qprof-gen and not using PGO at all. Optionally you could also provide it for /Qprof-use.

Thank you & best regards,

Georg Zitzlsberger

Yes, only Qprof-gen works. Sorry, I forgot to post the command line:

-O3 -QaxAVX -arch:SSE3 -Qparallel  -fp:precise -Qprof-gen -Qprof-dirc:/iccpgo -w

The above is the working command line. Below are the ones I've tried but don't work:

-O3 -QaxAVX -arch:SSE3 -Qparallel  -fp:precise -Qprof-use -Qprof-dirc:/iccpgo -w

-O3 -QaxAVX -arch:SSE3 -Qparallel  -fp:precise -w

[Georg_Z_Intel]

Thank you! Unfortunately it does not help. I'd expected some directives being defined. What I need is to fully follow the build steps: which files have been compiled and how Hence, the full build logs would be needed. I'm also not seeing how asm_enc_offsets.c is being compiled to produce the assemblies you sent me via asm-files.zip. The original asm_enc_offsets.c with all includes expanded reads like that: [cpp] ... // asm_offsets.h #if INLINE_ASM #define DEFINE(sym, val) asm("\n" #sym " EQU %0" : : "i" (val)); #define BEGIN int main(void) { #define END return 0; } #else #define DEFINE(sym, val) int sym = val; #define BEGIN #define END #endif ... // asm_enc_offsets.c BEGIN /* regular quantize */ DEFINE(vp8_block_coeff, offsetof(BLOCK, coeff)); DEFINE(vp8_block_zbin, offsetof(BLOCK, zbin)); ... DEFINE(vp8_common_mb_rows, offsetof(VP8_COMMON, mb_rows)); END ... [/cpp] The generated assembly reads like that: [plain] vp8_blockd_qcoeff EQU 0 vp8_block_src_diff EQU 0 ... vp8_common_mb_rows EQU 2536 END [/plain] That does not add up. Best regards, Georg Zitzlsberger

[Alex_K_6]

Here's a full build log of a failed build:

[Georg_Z_Intel]

Hello, I understand now, what's going on in your project build. The file asm_enc_offsets.c is compiled into asm_enc_offsets.obj. Some hand-made tool (host_obj_int_extract.exe) tries to extract symbol information from it. That obviously only works if the produced object file is a real/plain object file. However, when you use PGO, the object files created contain lots of other information (similar to IPO) and the hand-made tool cannot handle this. Solution: Build asm_enc_offsets.c without PGO (neither /Qprof-gen, nor /Qprof-use). You won't need it anyways for it as far as I can see. If you plan to use IPO, it's the same here, too. Don't apply IPO to asm_enc_offsets.c. Btw.: Should also be the same for asm_com_offsets.c and all others host_obj_int_extract.exe is using. Edit: As it turned out, the handling of /GL- by the compiler was not correct. PGO itself does not enable IPO. Only in the example discussed the current compiler (Intel(R) Composer XE 2013 Update 1) enabled IPO implicitly because of the /GL- issue, which should not be the case. See my reply below for more information. Best regards, Georg Zitzlsberger

[Alex_K_6]

Georg Zitzlsberger (Intel) wrote:

Hello,

I understand now, what's going on in your project build. The file asm_enc_offsets.c is compiled into asm_enc_offsets.obj. Some hand-made tool (host_obj_int_extract.exe) tries to extract symbol information from it. That obviously only works if the produced object file is a real/plain object file.
However, when you use PGO, the object files created contain lots of other information (similar to IPO) and the hand-made tool cannot handle this.

Solution: Build asm_enc_offsets.c without PGO (neither /Qprof-gen, nor /Qprof-use). You won't need it anyways for it as far as I can see.

If you plan to use IPO, it's the same here, too. Don't apply IPO to asm_enc_offsets.c.

Btw.: Should also be the same for asm_com_offsets.c and all others host_obj_int_extract.exe is using.

Best regards,

Georg Zitzlsberger

What's strange is that it doesn't compile if Qprof isn't defined at all, the only time it compiles is when Qprof-gen has been defined. This is what confuses me, why does Qprof-gen work, but not using Qprof at all not work? Even an empty command line doesn't compile. I thought there might be something that Qprof-gen does to ignore the undefined symbols (sort of like -FORCE).

[Georg_Z_Intel]

Hello, I cannot say without having any build log to reproduce the problem -- here: the case when PGO is not used at all. With the information I have at hand, I was able to properly create the file asm_enc_offsets.asm. This worked for both non-PGO and PGO's first phase (/Qprof-gen). As described, and expected, it only failed for PGO's second phase (/Qprof-use). It still might fail for the same reason for non-PGO if you're using IPO (as described above). You could also try to build with /Qipo- to be sure. If this does not help I'd expect a completely different problem. Please provide me at least the corresponding build log to figure it out. What do you mean by "empty command line"? Best regards, Georg Zitzlsberger

[Alex_K_6]

Georg Zitzlsberger (Intel) wrote:

Hello,

I cannot say without having any build log to reproduce the problem -- here: the case when PGO is not used at all.
With the information I have at hand, I was able to properly create the file asm_enc_offsets.asm. This worked for both non-PGO and PGO's first phase (/Qprof-gen). As described, and expected, it only failed for PGO's second phase (/Qprof-use).
It still might fail for the same reason for non-PGO if you're using IPO (as described above). You could also try to build with /Qipo- to be sure.

If this does not help I'd expect a completely different problem. Please provide me at least the corresponding build log to figure it out.
What do you mean by "empty command line"?

Best regards,

Georg Zitzlsberger

Here's the build log using just -O3 and -w. I'll try using -Qipo-

[Georg_Z_Intel]

Hello, just checked and it's IPO that's enabled. However, in a way that's not obvious right away. Microsoft Visual Studio* compilers know the option /GL which is global optimization (Microsofts way of doing IPO). If /GL is set with our compiler we enable IPO. However, there's a bug in the compiler driver that /GL- (note the minus) does not guarantee to turn off IPO. It erroneously enables it and also issues an incorrect warning message you didn't see because you turned off all warnings: icl: command line warning #10006: ignoring unknown option '/G-' Solution: Remove /GL- option. This does not have any side-effects if you use the Intel compiler as IPO is disabled by default. I'll let engineering know about the bug so it can be fixed. (edit: defect ticket DPD200317705) Best regards, Georg Zitzlsberger

[Alex_K_6]

Georg Zitzlsberger (Intel) wrote:

Hello,

just checked and it's IPO that's enabled. However, in a way that's not obvious right away. Microsoft Visual Studio* compilers know the option /GL which is global optimization (Microsofts way of doing IPO). If /GL is set with our compiler we enable IPO. However, there's a bug in the compiler driver that /GL- (note the minus) does not guarantee to turn off IPO. It erroneously enables it and also issues an incorrect warning message you didn't see because you turned off all warnings:
icl: command line warning #10006: ignoring unknown option '/G-'

Solution: Remove /GL- option. This does not have any side-effects if you use the Intel compiler as IPO is disabled by default.

I'll let engineering know about the bug so it can be fixed. (edit: defect ticket DPD200317705)

Best regards,

Georg Zitzlsberger

Thanks for your help! I found it odd that even stating -Qipo- didn't help at all.

[Georg_Z_Intel]

Hello, the order of the options is important. If /Qipo- comes before /GL[-] it is overwritten by the latter. In the current case both /GL (OK) and /GL- (bug) turn on IPO. If you put /Qipo- at the very end it turns off IPO for sure. Best regards, Georg Zitzlsberger

[SergeyKostrov]

>>the order of the options is important. If /Qipo- comes before /GL[-] it is overwritten by the latter. In the current case both >>/GL (OK) and /GL- (bug) turn on IPO. If you put /Qipo- at the very end it turns off IPO for sure. Thanks for the note, Georg!

[Georg_Z_Intel]

Hello, FYI: The problem with /GL- will be fixed with Intel(R) Composer XE 2013 Update 3 (did not make it into Update 2). Best regards, Georg Zitzlsberger