H.264 encoder is BROKEN (was: ippiTransformQuantFwdLumaDC4x4_H264 is buggy)

[registered_user]

as title says - in all versions till current 6.0.2.074.
The operations are performed in 16 bits registers, which isn't enough for (0,0) coefficient - the "main" DC of 16x16 intra mode. After hadamard transform this coefficient requires 17 bits - it's dynamic range is -65280 .. +65280. Of course it is saturated to -32768 .. +32767 instead, thus leading to corrupted output.

Obvious solution is to use right shift for coefficients during second stage of hadamard transform, so output fits 16 bits. Then of course right shift during quantization should be performed by 1 less.

ru.

[Ying_H_Intel]

Quoting - registered.user

as title says - in all versions till current 6.0.2.074.
The operations are performed in 16 bits registers, which isn't enough for (0,0) coefficient - the "main" DC of 16x16 intra mode. After hadamard transform this coefficient requires 17 bits - it's dynamic range is -65280 .. +65280. Of course it is saturated to -32768 .. +32767 instead, thus leading to corrupted output.

Obvious solution is to use right shift for coefficients during second stage of hadamard transform, so output fits 16 bits. Then of course right shift during quantization should be performed by 1 less.

ru.

Hi ru.

It seems there issimiliardiscussion regarding thisat
<<http://software.intel.com/en-us/forums/showthread.php?t=64409>>

I can't reproduce that issue there. Could you please provide us a piece of test code toillustrate the problem?

Thanks
Ying

[registered_user]

Quoting - (Intel)

Hi ru.

It seems there issimiliardiscussion regarding thisat
<<http://software.intel.com/en-us/forums/showthread.php?t=64409>>

I can't reproduce that issue there. Could you please provide us a piece of test code toillustrate the problem?

Thanks
Ying

Hi Ying,

you don't need the code to reproduce the problem - it is input data dependant. As I said dynamic range of (0,0) coefficient in DC block is higher than 16-bit register can cover. Here is the proof (with worst case data input):
let's assume input block is all 0s, and prediction is all 255's, which means difference is -255 and DC block contains values -255*16=-4080. After hadamard transform in ippiTransformQuantFwdLumaDC4x4_H264() (0,0) coefficient becomes -4080*16=-65280, which is 1 bit wider than 16-bit registers can cover.

You can feed this function with specified data and you'll see the problem. Hope this would help.

ru.

[registered_user]

removed double post

[registered_user]

It's almost two weeks passed since first post - it'd be really great if someone from Intel would at least conform that described problem is understood and is gonna be fixed. I'd appreciate if you give any hint regarding date of fix as well.

Thanks,
ru.

[pvonkaenel]

Quoting - registered.user

It's almost two weeks passed since first post - it'd be really great if someone from Intel would at least conform that described problem is understood and is gonna be fixed. I'd appreciate if you give any hint regarding date of fix as well.

Thanks,
ru.

Since you've discovered what appears to be a bug, your best bet is to submit it with premier support. I have always gotten an official response through that channel. I tend to use the forums for user and Intel guided assistance.

Peter

[registered_user]

Dear Intel support,

it's been more than 3 weeks already since the first post - is it that hard to verify the bug and confirm it?

ru.

[pvonkaenel]

Quoting - registered.user

Dear Intel support,

it's been more than 3 weeks already since the first post - is it that hard to verify the bug and confirm it?

ru.

I have always gotten a pretty quick responce from premier support (I assume you submitted the issue), but even after the initial response, it needs to work its way through their test/debug cycles. If you have some test data as you mentioned that generates the problem, they might be able to verify it faster.

Peter

[registered_user]

Quoting - pvonkaenel

I have always gotten a pretty quick responce from premier support (I assume you submitted the issue), but even after the initial response, it needs to work its way through their test/debug cycles. If you have some test data as you mentioned that generates the problem, they might be able to verify it faster.

Peter

I appreciate your input Peter, but the bug is described in depth here and it doesn't need any test data to verify - few lines of calculations of dynamic range (written few posts above!) make it all.

[Ying_H_Intel]

Quoting - registered.user

I appreciate your input Peter, but the bug is described in depth here and it doesn't need any test data to verify - few lines of calculations of dynamic range (written few posts above!) make it all.

Hello RU,

Sorry for the slow response.Right, it isa bug,the encodershould support larger data range prior the last right shift in DC transformation.As peter suggested, we usually track bugsthrough premier.intel.com system.So premier should be better place to submit a product defect if you have somein the future.For thisissue, Ihadescalated it to our developer team. I will keep you posted if any updates.

Best Regards,
Ying

[Chao_Y_Intel]

This problem is expected to be fixed in the IPP 6.1 release. Can you pleas check how it works for you now.

thanks,
Chao

[registered_user]

Quoting - Chao Yu (Intel)

This problem is expected to be fixed in the IPP 6.1 release. Can you pleas check how it works for you now.

thanks,
Chao

I don't have any test case atm to verify unfortunately. I'll report if I'll have any problems.

Thanks.