Intel® Distribution of OpenVINO™ Toolkit
Community support and discussions about the Intel® Distribution of OpenVINO™ toolkit, OpenCV, and all things computer vision-related on Intel® platforms.

FP16 vs FP32

username_007
Beginner
273 Views

I am trying to understand FP16 and FP32 results that I got. Shouldn't the latency be lower and throughput be higher for haif-precision floating point (FP16) compared to single-precision floating point (FP32). Same case for duration. The duration/executing time is higher for haif-precision floating point (FP16) compare to single-precision floating point (FP32). Shouldn't it be the opposite

My understanding was FP16 is less resource intensive to calculate as compared to FP32 so FP16 should be faster, but the results are opposite

 

exe1.PNGexe2.PNGLat.PNGthr.PNG

 

 

0 Kudos
5 Replies
username_007
Beginner
252 Views

after reading some search results on Google I understand that Intel cpu doesn’t have fp16 type and will call function to do fp16 calculation. Is this correct?

Iffa_Intel
Moderator
231 Views

Hi,

 

Since you are mentioning precisions and latency/throughput, I assume you are trying to assess your model's performance.

 

Indeed, FP32 would be more accurate compared to lower in size precision such as FP16 or even INT8. However, this also varies according to the hardware/device compatibility. Running FP32 on hardware might results in a faster inference rate compared to FP16 if the hardware actually prefers FP32 instead FP16.

 

You'll need to evaluate the hardware and use the precision that it prefers. You may refer to this supported formats section.

This Benchmark documentation might help to explain further.

 

Sincerely,

Iffa

 

 

 

 

username_007
Beginner
221 Views

how do i find out if a particulart CPU (Intel I5-10351G1 in this case) prefers FP32 instead of FP16?. Is there a spec sheet or do I look in the instruction set manual for this CPU?. I tried google, but couldn't find anything.

Iffa_Intel
Moderator
193 Views

Generally, according to the documentation that I shared previously, the CPU supports and prefers FP32 model format (precision).

 

OpenVINO CPU Plugin supports (latest):

  • Intel Xeon with Intel® Advanced Vector Extensions 2 (Intel® AVX2),
  • Intel® Advanced Vector Extensions 512 (Intel® AVX-512),
  • AVX512_BF16, Intel Core Processors with Intel AVX2,
  • Intel Atom Processors with Intel® Streaming SIMD Extensions (Intel® SSE)

Refer to: Supported Devices

 

Your CPU version should go well with FP32.

 

Sincerely,

Iffa

 

Iffa_Intel
Moderator
147 Views

Greetings,


Intel will no longer monitor this thread since we have provided a solution. If you need any additional information from Intel, please submit a new question.



Sincerely,

Iffa


Reply