Intel® oneAPI DL Framework Developer Toolkit
Get answers for developing new or customizing existing frameworks using common APIs.

Supported data types for backward convolution

francescobrozzu
Beginner
753 Views

Dear all,

I'm currently trying to implement and train a Deep Neural Network architecture using the oneDNN toolkit on Devcloud. So far I've been able to follow the example provided here: https://oneapi-src.github.io/oneDNN/cnn_training_bf16_cpp.html, however there are some parts in the backward pass implementation that are not completely clear to me. Specifically, when the gradients of the weights and biases are initialized it is not clear why the bias is being initialized with a type different from the one of the weights:

auto conv_user_diff_weights_memory = memory({{conv_weights_tz}, dt::f32, tag::nchw}, eng);
auto conv_diff_bias_memory = memory({{conv_bias_tz}, dt::f32, tag::x}, eng);

auto conv_bwd_src_md = memory::desc({conv_src_tz}, dt::bf16, tag::any);
auto conv_diff_weights_md = memory::desc({conv_weights_tz}, dt::bf16, tag::any);

auto conv_diff_bias_md = conv_diff_bias_memory.get_desc();

Additionally, looking at the documentation it seems that the bias has no data format guideline:

francescobrozzu_0-1617296369853.png

How is the backward pass supposed to be implemented? Should I simply use float32 type for all the gradients?

Best regards,

Francesco Brozzu

 

0 Kudos
1 Solution
Louie_T_Intel
Moderator
696 Views

Hi Brozzu,


For the codes attached, it actually maps to the "weights update" row, and conv_diff_bias should map to bias column in that row.

Therefore, both f32/bf16 work.


For the backward rows, bias column should be empty, because we compute the gradient wrt src (diff_src) and this computation does not involve the bias at all during the backward by data.





View solution in original post

3 Replies
JananiC_Intel
Moderator
728 Views

Hi,


Thanks for posting in Intel forums.


We will check on this internally and let you know.


Regards,

Janani Chandran



Louie_T_Intel
Moderator
712 Views

Hi Brozzu,


Thanks for your interests in oneDNN, and raised this question.

For conv forward/backward path, both f32 and bf16 should be supported, so you could use both.

Here is a comment for forward path.

https://github.com/oneapi-src/oneDNN/blob/master/examples/cnn_training_bf16.cpp#L100


In the meantime, let us double check why we don't have value for backward Bias data type in the table.

https://oneapi-src.github.io/oneDNN/dev_guide_convolution.html


thanks


Louie_T_Intel
Moderator
697 Views

Hi Brozzu,


For the codes attached, it actually maps to the "weights update" row, and conv_diff_bias should map to bias column in that row.

Therefore, both f32/bf16 work.


For the backward rows, bias column should be empty, because we compute the gradient wrt src (diff_src) and this computation does not involve the bias at all during the backward by data.





Reply