Supported data types for backward convolution

[francescobrozzu]

Dear all,

I'm currently trying to implement and train a Deep Neural Network architecture using the oneDNN toolkit on Devcloud. So far I've been able to follow the example provided here: https://oneapi-src.github.io/oneDNN/cnn_training_bf16_cpp.html, however there are some parts in the backward pass implementation that are not completely clear to me. Specifically, when the gradients of the weights and biases are initialized it is not clear why the bias is being initialized with a type different from the one of the weights:

auto conv_user_diff_weights_memory = memory({{conv_weights_tz}, dt::f32, tag::nchw}, eng);
auto conv_diff_bias_memory = memory({{conv_bias_tz}, dt::f32, tag::x}, eng);

auto conv_bwd_src_md = memory::desc({conv_src_tz}, dt::bf16, tag::any);
auto conv_diff_weights_md = memory::desc({conv_weights_tz}, dt::bf16, tag::any);

auto conv_diff_bias_md = conv_diff_bias_memory.get_desc();

Additionally, looking at the documentation it seems that the bias has no data format guideline:

How is the backward pass supposed to be implemented? Should I simply use float32 type for all the gradients?

Best regards,

Francesco Brozzu

[JananiC_Intel]

Hi,

Thanks for posting in Intel forums.

We will check on this internally and let you know.

Regards,

Janani Chandran

[Louie_T_Intel]

Hi Brozzu,

Thanks for your interests in oneDNN, and raised this question.

For conv forward/backward path, both f32 and bf16 should be supported, so you could use both.

Here is a comment for forward path.

https://github.com/oneapi-src/oneDNN/blob/master/examples/cnn_training_bf16.cpp#L100

In the meantime, let us double check why we don't have value for backward Bias data type in the table.

https://oneapi-src.github.io/oneDNN/dev_guide_convolution.html

thanks

[Louie_T_Intel · Accepted solution]

Hi Brozzu,

For the codes attached, it actually maps to the "weights update" row, and conv_diff_bias should map to bias column in that row.

Therefore, both f32/bf16 work.

For the backward rows, bias column should be empty, because we compute the gradient wrt src (diff_src) and this computation does not involve the bias at all during the backward by data.