Can MKL_DNN used for NER model with a bert?

[Hap_Zhang]

Hi, 

I tried to use mkl-dnn accelerate the inference of NER(Named Entity Recognition) model, however, some errors happened:

dnnl_verbose,exec,cpu,reorder,simple:any,undef,src_f32::blocked:a:f0 dst_f32::blocked:a:f0,,,5,0.0090332
Traceback (most recent call last):
  File "bert_for_token_classification_example_fast_dnnmkl.py", line 161, in <module>
    torch_outputs_ = model_(input_ids_.to(device), attention_mask_.to(device), token_type_ids_.to(device))
  File "/opt/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/miniconda3/lib/python3.7/site-packages/transformers/models/bert/modeling_bert.py", line 1701, in forward
    return_dict=return_dict,
  File "/opt/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/miniconda3/lib/python3.7/site-packages/transformers/models/bert/modeling_bert.py", line 944, in forward
    extended_attention_mask: torch.Tensor = self.get_extended_attention_mask(attention_mask, input_shape, device)
  File "/opt/miniconda3/lib/python3.7/site-packages/transformers/modeling_utils.py", line 289, in get_extended_attention_mask
    extended_attention_mask = attention_mask[:, None, None, :]
RuntimeError: opaque tensors do not have strides

I use the transformers(hugging face) with 4.6.1,  can someone help to point out the problem in it? Thanks a lot.

[Gopika_Intel]

Hi,

Thank you for posting in Intel Communities. We are trying to reproduce the issue from our end, we will get back to you as soon as we get an update.

Regards

Gopika

[Gopika_Intel]

Hi,

Could you please give us the following information so that we can try reproducing the issue from our end?

1. The pytorch version

2. Whether you are using quantized model or not

3. Steps to reproduce the issue

4. The model you are using

Regards

Gopika

[Hap_Zhang]

Hi, Gopika,

thank you for your reply！

1. The pytorch version

Reply:The pytorch version is 1.9.0

2. Whether you are using quantized model or not

Reply:not quantized model, just a model finetuned in BERT-Base, Chinese

3. Steps to reproduce the issue

Reply:

device = torch.device("cpu")
model_id = "bert-base-chinese"
config = PretrainedConfig.from_pretrained(model_id)
config.num_labels = 5
model = BertForTokenClassification.from_pretrained(model_id, config=config).to(device)
model.load_state_dict(torch.load("model-epoch49.pt", map_location=torch.device(device)))
model.eval()
tokenizer = BertTokenizer.from_pretrained(model_id)
sentence = "海底捞"
result = tokenizer(sentence, padding='max_length', max_length=512, truncation=True, return_tensors='pt').data
input_ids, token_type_ids, attention_mask = result.get('input_ids'), result.get('token_type_ids'), result.get('attention_mask')
input_ids_ = input_ids.float().to_mkldnn()
token_type_ids_ = token_type_ids.float().to_mkldnn()
attention_mask_ = attention_mask.float().to_mkldnn()
model_ = mkldnn.to_mkldnn(model)
torch_outputs_ = model_(input_ids_.to(device), attention_mask_.to(device), token_type_ids_.to(device))
torch_outputs = torch_outputs_.to_dense()

4. The model you are using

Reply:The "model-epoch49.pt" is 388M and exceed the maximum limit, so i can not attach here

[Gopika_Intel]

Hi,

The problem is that the opaque tensors do not have strides and it is not supported by the current version of ipex. Please try with the next release of ipex. Please refer: https://github.com/intel/intel-extension-for-pytorch for more information on ipex.

Regards

Gopika

[Gopika_Intel]

Hi,

Has the solution provided helped? Please let us know whether your query is resolved.

Regards

Gopika

[Gopika_Intel · Accepted solution]

Hi

We have not heard back you. As mentioned please try with the next release of ipex. If you need any additional information, please post a new question as this thread will no longer be monitored by Intel

Regards

Gopika