Downloader.py cannot download BERT

[Ray_Lo_Intel]

I think the binary is broken? It always fails as the Downloaded file is shorter*.

rlo@rlo-mac01 omz_demos_build % /opt/intel/openvino_2020.4.287/deployment_tools/open_model_zoo/tools/downloader/downloader.py --name "bert*" --output_dir ~/Documents/openvino_models
...
========== Downloading /Users/rlo/Documents/openvino_models/intel/bert-large-uncased-whole-word-masking-squad-fp32-0001/FP32/bert-large-uncased-whole-word-masking-squad-fp32-0001.bin
... 80%, 1053849 KB, 2988 KB/s, 352 seconds passed
########## Error: Downloaded file is shorter (1079141619 B) than expected (1335853368 B)

[Vladimir_Dudnik]

Hi Raymond,

BERT large model has more than 1 Gb in size, so downloader might experience networking issues. To make downloading more stable please use option to increase number of attempts downloader will try to load the model, as described in Model downloader usage section of OMZ tools downloader README.md file

By default, the script will attempt to download each file only once. You can use the --num_attempts option to change that and increase the robustness of the download process:

./downloader.py --all --num_attempts 5 # attempt each download five times

Regards,
  Vladimir

[Ray_Lo_Intel]

Can you test this command on your side? It seems to fail at the same location everytime. 

/opt/intel/openvino_2020.4.287/deployment_tools/open_model_zoo/tools/downloader/downloader.py --name "bert*" --output_dir ~/Documents/openvino_models

[Munesh_Intel · Accepted solution]

Hi Raymond,

Thanks for reaching out to us. I've just validated the download link, and managed to download the model using the following command:

./downloader.py --name bert* --num_attempts 5

I've attached the snapshot for your reference.

Regards,

Munesh

[Ray_Lo_Intel]

Got it.

I see that you also have the same 'drop off' at approximately the 1GB mark.

Apparently that attempt flag is a 'must' for downloading large files or it will always fail at the similar place. I think there is a size limitation somewhere in the script and it got cut every time. It is always around the 1GB zone. Why don't we just have that auto-attempt feature by default? 

rlo@rlo-mac01 Documents % /opt/intel/openvino_2020.4.287/deployment_tools/open_model_zoo/tools/downloader/downloader.py --name="bert*" --num_attempts 5 -o ~/Documents

################|| Downloading models ||################

========== Downloading /Users/rlo/Documents/intel/bert-large-uncased-whole-word-masking-squad-fp32-0001/FP32/bert-large-uncased-whole-word-masking-squad-fp32-0001.xml
... 100%, 934 KB, 43981 KB/s, 0 seconds passed

========== Downloading /Users/rlo/Documents/intel/bert-large-uncased-whole-word-masking-squad-fp32-0001/FP32/bert-large-uncased-whole-word-masking-squad-fp32-0001.bin
... 81%, 1056867 KB, 9966 KB/s, 106 seconds passed
########## Error: Downloaded file is shorter (1082232051 B) than expected (1335853368 B)
Will retry in 10 seconds...
... 100%, 1304544 KB, 42845 KB/s, 30 seconds passedd

[Munesh_Intel]

Hi Raymond,

We'll forward your improvement suggestion to the development team.

This thread will no longer be monitored since this issue has been resolved. If you need any additional information from Intel, please submit a new question. 

Regards,

Munesh