Intel® Distribution of OpenVINO™ Toolkit
Community assistance about the Intel® Distribution of OpenVINO™ toolkit, OpenCV, and all aspects of computer vision-related on Intel® platforms.

DL workbench accuracy tool is failing

raushan
Novice
2,628 Views

DL workbench accuracy tool is failing by giving status as Accuracy Tool Failed for my faster rcnn inception v2  coco tensorflow object detection model v 1.15. Please help with the Configuration. 

I have attached the screenshot of the DL workbench.

Labels (1)
0 Kudos
6 Replies
Iffa_Intel
Moderator
2,615 Views

Greetings,


You may try to follow these steps (there's also step by step video here): https://docs.openvinotoolkit.org/latest/workbench_docs_Workbench_DG_Int_8_Quantization.html


Sincerely,

iffa


0 Kudos
raushan
Novice
2,602 Views

Hi,

Thank you for the response.

when i'm trying to optimize model to INT8 using DL workbench, it's giving error as 'resize to input size possible, only for one input layer case'. i have attached screenshot also . Please help out in this issue.

Thanks

0 Kudos
Alexander_D_Intel1
2,422 Views

@raushan Calibration of Faster-RCNN with Default method works in DL Workbench 2021.3. AccuracyAware method still requires you to provide a valid Advanced accuracy config. 

0 Kudos
Iffa_Intel
Moderator
2,584 Views

Could you help to attach your faster rcnn graph and also your dataset,

So that I could replicate this?.



Sincerely,

Iffa


0 Kudos
Iffa_Intel
Moderator
2,572 Views

Greetings,


Thank you for your question. If you need any additional information from Intel, please submit a new question as Intel is no longer monitoring this thread.


Sincerely,

Iffa


0 Kudos
Alexander_D_Intel1
2,429 Views

Hi @raushan ,

 

Starting from 2021.3 (released at the end of Q1 in 2021) DL Workbench has two modes of accuracy measurement: basic and advanced. https://docs.openvinotoolkit.org/latest/workbench_docs_Workbench_DG_Measure_Accuracy.html

  • Basic mode provides simplified and reduced scope of accuracy settings and works for reduced range of models, for example basic mode supports models with only one input. At the same time
  • Advanced mode allows to provide a valid accuracy config for the Accuracy Checker tool that is used inside. Therefore, the list of models that you can measure accuracy for is much larger and, for examples, allows to measure accuracy for models with several inputs. 

In your particular case, in order to measure accuracy of Faster-RCNN model, you need to use Advanced mode and specify the input metadata layer in the configuration. Here is the example of configuration file that is valid for the following Faster-RCNN model (model with a name faster_rcnn_inception_v2_coco  from https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf1_detection_zoo.md):

 

name: faster_rcnn_inception_v2_coco
launchers:
  - adapter:
      type: ssd
    batch: 1
    device: CPU
    framework: dlsdk
    inputs:
      - name: image_info
        type: CONST_INPUT
        value:
          - - 600
            - 1024
            - 1
    model: $MODEL_PATH/faster_rcnn_inception_v2_coco.xml
    weights: $MODEL_PATH/faster_rcnn_inception_v2_coco.bin
datasets:
  - annotation_conversion:
      annotation_file: $DATASET_PATH/annotations/instances_val2017_200pictures.json
      converter: mscoco_detection
      has_background: true
      images_dir: $DATASET_PATH/images
      use_full_label_map: true
    data_source: $DATASET_PATH/images
    metrics:
      - max_detections: 100
        presenter: print_vector
        type: coco_precision
    name: dataset
    postprocessing:
      - dst_height: 600
        dst_width: 1024
        type: faster_rcnn_postprocessing_resize
    preprocessing:
      - aspect_ratio_scale: fit_to_window
        dst_height: 600
        dst_width: 1024
        type: resize
      - dst_height: 600
        dst_width: 1024
        pad_type: right_bottom
        type: padding
    subsample_size: 100%

 

Please note that you might want to specify additional parameters, refer to the Accuracy Checker documentation: https://github.com/openvinotoolkit/open_model_zoo/tree/master/tools/accuracy_checker

 

If you have any additional questions or proposals, please ask them via the official DL Workbench Discussion Forum: https://github.com/openvinotoolkit/workbench_feedback/discussions.

0 Kudos
Reply