DL workbench accuracy tool is failing

[raushan]

DL workbench accuracy tool is failing by giving status as Accuracy Tool Failed for my faster rcnn inception v2  coco tensorflow object detection model v 1.15. Please help with the Configuration. 

I have attached the screenshot of the DL workbench.

[Iffa_Intel]

Greetings,

You may try to follow these steps (there's also step by step video here): https://docs.openvinotoolkit.org/latest/workbench_docs_Workbench_DG_Int_8_Quantization.html

Sincerely,

iffa

[raushan]

Hi,

Thank you for the response.

when i'm trying to optimize model to INT8 using DL workbench, it's giving error as 'resize to input size possible, only for one input layer case'. i have attached screenshot also . Please help out in this issue.

Thanks

[Alexander_D_Intel1]

@raushan Calibration of Faster-RCNN with Default method works in DL Workbench 2021.3. AccuracyAware method still requires you to provide a valid Advanced accuracy config. 

[Iffa_Intel]

Could you help to attach your faster rcnn graph and also your dataset,

So that I could replicate this?.

Sincerely,

Iffa

[Iffa_Intel]

Greetings,

Thank you for your question. If you need any additional information from Intel, please submit a new question as Intel is no longer monitoring this thread.

Sincerely,

Iffa

[Alexander_D_Intel1]

Hi @raushan ,

Starting from 2021.3 (released at the end of Q1 in 2021) DL Workbench has two modes of accuracy measurement: basic and advanced. https://docs.openvinotoolkit.org/latest/workbench_docs_Workbench_DG_Measure_Accuracy.html

• Basic mode provides simplified and reduced scope of accuracy settings and works for reduced range of models, for example basic mode supports models with only one input. At the same time
• Advanced mode allows to provide a valid accuracy config for the Accuracy Checker tool that is used inside. Therefore, the list of models that you can measure accuracy for is much larger and, for examples, allows to measure accuracy for models with several inputs. 

In your particular case, in order to measure accuracy of Faster-RCNN model, you need to use Advanced mode and specify the input metadata layer in the configuration. Here is the example of configuration file that is valid for the following Faster-RCNN model (model with a name faster_rcnn_inception_v2_coco  from https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf1_detection_zoo.md):

name: faster_rcnn_inception_v2_coco
launchers:
  - adapter:
      type: ssd
    batch: 1
    device: CPU
    framework: dlsdk
    inputs:
      - name: image_info
        type: CONST_INPUT
        value:
          - - 600
            - 1024
            - 1
    model: $MODEL_PATH/faster_rcnn_inception_v2_coco.xml
    weights: $MODEL_PATH/faster_rcnn_inception_v2_coco.bin
datasets:
  - annotation_conversion:
      annotation_file: $DATASET_PATH/annotations/instances_val2017_200pictures.json
      converter: mscoco_detection
      has_background: true
      images_dir: $DATASET_PATH/images
      use_full_label_map: true
    data_source: $DATASET_PATH/images
    metrics:
      - max_detections: 100
        presenter: print_vector
        type: coco_precision
    name: dataset
    postprocessing:
      - dst_height: 600
        dst_width: 1024
        type: faster_rcnn_postprocessing_resize
    preprocessing:
      - aspect_ratio_scale: fit_to_window
        dst_height: 600
        dst_width: 1024
        type: resize
      - dst_height: 600
        dst_width: 1024
        pad_type: right_bottom
        type: padding
    subsample_size: 100%

Please note that you might want to specify additional parameters, refer to the Accuracy Checker documentation: https://github.com/openvinotoolkit/open_model_zoo/tree/master/tools/accuracy_checker

If you have any additional questions or proposals, please ask them via the official DL Workbench Discussion Forum: https://github.com/openvinotoolkit/workbench_feedback/discussions.