defining affinity of the layers with heterogeneus plugin

[uelor]

Hi:
I 'm trying to use the heterogeneus plugin to test the affinity of the layers using this tutorial:

https://docs.openvinotoolkit.org/latest/openvino_docs_IE_DG_supported_plugins_HETERO.html

but when I try to define some layer to specific hardware like the following code I have two issues:

res.supportedLayersMap["layerName"] = "CPU";
// set affinities to network
for (auto && layer : res.supportedLayersMap) {
    network.getLayerByName(layer->first)->affinity = layer->second;
}

The first one is about a deprectaion warning:
INFERENCE_ENGINE_DEPRECATED("Migrate to IR v10 and work with ngraph::Function directly. The method will be removed in 2020.3")

which I understand that to change the layer affinities I should use ngraph library such as in this sample:
https://github.com/openvinotoolkit/openvino/tree/master/inference-engine/samples/ngraph_function_creation_sample

The second one is, when I modify the affinity of specific layer when I try to query the network again with,

InferenceEngine::QueryNetworkResult res1 = core.QueryNetwork(cnnNetwork, device, { });​

This, layer affinity is not changed.

So my question are:
When a network is loaded in IR7 using read network, this network is read-only? I cannot modify any layer affinity unless I dont use ngraph graph builder?

Are there other samples that uses the heterogeneous plugin manual affinity?

Thanks in advance.

[Max_L_Intel]

Hi @uelor 

We are still looking into your issue. Meanwhile would you please let us know what OpenVINO version you use?

Also, as per Release Notes for latest OpenVINO releases,
>IRv7 is deprecated and support for this version will be removed in the next major version (v.2021.1). It means IRv7 and older cannot be read using Core::ReadNetwork. Users are recommended to migrate to IRv10 to prevent compatibility issues in the next release.

And since the manual affinity configuration is provided via core.ReadNetwork, hence this method might not work for IRv7 model. 
Have you tried to test manual affinity configuration for IRv10 model?

[uelor]

Hi my openVINO version is 2020.3. So I assume the default IR representation is IRV10.

I tried my the model with IRV7 and IRV10, but in both cases the result is the same.

After I assigned manually the affinity of specific layer I have not any error, but,  when I query the network with the new affinities ,using querynetwork function, the affinity of that layer is not changed.

[Max_L_Intel]

Hi @uelor 

Yes, by default in OpenVINO 2020.3 you should produce IRv10 models.

If manual affinity configuration doesn’t work for IRv10, then can you please attach a reproducer code and all the related data files that could help us to replicate this behavior from our end (model, sample code, all the files that you modified)?

Thank you.

[uelor]

Thank you for the answer @Max_L_Intel.

I have create a simple test, with HETERO:MYRIAD,CPU.
The model is created as IRV10 with FP16 precission, using model optimizer, is a mobilenet_v1_224x224.xml.
I tried to modify the affinity of 2 layers:
res.supportedLayersMap["MobilenetV1/MobilenetV1/Conv2d_9_pointwise/Relu6"] = "CPU";
res.supportedLayersMap["MobilenetV1/Predictions/Softmax"] = "CPU";

I have attached the model and the source code to reproduce the issue.
My biggest concern is that this line is not changing the model structure:
network.getLayerByName(layer.first.c_str())->affinity = layer.second.c_str();

#include <format_reader_ptr.h>


#include <inference_engine.hpp>
#include <limits>
#include <memory>
#include <samples/args_helper.hpp>
#include <samples/common.hpp>
#include <samples/slog.hpp>
#include <samples/classification_results.h>
#include <string>
#include <vector>
#include <inference_engine.hpp>

using namespace InferenceEngine;
using namespace ngraph;


int main(int argc, char* argv[]) {
    InferenceEngine::Core core;
    std::string model_path="/home/VICOMTECH/uelordi/projects/smacs/heterogeneus_execution/models/mobilenet_v1_FP16.xml";

    auto network = core.ReadNetwork(model_path);
    // This example demonstrates how to perform default affinity initialization and then
    // correct affinity manually for some layers
    const std::string device = "HETERO:MYRIAD,CPU";
    // QueryNetworkResult object contains map layer -> device
    InferenceEngine::QueryNetworkResult res = core.QueryNetwork(network, device, { });
    std::map<std::string, std::string>::iterator it;
    for ( it =  res.supportedLayersMap.begin();
                  it !=  res.supportedLayersMap.end(); it++ ) {
                slog::info << it->first  // string (key)
                          << ':'
                          << it->second   // string's value
                          << slog::endl ;
    }
    // update default affinities
   // this was defined as MYRIAD NOW IT SHOULD BE CHANGED TO CPU
    res.supportedLayersMap["MobilenetV1/MobilenetV1/Conv2d_9_pointwise/Relu6"] = "CPU";
    res.supportedLayersMap["MobilenetV1/Predictions/Softmax"] = "CPU";
    // set affinities to network
    for (auto && layer : res.supportedLayersMap) {
        network.getLayerByName(layer.first.c_str())->affinity = layer.second.c_str();
    }
    // load network with affinities set before
    InferenceEngine::QueryNetworkResult res1 = core.QueryNetwork(network, device, { });
      std::map<std::string, std::string>::iterator it1;
    for ( it1 =  res1.supportedLayersMap.begin();
                  it1 !=  res1.supportedLayersMap.end(); it1++ ) {
                slog::info << it1->first  // string (key)
                          << ':'
                          << it1->second   // string's value
                          << slog::endl ;
    }
    auto executable_network = core.LoadNetwork(network, device);
    return 0;
}

when I try to print the new affinities I see that both lists of affinities are the same, the changes has no make any effect.

output of first cnn layer query:

 Output of second cnn layer query: 

 Thanks in advance.

[Max_L_Intel]

Hi @uelor 

Thanks for providing all the additional details and reproducer files. We are investigating this.

[Max_L_Intel]

Hi @uelor 

We have to tell that QueryNetwork does not take affinities into account. It just reports a set of layers which supported by particular plugin. Manual affinities are only used in LoadNetwork when we load using HETERO:MYRIAD,CPU.

In your example QueryNetwork is being called twice, but the second call is not needed and it does not include user-set affinities.
Could you please try LoadNetwork and see whether the affinities are utilized?

We need to improve our documentation and state this explicitly that affinities are used only in LoadNetwork. 

[uelor · Accepted solution]

Hi @Max_L_Intel:

Your suggestion definitively works in my test:

I have taken the benchmarking example of OpenVINO benchmark app and I made the following test:

I made two tests:
1. - I forced to use CPU in all layers in heterogeneus plugin and

for (auto && layer : res.supportedLayersMap) {
cnnNetwork.getLayerByName(layer.first.c_str())->affinity = "CPU";
}

2.- I forced to use MYRIAD in all layers in heterogeneus plugin.

2.- for (auto && layer : res.supportedLayersMap) {
cnnNetwork.getLayerByName(layer.first.c_str())->affinity = "MYRIAD";
}

I also added the -pc flag to see the performance counts with this result:

in case of the CPU results i see that the peformance counts exec type is jit_avx2_1x1_FP32 style and in the myriad the output is more like the operation itself( such as Conv, CLamp, DpethConv).

The performance results indicate that this forcing is working:
CPU results:

latency: 3.79 ms

Throughput: 221.37FPS

MYRIAD RESULTS:

latency: 85.29ms
Throughput: 23.44 FPS.

below, the results in images.

CPU results:

 MYRIAD RESULTS:

So I assume that this heterogeneus plugin is working fine.
¿Would you need any more details about test?

Thanks @Max_L_Intel, for your support.

[Max_L_Intel]

Hi @uelor 

We are glad that our suggestions helped you to resolved this. Thank you for reporting this back to OpenVINO community!

We are also submitting the documentation change on QueryNetwork.

[Max_L_Intel]

This thread will no longer be monitored since this issue has been resolved.  If you need any additional information from Intel, please submit a new question.