Hi!
Thank you for the quick response. I created a dummy model to showcase the problem since I can't share the original one.
I'm attaching the following files:
- dummy.onnx model
- dummy fp32 IR files
- dummy quantized IR files
I used the command
mo --input_model onnx-parts/dummy.onnx --output_dir dummy
to convert the model from ONNX to OpenVINO fp32 IR.
I did PTQ with the following script:
import numpy as np
from addict import Dict
from openvino.tools.pot import IEEngine
from openvino.tools.pot import load_model, save_model
from openvino.tools.pot import compress_model_weights
from openvino.tools.pot import create_pipeline
from openvino.tools.pot import DataLoader
input_shape = (1,3,256,256)
class DummyDataLoader(DataLoader):
""" Loads images from a folder """
def __init__(self):
pass
def __len__(self):
return 1
def __getitem__(self, index):
return np.random.rand(*input_shape), None
model_xml = "./dummy/dummy.xml"
model_bin = "./dummy/dummy.bin"
MODEL_NAME = "dummy"
COMPRESSED_PATH = "dummy"
COMPRESSED_NAME = "q_dummy"
# Model config specifies the name of the model and paths to .xml and .bin files of the model.
model_config = {
"model_name": MODEL_NAME,
"model": model_xml,
"weights": model_bin,
}
# Engine config.
engine_config = Dict({"device": "CPU"})
algorithms = [
{
"name": "DefaultQuantization",
"params": {
'target_device': 'CPU',
'preset': 'mixed'
},
}
]
data_loader = DummyDataLoader()
# Step 2: Load a model.
print("Loading model...")
model = load_model(model_config=model_config)
print("Loaded model.")
# Step 3: Initialize the engine for metric calculation and statistics collection.
engine = IEEngine(config=engine_config, data_loader=data_loader)
# Step 4: Create a pipeline of compression algorithms and run it.
pipeline = create_pipeline(algorithms, engine)
compressed_model = pipeline.run(model=model)
# Step 5 (Optional): Compress model weights to quantized precision
# to reduce the size of the final .bin file.
compress_model_weights(compressed_model)
# Step 6: Save the compressed model to the desired path.
# Set save_path to the directory where the model should be saved.
compressed_model_paths = save_model(
model=compressed_model,
save_path=COMPRESSED_PATH,
model_name=COMPRESSED_NAME,
)
I also used the following script to compile the model.
from openvino.runtime import Core
compressed_model_xml = "dummy/q_dummy.xml"
compressed_model_bin = "dummy/q_dummy.bin"
ie = Core()
# Read and load a quantized model.
quantized_model = ie.read_model(
model=compressed_model_xml, weights=compressed_model_bin
)
quantized_compiled_model = ie.compile_model(model=quantized_model, device_name="CPU")
# quantized_compiled_model.inputs prints [<ConstOutput: names[x] shape[1,3,256,256] type: f32>]
# quantized_compiled_model.outputs prints [<ConstOutput: names[y] shape[1,3,256,256] type: f32>]
Additionally I want to add that I tried using the benchmark_app as well. The benchmark app converts the input to u8 in a preprocessing step but it does so only because of the shape of the input. I want my input to be u8 regardless of the shape and I also want to set that myself, and not have the benchmark_app do it.
I also have one new question.
In the zipped file you will also find a file q_dummy_graph.xml in the folder benchmark_app_output. This is the output graph I got by passing the option -exec_graph_path to the benchmark_app. I'm assuming this is the graph of the model optimized for execution on the device after calling compile_model. Can you please direct me to the part of the documentation that describes what happens during those steps?
Thank you for your help,
Karlo
