Hey,
We are trying to optimize our instances for coldstarts and found model caching like an immediate solution, however, both on servers and local deployments, the cold start performance for all models we test is consistently worse.
This is what our initialization code looks like:
core = ov.Core()
core.set_property({props.cache_dir: path_to_cache_dir})
compiled_model = core.compile_model(model=model, device_name=device_name)
After enabling cache, we can see that there is a blob file under the cache folder, however, the time it takes to load a model after caching is enabled increases by approximately 18% in our tests.
We are also trying to get rid of the non-cached files on instances where we are confident that a cached version of the model was previously generated by another service. It seems like this might not be possible, is that expected? Currently it seems like we would be using close to twice the amount of storage when caching is enabled.
At some point we considered that maybe caching wasn't supported in our test devices but this property is set to true:
caching_supported = 'EXPORT_IMPORT' in core.get_property("CPU", device.capabilities)
We suspect that the runtime is failing to load the cached version, however, no errors or traces are thrown so it's hard for us to diagnose what might be wrong or whether there might be a regression in the new SDK (2023.2).
Hardware used:
= Intel desktop CPUs (from 8th gen to 13th gen, i913900k)
= Undisclosed server CPUs
