How to speed up the startup time of model_server?

[linchpin]

Starting the service now takes at least 2.5 seconds(base on bert).
Is there any way to optimize the startup time, hopefully within 1 second? If the startup time is less than 1 second, it is possible to provide tremendous throughput by cloud serverless architecture.

[Wan_Intel]

Hi Linchpin,

Thanks for reaching out to us.

For your information, the smaller the model’s size, the faster the start-up time of the OpenVINO™ Model Server. We regret to inform you that the method to optimize the start-up time of the OpenVINO™ Model Server is not available.

bert-small-uncased-whole-word-masking-squad-int8-0002 vs face-detection-retail-0004:

Sorry for the inconvenience and thank you for your support.

Regards,

Wan

[linchpin]

Hi Wan,

Thank you for your experimental work.

Regards,

Lane

[Wan_Intel]

Hi Linchpin,

Thank you for your question.

If you need any additional information from Intel, please submit a new question as this thread is no longer being monitored.

Regards,

Wan