Intel® Distribution of OpenVINO™ Toolkit
Community assistance about the Intel® Distribution of OpenVINO™ toolkit, OpenCV, and all aspects of computer vision-related on Intel® platforms.
6503 Discussions

How to speed up the startup time of model_server?

linchpin
Beginner
676 Views

Starting the service now takes at least 2.5 seconds(base on bert).
Is there any way to optimize the startup time, hopefully within 1 second? If the startup time is less than 1 second, it is possible to provide tremendous throughput by cloud serverless architecture.

0 Kudos
3 Replies
Wan_Intel
Moderator
638 Views

Hi Linchpin,

Thanks for reaching out to us.

 

For your information, the smaller the model’s size, the faster the start-up time of the OpenVINO™ Model Server. We regret to inform you that the method to optimize the start-up time of the OpenVINO™ Model Server is not available.

 

bert-small-uncased-whole-word-masking-squad-int8-0002 vs face-detection-retail-0004:

size.png

start up time.png

 

Sorry for the inconvenience and thank you for your support.

 

 

Regards,

Wan

0 Kudos
linchpin
Beginner
602 Views

Hi Wan,

Thank you for your experimental work.

Regards,

Lane

0 Kudos
Wan_Intel
Moderator
583 Views

Hi Linchpin,

Thank you for your question.

If you need any additional information from Intel, please submit a new question as this thread is no longer being monitored.

 

 

Regards,

Wan

 

0 Kudos
Reply