Starting the service now takes at least 2.5 seconds(base on bert).
Is there any way to optimize the startup time, hopefully within 1 second? If the startup time is less than 1 second, it is possible to provide tremendous throughput by cloud serverless architecture.
Thanks for reaching out to us.
For your information, the smaller the model’s size, the faster the start-up time of the OpenVINO™ Model Server. We regret to inform you that the method to optimize the start-up time of the OpenVINO™ Model Server is not available.
Sorry for the inconvenience and thank you for your support.
Thank you for your question.
If you need any additional information from Intel, please submit a new question as this thread is no longer being monitored.