Facing Issue while retraining Person Vehicle Bike Detector Model in Dev Cloud

[Bhargavi_Ch_Intel]

Hi Team,

I have tried the steps from the below link for retraining the Person Vehicle Bike Detector Model with my own datasets(Taken 370 Images) by developing Jupyter Notebook File in Dev Cloud and able to do set up successfully in virtual environment on Dev Cloud.

https://github.com/opencv/openvino_training_extensions/tree/develop/tensorflow_toolkit/person_vehicle_bike_detector

I have changed the num_steps=50000 in config file and written a job script which includes the retraining command. I have submitted the job to the queue by using the below command for Retraining 50000 steps.

job_id_core = !qsub bicycle_retrain_job.sh -l nodes=1:idc001skl:tank-870:i5-6500te -N bicycle_train_core

After creating 1850 checkpoint, job file execution stopped automatically by throwing  an error “PBS: job killed: walltime 3639 exceeded limit 3600 “

I have attached Log File for your reference.

Please help us in resolving the issue.

TIA

Bhargavi

[Eltablawy__Alaa]

Hi Bhargavi,

Thanks for reaching out. Intel Devcloud for the Edge is not designed for DNN training. It has compute nodes to edge inference. The better choice for training would be Intel DevCloud for Data-centric Workloads — it has compute nodes more suitable for training workloads.

Regards,

Alaa