Intel® DevCloud
Help for those needing help starting or connecting to the Intel® DevCloud
1033 Discussions

Facing Issue while retraining Person Vehicle Bike Detector Model in Dev Cloud

Bhargavi_Ch_Intel
470 Views

Hi Team,

have tried the steps from the below link for retraining the Person Vehicle Bike Detector Model with my own datasets(Taken 370 Images) by developing Jupyter Notebook File in Dev Cloud and able to do set up successfully in virtual environment on Dev Cloud.

https://github.com/opencv/openvino_training_extensions/tree/develop/tensorflow_toolkit/person_vehicl...

I have changed the num_steps=50000 in config file and written a job script which includes the retraining command. I have submitted the job to the queue by using the below command for Retraining 50000 steps.

job_id_core = !qsub bicycle_retrain_job.sh -l nodes=1:idc001skl:tank-870:i5-6500te -N bicycle_train_core

After creating 1850 checkpoint, job file execution stopped automatically by throwing  an error “PBS: job killed: walltime 3639 exceeded limit 3600 “

I have attached Log File for your reference.

Please help us in resolving the issue.

TIA

Bhargavi

 

 

 

 

0 Kudos
1 Reply
Eltablawy__Alaa
436 Views

Hi Bhargavi,

Thanks for reaching out. Intel Devcloud for the Edge is not designed for DNN training. It has compute nodes to edge inference. The better choice for training would be Intel DevCloud for Data-centric Workloads — it has compute nodes more suitable for training workloads.

 

Regards,

Alaa

Reply