stuck in pending

[fa2233]

pending  

[fa2233]

why

[JinPei_N_Intel]

Hi,

could you please provide the error log?

Thanks

Best regards,

Jin Pei

[fa2233]

Attaching to dls_redis, dls_mongo, dls_mqtt, dlsui, dlscore, tls_nginx_1
dls_mqtt | 1597384522: mosquitto version 1.6.10 starting
dls_mqtt | 1597384522: Config loaded from /mosquitto/config/mosquitto.conf.
dls_mqtt | 1597384522: Opening ipv4 listen socket on port 1883.
dls_mqtt | 1597384522: Opening ipv6 listen socket on port 1883.
dls_mongo | {"t":{"$date":"2020-08-14T05:55:22.728+00:00"},"s":"I", "c":"CONTROL", "id":23285, "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
dls_redis | 1:C 14 Aug 2020 05:55:20.581 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
dls_redis | 1:C 14 Aug 2020 05:55:20.581 # Redis version=5.0.9, bits=64, commit=00000000, modified=0, pid=1, just started
dls_redis | 1:C 14 Aug 2020 05:55:20.581 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
dls_mongo | {"t":{"$date":"2020-08-14T05:55:22.812+00:00"},"s":"W", "c":"ASIO", "id":22601, "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
dlscore | [setupvars.sh] OpenVINO environment initialized
dls_redis | 1:M 14 Aug 2020 05:55:20.585 * Running mode=standalone, port=6379.
dls_redis | 1:M 14 Aug 2020 05:55:20.585 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
dls_redis | 1:M 14 Aug 2020 05:55:20.585 # Server initialized
dls_redis | 1:M 14 Aug 2020 05:55:20.585 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
dls_redis | 1:M 14 Aug 2020 05:55:20.585 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
dls_mongo | {"t":{"$date":"2020-08-14T05:55:22.813+00:00"},"s":"I", "c":"NETWORK", "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
dls_mongo | {"t":{"$date":"2020-08-14T05:55:22.814+00:00"},"s":"I", "c":"STORAGE", "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":1,"port":27017,"dbPath":"/data/db","architecture":"64-bit","host":"5d85fc50a8ea"}}
dls_mongo | {"t":{"$date":"2020-08-14T05:55:22.814+00:00"},"s":"I", "c":"CONTROL", "id":23403, "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"4.4.0","gitVersion":"563487e100c4215e2dce98d0af2a6a5a2d67c5cf","openSSLVersion":"OpenSSL 1.1.1 11 Sep 2018","modules":[],"allocator":"tcmalloc","environment":{"distmod":"ubuntu1804","distarch":"x86_64","target_arch":"x86_64"}}}}
dls_mongo | {"t":{"$date":"2020-08-14T05:55:22.814+00:00"},"s":"I", "c":"CONTROL", "id":51765, "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"18.04"}}}
dls_mongo | {"t":{"$date":"2020-08-14T05:55:22.814+00:00"},"s":"I", "c":"CONTROL", "id":21951, "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"net":{"bindIp":"*"}}}}
nginx_1 | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
nginx_1 | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
dls_mongo | {"t":{"$date":"2020-08-14T05:55:22.835+00:00"},"s":"I", "c":"STORAGE", "id":22270, "ctx":"initandlisten","msg":"Storage engine to use detected by data files","attr":{"dbpath":"/data/db","storageEngine":"wiredTiger"}}
dls_mongo | {"t":{"$date":"2020-08-14T05:55:22.835+00:00"},"s":"I", "c":"STORAGE", "id":22297, "ctx":"initandlisten","msg":"Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem","tags":["startupWarnings"]}
nginx_1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
dls_mongo | {"t":{"$date":"2020-08-14T05:55:22.835+00:00"},"s":"I", "c":"STORAGE", "id":22315, "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=7423M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],"}}
dls_mongo | {"t":{"$date":"2020-08-14T05:55:24.746+00:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1597384524:746659][1:0x7f0f336d5a80], txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 17 through 18"}}
dls_redis | 1:M 14 Aug 2020 05:55:20.593 * DB loaded from disk: 0.008 seconds
dls_redis | 1:M 14 Aug 2020 05:55:20.593 * Ready to accept connections
dls_mongo | {"t":{"$date":"2020-08-14T05:55:24.798+00:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1597384524:798867][1:0x7f0f336d5a80], txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 18 through 18"}}
dls_mongo | {"t":{"$date":"2020-08-14T05:55:24.926+00:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1597384524:926193][1:0x7f0f336d5a80], txn-recover: [WT_VERB_RECOVERY | WT_VERB_RECOVERY_PROGRESS] Main recovery loop: starting at 17/970880 to 18/256"}}
dls_mongo | {"t":{"$date":"2020-08-14T05:55:25.375+00:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1597384525:375582][1:0x7f0f336d5a80], txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 17 through 18"}}
dls_mongo | {"t":{"$date":"2020-08-14T05:55:25.701+00:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1597384525:701316][1:0x7f0f336d5a80], txn-recover: [WT_VERB_RECOVERY_PROGRESS] Recovering log 18 through 18"}}
dls_mongo | {"t":{"$date":"2020-08-14T05:55:25.763+00:00"},"s":"I", "c":"STORAGE", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"[1597384525:763012][1:0x7f0f336d5a80], txn-recover: [WT_VERB_RECOVERY | WT_VERB_RECOVERY_PROGRESS] Set global recovery timestamp: (0, 0)"}}
dls_mongo | {"t":{"$date":"2020-08-14T05:55:26.344+00:00"},"s":"I", "c":"STORAGE", "id":4795906, "ctx":"initandlisten","msg":"WiredTiger opened","attr":{"durationMillis":3509}}
dls_mongo | {"t":{"$date":"2020-08-14T05:55:26.344+00:00"},"s":"I", "c":"RECOVERY", "id":23987, "ctx":"initandlisten","msg":"WiredTiger recoveryTimestamp","attr":{"recoveryTimestamp":{"$timestamp":{"t":0,"i":0}}}}
dls_mongo | {"t":{"$date":"2020-08-14T05:55:26.543+00:00"},"s":"I", "c":"STORAGE", "id":22262, "ctx":"initandlisten","msg":"Timestamp monitor starting"}
dls_mongo | {"t":{"$date":"2020-08-14T05:55:26.662+00:00"},"s":"W", "c":"CONTROL", "id":22120, "ctx":"initandlisten","msg":"Access control is not enabled for the database. Read and write access to data and configuration is unrestricted","tags":["startupWarnings"]}
dls_mongo | {"t":{"$date":"2020-08-14T05:55:26.872+00:00"},"s":"I", "c":"STORAGE", "id":20536, "ctx":"initandlisten","msg":"Flow Control is enabled on this deployment"}
dls_mongo | {"t":{"$date":"2020-08-14T05:55:26.977+00:00"},"s":"I", "c":"FTDC", "id":20625, "ctx":"initandlisten","msg":"Initializing full-time diagnostic data capture","attr":{"dataDirectory":"/data/db/diagnostic.data"}}
dls_mongo | {"t":{"$date":"2020-08-14T05:55:26.979+00:00"},"s":"I", "c":"NETWORK", "id":23015, "ctx":"listener","msg":"Listening on","attr":{"address":"/tmp/mongodb-27017.sock"}}
dls_mongo | {"t":{"$date":"2020-08-14T05:55:26.979+00:00"},"s":"I", "c":"NETWORK", "id":23015, "ctx":"listener","msg":"Listening on","attr":{"address":"0.0.0.0"}}
dls_mongo | {"t":{"$date":"2020-08-14T05:55:26.979+00:00"},"s":"I", "c":"NETWORK", "id":23016, "ctx":"listener","msg":"Waiting for connections","attr":{"port":27017,"ssl":"off"}}
nginx_1 | 10-listen-on-ipv6-by-default.sh: Getting the checksum of /etc/nginx/conf.d/default.conf
nginx_1 | 10-listen-on-ipv6-by-default.sh: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
nginx_1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
nginx_1 | /docker-entrypoint.sh: Configuration complete; ready for start up
dls_mongo | {"t":{"$date":"2020-08-14T05:55:47.332+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"connection accepted","attr":{"remote":"172.18.0.6:45980","sessionId":1,"connectionCount":1}}
dls_mqtt | 1597384547: New connection from 172.18.0.6 on port 1883.
dls_mqtt | 1597384547: New client connected from 172.18.0.6 as dls_mqtt (p2, c1, k60).
dls_mongo | {"t":{"$date":"2020-08-14T05:55:47.333+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn1","msg":"client metadata","attr":{"remote":"172.18.0.6:45980","client":"conn1","doc":{"driver":{"name":"PyMongo","version":"3.10.1"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"4.15.0-112-generic"},"platform":"CPython 3.6.9.final.0"}}}
dlsui | LoopBack server listening @ https://0.0.0.0:8080/
dls_mongo | {"t":{"$date":"2020-08-14T05:55:51.676+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"connection accepted","attr":{"remote":"172.18.0.5:43382","sessionId":2,"connectionCount":2}}
dls_mongo | {"t":{"$date":"2020-08-14T05:55:51.678+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"connection accepted","attr":{"remote":"172.18.0.5:43394","sessionId":3,"connectionCount":3}}
dls_mongo | {"t":{"$date":"2020-08-14T05:55:51.689+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn2","msg":"client metadata","attr":{"remote":"172.18.0.5:43382","client":"conn2","doc":{"driver":{"name":"nodejs","version":"2.2.36"},"os":{"type":"Linux","name":"linux","architecture":"x64","version":"4.15.0-112-generic"},"platform":"Node.js v10.15.3, LE, mongodb-core: 2.1.20"}}}
dls_mongo | {"t":{"$date":"2020-08-14T05:55:51.691+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn3","msg":"client metadata","attr":{"remote":"172.18.0.5:43394","client":"conn3","doc":{"driver":{"name":"nodejs","version":"2.2.36"},"os":{"type":"Linux","name":"linux","architecture":"x64","version":"4.15.0-112-generic"},"platform":"Node.js v10.15.3, LE, mongodb-core: 2.1.20"}}}
dls_mongo | {"t":{"$date":"2020-08-14T05:55:51.711+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"connection accepted","attr":{"remote":"172.18.0.5:43396","sessionId":4,"connectionCount":4}}
dls_mongo | {"t":{"$date":"2020-08-14T05:55:51.712+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"connection accepted","attr":{"remote":"172.18.0.5:43398","sessionId":5,"connectionCount":5}}

[JinPei_N_Intel]

Hi,

can you capture the logs with the command line below when you are training?

docker-compose logs -f dlscore, and it will showing what is the progress of training and eventually what causes it terminate/failed.